====== Convert and Call Model through NPU ====== ===== Introduction ===== This document mainly introduces how to use the SDK to convert the user's own model and apply it to the NPU. Please review the document carefully before converting the reference document. ===== Prepare ===== 1. Train your own ''yolov3'' model. The training method and process can refer to the official: [[https://pjreddie.com/darknet/yolo|Darknet Yolo Page]], here we use the officially trained weights based on the coco data set. 2. Prepare SDK, app warehouse, and demo warehouse. Please refer to the SDK, app and demo documents respectively for how to obtain the corresponding code: - [[npu-sdk|NPU SDK Usage]] - [[npu-prebuilt-demo-usage|NPU Prebuilt Demo Usage]] - [[npu-app|Application Source Code]] ===== Conversion ===== The conversion is performed under the SDK. ```shell $ cd {workspace}/SDK/acuity-toolkit/conversion_scripts ``` ==== Modify file 0_import_model.sh ==== 1. Modify ''NAME'': ```shell NAME=mobilenet_tf --> NAME=yolov3 ``` 2. Comment out Tensorflow: ```shell $convert_tf \ --tf-pb ./model/mobilenet_v1.pb \ --inputs input \ --input-size-list '224,224,3' \ --outputs MobilenetV1/Logits/SpatialSqueeze \ --net-output ${NAME}.json \ --data-output ${NAME}.data ``` Modify to, ```shell #$convert_tf \ # --tf-pb ./model/mobilenet_v1.pb \ # --inputs input \ # --input-size-list '224,224,3' \ # --outputs MobilenetV1/Logits/SpatialSqueeze \ # --net-output ${NAME}.json \ # --data-output ${NAME}.data ``` 3. Comment Darknet: ```shell #$convert_darknet \ # --net-input xxx.cfg \ # --weight-input xxx.weights \ # --net-output ${NAME}.json \ # --data-output ${NAME}.data ``` Modify to, ```shell $convert_darknet \ --net-input path/to/yolov3.cfg \ --weight-input path/to/yolov3.weights \ --net-output ${NAME}.json \ --data-output ${NAME}.data ``` ==== Modify file 1_quantize_model.sh ==== 1. Modify ''NAME'': ```shell NAME=mobilenet_tf --> NAME=yolov3 ``` 2. Modify regression parameters: ```shell --channel-mean-value '128 128 128 128' \ ``` Modify to, ```shell --channel-mean-value '0 0 0 256' \ ``` 3. Modify ''validation_tf.txt'': Replace the image inside, ```shell $ cat ./data/validation_tf.txt ./space_shuttle_224.jpg, 813 ``` Modify to, ```shell path/to/416x416.jpg ``` The picture resolution here is the same as the configuration in the yolo cfg file. 4. Modify quant type: ```sh --quantized-dtype asymmetric_affine-u8 \ ``` Modify to, ```sh --quantized-dtype dynamic_fixed_point-i8 \ ``` ==== Modify file 2_export_case_code.sh ==== 1. Modify ''NAME'': ```shell NAME=mobilenet_tf --> NAME=yolov3 ``` 2. Modify regression parameters: ```shell --channel-mean-value '128 128 128 128' \ ``` Modify to, ```shell --channel-mean-value '0 0 0 256' \ ``` 3. Modify the RGB channel order: default channel is ''RGB'', ```shell --reorder-channel '0 1 2' \ ``` Modified to ''BGR'', ```shell --reorder-channel '2 1 0' \ ``` 4. Specified board model ''VIM3'': ```shell --optimize VIPNANOQI_PID0X88 \ ``` ''VIM3L'': ```shell --optimize VIPNANOQI_PID0X99 \ ``` ==== Compile And Get The Case Code ==== 1. Compile: ```shell $ bash 0_import_model.sh && bash 1_quantize_model.sh && bash 2_export_case_code.sh ``` 2. case code: The converted code is in the ''nbg_unify_yolov3'' directory, ```shell $ ls {workspace}/SDK/acuity-toolkit/conversion_scripts/nbg_unify_yolov3 BUILD main.c makefile.linux nbg_meta.json vnn_global.h vnn_post_process.c vnn_post_process.h vnn_pre_process.c vnn_pre_process.h vnn_yolov3.c vnn_yolov3.h yolov3.nb yolov3.vcxproj ``` ===== Compile ===== This part of the code is carried out in the ''aml_npu_app'' repository. Enter the directory of ''detect_yolo_v3''. ```shell $ cd {workspace}/aml_npu_app/detect_library/model_code/detect_yolo_v3 $ ls build_vx.sh include Makefile makefile.linux nn_data vnn_yolov3.c yolo_v3.c yolov3_process.c ``` ==== Replace VNN File ==== 1. Replace ''vnn_yolov3.h'' , ''vnn_post_process.h'', ''vnn_pre_process.h'' generated by SDK. ```shell $ cp {workspace}/SDK/acuity-toolkit/conversion_scripts/nbg_unify_yolov3/vnn_yolov3.h {workspace}/aml_npu_app/detect_library/model_code/detect_yolo_v3/include/vnn_yolov3.h $ cp {workspace}/SDK/acuity-toolkit/conversion_scripts/nbg_unify_yolov3/vnn_post_process.h {workspace}/aml_npu_app/detect_library/model_code/detect_yolo_v3/include/vnn_post_process.h $ cp {workspace}/SDK/acuity-toolkit/conversion_scripts/nbg_unify_yolov3/vnn_pre_process.h {workspace}/aml_npu_app/detect_library/model_code/detect_yolo_v3/include/vnn_pre_process.h ``` 2. Replace ''vnn_yolov3.c'' generated by SDK. ```shell $ cp {workspace}/SDK/acuity-toolkit/conversion_scripts/nbg_unify_yolov3/vnn_yolov3.c {workspace}/aml_npu_app/detect_library/model_code/detect_yolo_v3/vnn_yolov3.c ``` ==== Modify file yolov3_process.c ==== 1. Modify the class array: ```c static char *coco_names[] = {"person","bicycle","car","motorbike","aeroplane","bus","train","truck","boat","traffic light","fire hydrant","stop sign","parking meter","bench","bird","cat","dog","horse","sheep","cow","elephant","bear","zebra","giraffe","backpack","umbrella","handbag","tie","suitcase","frisbee","skis","snowboard","sports ball","kite","baseball bat","baseball glove","skateboard","surfboard","tennis racket","bottle","wine glass","cup","fork","knife","spoon","bowl","banana","apple","sandwich","orange","broccoli","carrot","hot dog","pizza","donut","cake","chair","sofa","pottedplant","bed","diningtable","toilet","tvmonitor","laptop","mouse","remote","keyboard","cell phone","microwave","oven","toaster","sink","refrigerator","book","clock" ,"vase","scissors","teddy bear","hair drier","toothbrush"}; ``` According to your training data set settings, if it is a coco data set, there is no need to modify it. 2. Modify ''yolo_v3_post_process_onescale'': Modify ''num_class'': ```c int num_class = 80; ``` The ''num_class'' here is the same as the number of classes in the training set. 3. Modified post-processing function ''yolov3_postprocess''. Modify ''num_class'' and ''size[3]''. ```c int num_class = 80; int size[3]={nn_width/32, nn_height/32,85*3}; ``` * The ''num_class'' here is the same as the number of classes in the training set. * The ''size[2]'' here is equal to ''(num_class + 5) * 3''. ==== Compile ==== Use the ''build_vx.sh'' script to compile the ''yolov3'' library, ```shell $ cd {workspace}/aml_npu_app/detect_library/model_code/detect_yolo_v3 $ ./build_vx.sh ``` The generated library is in the ''bin_r'' directory ```shell $ ls {workspace}/aml_npu_app/detect_library/model_code/detect_yolo_v3/bin_r libnn_yolo_v3.so vnn_yolov3.o yolo_v3.o yolov3_process.o ``` ===== Run ===== ==== Replace ==== 1. Replace ''yolov3'' library ```shell $ cp {workspace}/aml_npu_app/detect_library/model_code/detect_yolo_v3/bin_r/libnn_yolo_v3.so {workspace}/aml_npu_demo_binaries/detect_demo_picture/lib/libnn_yolo_v3.so ``` 2. Replace nb file VIM3: ```shell $ cp {workspace}/SDK/acuity-toolkit/conversion_scripts/nbg_unify_yolov3/yolov3.nb {workspace}/aml_npu_demo_binaries/detect_demo_picture/nn_data/yolov3_88.nb ``` VIM3L: ```shell $ cp {workspace}/SDK/acuity-toolkit/conversion_scripts/nbg_unify_yolov3/yolov3.nb {workspace}/aml_npu_demo_binaries/detect_demo_picture/nn_data/yolov3_99.nb ``` ### Run with board How to run the replaced ''aml_npu_demo_binaries'' on the board, please refer to:[[npu-prebuilt-demo-usage|NPU Prebuilt Demo Usage]].