~~tag> VIM3 VIM3L Amlogic NPU OpenCV~~ ====== Application Source Code ====== Guide to compiling the NPU application on the VIM3/3L. ===== Install OpenCV4 ===== Install the necessary library packages to build the OpenCV4 application. ```shell $ sudo apt update $ sudo apt install libopencv-dev python3-opencv ``` ===== Get Source Code ===== Get the source code: [[gh>khadas/aml_npu_app]] ```shell $ git clone https://github.com/khadas/aml_npu_app ``` ===== Source Description ===== ==== Base directory ==== Base directory of the cloned repository. ```shell $ cd aml_npu_app $ ls DDK_6.3.2 DDK_6.3.2.3 DDK_6.3.2.5 DDK_6.3.3.4 DDK_6.4.0.3 DDK_6.4.3 detect_library LICENSE NN_SLT ``` ^ File ^ Description ^ | DDK_xx | Library source code, xx is the version code | | detect_library | Application layer source code. the detect_library/model_code \\ is linked to the latest version of the DDK library | | NN_SLT | Separate source code directory for DnCnn model (no longer maintained). | | LICENSE | LICENSE file. | | ==== Detect library ==== Directory with various model specific application code. ```shell $ cd aml_npu_app/detect_library $ ls densenet_ctc_demo_picture inception model_code retinaface_demo_x11_usb sample_demo_fb sample_demo_x11 source_code yolo_demo_fb_mipi yolo_demo_fb_usb yolo_demo_x11_mipi yolo_demo_x11_usb yolov7_tiny_demo_x11_usb yolov8n_demo_x11_usb ``` ^ File ^ Description ^ | **model_code** | Load model and complete pre-processing and post-processing. | | **source_code** | Provide interfaces for demo to call model_code. | | **Other folders** | Application source code | ==== Model code ==== Directory containing all the detection library code. ```shell $ cd aml_npu_app/detect_library/model_code $ ls densenet_ctc detect_mtcnn detect_retinaface detect_yoloface detect_yolo_v2 detect_yolo_v3 detect_yolo_v3_tiny detect_yolo_v4 detect_yolo_v7_tiny detect_yolov8n facenet ``` Except for ''detect_mtcnn'', each directory represents a different model, and each model will be compiled into a library. ^ File ^ Description ^ | **densenet_ctc** | Character recognition, used to identify characters. The input is only a picture. | | **detect_retinaface** | Detect faces and the keypoints of faces. | | **detect_yoloface** | yoloface model, used to detect faces without keypoints. | | **detect_yolo_v2** | yolov2 model, for object detection. | | **detect_yolo_v3** | yolov3 model, for object detection. | | **detect_yolo_v3_tiny** | yolov3-tiny model, for object detection. | | **detect_yolo_v4** | yolov4 model, used to detect faces. | | **detect_yolo_v7_tiny** | yolov7_tiny model, for object detection. | | **detect_yolov8n** | yolov8n model, for object detection. The latest yolo model. | | **facenet** | Deprecated - No longer in use. | Take ''detect_yolo_v3'' as an example to illustrate the structure of each directory. ```shell $ cd aml_npu_app/detect_library/model_code/detect_yolo_v3 $ ls build_vx.sh include Makefile makefile.linux makefile.linux.def vnn_yolov3.c yolo_v3.c yolov3_process.c ``` ^ File ^ Description ^ | **build_vx.sh** | Compile script. | | **include** | The corresponding header files and all definitions will be placed in this directory. | | **Makefile** | Makefile file. | | **makefile.linux** | Make environment configuration file. | | **vnn_yolov3.c** | SDK The converted model processing file is mainly used to interface with the nb file. | | **yolo_v3.c** | Specify the called nb file, and define all the interfaces of the model call. | | **yolov3_process.c** | Mainly defines the pre-processing and post-processing of the model. | ==== Source code ==== Directory containing the application library source code. ```shell $ cd aml_npu_app/detect_library/source_code $ ls build_vx.sh detect.c detect_log.c include Makefile makefile.linux makefile.linux.def ``` ^ File ^ Description ^ | **build_vx.sh** | Compile script. | | **detect.c** | Provide interface for detect and call model_code. | | **detect_log.c** | About log level. | | **include** | The corresponding header files and all definitions will be placed in this directory. | | **Makefile** | Makefile file. | | **makefile.linux** | Make environment configuration file. | Enter the directory of the ''yolo_demo_x11_usb'': ```shell $ cd aml_npu_app/detect_library/yolo_demo_x11_usb $ ls build_vx.sh detect.h main.cpp makefile.linux makefile.linux.def makefile.target_name nn_detect_common.h nn_detect.h nn_detect_utils.h ``` ^ File ^ Description ^ | **build_vx.sh** | Compile script. | | **main.cpp** | Main function. | | **detect.h**, **nn_detect_common.h**, **nn_detect.h**, **nn_detect_utils.h** | The corresponding header files. | | **makefile.linux** | Make environment configuration file. | | **makefile.target_name** | The name of the compiled executable. | ===== Compile the application===== 1. Compile the model you want to use from the ''model_code'' directory. Here we shall take ''detect_yolo_v3'' as an example. ```shell $ cd aml_npu_app/detect_library/model_code/detect_yolo_v3 $ ls build_vx.sh include Makefile makefile.linux makefile.linux.def vnn_yolov3.c yolo_v3.c yolov3_process.c $ bash build_vx.sh $ ls bin_r build_vx.sh include linux_build_sample.log makefile.linux makefile.linux.def makefile.target_name vnn_yolov3.c yolo_v3.c yolov3_process.c ``` 2. The compiled library ''libnn_yolo_v3.so'' is generated in ''bin_r'' folder. Add it to the library path variable. ```shell $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)/bin_r/libnn_yolo_v3.so ``` 3. You will also need to build the contents of the ''source_code'' directory. ```shell $ cd aml_npu_app/detect_library/source_code $ ls build_vx.sh detect.c detect_log.c include Makefile makefile.linux makefile.linux.def $ bash build_vx.sh $ ls bin_r build_vx.sh detect.c detect_log.c include Makefile makefile.linux makefile.linux.def ``` 4. The compiled library ''libnn_yolo_v3.so'' is generated in bin_r folder. Add it to the library path variable. ```shell $ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$(pwd)/bin_r/libnn_detect.so ``` 5. Now, Build the example application. ```shell $ cd aml_npu_app/detect_library/yolo_demo_x11_usb $ ls build_vx.sh detect.h main.cpp makefile.linux makefile.linux.def makefile.target_name nn_detect_common.h nn_detect.h nn_detect_utils.h $ bash build_vx.sh $ ls bin_r_cv4 build_vx.sh detect.h linux_build_sample.log main.cpp makefile.linux makefile.linux.def makefile.target_name nn_detect_common.h nn_detect.h nn_detect_utils.h ``` If you use kernel are using version 5.15 or above, please remove the red lines in ''makefile.linux'' before running ''build_vx.sh''. ```diff LIBS += -L$(VIVANTE_SDK_LIB) -lOpenVX -lOpenVXU -lGAL -lovxlib -lArchModelSw -lNNArchPerf LIBS += -L../source_code/bin_r -lnn_detect -#LIBS +=-L$(LIB_DIR) -lstdc++ -LIBS += -lvpcodec -lamcodec -lamadec -lamvdec -lamavutils -lrt -lpthread -lge2d -lion ############################################################################# # Macros. PROGRAM = 1 CUR_SOURCE = ${wildcard *.c} ############################################################################# ``` 6. Create a folder named ''nn_data'' under ''bin_r_cv4'' and place the ''yolov3_88.nb'' model file in it. ```shell $ cd bin_r_cv4 $ mkdir nn_data $ cp -r xx/xx/yolov3_88.nb nn_data/ $ ls detect_demo_x11_usb main.o nn_data ``` ===== Run the application ===== Now you can run inference on captured video data using the model. ```shell $ ./detect_demo_x11_usb -m 2 -d /dev/video1 ``` ==== Application setup parameters ==== === Parameter to select the detection model === The parameter ''-m'' is for selecting the inference model, Here are the all the available models and their respective serial numbers. ^ Index ^ File ^ | **0** | detect_yoloface | | **1** | detect_yolo_v2 | | **2** | detect_yolo_v3 | | **3** | detect_yolo_v3_tiny | | **4** | detect_yolo_v4 | | **13** | detect_yolo_v7_tiny | | **14** | detect_yolov8n | | **15** | densenet_ctc | | **16** | detect_retinaface | For the above example, we used the serial number ''2'' which corresponds to the ''yolov3'' model. === Parameter to set the input as picture === If you use the model that takes input as a picture, change ''-d'' to ''-p''. Here is an example with ''densenet_ctc''. ```shell $ ./densenet_ctc_picture -m 15 -p ../KhadasTeam.png ```