Guide to compiling the NPU application on the VIM3/3L.

Install the necessary library packages to build the OpenCV4 application.

$ sudo apt update
$ sudo apt install libopencv-dev python3-opencv

Get the source code: khadas/aml_npu_app

$ git clone https://github.com/khadas/aml_npu_app

$ cd {workspace}/aml_npu_app
$ ls
DDK_6.3.2  DDK_6.3.2.3  DDK_6.3.2.5  DDK_6.3.3.4  DDK_6.4.0.3  DDK_6.4.3  detect_library  LICENSE  NN_SLT

• DDK_xxx - Different versions of library source code.
• detect_library - Application layer source code. detect_library/model_code link to the latest version of DDK.
• NN_SLT - Separate source code directory for DnCnn model (no longer maintained).
• LICENSE - LICENSE file.

$ cd {workspace}/aml_npu_app/detect_library
$ ls
densenet_ctc_demo_picture  inception  model_code  retinaface_demo_x11_usb  sample_demo_fb  sample_demo_x11  source_code  yolo_demo_fb_mipi  yolo_demo_fb_usb
yolo_demo_x11_mipi  yolo_demo_x11_usb  yolov7_tiny_demo_x11_usb  yolov8n_demo_x11_usb

• model_code - Load model and complete pre-processing and post-processing.
• source_code - Provide interfaces for demo to call model_code.
• Other folders - Load input and visualize the results.

Enter the directory of the library model_code, which is the directory of the DDK.

$ cd {workspace}/aml_npu_app/detect_library/model_code
$ ls
densenet_ctc  detect_mtcnn  detect_retinaface  detect_yoloface  detect_yolo_v2  detect_yolo_v3  detect_yolo_v3_tiny  detect_yolo_v4  detect_yolo_v7_tiny
detect_yolov8n  facenet

Except for detect_mtcnn, each directory represents a different model, and each model will be compiled into a library.

• densenet_ctc - Character recognition, used to identify characters. The input is only picture.
• detect_retinaface - Detect faces and the keypoints of faces.
• detect_yoloface - yoloface model, used to detect faces without keypoint.
• detect_yolo_v2 - yolov2 model, for object detection.
• detect_yolo_v3 - yolov3 model, for object detection.
• detect_yolo_v3_tiny - yolov3-tiny model, for object detection.
• detect_yolo_v4 - yolov4 model, used to detect faces.
• detect_yolo_v7_tiny - yolov7_tiny model, for object detection.
• detect_yolov8n - yolov8n model, for object detection. The latest yolo model.
• facenet - Has been deprecated.

Take detect_yolo_v3 as an example to illustrate the structure of each directory.

$ cd {workspace}/aml_npu_app/detect_library/model_code/detect_yolo_v3
$ ls
build_vx.sh  include  Makefile  makefile.linux  makefile.linux.def  vnn_yolov3.c  yolo_v3.c  yolov3_process.c

Main content description:

• build_vx.sh - Compile script.
• include - The corresponding header files and all definitions will be placed in this directory.
• Makefile - Makefile file.
• makefile.linux - Make environment configuration file.
• vnn_yolov3.c - SDK The converted model processing file is mainly used to interface with the nb file.
• yolo_v3.c - Specify the called nb file, and define all the interfaces of the model call.
• yolov3_process.c - Mainly defines the pre-processing and post-processing of the model.

Enter the directory of the source_code.

$ cd {workspace}/aml_npu_app/detect_library/source_code
$ ls
build_vx.sh  detect.c  detect_log.c  include  Makefile  makefile.linux  makefile.linux.def

Main content description:

• build_vx.sh - Compile script.
• detect.c - Provide interface for detect and call model_code.
• detect_log.c - About log level.
• include - The corresponding header files and all definitions will be placed in this directory.
• Makefile - Makefile file.
• makefile.linux - Make environment configuration file.

Enter the directory of the yolo_demo_x11_usb.

$ cd {workspace}/aml_npu_app/detect_library/yolo_demo_x11_usb
$ ls
build_vx.sh  detect.h  main.cpp  makefile.linux  makefile.linux.def  makefile.target_name  nn_detect_common.h  nn_detect.h  nn_detect_utils.h

• build_vx.sh - Compile script.
• main.cpp - Main function.
• detect.h, nn_detect_common.h, nn_detect.h, nn_detect_utils.h - The corresponding header files.
• makefile.linux - Make environment configuration file.
• makefile.target_name - The name of the compiled executable.

1. Compile the model you want to use from the model_code directory. Here we shall take detect_yolo_v3 as an example.

$ cd aml_npu_app/detect_library/model_code/detect_yolo_v3
$ ls
build_vx.sh  include  Makefile  makefile.linux  makefile.linux.def  vnn_yolov3.c  yolo_v3.c  yolov3_process.c
$ bash build_vx.sh
$ ls
bin_r  build_vx.sh  include  linux_build_sample.log  makefile.linux  makefile.linux.def  makefile.target_name  vnn_yolov3.c  yolo_v3.c  yolov3_process.c

2. The compiled library libnn_yolo_v3.so is generated in bin_r folder. Move the library file into the /usr/lib directory of the system.

$ sudo cp -r bin_r/libnn_yolo_v3.so /usr/lib

3. You will also need to build the contents of the source_code directory and move the library file into /usr/lib as well.

$ cd aml_npu_app/detect_library/source_code
$ ls
build_vx.sh  detect.c  detect_log.c  include  Makefile  makefile.linux  makefile.linux.def
$ bash build_vx.sh
$ ls
bin_r  build_vx.sh  detect.c  detect_log.c  include  Makefile  makefile.linux  makefile.linux.def
$ sudo cp -r bin_r/libnn_detect.so /usr/lib

4. Now, Build the example application

$ cd aml_npu_app/detect_library/yolo_demo_x11_usb
$ ls
build_vx.sh  detect.h  main.cpp  makefile.linux  makefile.linux.def  makefile.target_name  nn_detect_common.h  nn_detect.h  nn_detect_utils.h
$ bash build_vx.sh
$ ls
bin_r_cv4  build_vx.sh  detect.h  linux_build_sample.log  main.cpp  makefile.linux  makefile.linux.def  makefile.target_name  nn_detect_common.h  nn_detect.h  nn_detect_utils.h

5. Create a folder named nn_data under bin_r_cv4 and place the yolov3_88.nb model file in it.

$ cd bin_r_cv4
$ mkdir nn_data
$ cp -r xx/xx/yolov3_88.nb nn_data/
$ ls
detect_demo_x11_usb  main.o  nn_data

6. Finally you can run inference on captured video data using the model.

$ ./detect_demo_x11_usb -m 2 -d /dev/video1

The parameter -m is for selecting the inference model, Here are the all the available models and their respective serial numbers.

• 0 - detect_yoloface
• 1 - detect_yolo_v2
• 2 - detect_yolo_v3
• 3 - detect_yolo_v3_tiny
• 4 - detect_yolo_v4
• 13 - detect_yolo_v7_tiny
• 14 - detect_yolov8n
• 15 - densenet_ctc
• 16 - detect_retinaface

For the above example, we used the serial number 2 which corresponds to the yolov3 model.

If you use the model that takes input as a picture, change -d to -p. Here is an example with densenet_ctc.

$ ./densenet_ctc_picture -m 15 -p ../KhadasTeam.png