YOLOv8n OpenCV VIM4 Demo - 2

Download YOLOv8 official code ultralytics/ultralytics

$ git clone https://github.com/ultralytics/ultralytics

Refer README.md to train a YOLOv8n model. My version torch==1.10.1 and ultralytics==8.0.86.

Follow Docker official documentation to install Docker: Install Docker Engine on Ubuntu.

Follow the script below to get Docker image:

docker pull numbqq/npu-vim4

$ git lfs install
$ git lfs clone https://gitlab.com/khadas/vim4_npu_sdk.git
$ cd vim4_npu_sdk
$ ls
adla-toolkit-binary  adla-toolkit-binary-1.2.0.9  convert-in-docker.sh  Dockerfile  docs  README.md

• adla-toolkit-binary/docs - SDK documentations
• adla-toolkit-binary/bin - SDK tools required for model conversion
• adla-toolkit-binary/demo - Conversion examples

Get source khadas/vim4_npu_sdk.

$ git clone https://gitlab.com/khadas/vim4_npu_sdk

After training model, modify ultralytics/ultralytics/nn/modules/head.py as follows.

head.py

diff --git a/ultralytics/nn/modules/head.py b/ultralytics/nn/modules/head.py
index 0b02eb3..0a6e43a 100644
--- a/ultralytics/nn/modules/head.py
+++ b/ultralytics/nn/modules/head.py
@@ -42,6 +42,9 @@ class Detect(nn.Module):
 
     def forward(self, x):
         """Concatenates and returns predicted bounding boxes and class probabilities."""
+        if torch.onnx.is_in_onnx_export():
+            return self.forward_export(x)
+
         shape = x[0].shape  # BCHW
         for i in range(self.nl):
             x[i] = torch.cat((self.cv2[i](x[i]), self.cv3[i](x[i])), 1)
@@ -80,6 +83,15 @@ class Detect(nn.Module):
             a[-1].bias.data[:] = 1.0  # box
             b[-1].bias.data[:m.nc] = math.log(5 / m.nc / (640 / s) ** 2)  # cls (.01 objects, 80 classes, 640 img)
 
+    def forward_export(self, x):
+        results = []
+        for i in range(self.nl):
+            dfl = self.cv2[i](x[i]).contiguous()
+            cls = self.cv3[i](x[i]).contiguous()
+            results.append(torch.cat([cls, dfl], 1).permute(0, 2, 3, 1))
+            # results.append(torch.cat([cls, dfl], 1))
+        return tuple(results)
+

Create a python file written as follows to export ONNX model.

export.py

from ultralytics import YOLO
model = YOLO("./runs/detect/train/weights/best.pt")
results = model.export(format="onnx")

$ python export.py

Enter vim4_npu_sdk/demo and modify convert_adla.sh as follows.

convert_adla.sh

#!/bin/bash
 
ACUITY_PATH=../bin/
#ACUITY_PATH=../python/tvm/
adla_convert=${ACUITY_PATH}adla_convert
 
 
if [ ! -e "$adla_convert" ]; then
    adla_convert=${ACUITY_PATH}adla_convert.py
fi
 
$adla_convert --model-type onnx \
        --model ./model_source/yolov8n/yolov8n.onnx \
        --inputs "images" \
        --input-shapes  "3,640,640"  \
        --dtypes "float32" \
        --quantize-dtype int16 --outdir onnx_output  \
        --channel-mean-value "0,0,0,255"  \
        --source-file dataset.txt  \
        --batch-size 1 --target-platform PRODUCT_PID0XA003

Run convert_adla.sh to generate VIM4 model. The converted model is xxx.adla in onnx_output.

$ bash convert_adla.sh

Clone the source code from our khadas/vim4_npu_applications.

$ git clone https://github.com/khadas/vim4_npu_applications

$ sudo apt update
$ sudo apt install libopencv-dev python3-opencv cmake

Put yolov8n_int8.adla in vim4_npu_applications/yolov8n/data/.

# Compile
$ cd vim4_npu_applications/yolov8n
$ mkdir build
$ cd build
$ cmake ..
$ make
 
# Run
$ sudo ./yolov8n -m ../data/yolov8n_int8.adla -p ../data/horses.jpg

Put yolov8n_int8.adla in vim4_npu_applications/yolov8n_cap/data/.

# Compile
$ cd vim4_npu_applications/yolov8n_cap
$ mkdir build
$ cd build
$ cmake ..
$ make
 
# Run
$ sudo ./yolov8n_cap -m ../data/yolov8n_int8.adla -d 0 -w 1920 -h 1080

0 is camera device index.