Khadas Docs

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
~~tag> NPU PPOCR KSNN VIM4 ~~

**Doc for version ddk-3.4.7.7**

====== PPOCR KSNN Demo - 10 ======

{{indexmenu_n>10}}

===== Introduction =====

PPOCR is a state-of-the-art, highly efficient, open-source Optical Character Recognition system. It's designed to be practical, lightweight, and incredibly fast, making it ideal for deploying OCR capabilities directly on mobile or edge devices with limited computational resources (like smartphones or IoT devices), as well as high-volume cloud-based processing.

Inference results on VIM4.

{{:products:sbc:vim4:npu:ksnn:demo:ppcor-ksnn-demo-output.webp?800|}}

===== Download the model =====

Download the PPOCR model from [[gh>PaddlePaddle/PaddleOCR/blob/main/docs/ppocr/model_list.md|ppocr/model_list.md]]
 
{{:products:sbc:vim4:npu:ksnn:demo:ppocr-det-download.webp?800|}}

{{:products:sbc:vim4:npu:ksnn:demo:ppocr-rec-download.webp?800|}}

===== Convert the model =====

==== Get the conversion tool ====

You can find the SDK here: [[dl>products/vim4/tools/npu-sdk/]]

```shell
$ wget https://dl.khadas.com/products/vim4/tools/npu-sdk/vim4_npu_sdk-ddk-3.4.7.7-250508.tgz
$ tar xvzf vim4_npu_sdk-ddk-3.4.7.7-250508.tgz
$ cd vim4_npu_sdk-ddk-3.4.7.7-250508
$ ls
adla-toolkit-binary  adla-toolkit-binary-3.1.7.4  convert-in-docker.sh  Dockerfile  docs  README.md
```

  * ''docs'' - SDK documentations
  * ''adla-toolkit-binary/bin'' - SDK tools required for model conversion
  * ''adla-toolkit-binary/demo'' - Conversion examples
  * ''adla-toolkit-binary/python'' - KSNN conversion examples

<WRAP important>
If your kernel is older than 241129, please use branch npu-ddk-1.7.5.5.
</WRAP>

==== Convert ====

First, convert model from Paddle to ONNX. Please refer the two doc. [[gh>PaddlePaddle/Paddle2ONNX/blob/develop/README_en.md|Paddle2ONNX/blob/develop/README_en.md]], [[gh>PaddlePaddle/PaddleOCR/blob/main/docs/ppocr/infer_deploy/paddle2onnx.en.md|PaddleOCR/blob/main/docs/ppocr/infer_deploy/paddle2onnx.en.md]].

Our version is ''paddlepaddle==2.6.1'' and ''paddle2onnx==1.2.11'' and our covnert command.

```bash
$ paddle2onnx --model_dir ./ch_PP-OCRv4_det_infer --model_filename inference.pdmodel --params_filename inference.pdiparams --save_file ppocr_det.onnx
$ paddle2onnx --model_dir ./ch_PP-OCRv4_rec_infer --model_filename inference.pdmodel --params_filename inference.pdiparams --save_file ppocr_rec.onnx
```

Pull ''ppocr_det.onnx'' model and ''ppocr_rec'' model into ''vim4_npu_sdk/adla-toolkit-binary-3.1.7.4/python'' and modify ''ksnn_args.txt''.

```txt ppocr_det
--model-name ppocr_det 
--model-type onnx 
--model ./ppocr_det.onnx 
--inputs "x" 
--input-shapes  "3,736,736" 
--dtypes "float32" 
--quantize-dtype int8 
--outdir onnx_output 
--channel-mean-value "123.675,116.28,103.53,57.375" 
--source-file ocr_det_dataset.txt 
--iterations 1 
--batch-size 1 
--kboard VIM4 
--inference-input-type "float32" 
--inference-output-type "float32" 
```

```txt ppocr_rec
--model-name ppocr_rec 
--model-type onnx 
--model ./ppocr_rec.onnx 
--inputs "x" 
--input-shapes  "3,48,320" 
--dtypes "float32" 
--quantize-dtype int16 
--outdir onnx_output 
--channel-mean-value "127.5,127.5,127.5,128" 
--source-file ocr_rec_dataset.txt 
--iterations 1 
--batch-size 1 
--kboard VIM4 
--inference-input-type "float32" 
--inference-output-type "float32" 
```

```shell
$ ./convert-in-docker.sh ksnn
```

===== Run inference on the NPU by KSNN =====

==== Install KSNN ====

Download KSNN library and demo code. [[gh>khadas/ksnn-vim4]]

```shell
$ git clone https://github.com/khadas/ksnn-vim4
```

<WRAP important>
Only KSNN demo tag ddk-3.4.7.7 or higher supports PPOCR. Only firmware newer than 241129 supports this PPOCR demo.
</WRAP>

If you use Ubuntu 24.04, demo must run in python virtual environment.

```shell
$ sudo apt update
$ sudo apt install python3-venv
$ python3 -m venv myenv
$ source myenv/bin/activate
```

```shell
$ cd ksnn-vim4/ksnn
$ sudo apt update
$ sudo apt install python3-pip
$ pip3 install ksnn_vim4-1.4.1-py3-none-any.whl
$ pip3 install shapely pyclipper Pillow
```

==== Picture input demo ====

<WRAP important>
The demo use cn rec model. If you want to use other rec model, you need to modify the rec model output and character index dictionary.

```python ppocr-picture.py
# model input and output
det_input_size = (736, 736) # (model height, model width)
rec_input_size = ( 48, 320) # (model height, model width)
rec_output_size = (40, 97) # rec output
```

```python postprocess.py
character_str = ["blank"]
with open("./data/ppocr_keys_v1.txt", "rb") as fin: # the path for character index dictionary
    lines = fin.readlines()
    for line in lines:
        line = line.decode("utf-8").strip("\n").strip("\r\n")
        character_str.append(line)
character_str.append(" ")
ignored_token = [0]
```

You can find the dictionary txt file in ''PaddleOCR/ppocr/utils''
</WRAP>

```shell
$ cd ksnn-vim4/example/ppocr
$ export QT_QPA_PLATFORM=xcb
$ python3 ppocr-picture.py --det_model ./models/VIM4/ppocr_det_int8.adla --det_library ./libs/libnn_ppocr_det.so --rec_model ./models/VIM4/ppocr_rec_int16.adla --rec_library ./libs/libnn_ppocr_rec.so --picture ./data/test.png
```

=== Camera input demo ===

```shell
$ cd ksnn-vim4/example/ppocr
$ export QT_QPA_PLATFORM=xcb

# USB Camera
$ python3 ppocr-cap.py --det_model ./models/VIM4/ppocr_det_int8.adla --det_library ./libs/libnn_ppocr_det.so --rec_model ./models/VIM4/ppocr_rec_int16.adla --rec_library ./libs/libnn_ppocr_rec.so --type usb --device 0

# MIPI Camera
$ python3 ppocr-cap.py --det_model ./models/VIM4/ppocr_det_int8.adla --det_library ./libs/libnn_ppocr_det.so --rec_model ./models/VIM4/ppocr_rec_int16.adla --rec_library ./libs/libnn_ppocr_rec.so --type mipi --device 63
```

''0'' is the camera device index.

<WRAP important>
If you want to use MIPI Camera, you should remove OpenCV in python lib and then us apt install OpenCV.

```
$ pip3 uninstall opencv-python
$ sudo apt install python3-opencv
```

If you use virtual environment, you need to download the opencv source code and compile the GSTREAMER function manually. (If you do not know how to compile, you can ask help in [[https://forum.khadas.com/ | KHADAS FORUM]]).
</WRAP>
Khadas Docs

User Tools

Site Tools

Page Tools
