【有奖征文】EdgeX + OpenVINO 实现边缘智能 AI 应用

openlab_96bf3613 更新于 14天前

架构设计图


本系统设计硬件设计采用 Intel® CPU(iGPU) + GPU[可选] ,CPU 必须为 Intel® 架构,独立 GPU 可根据实际需要灵活扩展。

本系统全部采用 Docker 微服务运行,描述如下:

图例中,① 作为 AI 推理服务器,运行 OpenVINO™ Model Server 容器;
图例中,② 作为流媒体服务器,负责流媒体的编解码,实时查看视频流,运行亿琪软件产品 YiMEDIA,同样也采用容器运行;
图例中,③ 作为边缘计算,运行 EdgeXFoundry 容器,负责整个系统的协调和业务驱动;
应用配置
硬件环境


软件环境

采用 Ubuntu 22.04 作为 Docker 宿主主机,并且已经成功完成 Docker 运行环境的安装和测试。

<font size="3"># uname -a
Linux YiFUSION-N100 6.5.0-21-generic #21~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 9 13:32:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

# l**_release -a
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy</font>


采用 CPU 内置 iGPU 作为推理

<font size="3"># ll /dev/dri/
total 0
drwxr-xr-x 3 root root 100 2月 23 17:34 ./
drwxr-xr-x 20 root root 4620 2月 23 20:19 ../
crw-rw----+ 1 root video 226, 0 2月 23 20:58 card0
crw-rw----+ 1 root render 226, 128 2月 23 17:39 renderD128</font>


OpenVINO Model Server 服务
模型库配置:

name: 模型名称
base_path: 模型文件路径
target_device: 目标推理设备
layout: 需要的图片格式(可选)

<font size="3"># more models/config.json
{
  "model_config_list":[
      {
        "config":{
            "name":"ssdlite_mobilenet_v2",
            "base_path":"/models/ssdlite_mobilenet_v2",
            "target_device": "GPU"
        }
      },
  ]
}</font>

既可以用云存储作为模型库存放位置,也可以使用本地磁盘作为模型库存放位置(本例子使用)。

一级目录是对应的模型名称
二级目录是模型的版本,比如:1,2...
xml 是 OpenVINO™ 模型库相关配置,bin 是 OpenVINO™ 模型 IR 文件
models 目录结构如下:

model****r/>├── ssdlite_mobilenet_v2
│ └── 1
│ ├── coco_91cl_bkgr.txt
│ ├── ssdlite_mobilenet_v2.bin
│ ├── ssdlite_mobilenet_v2.mapping
│ └── ssdlite_mobilenet_v2.xml
启动 Model Server 容器

<font size="3">docker run -d \
--name model_server \
--network edgex_edgex-network \
--device /dev/dri \
-v $(pwd)/models:/models \
-p 9000:9000 \
-p 8000:8000 \
openvino/model_server:latest-gpu \
--config_path /models/config.json \
--port 9000 \
--rest_port 8000 \
--log_level DEBUG</font>

通过 Kserve RESTful API 访问 OVMS 工作状态,这里访问已经启动的模型:ssdlite_mobilenet_v2。

<font size="3">curl http://192.168.198.8:8000/v1/config
{
  "ssdlite_mobilenet_v2": {
    "model_version_status": [
      {
        "version": "1",
        "state": "AVAILABLE",
        "status": {
          "error_code": "OK",
          "error_message": "OK"
        }
      }
    ]
  }
}</font>


同样通过 Kserve RESTful API 可以访问模型的详细输入和输出参数,主要是 inputs 和 outputs 里面的 dim。

inputs: 需要输入图片名称是 image_tensor,格式是 DT_INT8,可以用 byte。图片纬度是 [1,300,300,3], width:300, height:300, color:3;
outputs: 输出格式是一个四维矩阵 [1,1,100,7],只需要后面两个维度:rows:100, cols:7;

<font size="3">{
  "modelSpec": {
    "name": "ssdlite_mobilenet_v2",
    "signatureName": "",
    "version": "1"
  },
  "metadata": {
    "signature_def": {
      "@type": "type.googleapis.com/tensorflow.serving.SignatureDefMap",
      "signatureDef": {
        "serving_default": {
          "inputs": {
            "image_tensor": {
              "dtype": "DT_UINT8",
              "tensorShape": {
                "dim": [
                  {
                    "size": "1",
                    "name": ""
                  },
                  {
                    "size": "300",
                    "name": ""
                  },
                  {
                    "size": "300",
                    "name": ""
                  },
                  {
                    "size": "3",
                    "name": ""
                  }
                ],
                "unknownRank": false
              },
              "name": "image_tensor"
            }
          },
          "outputs": {
            "detection_boxes": {
              "dtype": "DT_FLOAT",
              "tensorShape": {
                "dim": [
                  {
                    "size": "1",
                    "name": ""
                  },
                  {
                    "size": "1",
                    "name": ""
                  },
                  {
                    "size": "100",
                    "name": ""
                  },
                  {
                    "size": "7",
                    "name": ""
                  }
                ],
                "unknownRank": false
              },
              "name": "detection_boxes"
            }
          },
          "methodName": "",
          "defaults": {
 
          }
        }
      }
    }
  }
}</font>


笔者采用亿琪软件公司自己的流媒体服务器软件:YiMEDIA,来实现视频流的编解码。

当然,用户也可以使用一些其他框架来支撑,比如:

SRS(Simple Realtime Server)
go2rtc
MediaMTX (formerly rtsp-simple-server)


EdgeX Foundry 服务
根据 EdgeXFoundry edgex-compose 仓库手册,创建自己的 EdgeX 运行环境。本测试案例,只需要核心服务和 mqtt-broker 服务即可。

Device Service for ONVIF Camera 是否需要使用,取决于是否由 EdgeX 来管理网络摄像头(IPC),本例中暂时未启用此服务,采用手动设置。

这是一个系列微服务,包含了各种算法库,支持算法调用和推理。

SSDLITE_MOBILENET_V2:提交一张图片,返回目标识别结果;

deviceList:
 
- name: OpenVINO-Object-Detection
  profileName: OpenVINO-Device
  description: Example of OpenVINO Device
  labels: [AI]
  protocols:
    ovms:
      Host: 192.168.198.8
      Port: 9000
      Model: ssdlite_mobilenet_v2
      Version: 1
      Uri: rtsp://192.168.123.12:18554/test
      Snapshot: false
      Record: false
      Score: 0.3


这是一个设备服务基础配置例子,用户可根据实际业务需求配置:

Host: OVMS 服务地址
Port: OVMS 服务端口
Model: 模型名称
Version: 模型版本
Uri: IPC 摄像头流地址
Snapshot: 是否建立快照
Record: 是否记录推理视频
Score: 推理结果最低信任度
以下是完整的 EdgeX metadata:

{
  "apiVersion": "v3",
  "statusCode": 200,
  "totalCount": 1,
  "devices": [
    {
      "created": 1708691437289,
      "modified": 1708693207326,
      "id": "6b02c540-eeae-4b61-b741-cdefcf93fd09",
      "name": "OpenVINO-Object-Detection",
      "description": "Example of OpenVINO Device",
      "adminState": "UNLOCKED",
      "operatingState": "UP",
      "labels": ["AI"],
      "serviceName": "device-openvino-ssdlite-mobilenet-v2",
      "profileName": "OpenVINO-Device",
      "autoEvents": [{}],
      "protocols": {
        "ovms": {
          "Host": "192.168.198.8",
          "Model": "ssdlite_mobilenet_v2",
          "Port": 9000,
          "Record": false,
          "Snapshot": false,
          "Uri": "rtsp://192.168.123.12:18554/test",
          "Version": 1,
          "Score": 0.3
        }
      }
    }
  ]
}


验证配置
所有的容器都运行成功后,可看到类似以下的结果:

# docker ps --format 'table {{.Image}}\t{{.Names}}'
 
IMAGE                                                NAMES
①
openvino/model_server:latest-gpu                     model_server
 
②
yiqisoft/YiMEDIA                                     yimedia
 
③
edgexfoundry/device-openvino-ssdlite-mobilenet-v2:0.0.0-dev       edgex-device-openvino-resnet
edgexfoundry/app-service-configurable:3.1.0                       edgex-app-rules-engine
edgexfoundry/core-data:3.1.0                                      edgex-core-data
edgexfoundry/core-command:3.1.0                                   edgex-core-command
edgexfoundry/support-scheduler:3.1.0                              edgex-support-scheduler
edgexfoundry/core-common-config-bootstrapper:3.1.0                edgex-core-common-config-bootstrapper
edgexfoundry/support-notifications:3.1.0                          edgex-support-notifications
edgexfoundry/core-metadata:3.1.0                                  edgex-core-metadata
eclipse-mosquitto:2.0.18                                          edgex-mqtt-broker
edgexfoundry/edgex-ui:3.1.0                                       edgex-ui-go
redis:7.0.14-alpine                                               edgex-redis
hashicorp/consul:1.16.2                                           edgex-core-consul
OpenVINO Model Server 日志
 
# docker logs -f model_server
 
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][kfs_grpc_inference_service.cpp:251] Processing gRPC request for model: ssdlite_mobilenet_v2; version: 1
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][kfs_grpc_inference_service.cpp:290] ModelInfer requested name: ssdlite_mobilenet_v2, version: 1
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][modelmanager.cpp:1519] Requesting model: ssdlite_mobilenet_v2; version: 1.
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][modelinstance.cpp:1054] Model: ssdlite_mobilenet_v2, version: 1 already loaded
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][predict_request_validation_utils.cpp:1035] [servable name: ssdlite_mobilenet_v2 version: 1] Validating request containing binary image input: name: image_tensor
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][modelinstance.cpp:1234] Getting infer req duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.022 ms
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][modelinstance.cpp:1242] Preprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.941][131][serving][debug][deserialization.hpp:449] Request contains input in native file format: image_tensor
model_server  | [2024-04-11 04:29:29.943][131][serving][debug][modelinstance.cpp:1252] Deserialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 2.492 ms
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][kfs_grpc_inference_service.cpp:251] Processing gRPC request for model: ssdlite_mobilenet_v2; version: 1
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][kfs_grpc_inference_service.cpp:290] ModelInfer requested name: ssdlite_mobilenet_v2, version: 1
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][modelmanager.cpp:1519] Requesting model: ssdlite_mobilenet_v2; version: 1.
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][modelinstance.cpp:1054] Model: ssdlite_mobilenet_v2, version: 1 already loaded
model_server  | [2024-04-11 04:29:29.948][128][serving][debug][predict_request_validation_utils.cpp:1035] [servable name: ssdlite_mobilenet_v2 version: 1] Validating request containing binary image input: name: image_tensor
model_server  | [2024-04-11 04:29:29.951][131][serving][debug][modelinstance.cpp:1260] Prediction duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 7.547 ms
model_server  | [2024-04-11 04:29:29.951][131][serving][debug][modelinstance.cpp:1269] Serialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.226 ms
model_server  | [2024-04-11 04:29:29.951][131][serving][debug][modelinstance.cpp:1277] Postprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.951][131][serving][debug][kfs_grpc_inference_service.cpp:271] Total gRPC request processing time: 10.4070 ms
model_server  | [2024-04-11 04:29:29.951][128][serving][debug][modelinstance.cpp:1234] Getting infer req duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 2.850 ms
model_server  | [2024-04-11 04:29:29.951][128][serving][debug][modelinstance.cpp:1242] Preprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.951][128][serving][debug][deserialization.hpp:449] Request contains input in native file format: image_tensor
model_server  | [2024-04-11 04:29:29.953][128][serving][debug][modelinstance.cpp:1252] Deserialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 2.192 ms
model_server  | [2024-04-11 04:29:29.961][128][serving][debug][modelinstance.cpp:1260] Prediction duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 7.519 ms
model_server  | [2024-04-11 04:29:29.961][128][serving][debug][modelinstance.cpp:1269] Serialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.052 ms
model_server  | [2024-04-11 04:29:29.961][128][serving][debug][modelinstance.cpp:1277] Postprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.961][128][serving][debug][kfs_grpc_inference_service.cpp:271] Total gRPC request processing time: 12.787 ms
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][kfs_grpc_inference_service.cpp:251] Processing gRPC request for model: ssdlite_mobilenet_v2; version: 1
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][kfs_grpc_inference_service.cpp:290] ModelInfer requested name: ssdlite_mobilenet_v2, version: 1
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][modelmanager.cpp:1519] Requesting model: ssdlite_mobilenet_v2; version: 1.
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][modelinstance.cpp:1054] Model: ssdlite_mobilenet_v2, version: 1 already loaded
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][predict_request_validation_utils.cpp:1035] [servable name: ssdlite_mobilenet_v2 version: 1] Validating request containing binary image input: name: image_tensor
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][modelinstance.cpp:1234] Getting infer req duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.004 ms
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][modelinstance.cpp:1242] Preprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.973][132][serving][debug][deserialization.hpp:449] Request contains input in native file format: image_tensor
model_server  | [2024-04-11 04:29:29.976][132][serving][debug][modelinstance.cpp:1252] Deserialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 2.705 ms
model_server  | [2024-04-11 04:29:29.983][132][serving][debug][modelinstance.cpp:1260] Prediction duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 7.461 ms
model_server  | [2024-04-11 04:29:29.983][132][serving][debug][modelinstance.cpp:1269] Serialization duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.049 ms
model_server  | [2024-04-11 04:29:29.983][132][serving][debug][modelinstance.cpp:1277] Postprocessing duration in model ssdlite_mobilenet_v2, version 1, nireq 0: 0.000 ms
model_server  | [2024-04-11 04:29:29.983][132][serving][debug][kfs_grpc_inference_service.cpp:271] Total gRPC request processing time: 10.4050 ms


可以看到推理使用设备:Used device: CPU,也可以在 config.json 配置文件中修改为: AUTO 或 GPU
整个请求所花的时间:Total gRPC request processing time: 10.4050 ms,综合平均值,也就是10-15ms 之间,性能非常好。
推理结果
通过 Web MJPEG 方式实时查看推理结果。

0个评论