#AI挑战营终点站#RV1106使用rknn进行MNIST多数字实时识别
<div class='showpostmsg'> 本帖最后由 LitchiCheng 于 2024-6-1 00:15 编辑<article data-content="[{"type":"block","id":"3060-1621846615933","name":"paragraph","data":{},"nodes":[{"type":"text","id":"p5PQ-1621846617594","leaves":[{"text":"首先给出工程打包仓库地址，参考官网yolov5-rtsp示例","marks":[]}]}],"state":{}},{"type":"block","id":"hdNG-1717167860102","name":"paragraph","data":{},"nodes":[{"type":"text","id":"IVLT-1717167860101","leaves":[{"text":"git@github.com:LitchiCheng/RV1106_Linux.git","marks":[]}]}],"state":{}},{"type":"block","id":"8qby-1717167930539","name":"paragraph","data":{},"nodes":[{"type":"text","id":"b5Rq-1717167930538","leaves":[{"text":"包含内容如下：","marks":[]}]}],"state":{}},{"type":"block","id":"uLDV-1717167940849","name":"paragraph","data":{"style":{}},"nodes":[{"type":"text","id":"uVuJ-1717167940848","leaves":[{"text":"1. v4l2封装调用库工程","marks":[]}]}],"state":{"index":1}},{"type":"block","id":"XEJd-1717167965202","name":"paragraph","data":{"style":{}},"nodes":[{"type":"text","id":"lAiq-1717167965201","leaves":[{"text":"2. st7735 LCD播放badapple工程","marks":[]}]}],"state":{"index":2}},{"type":"block","id":"ngfY-1717168010718","name":"paragraph","data":{"style":{}},"nodes":[{"type":"text","id":"qLCz-1717168010717","leaves":[{"text":"3. MNIST多数字实时识别工程","marks":[]}]}],"state":{"index":3}},{"type":"block","id":"ozVN-1717168032037","name":"paragraph","data":{"style":{}},"nodes":[{"type":"text","id":"SFFK-1717168032036","leaves":[{"text":"4. rkrtsp推流测试工程","marks":[]}]}],"state":{"index":4}},{"type":"block","id":"4mMn-1717168048058","name":"paragraph","data":{"style":{}},"nodes":[{"type":"text","id":"RYT1-1717168048057","leaves":[{"text":"5. rkmpi csi摄像头测试工程","marks":[]}]}],"state":{"index":5}},{"type":"block","id":"vnlN-1717167990229","name":"paragraph","data":{"style":{}},"nodes":[{"type":"text","id":"Ug2p-1717167990228","leaves":[{"text":"6. yolov5单帧识别工程","marks":[]}]}],"state":{"index":6}},{"type":"block","id":"NL4r-1717167859041","name":"paragraph","data":{},"nodes":[{"type":"text","id":"cvTj-1717167859040","leaves":[{"text":"","marks":[]}]}],"state":{}},{"type":"block","id":"csV2-1717168114523","name":"paragraph","data":{},"nodes":[{"type":"text","id":"ZoCV-1717168114522","leaves":[{"text":"如下直接介绍MNIST多数字实时识别思路：","marks":[]}]}],"state":{}},{"type":"block","id":"ZjG7-1717168134158","name":"list-item","data":{"listId":"Txts-1717168135041","listType":"ordered","listLevel":1,"style":{}},"nodes":[{"type":"text","id":"ieFv-1717168134156","leaves":[{"text":"使用rkmpi库进行frame的抓取","marks":[]}]}],"state":{"index":1}},{"type":"block","id":"Tzxp-1717168162563","name":"list-item","data":{"listId":"Txts-1717168135041","listType":"ordered","listLevel":1,"style":{}},"nodes":[{"type":"text","id":"d6Il-1717168162561","leaves":[{"text":"使用opencv-mobile库进行单帧frame进行处理，通过二值化、反相、腐蚀、膨胀等处理，获得多个数字的框选位置","marks":[]}]}],"state":{"index":2}},{"type":"block","id":"04Xj-1717168257446","name":"list-item","data":{"listId":"Txts-1717168135041","listType":"ordered","listLevel":1,"style":{}},"nodes":[{"type":"text","id":"Dwnn-1717168257445","leaves":[{"text":"使用mnist训练且转换后的rknn模型单独对框选位置的图片进行推理，得到该框选位置的概率以及数字","marks":[]}]}],"state":{"index":3}},{"type":"block","id":"ZP0w-1717168297783","name":"list-item","data":{"listId":"Txts-1717168135041","listType":"ordered","listLevel":1,"style":{}},"nodes":[{"type":"text","id":"YiCj-1717168297782","leaves":[{"text":"在frame上通过opencv-mobile进行画框以及标记数字及概率","marks":[]}]}],"state":{"index":4}},{"type":"block","id":"qJaY-1717168329672","name":"list-item","data":{"listId":"Txts-1717168135041","listType":"ordered","listLevel":1,"style":{}},"nodes":[{"type":"text","id":"3GtB-1717168329671","leaves":[{"text":"使用rkrtsp进行推流","marks":[]}]}],"state":{"index":5}},{"type":"block","id":"spyG-1717168348941","name":"paragraph","data":{},"nodes":[{"type":"text","id":"JMI8-1717168348940","leaves":[{"text":"","marks":[]}]}],"state":{}},{"type":"block","id":"Zaol-1717168349759","name":"paragraph","data":{},"nodes":[{"type":"text","id":"guSl-1717168349758","leaves":[{"text":"如下为代码关键部分","marks":[]}]}],"state":{}},{"type":"block","id":"fMuQ-1717168364104","name":"paragraph","data":{},"nodes":[{"type":"text","id":"gn4t-1717168364103","leaves":[{"text":"一、获取frame、处理、推理、标记","marks":[]}]}],"state":{}}]">
<p> </p>
<p>首先给出工程打包仓库地址，参考官网yolov5-rtsp示例</p>
<p>git@github.com:LitchiCheng/RV1106_Linux.git</p>
<p>包含内容如下：</p>
<p>1. v4l2封装调用库工程</p>
<p>2. st7735 LCD播放badapple工程</p>
<p>3. MNIST多数字实时识别工程</p>
<p>4. rkrtsp推流测试工程</p>
<p>5. rkmpi csi摄像头测试工程</p>
<p>6. yolov5单帧识别工程</p>
<p> </p>
<p>RV1106其他的测试帖子链接：</p>
<p><a href="https://bbs.eeworld.com.cn/thread-1270526-1-1.html" target="_blank">LUCKFOX Pico rv1106开发板烧录镜像教程</a></p>
<p><a href="https://bbs.eeworld.com.cn/thread-1270648-1-1.html" target="_blank">rv1106开发板上使用yolov5转换rknn模型进行图像识别的推理实测</a></p>
<p><a href="https://bbs.eeworld.com.cn/thread-1271075-1-1.html" target="_blank">rv1106开发板配置TypeC接口USB Host并识别USB设备</a></p>
<p><a href="https://bbs.eeworld.com.cn/thread-1271088-1-1.html" target="_blank">rv1106开发板buildroot下使用v4l2和fswebcam进行USB摄像头测试</a></p>
<p><a href="https://bbs.eeworld.com.cn/thread-1271609-1-1.html" target="_blank">RV1106手把手教你：yolov5图像识别模型从pt转换到onnx再到rknn</a></p>
<p><a href="https://bbs.eeworld.com.cn/thread-1271898-1-1.html" target="_blank">RV1106手把手教你：ffmpeg无界面使用USB摄像头录制视频</a></p>
<p><a href="https://bbs.eeworld.com.cn/thread-1272578-1-1.html" target="_blank">RV1106 Badapple纯享版</a></p>
<p><a href="https://bbs.eeworld.com.cn/thread-1272577-1-1.html" target="_blank">RV1106手把手教你：使用ffmpeg、framebuffer在ST7735屏幕上播放bad apple</a></p>
<p><a href="https://bbs.eeworld.com.cn/thread-1274253-1-1.html" target="_blank">RV1106手把手教你：惊呆了！USB摄像头秒变AI助手，rknn轻松拍照做yolov5推理！</a> </p>
<p><a href="https://bbs.eeworld.com.cn/thread-1278341-1-1.html" target="_blank">#AI挑战营第一站#pytorch训练MNIST数据集实现手写数字识别</a></p>
<p> </p>
<p>如下直接介绍MNIST多数字实时识别思路：</p>
<ol yne-block-type="list">
<li>使用rkmpi库进行frame的抓取</li>
<li>使用opencv-mobile库进行单帧frame进行处理，通过二值化、反相、腐蚀、膨胀等处理，获得多个数字的框选位置</li>
<li>使用mnist训练且转换后的rknn模型单独对框选位置的图片进行推理，得到该框选位置的概率以及数字</li>
<li>在frame上通过opencv-mobile进行画框以及标记数字及概率</li>
<li>使用rkrtsp进行推流</li>
</ol>
<p>如下为代码关键部分</p>
<p>一、获取frame、处理、推理、标记</p>
<pre>
<code>void *data = RK_MPI_MB_Handle2VirAddr(stVpssFrame.stVFrame.pMbBlk);
cv::Mat frame(height,width,CV_8UC3,data);
std::vector<cv::Rect> rects;
std::vector<detect_result> detect_results;
std::vector<cv::Mat> sub_pics;
rects = find_contour(frame, sub_pics);
for(int i = 0; i < rects.size(); i++){
if (rects.area() > 0){
inference_mnist_model(&rknn_app_ctx, sub_pics, detect_results);
if (!detect_results.empty()){
detect_result result = detect_results.back();
//绿框，thickness为1
cv::rectangle(frame, rects, cv::Scalar(0, 255, 0), 1);
//红字，fontscale为1，thickness为1
cv::putText(frame, std::to_string(result.num), cv::Point(rects.x, rects.y + 10),
cv::FONT_HERSHEY_SIMPLEX, 1, cv::Scalar(255, 0, 0), 1);
//蓝字，fontscale为1，thickness为1
cv::putText(frame, std::to_string(result.probability), cv::Point(rects.x+ 30, rects.y + 10),
cv::FONT_HERSHEY_SIMPLEX, 1, cv::Scalar(0, 0, 255), 1);
detect_results.pop_back();
}
}
}</code></pre>
<article data-content="[{"type":"block","id":"rGQG-1717168455903","name":"paragraph","data":{},"nodes":[{"type":"text","id":"jxT3-1717168455904","leaves":[{"text":"二、单帧图像处理","marks":[]}]}],"state":{}}]">
<p>二、单帧图像处理</p>
<pre>
<code>std::vector<cv::Rect> find_contour(const cv::Mat &image, std::vector<cv::Mat>& sub_pics) {
// 预处理图像
cv::Mat gray, edged, org_clone;
org_clone = image.clone();
// 转成灰度图
cv::cvtColor(image, gray, cv::COLOR_BGR2GRAY);
// 反相灰度图
edged = ~gray;
cv::Mat kernel = cv::getStructuringElement(cv::MORPH_RECT, cv::Size(3, 3));
// 边缘膨胀
cv::dilate(edged, edged, kernel);
// 腐蚀
cv::erode(edged, edged, kernel);
// 二值化
cv::threshold(edged, edged, 127, 0, cv::THRESH_TOZERO);
// 用来存找到的轮廓
std::vector<std::vector<cv::Point>> contours;
cv::findContours(edged, contours, cv::RETR_EXTERNAL, cv::CHAIN_APPROX_SIMPLE);
std::vector<cv::Rect> borders;
for(auto contour: contours){
cv::Rect bounding_box = cv::boundingRect(contour);
//当前矩形框大于30才使用
if (cv::contourArea(contour) > 30) {
//扩大矩形框，以防数字被截断
bounding_box.x = std::max(0, bounding_box.x - 10);
bounding_box.y = std::max(0, bounding_box.y - 10);
bounding_box.width = std::min(image.cols - bounding_box.x, bounding_box.width + 20);
bounding_box.height = std::min(image.rows - bounding_box.y, bounding_box.height + 20);
borders.push_back(bounding_box);
cv::Mat postsub;
cv::Mat sub = edged(bounding_box);
// 扩大数字轮廓
cv::threshold(sub, sub, 0, 255, cv::THRESH_BINARY | cv::THRESH_OTSU);
// // 将图片大小调整为28
cv::resize(sub, postsub, cv::Size(28, 28), 0, 0, cv::INTER_AREA);
sub_pics.push_back(postsub);
}
}
return borders;
}</code></pre>
<article data-content="[{"type":"block","id":"3Cor-1717168548870","name":"paragraph","data":{},"nodes":[{"type":"text","id":"342b-1717168548869","leaves":[{"text":"三、推理函数","marks":[]}]}],"state":{}}]">
<p>三、推理函数</p>
<pre>
<code>static float deqnt_affine_to_f32(int8_t qnt, int32_t zp, float scale) { return ((float)qnt - (float)zp) * scale; }
int inference_mnist_model(rknn_app_context_t* app_ctx, cv::Mat &frame, std::vector<detect_result>& results)
{
int ret;
int width = app_ctx->input_attrs.dims;
int stride = app_ctx->input_attrs.w_stride;
if (width == stride){
memcpy(app_ctx->input_mems->virt_addr, frame.data, width * app_ctx->input_attrs.dims * app_ctx->input_attrs.dims);
}else{
int height = app_ctx->input_attrs.dims;
int channel = app_ctx->input_attrs.dims;
// copy from src to dst with stride
uint8_t *src_ptr = frame.data;
uint8_t *dst_ptr = (uint8_t *)app_ctx->input_mems->virt_addr;
// width-channel elements
int src_wc_elems = width * channel;
int dst_wc_elems = stride * channel;
for (int h = 0; h < height; ++h){
memcpy(dst_ptr, src_ptr, src_wc_elems);
src_ptr += src_wc_elems;
dst_ptr += dst_wc_elems;
}
}
ret = rknn_run(app_ctx->rknn_ctx, nullptr);
if (ret < 0) {
printf("rknn_run fail! ret=%d\n", ret);
return -1;
}
#define DETECT_NUM_SIZE 10
// Post Process
uint8_t*output= (uint8_t*)malloc(sizeof(uint8_t) * DETECT_NUM_SIZE);
float *out_fp32 = (float*)malloc(sizeof(float) * DETECT_NUM_SIZE);
output = (uint8_t *)app_ctx->output_mems->virt_addr;
int32_t zp =app_ctx->output_attrs.zp;
float scale = app_ctx->output_attrs.scale;
//反量化为浮点数
for(int i = 0; i < DETECT_NUM_SIZE; i ++)
out_fp32 = deqnt_affine_to_f32(output,zp,scale);
//归一化
float sum = 0;
for(int i = 0; i < DETECT_NUM_SIZE; i++)
sum += out_fp32 * out_fp32;
float norm = sqrt(sum);
for(int i = 0; i < DETECT_NUM_SIZE; i++)
out_fp32 /= norm;
//对概率进行排序
float max_probability = -1.0;
int detect_num = -1;
for (int i = 0; i < DETECT_NUM_SIZE; ++i){
if (out_fp32 > max_probability){
max_probability = out_fp32;
detect_num = i;
}
}
//结果入列
results.push_back({detect_num, max_probability});
return 0;
}</code></pre>
<article data-content="[{"type":"block","id":"7iob-1717168598344","name":"paragraph","data":{},"nodes":[{"type":"text","id":"yd6D-1717168598342","leaves":[{"text":"工程编译","marks":[]}]}],"state":{}}]">
<p>工程编译</p>
<pre>
<code>cd MNIST
mkdri build
cd build
cmake ..
make
make install
scp -r ../rtsp_mnist_test root@xxx.xxx.xxx.xxx:~</code></pre>
<p> </p>
<p>目标板运行</p>
<pre>
<code>killall rkipc
cd rtsp_mnist_test
./rtsp_mnist model/mnist.rknn</code></pre>
<p> VLC拉流实时显示</p>
<article data-content="[{"type":"block","id":"lzWO-1717168788687","name":"paragraph","data":{},"nodes":[{"type":"text","id":"TL0A-1717168788685","leaves":[{"text":"VLC拉流","marks":[]}]}],"state":{}}]">
<p> </p>
<p> </p>
<p> </p>
<p>视频分享</p>
<p><iframe allowfullscreen="true" frameborder="0" height="450" src="//player.bilibili.com/player.html?bvid=1Sy411h7ZN&page=1" style="background:#eee;margin-bottom:10px;" width="700"></iframe><br />
</p>
</article>
</article>
</article>
</article>
</article>
</div><script> var loginstr = '<div class="locked">查看本帖全部内容，请<a href="javascript:;" style="color:#e60000" class="loginf">登录</a>或者<a href="https://bbs.eeworld.com.cn/member.php?mod=register_eeworld.php&action=wechat" style="color:#e60000" target="_blank">注册</a></div>';
if(parseInt(discuz_uid)==0){
(function($){
var postHeight = getTextHeight(400);
$(".showpostmsg").html($(".showpostmsg").html());
$(".showpostmsg").after(loginstr);
$(".showpostmsg").css({height:postHeight,overflow:"hidden"});
})(jQuery);
} </script><script type="text/javascript">(function(d,c){var a=d.createElement("script"),m=d.getElementsByTagName("script"),eewurl="//counter.eeworld.com.cn/pv/count/";a.src=eewurl+c;m.parentNode.insertBefore(a,m)})(document,523)</script> <p>赞，学习了</p>
页:
[1]