# Jetson-Opencv-Gstreamer-Ncnn **Repository Path**: jia0510/jetson-opencv-gstreamer-ncnn ## Basic Information - **Project Name**: Jetson-Opencv-Gstreamer-Ncnn - **Description**: 本着共享的原则，为了给大家避坑，由于在边缘计算中用到视频硬解码，用到NVIDIA公司gstream插件，该插件已经结合在OPENCV中，只需要在编译时打开该功能，设置相关参数，做AI识别时用到腾讯公司ncnn向前推理框架。 - **Primary Language**: C++ - **License**: Apache-2.0 - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 1 - **Forks**: 1 - **Created**: 2021-03-19 - **Last Updated**: 2024-03-01 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # Jetson-Opencv-Gstreamer-Ncnn-RTMP 本着共享的原则，为了给大家避坑，本项目在jetson nano上对海康摄像头RTSP流通过Gstreamer管道进行硬解码，做AI识别时用到腾讯公司ncnn向前推理框架，输出结果后，通过Gstreamer压缩H.264后推出RTMP流，具体过程看yolov4代码。 由于Gstream已经结合在OPENCV中，只需要在编译时打开该功能，设置WITH_GSTREAMER=ON即可，如果想要python也能使用，编译同样要打开参数BUILD_opencv_python3=ON； # Gstreamer介绍 Gstreamer是一个用于开发流式多媒体应用的开源框架，采用了基于插件（plugin）和管道（pipeline）的体系结构，框架中的所有的功能模块都被实现成可以插拔的组件（component）， 并且能够很方便地安装到任意一个管道上。由于所有插件都通过管道机制进行统一的数据交换，因此很容易利用已有的各种插件“组装”出一个功能完善的多媒体应用程序。可以自行百度了解。 # 安装介绍 + Gstreamer 1.10.4 + opencv4.5.2 + NCNN最新版 + Jetson nano arch64 + Cuda10.1 此项Jetson中默认已安装 # 安装教程 # 1. Gstreamer 1.10.4安装 在jetson上通过以下命令流程安装 Gstreamer-1.0 平台: ``` sudo add-apt-repository universe sudo add-apt-repository multiverse sudo apt-get update sudo apt-get install gstreamer1.0-tools gstreamer1.0-alsa gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav sudo apt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-good1.0-dev libgstreamer-plugins-bad1.0-dev ``` 通过以下命令检查 Gstreamer-1.0 版本: gst-inspect-1.0 --version 通过以下命令通过NVIDIA 加速进行硬解码 ``` gst-launch-1.0 rtspsrc location=rtsp://admin:ubo12345@192.168.1.100:554/h264/ch1/main/av_stream latency=0 drop-on-latency=true max-size-buffers=0 ! decodebin ! nvvidconv flip-method=0 ! nvoverlaysink overlay-x=0 overlay-y=0 overlay-w=1280 overlay-h=720 sync=false -e ``` 通过sudo jtop查看cpu gpu使用情况 ``` sudo pip3 install jetson-stats ``` # 2. opencv4.5.2安装 自行下载安装包：https://github.com/opencv/opencv/releases/tag/4.5.2 如需卸载请使用以下命令： ``` sudo rm -r /usr/local/include/opencv2 /usr/local/include/opencv /usr/include/opencv /usr/include/opencv2 /usr/local/share/opencv /usr/local/share/OpenCV /usr/share/opencv /usr/share/OpenCV /usr/local/bin/opencv* /usr/local/lib/libopencv* ``` OpenCV依赖库的安装 ``` sudo apt-get install cmake sudo apt-get install libtbb-dev sudo apt-get install build-essential libgtk2.0-dev libavcodec-dev sudo apt-get install libavformat-dev libjpeg.dev libtiff4.dev sudo apt-get install libswscale-dev libjasper-dev sudo apt-get install libjpeg62-dev sudo apt-get install v4l2ucp v4l-utils sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler sudo apt-get install --no-install-recommends libboost-all-dev sudo apt-get install libopenblas-dev liblapack-dev libatlas-base-dev sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev ``` 进入opencv-4.5.2目录，创建build文件 ``` mkdir build cd build ``` 执行cmake命令 ``` cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D INSTALL_C_EXAMPLES=OFF -D WITH_GSTREAMER=ON -D WITH_GTK_2_X=ON -D WITH_GTHREAD=ON -D WITH_TBB=ON -D WITH_OPENGL=ON -DBUILD_opencv_python3=ON -D OPENCV_GENERATE_PKGCONFIG=1 .. ``` 执行cmake命令-带扩展库 ``` cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-4.5.2/modules/ -D INSTALL_C_EXAMPLES=OFF -D WITH_GSTREAMER=ON -D WITH_GTK_2_X=ON -D WITH_GTHREAD=ON -D WITH_TBB=ON -D WITH_OPENGL=ON -DBUILD_opencv_python3=ON -D OPENCV_GENERATE_PKGCONFIG=1 .. ``` 编译并安装进local目录 ``` make -j 4 # -j4 四核编译 sudo make install (注意一定要加sudo) pkg-config --modversion opencv4 # 查看OpenCV版本 ``` 路径配置 将OpenCV的库添加到路径，从而可以让系统找到 ``` sudo gedit /etc/ld.so.conf.d/opencv.conf ``` 执行此命令后打开的可能是一个空白的文件，不用管，只需要在文件末尾添加 ``` /usr/local/lib ``` 然后保存退出，执行如下命令使得刚才的配置路径生 ``` sudo ldconfig ``` 配置bash，执行如下命 ``` sudo gedit /etc/bash.bashrc ``` 在末尾添 ``` export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig ``` 执行命令 ``` sudo source /etc/bash.bashrc ``` 测试cpp文件 ``` g++ test_gstream.cpp `pkg-config --cflags --libs opencv4` --std=c++11 ``` 测试python文件 ``` python3 test-gstreamer-tegra-rtsp.py ``` # 3. Ncnn在Jetson安装 更新一下环境 ``` $ sudo apt-get update $ sudo apt-get upgrade ``` 安装依赖项 ``` $ sudo apt-get install build-essential git cmake $ sudo apt-get install libprotobuf-dev protobuf-compiler libvulkan-dev vulkan-utils libopencv-dev ``` 安装一些依赖项，cmake、protobuf、vulkan等等； 下载ncnn源码 ``` $ git clone --depth=1 https://github.com/Tencent/ncnn.git ``` 用git clone --depth=1的好处是限制 clone 的深度，不会下载 Git 协作的历史记录，这样可以大大加快克隆的速度 下载glslang ``` $ cd ncnn $ git submodule update --depth=1 --init ``` 克隆项目后，默认子模块目录下无任何内容。需要在项目根目录执行git submodule update完成子模块的下载 ncnn中的submodule指的是glslang（我并不知道它是个什么东东。。An OpenGL and OpenGL ES shader front end and validator） 开始编译 ``` $ mkdir build $ cd build $ cmake -DCMAKE_TOOLCHAIN_FILE=../toolchains/jetson.toolchain.cmake -DNCNN_BUILD_EXAMPLES=ON -DNCNN_VULKAN=ON -DNCNN_SYSTEM_GLSLANG=ON -DNCNN_SHARED_LIB=ON -DNCNN_BUILD_TOOLS=ON -DCMAKE_BUILD_TYPE=Release .. $ make -j4 $ sudo make install ``` 以上，在jetson nano中完成了ncnn的配置，其中设置-DNCNN_VULKAN=ON，vulkan会 利用GPU加速，加快运行结果。 模型转换Convert cfg and weights: ``` $ ./darknet2ncnn yolov4-tiny.cfg yolov4-tiny.weights yolov4-tiny.param yolov4-tiny.bin 1 ``` 如果成功将会输出: ``` Loading cfg... WARNING: The ignore_thresh=0.700000 of yolo0 is too high. An alternative value 0.25 is written instead. WARNING: The ignore_thresh=0.700000 of yolo1 is too high. An alternative value 0.25 is written instead. Loading weights... Converting model... 83 layers, 91 blobs generated. NOTE: The input of darknet uses: mean_vals=0 and norm_vals=1/255.f. NOTE: Remeber to use ncnnoptimize for better performance. ``` 模型优化Optimize graphic ``` $ ./ncnnoptimize yolov4-tiny.param yolov4-tiny.bin yolov4-tiny-opt.param yolov4-tiny-opt.bin 0 ``` # 使用说明 1. 将以上yolov4.cpp文件放入ncnn中example再次进行编译，之后运行build目录./yolov4。 # 参考 + 1.TX2+GStreamer+OpenCV读取保存rtsp视频流数据 https://blog.csdn.net/zong596568821xp/article/details/80405816 + 2.Jetson之GStreamer+OpenCV读取显示摄像头https://zongxp.blog.csdn.net/article/details/80306987 + 3.在jetson nano上配置ncnn https://zhuanlan.zhihu.com/p/285594861 + 4.Jetson之GStreamer+OpenCV读取显示摄像头https://blog.csdn.net/zong596568821xp/article/details/80306987 + 5.使用Gstreamer处理RTSP视频流 https://blog.csdn.net/han2529386161/article/details/102724856?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-3.control&dist_request_id=&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-3.control + 6.Ubuntu下OpenCV环境配置https://blog.csdn.net/zong596568821xp/article/details/80393810 + 7.gstreamer_learn https://gitee.com/tosonw/gstreamer_learn?_from=gitee_search + 8.Jetson之opencv硬件编码输出rtsp https://blog.csdn.net/zong596568821xp/article/details/108492308?spm=1001.2014.3001.5501 + 9.gstreamer读取USB摄像头H264帧并用rtmp推流 https://blog.csdn.net/zhaoyun_zzz/article/details/86496621 +10.Darknet To NCNN https://github.com/Tencent/ncnn/tree/master/tools/darknet