# MultiSourceReconstruction **Repository Path**: remvs/msr ## Basic Information - **Project Name**: MultiSourceReconstruction - **Description**: 多源数据三维重建 - **Primary Language**: Python - **License**: BSD-3-Clause - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 2 - **Forks**: 2 - **Created**: 2021-07-08 - **Last Updated**: 2022-10-29 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # 多源数据的注册与融合 本工程受**国家重点研发计划“云计算与大数据”重点专项项目——“多源数据驱动的智能化高效场景建模与绘制引擎”(2017YFB1002601)**资助，为该项目中课题一“复杂场景多源数据分析融合与联合优化”的子课题。 ![pipeline](output/calib_dataflow.png) ## 数据 1. 图片数据有无人机航拍的数据图片，和地面单反相机拍摄的图片。 2. 激光点云数据来自于车载激光雷达，32线激光，Ladybug全景相机。 ## 安装依赖 * python ```bash conda create -n recons python=3.6 conda activate recons pip install numpy scipy matplotlib opencv-python plyfile python-pcl exifread pip install flask flask-wtf ``` * cmake 3.20 ```bash wget -c https://github.com/Kitware/CMake/releases/download/v3.20.4/cmake-3.20.4.tar.gz tar -zxvf cmake-3.20.4.tar.gz cd cmake-3.20.4 ./bootstrap make -j sudo make install ``` * CUDA >= 10.1 ```bash # ref: ``` * Eigen >= 3.3 ```bash wget -c https://gitlab.com/libeigen/eigen/-/archive/3.3.9/eigen-3.3.9.tar.gz tar -zxvf eigen-3.3.9.tar.gz cd eigen-3.3.9 mkdir build cd build cmake .. sudo make install ``` * OpenCV 3.4.14 ```bash # ref: https://docs.opencv.org/master/d7/d9f/tutorial_linux_install.html # support gui sudo apt-get install libgtk2.0-dev libgtk-3-dev # support ffmpeg sudo apt install ffmpeg libavcodec-dev libavformat-dev libavutil-dev libswscale-dev libavresample-dev libdc1394-22-dev libv4l-dev # build opencv wget -c https://github.com/opencv/opencv/archive/3.4.14.zip unzip 3.4.14.zip cd opencv-3.4.14 mkdir build cd build cmake .. make -j sudo make install ## trouble shooting: ippicv # disable ippicv cmake -DWITH_IPP=OFF .. # or manual download cat CMakeDownloadLog.txt # ref: https://yinguobing.com/cache-ippicv-manually/ ``` * Colmap ```bash # ref: https://colmap.github.io/install.html # dependencies sudo apt install \ git \ cmake \ build-essential \ libboost-program-options-dev \ libboost-filesystem-dev \ libboost-graph-dev \ libboost-system-dev \ libboost-test-dev \ libeigen3-dev \ libsuitesparse-dev \ libfreeimage-dev \ libgoogle-glog-dev \ libgflags-dev \ libglew-dev \ qtbase5-dev \ libqt5opengl5-dev \ libcgal-dev \ libcgal-qt5-dev # ceres-solver sudo apt-get install libatlas-base-dev libsuitesparse-dev git clone https://ceres-solver.googlesource.com/ceres-solver cd ceres-solver git checkout $(git describe --tags) # Checkout the latest release mkdir build cd build cmake .. -DBUILD_TESTING=OFF -DBUILD_EXAMPLES=OFF make -j sudo make install # trouble shooting: hang on add commit hook to gerrit # comment the following two lines in ../CMakeLists.txt #include(AddGerritCommitHook) #add_gerrit_commit_hook(${Ceres_SOURCE_DIR} ${Ceres_BINARY_DIR}) # trouble shooting: undefined reference to libtiff4.0 if using anaconda conda uninstall libtiff # trouble shooting: No rule to make target 'libmkl_intel_lp64.so', needed by 'src/exe/colmap'. conda install libmkl # colmap git clone https://github.com/colmap/colmap.git cd colmap git checkout dev mkdir build cd build cmake .. make -j sudo make install # check colmap -h colmap gui ``` * cloudcompare ```bash sudo snap install core sudo snap install cloudcompare ``` * OpenGL ```bash sudo apt install build-essential libgl1-mesa-dev libglu1-mesa-dev freeglut3-dev libglm-dev sudo apt install libglew-dev libglfw3 libglfw3-dev ``` * Glog ```bash sudo apt install libgoogle-glog-dev ``` * Qt 5.12 ```bash # use TUNA mirror wget -c https://mirrors.tuna.tsinghua.edu.cn/qt/official_releases/qt/5.12/5.12.11/qt-opensource-linux-x64-5.12.11.run chmod u+x ./qt-opensource-linux-x64-5.12.11.run # need desktop sudo ./qt-opensource-linux-x64-5.12.11.run ``` * PCL 1.11 ```bash # dependencies sudo apt install \ libeigen3-dev \ libflann-dev \ libglu1-mesa-dev \ freeglut3-dev \ mesa-common-dev \ libboost-all-dev \ libusb-1.0-0-dev \ libusb-dev \ libopenni-dev \ libopenni2-dev \ libpcap-dev \ libpng-dev \ mpi-default-dev \ openmpi-bin \ openmpi-common \ libqhull-dev \ libgtest-dev # download PCL PCL_VERSION="1.11.0" wget -c https://github.com/PointCloudLibrary/pcl/archive/pcl-${PCL_VERSION}.tar.gz tar -xf pcl-${PCL_VERSION}.tar.gz cd pcl-pcl-${PCL_VERSION} # build mkdir build cd build cmake .. -DCMAKE_BUILD_TYPE=Release -DVTK_RENDERING_BACKEND=OpenGL2 make -j sudo make install sudo apt install pcl-tools unset PCL_VERSION ``` * Sophus latest ```bash git clone https://github.com/strasdat/Sophus.git cd Sophus mkdir build cd build cmake .. make -j sudo make install ``` * fmt 7.1.3 ```bash wget -c https://github.com/fmtlib/fmt/archive/refs/tags/7.1.3.tar.gz tar -zxvf 7.1.3.tar.gz cd fmt-7.1.3 mkdir build cd build cmake -DBUILD_SHARED_LIBS=TRUE -DBUILD_SHARED_LIBS=TRUE .. ``` ### Install * VelodyneLadybugFusion ```bash cd VelodyneLadybugFusion mkdir build cd build cmake .. make -j ``` * GPS2XYZ ```bash cd GPS2XYZ mkdir build cd build cmake .. make -j ``` * PSDF ```bash cd PSDF mkdir build cd build cmake .. make -j ``` ## 流程 ### 图片的三维重建 >输入：多视角图片 (data/image_data/images/) > >输出：稀疏模型(data/image_data/sparse/)、稠密模型(data/image_data/dense/) 使用colmap对图片进行重建，使用脚本reconstruction.sh。修改reconstruction.sh里的DATASET_PATH和EXE_PATH，分别是重建数据的根目录和colmap可执行文件的目录。 如图，将重建所需的图片放到images文件夹下，其余文件夹都是程序自动生成的。sparse文件夹下是稀疏重建结果，dense文件夹下是稠密重建结果。 ```shell ./run_colmap.sh ``` ### 激光点云配准（未开源） >输入：每帧的激光点云(data/lidar_data/.pcap文件)、Ladybug全景相机视频(ladybug.avi)、车辆导航数据(.nav文件) > >输出：配准且GPS对齐的激光点云(output/registered_lidar_points.ply) 将上述文件放在指定文件夹，运行程序VelodyneLadybugFusion程序即可。 ```shell ./lidar.sh ``` ### 多源数据点云配准 >输入：多视角图片(data/image_data/images/)，稀疏模型(data/image_data/sparse/)，稠密模型(data/image_data/dense/) > >输出：相机的GPS信息(output/img_gps.txt)、相机的本地坐标(output/img_local_xyz.txt)、GPS对齐的稠密模型(output/geo_registered_fused.ply)、调整颜色后的激光点云模型(output/registered_adjusted_lidar_points.ply) ```shell ./geo_register.sh ``` ### 深度图融合（PSDF未开源） >输入：多视角图片(data/image_data/images/)，稠密模型(data/image_data/dense/)，调整颜色后的激光点云模型(output/registered_adjusted_lidar_points.ply) > >输出：融合后的Mesh模型(output/mesh.ply)，以及置信度模型(output/mesh_conf.ply) 1. 从colmap中获取深度图和位姿。使用main.py，输出文件和相机内参 2. 打开PSDF/config/tum1.yml文件，更改datset_path为文件目录，更改相机内参 3. 运行PSDF程序 ```shell ./psdf_recons.sh ``` ## 重建结果 在线重建结果使用PSDF进行体素更新并抽取mesh。重建mesh和置信度如下: ![psdf_mesh](./output/psdf_mesh_animation.gif) ![mesh_conf](./output/conf_animation.gif)