# morefusion
**Repository Path**: null_035_6851/morefusion
## Basic Information
- **Project Name**: morefusion
- **Description**: MoreFusion: Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusion
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 1
- **Forks**: 0
- **Created**: 2021-12-07
- **Last Updated**: 2022-04-01
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
MoreFusion
Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusion
MoreFusion is an object-level reconstruction system that builds a map with
known-shaped objects, exploiting volumetric reconstruction of detected
objects in a real-time, incremental scene reconstruction senario. The key
components are:
- **Occupancy-based volumetric reconstruction** of detected objects for model
alignment in the later stage;
- **Volumetric pose prediction** that exploits volumetric reconstruction and
CNN feature extraction from the image observation;
- **Joint pose refinement of objects** based on geometric consistency
among objects and impenetrable space.
##
## Installation
There're several options for installation:
- [Python project only](#python-project-only): Python framework for pose
estimation (e.g., training, inference, refinement).
- [ROS project for camera demonstration](#ros-project-for-camera-demonstration): ROS framework for
object-level mapping with live cameras.
- [ROS project for robotic demonstration](#ros-project-for-robotic-demonstration): ROS framework for
robotic demonstration with object-level mapping.
### Python project only
```bash
make install
```
### ROS project for camera demonstration
```bash
mkdir -p ~/ros_morefusion/src
cd ~/ros_morefusion/src
git clone https://github.com/wkentaro/morefusion.git
cd morefusion
make install
cd ~/ros_morefusion
ln -s src/ros/*.sh .
./rosdep_install.sh
./catkin_build.robot_agent.sh
source .autoenv.zsh
```
### ROS project for robotic demonstration
- `robot-agent`: A computer for visual processing.
- `robot-node`: A computer with real-time OS for Panda robot.
#### @robot-agent
Same as above instruction:
ROS project for camera demonstration.
#### @robot-node
```bash
mkdir -p ~/ros_morefusion/src
cd ~/ros_morefusion/src
git clone https://github.com/wkentaro/morefusion.git
cd ~/ros_morefusion
ln -s src/ros/*.sh .
./catkin_build.robot_node.sh
source devel/setup.bash
rosrun franka_control_custom create_udev_rules.sh
```
## Usage
### Training & Inference
Pre-trained models are provided in the demos as following, so this process is
optional to run the [demos](#full-demonstration).
- [Pre-trained instance segmentation model](https://github.com/wkentaro/morefusion/blob/a2b9eb012fa3e67c86f81c6db872f5728699f0b9/ros/morefusion_panda_ycb_video/nodes/mask_rcnn_instance_segmentation.py#L25-L29)
- [Pre-trained 6D pose prediction model](https://github.com/wkentaro/morefusion/blob/a2b9eb012fa3e67c86f81c6db872f5728699f0b9/ros/morefusion_panda_ycb_video/nodes/singleview_3d_pose_estimation.py#L31-L48)
#### Instance Segmentation
```bash
cd examples/ycb_video/instance_segm
./download_dataset.py
mpirun -n 4 python train_multi.py # 4-gpu training
./image_demo.py --model logs/XXX/XXX.npz
```
#### 6D pose prediction
```bash
# baseline model (point-cloud-based)
cd examples/ycb_video/singleview_pcd
./download_dataset.py
./train.py --gpu 0 --centerize-pcd --pretrained-resnet18 # 1-gpu
mpirun -n 4 ./train.py --multi-node --centerize-pcd --pretrained-resnet18 # 4-gpu
# volumetric prediction model (3D-CNN-based)
cd examples/ycb_video/singleview_3d
./download_dataset.py
./train.py --gpu 0 --centerize-pcd --pretrained-resnet18 --with-occupancy # 1-gpu
mpirun -n 4 ./train.py --multi-node --pretrained-resnet18 --with-occupancy # 4-gpu
mpirun -n 4 ./train.py --multi-node --pretrained-resnet18 # w/o occupancy
# inference
./download_pretrained_model.py # for downloading pretrained model
./demo.py logs/XXX/XXX.npz
./evaluate.py logs/XXX
```
### Joint pose refinement
```bash
cd examples/ycb_video/pose_refinement
./check_icp_vs_icc.py # press [s] to start
```
### Camera demonstration
#### Static Scene
```bash
# using orb-slam2 for camera tracking
roslaunch morefusion_panda_ycb_video rs_rgbd.launch
roslaunch morefusion_panda_ycb_video rviz_static.desk.launch
roslaunch morefusion_panda_ycb_video setup_static.desk.launch
```
Figure 1. Static Scene Reconstruction with the Human Hand-mounted Camera.
```bash
# using robotic kinematics for camera tracking
roslaunch morefusion_panda_ycb_video rs_rgbd.robot.launch
roslaunch morefusion_panda_ycb_video rviz_static.robot.launch
roslaunch morefusion_panda_ycb_video setup_static.robot.launch
```
Figure 2. Static Scene Reconstruction with the Robotic Hand-mounted Camera.
#### Dynamic Scene
```bash
roslaunch morefusion_panda_ycb_video rs_rgbd.launch
roslaunch morefusion_panda_ycb_video rviz_dynamic.desk.launch
roslaunch morefusion_panda_ycb_video setup_dynamic.desk.launch
roslaunch morefusion_panda_ycb_video rs_rgbd.robot.launch
roslaunch morefusion_panda_ycb_video rviz_dynamic.robot.launch
roslaunch morefusion_panda_ycb_video setup_dynamic.robot.launch
```
Figure 3. Dynamic Scene Reconstruction with the Human Hand-mounted Camera.
### Robotic Demonstration
#### Robotic Pick-and-Place
```bash
robot-agent $ sudo ntpdate 0.uk.pool.ntp.org # for time synchronization
robot-node $ sudo ntpdate 0.uk.pool.ntp.org # for time synchronization
robot-node $ roscore
robot-agent $ roslaunch morefusion_panda panda.launch
robot-node $ roslaunch morefusion_panda_ycb_video rs_rgbd.robot.launch
robot-node $ roslaunch morefusion_panda_ycb_video rviz_static.launch
robot-node $ roslaunch morefusion_panda_ycb_video setup_static.robot.launch TARGET:=2
robot-node $ rosrun morefusion_panda_ycb_video robot_demo_node.py
>>> ri.run()
```
Figure 4. Targetted Object Pick-and-Place. (a) Scanning the Scene; (b) Removing Distractor Objects; (c) Picking Target Object.
## Citation
If you find MoreFusion useful, please consider citing the paper as:
```
@inproceedings{Wada:etal:CVPR2020,
title={{MoreFusion}: Multi-object Reasoning for {6D} Pose Estimation from Volumetric Fusion},
author={Kentaro Wada and Edgar Sucar and Stephen James and Daniel Lenton and Andrew J. Davison},
booktitle={Proceedings of the {IEEE} Conference on Computer Vision and Pattern Recognition ({CVPR})},
year={2020},
}
``` 
