## β TODO List
- [x] **ZEB**: **Z**ero-shot **E**valuation **B**enchmark
- [x] 3D Reconstruction
- [ ] Inference code
- [ ] gim_roma
- [x] gim_dkm
- [x] gim_loftr
- [x] gim_lightglue
- [ ] Training code
> We are actively continuing with the remaining open-source work and appreciate everyone's attention.
## π€ Online demo
Go to [Huggingface](https://huggingface.co/spaces/xuelunshen/gim-online) to quickly try our model online.
## βοΈ Environment
I set up the running environment on a new machine using the commands listed below.
[ Click to show commands ]
```bash
conda create -n gim python=3.9
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch -c conda-forge
conda install xformers -c xformers
pip install albumentations==1.0.1 --no-binary=imgaug,albumentations
pip install colour-demosaicing==0.2.2
pip install pytorch-lightning==1.5.10
pip install opencv-python==4.5.3.56
pip install imagesize==1.2.0
pip install kornia==0.6.10
pip install einops==0.3.0
pip install loguru==0.5.3
pip install joblib==1.0.1
pip install yacs==0.1.8
pip install h5py==3.1.0
pip install matplotlib
pip install omegaconf
pip install triton
```
## π¨ Usage
1. Clone the repository
```bash
git clone https://github.com/xuelunshen/gim.git
cd gim
```
2. Download `gim_dkm` model weight from [Google Drive](https://drive.google.com/file/d/1gk97V4IROnR1Nprq10W9NCFUv2mxXR_-/view?usp=sharing) or [OneDrive](https://stuxmueducn-my.sharepoint.com/:u:/g/personal/xuelun_stu_xmu_edu_cn/EdJOibZ8VABOoKoyOHWo8ZEBHd_MyHbSvhRyT_o40SIPGA?e=GCjGZE)
3. Put it on the folder `weights`
4. Run the following commands
[ Click to show commands ]
```bash
python demo.py --model gim_dkm
```
or
```bash
python demo.py --model gim_loftr
```
or
```bash
python demo.py --model gim_lightglue
```
5. The code will match `a1.png` and `a2.png` in the folder `assets/demo`,and output `a1_a2_match.png` and `a1_a2_warp.png`.
[ Click to show
a1.png
and
a2.png ]
[ Click to show
a1_a2_match.png ]
a1_a2_match.png is a visualization of the match between the two images
[ Click to show
a1_a2_warp.png ]
a1_a2_warp.png shows the effect of projecting image a2 onto image a1 using homography
There are more images in the `assets/demo` folder, you can try them out.
[ Click to show other images ]
## π 3D Reconstruction
The code for 3D reconstruction in this repository is implemented based on [hloc](https://github.com/cvg/Hierarchical-Localization).
First, install [colmap](https://colmap.github.io/) and [pycolmap](https://github.com/colmap/pycolmap) according to hloc's README.
Then, download the [semantic-segmentation](https://github.com/CSAILVision/semantic-segmentation-pytorch)'s model parameters from [Google Drive](https://drive.google.com/file/d/1YswCj58VuVhqEpMKQ_k0QJb3_mMdpF8M/view?usp=sharing) or [OneDrive](https://stuxmueducn-my.sharepoint.com/:u:/g/personal/xuelun_stu_xmu_edu_cn/EUR_XMay5b5FtWelmqXiLi4Bcnv4G1w5b2aYjhqS-Ds_ow) and put the model parameters in the folder `weights`.
Next, create some folders. If you want to reconstruct a room in 3D, run the following command:
```bash
mkdir -p inputs/room/images
```
Then, put images of the room to be reconstructed in 3D into the `images` folder.
Finally, run the following command to perform a 3D reconstruction:
```bash
sh reconstruction.sh room
```
> Tips:\
> At present, the code for 3D reconstruction defaults to pairing all images pairwise, and then performing image matching and reconstruction,\
> For better reconstruction results, it is recommended to modify the code according to the actual situation and adjust the paired images.
## π ZEB: Zero-shot Evaluation Benchmark
1. Create a folder named **`zeb`**.
2. Download zip archives containing the **ZEB** data from [the URL](https://stuxmueducn-my.sharepoint.com/:f:/g/personal/lizijun_stu_xmu_edu_cn/EmHLjQpbpDRKmiED88rxGl4BFIkSp7vAzXifwXtvVbQA9w?e=ey8WVk), put it into the **`zeb`** folder and unzip zip archives.
3. Run the following commands
[ Click to show commands ]
The number **1** below represents the number of GPUs you want to use. If you want to use **2 GPUs**, change the number **1** to **2**.
```bash
sh TEST_GIM_DKM.sh 1
```
or
```bash
sh TEST_GIM_LOFTR.sh 1
```
or
```bash
sh TEST_GIM_LIGHTGLUE.sh 1
```
or
```bash
sh TEST_ROOT_SIFT.sh 1
```
4. Run the command `python check.py` to check if everything outputs `"Good"`.
5. Run the command `python analysis.py --dir dump/zeb --wid gim_dkm --version 100h --verbose` to get result.
6. Paste the **ZEB** result to the Excel file named `zeb.xlsx`.
[ Click to show π ZEB Result ]
> The data in this table comes from the **ZEB**: Zero-shot Evaluation Benchmark for Image Matching proposed in the paper. This benchmark consists of 12 public datasets that cover a variety of scenes, weather conditions, and camera models, corresponding to the 12 test sequences starting from GL3 in the table.
| |
## π Citation
If the paper and code from `gim` help your research, we kindly ask you to give a citation to our paper β€οΈ. Additionally, if you appreciate our work and find this repository useful, giving it a star βοΈ would be a wonderful way to support our work. Thank you very much.
```bibtex
@inproceedings{
xuelun2024gim,
title={GIM: Learning Generalizable Image Matcher From Internet Videos},
author={Xuelun Shen and Zhipeng Cai and Wei Yin and Matthias MΓΌller and Zijun Li and Kaixuan Wang and Xiaozhi Chen and Cheng Wang},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024}
}
```
## π Star History
## License
This repository is under the MIT License. This content/model is provided here for research purposes only. Any use beyond this is your sole responsibility and subject to your securing the necessary rights for your purpose.
