# GLU-Net
**Repository Path**: yrccry/GLU-Net
## Basic Information
- **Project Name**: GLU-Net
- **Description**: Official implementation of the paper GLU-Net (CVPR2020-Oral)
- **Primary Language**: Unknown
- **License**: GPL-3.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No
## Statistics
- **Stars**: 0
- **Forks**: 0
- **Created**: 2021-07-02
- **Last Updated**: 2021-07-02
## Categories & Tags
**Categories**: Uncategorized
**Tags**: None
## README
# Official implementation of GLUNet
This is the official implementation of our paper :
**GLU-Net: Global-Local Universal Network for dense flow and correspondences (CVPR 2020-Oral).**
Authors: [Prune Truong](https://prunetruong.com/), [Martin Danelljan](https://martin-danelljan.github.io/) and Radu Timofte
\[[Paper](https://arxiv.org/abs/1912.05524)\]
\[[Website](https://prunetruong.com/research/glu-net)\]
\[[Poster](https://drive.google.com/file/d/1pS_OMZ83EG-oalD-30vDa3Ru49GWi-Ky/view?usp=sharing)\]
\[[Oral Video](https://www.youtube.com/watch?v=xB2gNx8f8Xc&feature=emb_title)\]
\[[Teaser Video](https://www.youtube.com/watch?v=s5OUdkM9QLo)\]
Check out our related publication [**GOCor**](https://arxiv.org/abs/2009.07823) ([website](https://prunetruong.com/research/gocor))
and corresponding code [here](https://github.com/PruneTruong/GOCor) !
For an improved version of a dense correspondence network, also predicting a confidence mask, check out [**PDCNet**](https://arxiv.org/abs/2101.01710) ([website](https://prunetruong.com/research/pdcnet)) and code [here](https://github.com/PruneTruong/DenseMatching).
For any questions, issues or recommendations, please contact Prune at prune.truong@vision.ee.ethz.ch
If our project is helpful for your research, please consider citing :
```bash
@inproceedings{GLUNet_Truong_2020,
title = {{GLU-Net}: Global-Local Universal Network for dense flow and correspondences},
author = {Prune Truong and
Martin Danelljan and
Radu Timofte},
year = {2020},
booktitle = {2020 {IEEE} Conference on Computer Vision and Pattern Recognition, {CVPR} 2020}
}
```
# Network
Our model GLU-Net is illustrated below:

The models, evaluation and training codes for Local-Net (a 3 level pyramidal network with only local correlations), Global-Net (a 3 levels pyramidal network with a single
global correlation followed by concatenation of feature maps) and GLOCAL-Net (a combination of the two, a 3 levels pyramidel network
with a single global correlation followed by two local correlation layers) are also available for reference.
They are all illustrated below:

For more details, refer to our [paper](https://arxiv.org/abs/1912.05524)
# Table of Content
1. [Installation](#Installation)
2. [Test on your own image pairs !](#test_pairs)
3. [Datasets downloading](#download_dataset)
1. [Training datasets](#training_dataset)
2. [Testing datasets](#testing_dataset)
4. [Training](#training)
5. [Evaluation](#evaluation)
1. [Performance on geometric matching dataset](#geometric_matching)
2. [Performance on semantic matching dataset](#semantic_matching)
3. [Performance on optical flow dataset](#OF)
6. [Acknowledgement](#acknowledgement)
7. [Changelog](#changelog)
#
# 1. Installation
Note that the models were trained with **torch 1.0**. Torch versions up to 1.7 were tested for inference on testing datasets but NOT for training,
so I cannot guarantee that the models train smoothly for higher torch versions.
* Create and activate conda environment with Python 3.x
```bash
conda create -n GLUNet_env python=3.7
conda activate GLUNet_env
```
* Install all dependencies (except for cupy, see below) by running the following command:
```bash
pip install -r requirements.txt
```
**ATTENTION**, CUDA is required to run the code. Indeed, the correlation layer is implemented in CUDA using CuPy,
which is why CuPy is a required dependency. It can be installed using pip install cupy or alternatively using one of the
provided binary packages as outlined in the CuPy repository. The code was developed using Python 3.7 & PyTorch 1.0 & CUDA 9.0,
which is why I installed cupy for cuda90. For another CUDA version, change accordingly.
```bash
pip install cupy-cuda90==7.8.0 --no-cache-dir
```
There are some issues with latest versions of cupy. So for all cuda, install cupy version 7.8.0. For example, on cuda10,
```bash
pip install cupy-cuda100==7.8.0 --no-cache-dir
```
* **Download an archive with pre-trained models [click](https://drive.google.com/open?id=15yXIi8kJbCyXCAHzg-UbMQ6RD2lJ1nOi) and extract it to the project folder**
# 2. Test on your own image pairs !
One can test GLU-Net on a pair of images using test_GLUNet.py and the provided trained model weights.
The inputs are the paths to the source and target images. They are then passed
to the network which outputs the corresponding flow field relating the target to the source image. The source is then warped according to
the estimated flow, and a figure is saved.
For this pair of images (provided to check that the code is working properly), the output is:
```bash
python test_GLUNet.py --path_source_image images/yosemite_source.png --path_target_image images/yosemite_target.png --write_dir evaluation/
additional optional arguments:
--pre_trained_models_dir (default is pre_trained_models/)
--pre_trained_model (default is DPED_CityScape_ADE)
```

Another example and output (attention large images):
```bash
python test_GLUNet.py --path_source_image images/hp_source.png --path_target_image images/hp_target.png --write_dir evaluation/
optional arguments:
* --pre_trained_models_dir : Directory containing the pre-trained-models (default is pre_trained_models/)
* --pre_trained_model: Name of the pre-trained-model (default= DPED_CityScape_ADE )
```

# 3. Datasets downloading
## 3.1. Training datasets
For the training, we use a combination of the DPED, CityScapes and ADE-20K datasets.
The DPED training dataset is composed of only approximately 5000 sets of images taken by four different cameras.
We use the images from two cameras, resulting in around 10,000 images.
CityScapes additionally adds about 23,000 images.
We complement with a random sample of ADE-20K images with a minimum resolution of 750 x 750.
It results in 40.000 original images, used to create pairs of training images by applying geometric transformations to them.
The path to the original images as well as the geometric transformation parameters are given in the csv files
'datasets/csv_files/homo_aff_tps_train_DPED_CityScape_ADE.csv' and 'datasets/csv_files/homo_aff_tps_test_DPED_CityScape_ADE.csv'.
**Apparently, the structure of the ADE-20K dataset has changed and the provided paths in the csv files are not valid anymore**.
I am working on a fix for the ADE-20K images.
In the meantime, use 'datasets/csv_files/homo_aff_tps_train_DPED_CityScape.csv' and 'datasets/csv_files/homo_aff_tps_test_DPED_CityScape.csv' to exclude the ADE images (you don't need to download the ADE-20K dataset in that case).
Resulting training data is 31K images. Performance of the resulting trained model might be a bit different.
* Download the [DPED dataset](http://people.ee.ethz.ch/~ihnatova/) (54 GB) ==> images are created in original_images/
* Download the [CityScapes dataset](https://www.cityscapes-dataset.com/)
- download 'leftImg8bit_trainvaltest.zip' (11GB, left 8-bit images - train, val, and test sets', 5000 images) ==> images are created in CityScape/
- download leftImg8bit_trainextra.zip (44GB, left 8-bit images - trainextra set, 19998 images) ==> images are created in CityScape_extra/
* Download the [ADE-20K dataset](https://groups.csail.mit.edu/vision/datasets/ADE20K/) (3.8 GB, 20.210 images) ==> images are created in ADE20K_2016_07_26/
Put all the datasets in the same directory.
As illustration, your root training directory should be organised as follows:
/training_datasets/
original_images/
CityScape/
CityScape_extra/
ADE20K_2016_07_26/
**Optional: To save the synthetic image pairs and flows to disk**
During training, from this set of original images, the pairs of synthetic images are created on the fly at each epoch.
However, this dataset generation takes time and since no augmentation is applied at each epoch, one can also create the dataset in advance
and save it to disk. During training, the image pairs composing the training datasets are then just loaded from the disk
before passing through the network, which is a lot faster.
To generate the training dataset and save it to disk:
```bash
python save_training_dataset_to_disk.py --image_data_path /directory/to/original/training_datasets/
--csv_path datasets/csv_files/homo_aff_tps_train_DPED_CityScape_ADE.csv --save_dir /path/to/save_dir --plot True
```
It will create the images pairs and corresponding flow fields in save_dir/images and save_dir/flow respectively.
## 3.2. Testing datasets
The testing datasets are available at the following links:
* HPatches dataset: Download HPatches dataset (Full image sequences). The dataset is available [here](https://github.com/hpatches/hpatches-dataset) at the end of the page.
The corresponding csv files for each viewpoint ID with the path to the images and the homography parameters relating the pairs are listed in /datasets/csv_files/
* TSS dataset: The dataset is available [here](https://taniai.space/projects/cvpr16_dccs/) (1.2 GB)
* KITTI datasets: Both KITTI-2012 and KITTI-2015 are available [here](http://www.cvlibs.net/datasets/kitti/eval_flow.php)
* ETH3D dataset: See below in section 5.1
# 4.0 Training
Training files for GLUNet (and its variants, including Semantic-GLU-Net), GLOCAL-Net, LocalNet and GlobalNet are available.
**This will create the synthetic training and evaluation pairs along with the ground-truth on the fly !**
```bash
python train_GLUNet.py --name_exp GLUNet_train --training_data_dir /path/to/directory/original_images-for-training/ --evaluation_data_dir /path/to/directory/original_images-for-evaluation/
if the network is already pretrained and the user wants to start the training from an old weight file
* --pretrained /path/to/pretrained_file.pth
```
**To load the pre-saved synthetic training and evaluation image pairs and ground truth flow fields instead (created earlier and saved to disk):**
```bash
python train_GLUNet.py --name_exp GLUNet_train --pre_loaded_training_dataset True --training_data_dir /path/to/directory/synthetic_training_image_pairs_and_flows/
--evaluation_data_dir /path/to/directory/synthetic_validation_image_pairs_and_flows/
if the network is already pretrained and the user wants to start the training from an old weight file
* --pretrained /path/to/pretrained_file.pth
```
In the training files, one can modify all the parameters of the network. The default ones are for GLU-Net.
# 5. Evaluation
# 5.1. Performance on geometric matching dataset
In the case of geometric matching, pairs of images present different viewpoints of the same scene.
## HPATCHES (original size and resized to 240x240)
To test on the HPatches dataset, HP-240 (images and flow rescaled to 240x240) and HP (original)
```bash
python eval.py --model GLUNet --pre_trained_models DPED_CityScape_ADE --dataset HPatchesdataset --data_dir /directory/to/hpatches --save_dir /directory/to/save_dir
optional argument:
* --hpatches_original_size To test on the original image size, True or False (default to False)
```
Out of the 120 sequences of HPatches, we only evaluate on the 59 sequences in HP labelled with v_X, which have viewpoint changes,
thus excluding the ones labelled i_X, which only have illumination changes.
/ETH3D/
multiview_testing/
lakeside/
sand_box/
storage_room/
storage_room_2/
tunnel/
multiview_training/
delivery_area/
electro/
forest/
playground/
terrains/
info_ETH3D_files/
The organisation of your directories is important, since the bundle files contain the relative paths to the images, from the ETH3D root folder.