# icrmi

**Repository Path**: quantumbolt/icrmi

## Basic Information

- **Project Name**: icrmi
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2024-03-19
- **Last Updated**: 2024-03-25

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README


# Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning

This repository contains the code for the ArXiv preprint [Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning](https://arxiv.org/abs/2402.17510),  by [Maurits Bleeker](https://mauritsbleeker.github.io)<sup>1</sup>, [Mariya Hendriksen](https://mariyahendriksen.github.io)<sup>1</sup>, [Andrew Yates](https://andrewyates.net)<sup>1</sup>, and [Maarten de Rijke](https://staff.fnwi.uva.nl/m.derijke/)<sup>1</sup>.

The implementation builds upon the codebase of [Latent Target Decoding](https://github.com/MauritsBleeker/reducing-predictive-feature-suppression/).

<sup>1</sup>University of Amsterdam, The Netherlands

## Requirements

To set up the environment, install the requirements using the provided YAML file:

```angular2html
conda env create --file environment.yaml
```

This command will create a conda environment `contrastive-shortcuts`. Activate the created environment:

```angular2html
source activate contrastive-shortcuts
```

## Training the models 

For local development, execute the following command:

```angular2html
python src/training.py --yaml_file src/configs/{f30k, coco}/development_local.yaml
```

To train a model run `python src/training.py` and provide a base config in YAML format using `--yaml_file <config path.yaml>`. 

Hyperparameters can be overridden using command line flags. For example:
```angular2html
python src/training.py --yaml_file src/configs/f30k/development_local.yaml --experiment.wandb_project <your project name>
```

The recommended approach is to have a fixed base config for each experiment and only modify specific hyperparameters for different training/evaluation settings. 

All training and evaluation were conducted using a SLURM-based scheduling system.


## Data path

1. coco

```
/home/LAB/gongtx/data/coco
```

2. flickr30k

```
/home/LAB/gongtx/data/flickr30k
```



### Data loading and preparation

We implemented a PyTorch Dataloader class that loads the images from the memory of the compute node the training runs on. The captions are loaded from either the Flickr30k or MS-COCO annotation file.

Update the *.yaml config with the right file paths.

```angular2html
img_path:
annotation_file:
annotation_path:
```


### Vocabulary class

To create the vocabulary class, run:

```angular2html
python utils/vocab.py 
```
With the appropriate input flags.


### Job files

Job and hyperparameter files to reproduce experiments can be found in `src/jobs/{coco, f30k}/`.

The shortcut experiments (Section 4) are available in the `shortcuts` folder, the LTD experiments in the `LTD` folder, and the IFM experiments in the 'IFM' folder (Section 6).

## Evaluation

To reproduce results from Section 3, run the following evaluation script (ensure correct file paths).

```angular2html
sbatch src/jobs/{coco, f30k}/snellius/shortcuts/{clip, vse}/{clip, vse}_{coco, f30k}_shortcut_experiments_eval.job
```

Next, copy all the RSUM values to `notebooks/visualizations/visualization.ipynb` to generate the plot.

The results from Section 6 are generated by using `notebooks/Evaluation.ipynb`.