FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding (CVPR'2021)

Abstract

Emerging interests have been brought to recognize previously unseen objects given very few training examples, known as few-shot object detection (FSOD). Recent researches demonstrate that good feature embedding is the key to reach favorable few-shot learning performance. We observe object proposals with different Intersection-of-Union (IoU) scores are analogous to the intra-image augmentation used in contrastive approaches. And we exploit this analogy and incorporate supervised contrastive learning to achieve more robust objects representations in FSOD. We present Few-Shot object detection via Contrastive proposals Encoding (FSCE), a simple yet effective approach to learning contrastive-aware object proposal encodings that facilitate the classification of detected objects. We notice the degradation of average precision (AP) for rare objects mainly comes from misclassifying novel instances as confusable classes. And we ease the misclassification issues by promoting instance level intra-class compactness and inter-class variance via our contrastive proposal encoding loss (CPE loss). Our design outperforms current state-ofthe-art works in any shot and all data splits, with up to +8.8% on standard benchmark PASCAL VOC and +2.7% on challenging COCO benchmark. Code is available at: https://github.com/bsun0802/FSCE.git

Citation

@inproceedings{sun2021fsce,
    title={FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding},
    author={Sun, Bo and Li, Banghuai and Cai, Shengcai and Yuan, Ye and Zhang, Chi},
    booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)},
    year={2021}
}

Note: ALL the reported results use the data split released from fsce official repo, unless stated otherwise. Currently, each setting is only evaluated with one fiNed few shot dataset. Please refer to here to get more details about the dataset and data preparation.

How to reproduce FSCE

Following the original implementation, it consists of 3 steps:

• Step1: Base training

  • use all the images and annotations of base classes to train a base model.

• Step2: Reshape the bbox head of base model:

  • create a new bbox head for all classes fine-tuning (base classes + novel classes)
  • the weights of base class in new bbox head directly use the original one as initialization.
  • the weights of novel class in new bbox head use random initialization.

• Step3: Few shot fine-tuning:

  • use the base model from step2 as model initialization and further fine tune the bbox head with few shot datasets.

An example of VOC split1 1 shot setting with 8 gpus

# step1: base training for voc split1
bash ./tools/detection/dist_train.sh \
    configs/detection/fsce/voc/split1/fsce_r101_fpn_voc-split1_base-training.py 8

# step2: reshape the bbox head of base model for few shot fine-tuning
python -m tools.detection.misc.initialize_bbox_head \
    --src1 work_dirs/fsce_r101_fpn_voc-split1_base-training/latest.pth \
    --method randinit \
    --save-dir work_dirs/fsce_r101_fpn_voc-split1_base-training

# step3: few shot fine-tuning
bash ./tools/detection/dist_train.sh \
    configs/detection/fsce/voc/split1/fsce_r101_fpn_voc-split1_1shot-fine-tuning.py 8

Note:

• The default output path of the reshaped base model in step2 is set to work_dirs/{BASE TRAINING CONFIG}/base_model_random_init_bbox_head.pth. When the model is saved to different path, please update the argument load_from in step3 few shot fine-tune configs instead of using resume_from.
• To use pre-trained checkpoint, please set the load_from to the downloaded checkpoint path.

Results on VOC dataset

Base Training

Note:

• All the base training configs is the same as TFA. Therefore, the few shot fine-tuning can directly reuse the reshaped base model of fsce by creating a symlink or copying the whole checkpoint to the corresponding folder. Also, the released base training checkpoint is the same as the TFA, too.
• The performance of the same few shot setting using different base training models can be dramatically unstable (AP50 can fluctuate by 5.0 or more), even their mAP on base classes are very close.
• Temporally, the solution to getting a good base model is training the base model with different random seed. Also, the random seed used in this code base may not the optimal one, and it is possible to get the higher results by using other random seeds. However, using the same random seed still can not guarantee the identical result each time, as some nondeterministic CUDA operations. We will continue to investigate and improve it.
• To reproduce the reported few shot results, it is highly recommended using the released step2 model for few shot fine-tuning.
• The difficult samples will be used in base training, but not be used in few shot setting.

Few Shot Fine-tuning

Note:

• Following the original implementation, the contrastive loss only is added at VOC 3/5/10 shot setting, while in VOC 1/2 shot setting only the fc_cls and fc_reg layers are fine-tuned.
• Some arguments of configs are different from the official codes, for example, the official codes use aug test in some settings, while all the results reported above do not use aug_test.

Results on COCO dataset

Base Training

Few Shot Fine-tuning

Note:

• Some arguments of configs are different from the official codes, for example, the official codes use aug test in some settings, while all the results reported above do not use aug_test.