1 Star 1 Fork 0

hzw_ai / ACGA

Gitee — Enterprise-level DevOps R&D management platform
Join Gitee
Clone or download
Cancel
Notice: Creating folder will generate an empty file .keep, because not support in Git
Loading...
README.md

Code for the following paper (ICCV19 ACGA)

Hao Wang, Cheng Deng*, Junchi Yan, Dacheng Tao. "Asymmetric Cross-Guided Attention Network for Actor and Action Video Segmentation From Natural Language Query", ICCV, 2019. [pdf]

Introduction

Actor and action video segmentation from natural language query aims to selectively segment the actor and its action in a video based on an input textual description. Previous works mostly focus on learning simple correlation between two heterogeneous features of vision and language via dynamic convolution or fully convolutional classification. However, they ignore the linguistic variation of natural language query and have difficulty in modeling global visual context, which leads to unsatisfactory segmentation performance. To address these issues, we propose an asymmetric cross-guided attention network for actor and action video segmentation from natural language query. Specifically, we frame an asymmetric cross-guided attention network, which consists of vision guided language attention to reduce the linguistic variation of input query and language guided vision attention to incorporate query-focused global visual context simultaneously. Moreover, we adopt multiresolution fusion scheme and weighted loss for foreground and background pixels to obtain further performance improvement. Extensive experiments on Actor-Action Dataset Sentences and J-HMDB Sentences show that our proposed approach notably outperforms state-of-the-art methods.

image

Run

Environment: Python 3.8.2, Pytorch 1.4.0, CUDA 10.1, 1 GPU (>11GB), RAM (>160GB), ROM (>200GB)

Data:
A2D extracted feature (BaiduNetDisk 163GB, OneDrive 163GB, passwd: abzr), Split (BaiduNetDisk, OneDrive passwd: 8ehb)
JHMDB extracted feature (BaiduNetDisk 23GB, OneDrive 23GB, passwd: d7vw), Split (BaiduNetDisk, OneDrive passed: p0ak)
pretrained model (BaiduNetDisk 666MB, OneDrive 666MB, passwd: jtzu)

cat a2d_data.tara* >> a2d_data.tar
tar xvf a2d_data.tar -C dataset/A2D/preprocessed/
cat jhmdb_data.tara* >> jhmdb_data.tar
tar xvf jhmdb_data.tar -C dataset/JHMDB/preprocessed/
tar xvf ckpt.tar -C checkpoint/ACGA/

Testing

CUDA_VISIBLE_DEVICES=0 PYTHONPATH=`pwd` <Your Python Path> main.py --dataset A2D --testing
CUDA_VISIBLE_DEVICES=0 PYTHONPATH=`pwd` <Your Python Path> main.py --dataset JHMDB --testing

Results (Note: slightly different with the paper after we clean and refactor the code)

A2D

Method P@0.5 P@0.6 P@0.7 P@0.8 P@0.9 mAP@0.5:0.95 Overall IoU Mean IoU
Hu et al. 34.8 23.6 13.3 3.3 0.1 13.2 47.4 35.0
Li et al. 38.7 29.0 17.5 6.6 0.1 16.3 51.5 35.4
Gavrilyuk et al. 47.5 34.7 21.1 8.0 0.2 19.8 53.6 42.1
ACGA (ours) 56.7 47.6 33.8 16.9 1.9 28.5 61.3 49.8

JHMDB

Method P@0.5 P@0.6 P@0.7 P@0.8 P@0.9 mAP@0.5:0.95 Overall IoU Mean IoU
Hu et al. 63.3 35.0 8.5 0.2 0.0 17.8 54.6 52.8
Li et al. 57.8 33.5 10.3 0.6 0.0 17.3 52.9 49.1
Gavrilyuk et al. 69.9 46.0 17.3 1.4 0.0 23.3 54.1 54.2
ACGA (ours) 75.4 58.5 29.4 3.6 0.0 29.3 57.8 58.8

Training from scratch

CUDA_VISIBLE_DEVICES=0 PYTHONPATH=`pwd` <Your Python Path> main.py --dataset A2D

Reference

If you found this code useful, please cite the following paper:

@inproceedings{wang2019asymmetric,
  title={Asymmetric Cross-Guided Attention Network for Actor and Action Video Segmentation From Natural Language Query},
  author={Wang, Hao and Deng, Cheng and Yan, Junchi and Tao, Dacheng},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  pages={3939--3948},
  year={2019}
}

Apache License 2.0

Repository Comments ( 0 )

Sign in for post a comment

About

PyTorch Implementation for "Asymmetric Cross-Guided Attention Network for Actor and Action Video Segmentation From Natural Language Query" expand collapse
Apache-2.0
Cancel

Releases

No release

Contributors

All

Activities

load more
can not load any more
1
https://gitee.com/hzw-ai/ACGA.git
git@gitee.com:hzw-ai/ACGA.git
hzw-ai
ACGA
ACGA
master

Search

181749 a2d7925e 1850385 181749 9f8568a7 1850385