The pytorch implementation of our CVPR 2023 paper Conditional Image-to-Video Generation with Latent Flow Diffusion Models.
[Updated on 07/08/2023] Added multi-GPU training codes for MHAD dataset.
[Updated on 05/12/2023] Released a testing demo for NATOPS dataset.
[Updated on 03/31/2023] Added the illustration of training a LFDM for NATOPS dataset.
[Updated on 03/27/2023] Added the illustration of training a LFDM for MHAD dataset.
[Updated on 03/27/2023] Released a testing demo for MHAD dataset.
[Updated on 03/26/2023] Added the illustration of training a LFDM for MUG dataset.
[Updated on 03/26/2023] Now our paper is available on arXiv.
[Updated on 03/20/2023] Released a testing demo for MUG dataset.
All the subjects of the following videos are unseen during the training.
Some generated video results on MUG dataset.
Some generated video results on MHAD dataset.
Some generated video results on NATOPS dataset.
Applied LFDM trained on MUG to FaceForensics dataset.
Dataset | Model | Frame Sampling | Link (Google Drive) |
---|---|---|---|
MUG | LFAE | - | https://drive.google.com/file/d/1dRn1wl5TUaZJiiDpIQADt1JJ0_q36MVG/view?usp=share_link |
MUG | DM | very_random | https://drive.google.com/file/d/1lPVIT_cXXeOVogKLhD9fAT4k1Brd_HHn/view?usp=share_link |
MHAD | LFAE | - | https://drive.google.com/file/d/1AVtpKbzqsXdIK-_vHUuQQIGx6Wa5PxS0/view?usp=share_link |
MHAD | DM | random | https://drive.google.com/file/d/1BoFPQAeOuHE5wt7h-chhYAO-dU0B1p2y/view?usp=share_link |
NATOPS | LFAE | - | https://drive.google.com/file/d/10iyzoYqSwzQ3fZgb6oh3Uay-P7k2A12s/view?usp=share_link |
NATOPS | DM | random | https://drive.google.com/file/d/1lSLSzS_KyGvJ7dW3l5hLJLR9k2k8LoU3/view?usp=share_link |
MUG Dataset
python -u demo/demo_mug.py
to generate the example videos. Please set the paths in the code files and config file config/mug128.yaml
if needed. The pretrained models for MUG dataset have released.MHAD Dataset
python -u demo/demo_mhad.py
to generate the example videos. Please set the paths in the code files and config file config/mhad128.yaml
if needed. The pretrained models for MHAD dataset have released.NATOPS Dataset
python -u demo/demo_natops.py
to generate the example videos. Please set the paths in the code files and config file config/natops128.yaml
if needed. The pretrained models for NATOPS dataset have released.The training of our LFDM includes two stages: 1. train a latent flow autoencoder (LFAE) in an unsupervised fashion. To accelerate the training, we initialize LFAE with the pretrained models provided by MRAA, which can be found in their github; 2. train a diffusion model (DM) on the latent space of LFAE.
MUG Dataset
preprocessing/preprocess_MUG.py
.python -u LFAE/run_mug.py
to train the LFAE. Please set the paths and config file config/mug128.yaml
if needed.python -u LFAE/test_flowautoenc_mug.py
.python -u DM/train_video_flow_diffusion_mug.py
to train the DM. Please set the paths and config file config/mug128.yaml
if needed.python -u DM/test_video_flow_diffusion_mug.py
.MHAD Dataset
preprocessing/preprocess_MHAD.py
.python -u LFAE/run_mhad.py
to train the LFAE. Please set the paths and config file config/mhad128.yaml
if needed.python -u LFAE/test_flowautoenc_mhad.py
.python -u DM/train_video_flow_diffusion_mhad.py
to train the DM. Please set the paths and config file config/mhad128.yaml
if needed.python -u DM/test_video_flow_diffusion_mhad.py
.NATOPS Dataset
preprocessing/preprocess_NATOPS.py
.python -u LFAE/run_natops.py
to train the LFAE. Please set the paths and config file config/natops128.yaml
if needed.python -u LFAE/test_flowautoenc_natops.py
.python -u DM/train_video_flow_diffusion_natops.py
to train the DM. Please set the paths and config file config/natops128.yaml
if needed.python -u DM/test_video_flow_diffusion_natops.py
.If you find our approaches useful in your research, please consider citing:
@inproceedings{ni2023conditional,
title={Conditional Image-to-Video Generation with Latent Flow Diffusion Models},
author={Ni, Haomiao and Shi, Changhao and Li, Kai and Huang, Sharon X and Min, Martin Renqiang},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={18444--18455},
year={2023}
}
For questions with the code, please feel free to open an issue or contact me: homerhm.ni@gmail.com
Part of our code was borrowed from MRAA, VDM, and LDM. We thank the authors of these repositories for their valuable implementations.
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。