1 Star 0 Fork 0

SaiSo/ComfyUI-Frame-Interpolation

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
MIT

ComfyUI Frame Interpolation (ComfyUI VFI) (WIP)

A custom node set for Video Frame Interpolation in ComfyUI.

Nodes

  • KSampler Gradually Adding More Denoise (efficient)
  • GMFSS Fortuna VFI
  • IFRNet VFI
  • IFUnet VFI
  • M2M VFI
  • RIFE VFI (4.0 - 4.9) (Note that option fast_mode won't do anything from v4.5+ as contextnet is removed)
  • FILM VFI
  • Sepconv VFI
  • AMT VFI
  • Make Interpolation State List
  • STMFNet VFI (requires at least 4 frames, can only do 2x interpolation for now)
  • FLAVR VFI (same conditions as STMFNet)

Install

ComfyUI Manager

Incompatibile issue with it is now fixed

Following this guide to install this extension

https://github.com/ltdrdata/ComfyUI-Manager#how-to-use

Command-line

Windows

Run install.bat

For Window users, if you are having trouble with cupy, please run install.bat instead of install-cupy.py or python install.py.

Linux

Open your shell app and start venv if it is used for ComfyUI. Run:

python install.py

Support for non-CUDA device (experimental)

If you don't have a NVidia card, you can try taichi ops backend powered by Taichi Lang

On Windows, you can install it by running install.bat or pip install taichi on Linux

Then change value of ops_backend from cupy to taichi in config.yaml

If NotImplementedError appears, a VFI node in the workflow isn't supported by taichi

Usage

All VFI nodes can be accessed in category ComfyUI-Frame-Interpolation/VFI if the installation is successful and require a IMAGE containing frames (at least 2, or at least 4 for STMF-Net/FLAVR).

Regarding STMFNet and FLAVR, if you only have two or three frames, you should use: Load Images -> Other VFI node (FILM is recommended in this case) with multiplier=4 -> STMFNet VFI/FLAVR VFI

clear_cache_after_n_frames is used to avoid out-of-memory. Decreasing it makes the chance lower but also increases processing time.

It is recommended to use LoadImages (LoadImagesFromDirectory) from ComfyUI-Advanced-ControlNet and ComfyUI-VideoHelperSuite along side with this extension.

Example

Simple workflow

Workflow metadata isn't embeded Download these two images anime0.png and anime1.png and put them into a folder like E:\test in this image.

Complex workflow

It's used in AnimationDiff (can load workflow metadata)

Credit

Big thanks for styler00dollar for making VSGAN-tensorrt-docker. About 99% the code of this repo comes from it.

Citation for each VFI node:

GMFSS Fortuna

The All-In-One GMFSS: Dedicated for Anime Video Frame Interpolation

https://github.com/98mxr/GMFSS_Fortuna

IFRNet

@InProceedings{Kong_2022_CVPR, 
  author = {Kong, Lingtong and Jiang, Boyuan and Luo, Donghao and Chu, Wenqing and Huang, Xiaoming and Tai, Ying and Wang, Chengjie and Yang, Jie}, 
  title = {IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation}, 
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, 
  year = {2022}
}

IFUnet

RIFE with IFUNet, FusionNet and RefineNet

https://github.com/98mxr/IFUNet

M2M

@InProceedings{hu2022m2m,
    title={Many-to-many Splatting for Efficient Video Frame Interpolation},
    author={Hu, Ping and Niklaus, Simon and Sclaroff, Stan and Saenko, Kate},
    journal={CVPR},
    year={2022}
    }

RIFE

@inproceedings{huang2022rife,
  title={Real-Time Intermediate Flow Estimation for Video Frame Interpolation},
  author={Huang, Zhewei and Zhang, Tianyuan and Heng, Wen and Shi, Boxin and Zhou, Shuchang},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  year={2022}
}

FILM

Frame interpolation in PyTorch

@inproceedings{reda2022film,
 title = {FILM: Frame Interpolation for Large Motion},
 author = {Fitsum Reda and Janne Kontkanen and Eric Tabellion and Deqing Sun and Caroline Pantofaru and Brian Curless},
 booktitle = {European Conference on Computer Vision (ECCV)},
 year = {2022}
}
@misc{film-tf,
  title = {Tensorflow 2 Implementation of "FILM: Frame Interpolation for Large Motion"},
  author = {Fitsum Reda and Janne Kontkanen and Eric Tabellion and Deqing Sun and Caroline Pantofaru and Brian Curless},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/google-research/frame-interpolation}}
}

Sepconv

[1]  @inproceedings{Niklaus_WACV_2021,
         author = {Simon Niklaus and Long Mai and Oliver Wang},
         title = {Revisiting Adaptive Convolutions for Video Frame Interpolation},
         booktitle = {IEEE Winter Conference on Applications of Computer Vision},
         year = {2021}
     }
[2]  @inproceedings{Niklaus_ICCV_2017,
         author = {Simon Niklaus and Long Mai and Feng Liu},
         title = {Video Frame Interpolation via Adaptive Separable Convolution},
         booktitle = {IEEE International Conference on Computer Vision},
         year = {2017}
     }
[3]  @inproceedings{Niklaus_CVPR_2017,
         author = {Simon Niklaus and Long Mai and Feng Liu},
         title = {Video Frame Interpolation via Adaptive Convolution},
         booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
         year = {2017}
     }

AMT

@inproceedings{licvpr23amt,
   title={AMT: All-Pairs Multi-Field Transforms for Efficient Frame Interpolation},
   author={Li, Zhen and Zhu, Zuo-Liang and Han, Ling-Hao and Hou, Qibin and Guo, Chun-Le and Cheng, Ming-Ming},
   booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
   year={2023}
}

ST-MFNet

@InProceedings{Danier_2022_CVPR,
    author    = {Danier, Duolikun and Zhang, Fan and Bull, David},
    title     = {ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {3521-3531}
}

FLAVR

@article{kalluri2021flavr,
  title={FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation},
  author={Kalluri, Tarun and Pathak, Deepak and Chandraker, Manmohan and Tran, Du},
  booktitle={arxiv},
  year={2021}
}
MIT License Copyright (c) 2023 Fannovel16 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

简介

暂无描述 展开 收起
README
MIT
取消

发行版

暂无发行版

贡献者

全部

近期动态

不能加载更多了
马建仓 AI 助手
尝试更多
代码解读
代码找茬
代码优化
1
https://gitee.com/saiso/ComfyUI-Frame-Interpolation.git
git@gitee.com:saiso/ComfyUI-Frame-Interpolation.git
saiso
ComfyUI-Frame-Interpolation
ComfyUI-Frame-Interpolation
main

搜索帮助