8 Star 9 Fork 20

Yanlq / zjut_mindvideo

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
Apache-2.0

MindVideo

docs license open issuesPRs

English|中文

简介

MindVideo 是一个基于MindSpore的计算机视觉研究和开发的开源视频工具箱。它收集了一系列经典的和SoTA的视觉模型,如C3D和ARN,以及它们的预训练权重和训练策略。通过解耦模块的设计,很容易将MindVideo 应用于或适应你自己的CV任务。

主要特点

  • 模块化设计

我们将视频框架分解成不同的组件,人们可以通过结合不同的模块轻松构建一个定制的视频框架。

ModularDesign.png

目前,MindVideo支持动作识别、视频跟踪、视频分割。

result.gif

result.png

MOT17_09_SDP.gif

性能结果

用MindVideo训练的模型的性能总结在benchmark.md中,其中训练配方和权重都是可用的。

安装

依赖

使用以下指令来安装依赖:

git clone https://gitee.com/ZJUT-ERCISS/zjut_mindvideo.git
cd zjut_mindvideo

# If you use vistr, the version of Python should be 3.7
# Please first install mindspore according to instructions on the official website: https://www.mindspore.cn/install

pip install -r requirements.txt
pip install -e .

数据集准备

MindVideo 支持的数据集可以从以下链接下载:

然后将所有的训练和评估数据放入一个目录中,然后将"data_root"改为data.json中的该目录。

"data_root": "/home/publicfile/dataset/tracking"

mindvideo中, 根据所使用的每个数据集的所有数据处理方法都可以在data 文件夹中找到。

快速开始

运行

每个mindvideo支持的模型都为初学者提供了可运行模块。安装MindSpore以及本仓库需要的依赖后,在tutorials 文件夹下,能够找到对应每个模型名称的文件夹,其中专门为初学者设计了可供学习的模块,可以打开.ipynb文件并运行代码。我们也支持一些参数配置以快速启动,在处理包含每个模型所需参数的YAML文件时,可使用tools 文件夹下所有模型的训练和推理接口,对于这种方法,以I3D为例,只需运行以下命令进行训练:

cd tools/classification
python train.py -c ../../mindvideo/config/i3d/i3d_rgb.yaml

使用以下指令来评估:

cd tools/classification
python eval.py -c ../../mindvideo/config/i3d/i3d_rgb.yaml

使用以下指令来推理:

cd tools/classification
python infer.py -c ../../mindvideo/config/i3d/i3d_rgb.yaml

同时, paperswithcode 是浏览mindvideo模型的良好资源,不同的模型都可以在以下链接找到:

Model Link
ARN https://paperswithcode.com/paper/few-shot-action-recognition-via-improved#code
C3D https://paperswithcode.com/paper/learning-spatiotemporal-features-with-3d#code
Fairmot https://paperswithcode.com/paper/a-simple-baseline-for-multi-object-tracking#code
I3D https://paperswithcode.com/paper/quo-vadis-action-recognition-a-new-model-and#code
Nonlocal https://paperswithcode.com/paper/non-local-neural-networks
R(2+1)D https://paperswithcode.com/paper/a-closer-look-at-spatiotemporal-convolutions
Vist https://paperswithcode.com/paper/video-swin-transformer#code
X3D https://paperswithcode.com/paper/x3d-expanding-architectures-for-efficient
Vistr https://paperswithcode.com/paper/end-to-end-video-instance-segmentation-with#code

模型检查点

以下使预训练模型的下载链接:

Model Link
ARN arn.ckpt
C3D c3d.ckpt
Fairmot fairmot_dla34-30_886.ckpt
I3D i3d_rgb_kinetics400.ckpt
Nonlocal nonlocal_mindspore.ckpt
R(2+1)D r2plus1d18_kinetic400.ckpt
Vistr vistr_r50_all.ckpt
X3D x3d_l_kinetics400.ckpt
x3d_m_kinetics400.ckpt
x3d_s_kinetics400.ckpt
x3d_xs_kinetics400.ckpt
Vist ms_swin_base_patch244_window877_kinetics400_22k.ckpt
ms_swin_small_patch244_window877_kinetics400_1k.ckpt
ms_swin_tiny_patch244_window877_kinetics400_1k.ckpt

Model List

  • C3D 动作识别

  • I3D 动作识别

  • X3D 动作识别

  • R(2+1)d 动作识别

  • NonLocal 动作识别

  • ViST 动作识别

  • fairMOT 单次学习跟踪

  • VisTR实例分割

  • ARN 少样本动作识别

    主干分支的工作原理是 MindSpore 1.5+.

文档建立

API文档教程

  1. 克隆mindvideo
git clone https://gitee.com/ZJUT-ERCISS/zjut_mindvideo.git
cd zjut_mindvideo
  1. 安装文档的依赖
pip install -r requirements.txt
  1. 建立文档
make html
  1. 通过浏览器打开 build/html/index.html

支持算法

支持算法:

  • 动作识别
  • 视频跟踪
  • 视频实例分割

基本结构

MindVideo是一个基于MindSpore的Python包,提供以下高级功能:

  • c3d和resnet系列等模型的基础骨干。.
  • 面向领域的丰富数据集界面
  • 丰富的可视化IO接口.

BaseArchitecture.png

反馈与联系

动态版本仍在开发中,如果您发现任何问题或对新功能有想法,请不要犹豫,通过 Gitee Issues与我们联系

贡献

我们感谢所有对改善 MindVideo的贡献。请参考CONTRIBUTING.md的贡献指南

许可证

这个项目是在Apache 2.0 license下发布的。

致谢

MindSpore 是一个开源项目,欢迎任何贡献和反馈。我们希望这个工具箱和基准能够通过提供一个灵活和标准化的工具箱,重新实施现有的方法和开发他们自己的新的计算机视觉方法,为日益增长的研究界服务。贡献人在 CONTRIBUTERS.md中列出

引用

如果你觉得mindvideo对你的项目有帮助,请考虑引用:

@misc{MindVideo 2022,
    title={{MindVideo}:MindVideo Toolbox and Benchmark},
    author={MindVideo Contributors},
    howpublished = {\url{https://gitee.com/ZJUT-ERCISS/zjut_mindvideo}},
    year={2022}
}
Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

简介

zjut mindvideo 展开 收起
Apache-2.0
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
Python
1
https://gitee.com/ZJUT-ERCISS/zjut_mindvideo.git
git@gitee.com:ZJUT-ERCISS/zjut_mindvideo.git
ZJUT-ERCISS
zjut_mindvideo
zjut_mindvideo
master

搜索帮助

53164aa7 5694891 3bd8fe86 5694891