2 Star 3 Fork 0

Tencent / Forward

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
贡献代码
同步代码
取消
提示: 由于 Git 不支持空文件夾,创建文件夹后会生成空的 .keep 文件
Loading...
README
MIT

Forward 深度学习推理加速框架

License Build Status



[English Version]

什么是 Forward

Forward 是一款腾讯研发的 GPU 高性能推理加速框架。它提出了一种解析方案,可直接加载主流框架模型(Tensorflow / PyTorch / Keras / ONNX)转换成 TensorRT 推理加速引擎,帮助用户节省中间繁杂的模型转换或网络构建步骤。相对于直接使用 TensorRT,Forward 更易用以及更容易扩展支持更多模型和算子。目前,Forward 除了覆盖支持主流的 CV,NLP 及推荐领域的深度学习模型外,还支持一些诸如 BERT,FaceSwap,StyleTransfer 这类高级模型。

为什么选择 Forward

  • 模型性能优化高:基于 TensorRT API 开发网络层级的支持,保证对于通用网络层级的推理性能优化处于最优级别;
  • 模型支持范围广:除了通用的 CV,NLP,及推荐类模型,还支持一些诸如 BERT,FaceSwap,StyleTransfer 这类高级模型;
  • 多种推理模式:支持 FLOAT / HALF / INT8 推理模式;
  • 接口简单易用:直接导入已训练好的 Tensorflow(.pb) / PyTorch(.pth) / Keras(.h5) / ONNX(.onnx) 模型文件,隐式转换为高性能的推理 Engine 进行推理加速;
  • 支持自研扩展:可根据业务模型扩展支持自定义网络层级
  • 支持 C++ 和 Python 接口调用

快速上手 Forward

环境依赖

  • NVIDIA CUDA >= 10.0, CuDNN >= 7 (推荐 CUDA 10.2 以上)
  • TensorRT >= 7.0.0.11 (推荐 TensorRT-7.2.1.6)
  • CMake >= 3.12.2
  • GCC >= 5.4.0, ld >= 2.26.1
  • PyTorch >= 1.7.0
  • TensorFlow >= 1.15.0 (若使用 Linux 操作系统,需额外下载 Tensorflow 1.15.0,并将解压出来的 .so 文件拷贝至 Forward/source/third_party/tensorflow/lib 目录下)
  • Keras HDF5 (从 Forward/source/third_party/hdf5 源码构建)

项目构建

使用 CMake 进行构建生成 Makefiles 或者 Visual Studio 项目。根据使用目的,Forward 可构建成适用于不同框架的库,如 Fwd-Torch、Fwd-Python-Torch、Fwd-Tf、Fwd-Python-Tf、Fwd-Keras、Fwd-Python-Keras、Fwd-Onnx 和 Fwd-Python-Onnx。

以 Linux 平台构建 Fwd-Tf 为例,

步骤一:克隆项目

1 git clone https://github.com/Tencent/Forward.git

步骤二:下载 Tensorflow 1.15.0(仅在 Linux 平台使用 Tensorflow 框架推理时需要)

1 cd Forward/source/third_party/tensorflow/
2 wget https://github.com/neargye-forks/tensorflow/releases/download/v1.15.0/libtensorflow-cpu-linux-x86_64-1.15.0.tar.gz
3 tar -xvf libtensorflow-gpu-linux-x86_64-1.15.0.tar.gz

步骤三:创建 build 文件夹

1 cd ~/Forward/
2 rm -rf build
3 mkdir -p build
4 cd build/

步骤四:使用 cmake 生成构建关系,需指定 TensorRT_ROOT 安装路径

1 cmake ..  -DTensorRT_ROOT=<path_to_TensorRT> -DENABLE_TENSORFLOW=ON -DENABLE_UNIT_TESTS=ON

步骤五:使用 make 构建项目

1 make -j

步骤六:运行 unit_test 验证项目是否构建成功

cd bin/
./unit_test --gtest_filter=TestTfNodes.*

# 出现已下提示表示项目构建成
# [       OK ] TestTfNodes.ZeroPadding (347 ms)
# [----------] 22 tests from TestTfNodes (17555 ms total)

# [----------] Global test environment tear-down
# [==========] 22 tests from 1 test case ran. (17555 ms total)
# [  PASSED  ] 22 tests.

更多构建流程可参考 CMake 构建流程

Forward-Cpp 使用

参考 Demo for using Forward-Cpp in Linux

Forward-Python 使用

参考 Demo for using Forward-Python

Forward-Bert 使用

Refer to Demo for using Forward-Bert

更多使用方法

注意: 模型输入名可通过模型查看器来查看, 例如用 Netron 查看。

Logging 日志

Forward 使用 easylogging++ 作为日志功能,并使用 forward_log.conf 作为日志配置文件。

  • 若工作目录中存在 forward_log.conf 文件,Forward 将使用该配置文件,更多内容可参考 Using-configuration-file
  • 若工作目录中不存在 forward_log.conf 文件,Forward 将使用默认配置,并将日志记录到 logs/myeasylog.log

forward_log.conf 文件配置样例

* GLOBAL:
  FORMAT               =  "[%level] %datetime %fbase(%line): %msg"
  FILENAME             =  "Forward.log"
  ENABLED              =  true
  TO_FILE              =  true
  TO_STANDARD_OUTPUT   =  true
  PERFORMANCE_TRACKING =  true
  MAX_LOG_FILE_SIZE    =  2097152 ## 2MB - Comment starts with two hashes (##)
  LOG_FLUSH_THRESHOLD  =  100 ## Flush after every 100 logs

模型和算子支持

当前 Forward 的模型与算子支持如下所示,如有需要添加更多支持的,欢迎联系添加 Issue 反馈。如需要自行扩展添加支持的,可参考 开源共建:扩展添加支持操作的流程

模型

算子

参考资料

  1. 推理流程构建过程
  2. 推理引擎使用方法
  3. 工具与测试
  4. 常见问题

贡献

  1. 联系进入开源共建交流讨论群,QQ 群:776314438
  2. 请参考 CONTRIBUTING.md 进行开源共建。

Aster JIAN

Zexi YUAN

Ao LI

Paul LU

Zhaoyi LUO

Jett Hu

Ryosuke1eep

感谢所有贡献者,欢迎更多人加入一起贡献。

许可证

详情见 LISENCE

Tencent is pleased to support the open source community by making Forward-深度学习推理加速框架 available. Copyright (C) 2021 THL A29 Limited, a Tencent company. All rights reserved. The below software in this distribution may have been modified by THL A29 Limited (“Tencent Modifications”). All Tencent Modifications are Copyright (C) THL A29 Limited. Forward-深度学习推理加速框架 is licensed under the Apache License Version 2.0 except for the third-party components listed below. Terms of the Apache License Version 2.0: --------------------------------------------- Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. “License” shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. “Licensor” shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. “Legal Entity” shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, “control” means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. “You” (or “Your”) shall mean an individual or Legal Entity exercising permissions granted by this License. “Source” form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. “Object” form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. “Work” shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). “Derivative Works” shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. “Contribution” shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, “submitted” means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as “Not a Contribution.” “Contributor” shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: You must give any other recipients of the Work or Derivative Works a copy of this License; and You must cause any modified files to carry prominent notices stating that You changed the files; and You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and If the Work includes a “NOTICE” text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS Other dependencies and licenses: Open Source Software licensed under the BSD 3-Clause license: The below software in this distribution may have been modified by THL A29 Limited (“Tencent Modifications”). All Tencent Modifications are Copyright (C) 2021 THL A29 Limited. --------------------------------------------- 1. pybind11 Copyright (c) 2016 Wenzel Jakob <wenzel.jakob@epfl.ch>, All rights reserved. 2. cub Copyright (c) 2010-2011, Duane Merrill. All rights reserved. Copyright (c) 2011-2018, NVIDIA CORPORATION. All rights reserved. 3. googletest Copyright 2008, Google Inc. All rights reserved. Please note this software may have been modified by Tencent. Terms of the BSD 3-Clause License: --------------------------------------------- Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Open Source Software licensed under the MIT license: The below software in this distribution may have been modified by THL A29 Limited (“Tencent Modifications”). All Tencent Modifications are Copyright (C) 2021 THL A29 Limited. --------------------------------------------- 1. easylogging Copyright (c) 2017 muflihun.com Terms of the MIT License: --------------------------------------------- Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. Open Source Software Licensed under the MIT and Other Licenses of the Third-Party Components therein: --------------------------------------------- 1. json Copyright (c) 2013-2019 Niels Lohmann A copy of MIT license is included in this file. For amalgamate: Acopy of BSD 3-Clause license is included in this file. For cpplint: Acopy of BSD 3-Clause license is included in this file. Open Source Software Licensed under the MIT and the Apache version 2.0 license: --------------------------------------------- 1. hello_tf_c_api Copyright (c) 2018 - 2020 Daniil Goncharov A copy of MIT license is included in this file. For tensorflow: A copy of Apache License Version 2.0 is included in this file.{"code":0,"data":null,"message":null,"timestamp":1615171086405}

简介

Forward 深度学习推理加速框架 展开 收起
Python
MIT
取消

发行版

暂无发行版

贡献者

全部

近期动态

加载更多
不能加载更多了
Python
1
https://gitee.com/mirrors_Tencent/Forward.git
git@gitee.com:mirrors_Tencent/Forward.git
mirrors_Tencent
Forward
Forward
master

搜索帮助