From 6fdca1a2b0e907463e687579d6a9cefdc60cf800 Mon Sep 17 00:00:00 2001 From: Xinrui Chen Date: Tue, 20 May 2025 20:25:35 +0800 Subject: [PATCH] [Docs] Update the README_CN.md and README.md --- README.md | 277 ++++++++++++++++++--------------------------------- README_CN.md | 271 +++++++++++++++++-------------------------------- 2 files changed, 191 insertions(+), 357 deletions(-) diff --git a/README.md b/README.md index 155e10064..f71cc920b 100644 --- a/README.md +++ b/README.md @@ -6,79 +6,85 @@ ## 1. Introduction -The goal of the MindFormers suite is to build a full-process development suite for foundation model training, fine-tuning, evaluation, inference, and deployment. It provides mainstream Transformer-based pre-trained models and SOTA downstream task applications in the industry, covering various parallel features. It is expected to help users easily implement foundation model training and innovative R&D. +The goal of the MindSpore Transformers suite is to build a full-process development suite for Large model pre-training, fine-tuning, evaluation, inference, and deployment. It provides mainstream Transformer-based Large Language Models (LLMs) and Multimodal Models (MMs). It is expected to help users easily realize the full process of large model development. -Based on MindSpore's built-in parallel technology and component-based design, the MindFormers suite has the following features: +Based on MindSpore's built-in parallel technology and component-based design, the MindSpore Transformers suite has the following features: -- Seamless switch from single-device to large-scale cluster training with just one line of code -- Flexible and easy-to-use personalized parallel configuration -- Automatic topology awareness, efficiently combining data parallelism and model parallelism strategies -- One-click launch for single-device/multi-device training, fine-tuning, evaluation, and inference for any task -- Support for users to configure any module in a modular way, such as optimizers, learning strategies, and network assembly -- High-level usability APIs such as Trainer, pipeline, and AutoClass. -- Built-in SOTA weight auto-download and loading functionality -- Seamless migration and deployment support for AI computing centers +- One-click initiation of single or multi card pre-training, fine-tuning, evaluation, inference, and deployment processes for large models; +- Provides rich multi-dimensional hybrid parallel capabilities for flexible and easy-to-use personalized configuration; +- System-level deep optimization on large model training and inference, native support for ultra-large-scale cluster efficient training and inference, rapid fault recovery; +- Support for configurable development of task components. Any module can be enabled by unified configuration, including model network, optimizer, learning rate policy, etc.; +- Provide real-time visualization of training accuracy/performance monitoring indicators. -For details about MindFormers tutorials and API documents, see **[MindFormers Documentation](https://www.mindspore.cn/mindformers/docs/en/dev/index.html)**. The following are quick jump links to some of the key content: +For details about MindSpore Transformers tutorials and API documents, see **[MindSpore Transformers Documentation](https://www.mindspore.cn/mindformers/docs/en/r1.5.0/index.html)**. The following are quick jump links to some of the key content: -- [Calling Source Code to Start](https://www.mindspore.cn/mindformers/docs/en/dev/quick_start/source_code_start.html) -- [Pre-training](https://www.mindspore.cn/mindformers/docs/en/dev/usage/pre_training.html) -- [Fine-Tuning](https://www.mindspore.cn/mindformers/docs/en/dev/usage/sft_tuning.html) -- [MindIE Service Deployment](https://www.mindspore.cn/mindformers/docs/en/dev/usage/mindie_deployment.html) +- 📝 [Quick Launch](https://www.mindspore.cn/mindformers/docs/en/dev/quick_start/source_code_start.html) +- 📝 [Pre-training](https://www.mindspore.cn/mindformers/docs/en/dev/usage/pre_training.html) +- 📝 [Fine-Tuning](https://www.mindspore.cn/mindformers/docs/en/dev/usage/sft_tuning.html) +- 📝 [Evaluation](https://www.mindspore.cn/mindformers/docs/en/dev/usage/evaluation.html) +- 📝 [Service-oriented Deployment](https://www.mindspore.cn/mindformers/docs/en/dev/usage/mindie_deployment.html) -If you have any suggestions on MindFormers, contact us through an issue, and we will address it promptly. +If you have any suggestions on MindSpore Transformers, contact us through an issue, and we will address it promptly. ### Models List -The following table lists models supported by MindFormers. - -| Model | Specifications | Model Type | Latest Version | -|:--------------------------------------------------------------------------------------------------------|:------------------------------|:----------------:|:----------------------:| -| [CodeLlama](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/codellama.md) | 34B | Dense LLM | In-development version | -| [CogVLM2-Image](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/cogvlm2_image.md) | 19B | MM | In-development version | -| [CogVLM2-Video](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/cogvlm2_video.md) | 13B | MM | In-development version | -| [DeepSeek-V3](https://gitee.com/mindspore/mindformers/blob/dev/research/deepseek3) | 671B | Sparse LLM | In-development version | -| [DeepSeek-V2](https://gitee.com/mindspore/mindformers/blob/dev/research/deepseek2) | 236B | Sparse LLM | In-development version | -| [DeepSeek-Coder-V1.5](https://gitee.com/mindspore/mindformers/blob/dev/research/deepseek1_5) | 7B | Dense LLM | In-development version | -| [DeepSeek-Coder](https://gitee.com/mindspore/mindformers/blob/dev/research/deepseek) | 33B | Dense LLM | In-development version | -| [GLM4](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/glm4.md) | 9B | Dense LLM | In-development version | -| [GLM3-32K](https://gitee.com/mindspore/mindformers/blob/dev/research/glm32k) | 6B | Dense LLM | In-development version | -| [GLM3](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/glm3.md) | 6B | Dense LLM | In-development version | -| [InternLM2](https://gitee.com/mindspore/mindformers/blob/dev/research/internlm2) | 7B/20B | Dense LLM | In-development version | -| [Llama3.1](https://gitee.com/mindspore/mindformers/blob/dev/research/llama3_1) | 8B/70B | Dense LLM | In-development version | -| [Llama3](https://gitee.com/mindspore/mindformers/blob/dev/research/llama3) | 8B/70B | Dense LLM | In-development version | -| [Llama2](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/llama2.md) | 7B/13B/70B | Dense LLM | In-development version | -| [Mixtral](https://gitee.com/mindspore/mindformers/blob/dev/research/mixtral) | 8x7B | Sparse LLM | In-development version | -| [Qwen2](https://gitee.com/mindspore/mindformers/blob/dev/research/qwen2) | 0.5B/1.5B/7B/57B/57B-A14B/72B | Dense/Sparse LLM | In-development version | -| [Qwen1.5](https://gitee.com/mindspore/mindformers/blob/dev/research/qwen1_5) | 7B/14B/72B | Dense LLM | In-development version | -| [Qwen-VL](https://gitee.com/mindspore/mindformers/blob/dev/research/qwenvl) | 9.6B | MM | In-development version | -| [Whisper](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/whisper.md) | 1.5B | MM | In-development version | -| [Yi](https://gitee.com/mindspore/mindformers/blob/dev/research/yi) | 6B/34B | Dense LLM | In-development version | -| [Baichuan2](https://gitee.com/mindspore/mindformers/blob/r1.3.0/research/baichuan2/baichuan2.md) | 7B/13B | Dense LLM | 1.3.2 | -| [GLM2](https://gitee.com/mindspore/mindformers/blob/r1.3.0/docs/model_cards/glm2.md) | 6B | Dense LLM | 1.3.2 | -| [GPT2](https://gitee.com/mindspore/mindformers/blob/r1.3.0/docs/model_cards/gpt2.md) | 124M/13B | Dense LLM | 1.3.2 | -| [InternLM](https://gitee.com/mindspore/mindformers/blob/r1.3.0/research/internlm/internlm.md) | 7B/20B | Dense LLM | 1.3.2 | -| [Qwen](https://gitee.com/mindspore/mindformers/blob/r1.3.0/research/qwen/qwen.md) | 7B/14B | Dense LLM | 1.3.2 | -| [CodeGeex2](https://gitee.com/mindspore/mindformers/blob/r1.1.0/docs/model_cards/codegeex2.md) | 6B | Dense LLM | 1.1.0 | -| [WizardCoder](https://gitee.com/mindspore/mindformers/blob/r1.1.0/research/wizardcoder/wizardcoder.md) | 15B | Dense LLM | 1.1.0 | -| [Baichuan](https://gitee.com/mindspore/mindformers/blob/r1.0/research/baichuan/baichuan.md) | 7B/13B | Dense LLM | 1.0 | -| [Blip2](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/blip2.md) | 8.1B | MM | 1.0 | -| [Bloom](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/bloom.md) | 560M/7.1B/65B/176B | Dense LLM | 1.0 | -| [Clip](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/clip.md) | 149M/428M | MM | 1.0 | -| [CodeGeex](https://gitee.com/mindspore/mindformers/blob/r1.0/research/codegeex/codegeex.md) | 13B | Dense LLM | 1.0 | -| [GLM](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/glm.md) | 6B | Dense LLM | 1.0 | -| [iFlytekSpark](https://gitee.com/mindspore/mindformers/blob/r1.0/research/iflytekspark/iflytekspark.md) | 13B | Dense LLM | 1.0 | -| [Llama](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/llama.md) | 7B/13B | Dense LLM | 1.0 | -| [MAE](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/mae.md) | 86M | MM | 1.0 | -| [Mengzi3](https://gitee.com/mindspore/mindformers/blob/r1.0/research/mengzi3/mengzi3.md) | 13B | Dense LLM | 1.0 | -| [PanguAlpha](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/pangualpha.md) | 2.6B/13B | Dense LLM | 1.0 | -| [SAM](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/sam.md) | 91M/308M/636M | MM | 1.0 | -| [Skywork](https://gitee.com/mindspore/mindformers/blob/r1.0/research/skywork/skywork.md) | 13B | Dense LLM | 1.0 | -| [Swin](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/swin.md) | 88M | MM | 1.0 | -| [T5](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/t5.md) | 14M/60M | Dense LLM | 1.0 | -| [VisualGLM](https://gitee.com/mindspore/mindformers/blob/r1.0/research/visualglm/visualglm.md) | 6B | MM | 1.0 | -| [Ziya](https://gitee.com/mindspore/mindformers/blob/r1.0/research/ziya/ziya.md) | 13B | Dense LLM | 1.0 | -| [Bert](https://gitee.com/mindspore/mindformers/blob/r0.8/docs/model_cards/bert.md) | 4M/110M | Dense LLM | 0.8 | +The following table lists models supported by MindSpore Transformers. + +| Model | Specifications | Model Type | Latest Version | +|:--------------------------------------------------------------------------------------------------------|:------------------------------|:----------------:|:-----------------------------:| +| [DeepSeek-V3](https://gitee.com/mindspore/mindformers/blob/dev/research/deepseek3) | 671B | Sparse LLM | In-development version, 1.5.0 | +| [GLM4](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/glm4.md) | 9B | Dense LLM | In-development version, 1.5.0 | +| [Llama3.1](https://gitee.com/mindspore/mindformers/blob/dev/research/llama3_1) | 8B/70B | Dense LLM | In-development version, 1.5.0 | +| [Qwen2.5](https://gitee.com/mindspore/mindformers/blob/dev/research/qwen2_5) | 0.5B/1.5B/7B/14B/32B/72B | Dense LLM | In-development version, 1.5.0 | +| [TeleChat2](https://gitee.com/mindspore/mindformers/blob/dev/research/telechat2) | 7B/35B/115B | Dense LLM | In-development version, 1.5.0 | +| [CodeLlama](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/codellama.md) | 34B | Dense LLM | 1.5.0 | +| [CogVLM2-Image](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/cogvlm2_image.md) | 19B | MM | 1.5.0 | +| [CogVLM2-Video](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/cogvlm2_video.md) | 13B | MM | 1.5.0 | +| [DeepSeek-V2](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/deepseek2) | 236B | Sparse LLM | 1.5.0 | +| [DeepSeek-Coder-V1.5](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/deepseek1_5) | 7B | Dense LLM | 1.5.0 | +| [DeepSeek-Coder](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/deepseek) | 33B | Dense LLM | 1.5.0 | +| [GLM3-32K](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/glm32k) | 6B | Dense LLM | 1.5.0 | +| [GLM3](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/glm3.md) | 6B | Dense LLM | 1.5.0 | +| [InternLM2](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/internlm2) | 7B/20B | Dense LLM | 1.5.0 | +| [Llama3.2](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/llama3_2.md) | 3B | Dense LLM | 1.5.0 | +| [Llama3.2-Vision](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/mllama.md) | 11B | MM | 1.5.0 | +| [Llama3](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/llama3) | 8B/70B | Dense LLM | 1.5.0 | +| [Llama2](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/llama2.md) | 7B/13B/70B | Dense LLM | 1.5.0 | +| [Mixtral](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/mixtral) | 8x7B | Sparse LLM | 1.5.0 | +| [Qwen2](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/qwen2) | 0.5B/1.5B/7B/57B/57B-A14B/72B | Dense/Sparse LLM | 1.5.0 | +| [Qwen1.5](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/qwen1_5) | 7B/14B/72B | Dense LLM | 1.5.0 | +| [Qwen-VL](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/qwenvl) | 9.6B | MM | 1.5.0 | +| [TeleChat](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/telechat) | 7B/12B/52B | Dense LLM | 1.5.0 | +| [Whisper](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/whisper.md) | 1.5B | MM | 1.5.0 | +| [Yi](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/yi) | 6B/34B | Dense LLM | 1.5.0 | +| [YiZhao](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/yizhao) | 12B | Dense LLM | 1.5.0 | +| [Baichuan2](https://gitee.com/mindspore/mindformers/blob/r1.3.0/research/baichuan2/baichuan2.md) | 7B/13B | Dense LLM | 1.3.2 | +| [GLM2](https://gitee.com/mindspore/mindformers/blob/r1.3.0/docs/model_cards/glm2.md) | 6B | Dense LLM | 1.3.2 | +| [GPT2](https://gitee.com/mindspore/mindformers/blob/r1.3.0/docs/model_cards/gpt2.md) | 124M/13B | Dense LLM | 1.3.2 | +| [InternLM](https://gitee.com/mindspore/mindformers/blob/r1.3.0/research/internlm/internlm.md) | 7B/20B | Dense LLM | 1.3.2 | +| [Qwen](https://gitee.com/mindspore/mindformers/blob/r1.3.0/research/qwen/qwen.md) | 7B/14B | Dense LLM | 1.3.2 | +| [CodeGeex2](https://gitee.com/mindspore/mindformers/blob/r1.1.0/docs/model_cards/codegeex2.md) | 6B | Dense LLM | 1.1.0 | +| [WizardCoder](https://gitee.com/mindspore/mindformers/blob/r1.1.0/research/wizardcoder/wizardcoder.md) | 15B | Dense LLM | 1.1.0 | +| [Baichuan](https://gitee.com/mindspore/mindformers/blob/r1.0/research/baichuan/baichuan.md) | 7B/13B | Dense LLM | 1.0 | +| [Blip2](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/blip2.md) | 8.1B | MM | 1.0 | +| [Bloom](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/bloom.md) | 560M/7.1B/65B/176B | Dense LLM | 1.0 | +| [Clip](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/clip.md) | 149M/428M | MM | 1.0 | +| [CodeGeex](https://gitee.com/mindspore/mindformers/blob/r1.0/research/codegeex/codegeex.md) | 13B | Dense LLM | 1.0 | +| [GLM](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/glm.md) | 6B | Dense LLM | 1.0 | +| [iFlytekSpark](https://gitee.com/mindspore/mindformers/blob/r1.0/research/iflytekspark/iflytekspark.md) | 13B | Dense LLM | 1.0 | +| [Llama](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/llama.md) | 7B/13B | Dense LLM | 1.0 | +| [MAE](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/mae.md) | 86M | MM | 1.0 | +| [Mengzi3](https://gitee.com/mindspore/mindformers/blob/r1.0/research/mengzi3/mengzi3.md) | 13B | Dense LLM | 1.0 | +| [PanguAlpha](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/pangualpha.md) | 2.6B/13B | Dense LLM | 1.0 | +| [SAM](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/sam.md) | 91M/308M/636M | MM | 1.0 | +| [Skywork](https://gitee.com/mindspore/mindformers/blob/r1.0/research/skywork/skywork.md) | 13B | Dense LLM | 1.0 | +| [Swin](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/swin.md) | 88M | MM | 1.0 | +| [T5](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/t5.md) | 14M/60M | Dense LLM | 1.0 | +| [VisualGLM](https://gitee.com/mindspore/mindformers/blob/r1.0/research/visualglm/visualglm.md) | 6B | MM | 1.0 | +| [Ziya](https://gitee.com/mindspore/mindformers/blob/r1.0/research/ziya/ziya.md) | 13B | Dense LLM | 1.0 | +| [Bert](https://gitee.com/mindspore/mindformers/blob/r0.8/docs/model_cards/bert.md) | 4M/110M | Dense LLM | 0.8 | + +The model maintenance strategy follows the [Life Cycle And Version Matching Strategy](#4-life-cycle-and-version-matching-strategy) of the corresponding latest supported version. ## 2. Installation @@ -88,20 +94,22 @@ Currently, the [Atlas 800T A2](https://www.hiascend.com/hardware/ai-server?tag=9 Python 3.11.4 is recommended for the current suite. -| MindFormers | MindSpore | CANN | Driver/Firmware | Image Link | +| MindSpore Transformers | MindSpore | CANN | Driver/Firmware | Image Link | |:----------------------:|:----------------------:|:----------------------:|:----------------------:|:------------:| | In-development version | In-development version | In-development version | In-development version | Not involved | Historical Version Supporting Relationships: -| MindFormers | MindSpore | CANN | Driver/Firmware | Image Link | -|:-----------:|:--------------------------------------------:|:------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------:|:--------------------------------------------------------------------:| -| r1.3.0 | [2.4.0](https://www.mindspore.cn/install/en) | [8.0.RC3.beta1](https://www.hiascend.com/developer/download/community/result?module=cann&cann=8.0.RC3.beta1) | [24.1.RC3](https://www.hiascend.com/hardware/firmware-drivers/community) | [Link](http://mirrors.cn-central-221.ovaijisuan.com/detail/154.html) | -| r1.2.0 | [2.3.0](https://www.mindspore.cn/install/en) | [8.0.RC2.beta1](https://www.hiascend.com/developer/download/community/result?module=cann&cann=8.0.RC2.beta1) | [24.1.RC2](https://www.hiascend.com/hardware/firmware-drivers/community) | [Link](http://mirrors.cn-central-221.ovaijisuan.com/detail/138.html) | +| MindSpore Transformers | MindSpore | CANN | Driver/Firmware | Image Link | +|:----------------------:|:---------------------------------------------:|:--------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------:|:--------------------------------------------------------------------:| +| 1.5.0 | [2.6.0-rc1](https://www.mindspore.cn/install) | [8.1.RC1](https://www.hiascend.com/document/detail/en/canncommercial/81RC1/softwareinst/instg/instg_0000.html) | [25.0.RC1](https://www.hiascend.com/document/detail/en/canncommercial/81RC1/softwareinst/instg/instg_0000.html) | Coming soon | +| 1.3.2 | [2.4.10](https://www.mindspore.cn/versions) | [8.0.0](https://www.hiascend.com/document/detail/en/canncommercial/800/softwareinst/instg/instg_0000.html) | [24.1.0](https://www.hiascend.com/document/detail/en/canncommercial/800/softwareinst/instg/instg_0000.html) | [Link](http://mirrors.cn-central-221.ovaijisuan.com/detail/168.html) | +| 1.3.0 | [2.4.0](https://www.mindspore.cn/versions) | [8.0.RC3](https://www.hiascend.com/document/detail/en/canncommercial/80RC3/softwareinst/instg/instg_0000.html) | [24.1.RC3](https://www.hiascend.com/document/detail/en/canncommercial/80RC3/softwareinst/instg/instg_0000.html) | [Link](http://mirrors.cn-central-221.ovaijisuan.com/detail/154.html) | +| 1.2.0 | [2.3.0](https://www.mindspore.cn/versions) | [8.0.RC2](https://www.hiascend.com/document/detail/en/canncommercial/80RC2/softwareinst/instg/instg_0000.html) | [24.1.RC2](https://www.hiascend.com/document/detail/en/canncommercial/80RC2/softwareinst/instg/instg_0000.html) | [Link](http://mirrors.cn-central-221.ovaijisuan.com/detail/138.html) | ### Installation Using the Source Code -Currently, MindFormers can be compiled and installed using the source code. You can run the following commands to install MindFormers: +Currently, MindSpore Transformers can be compiled and installed using the source code. You can run the following commands to install MindSpore Transformers: ```shell git clone -b dev https://gitee.com/mindspore/mindformers.git @@ -111,121 +119,30 @@ bash build.sh ## 3. User Guide -MindFormers supports model pre-training, fine-tuning, inference, and evaluation. You can click a model name in [Supported Models](#supported-models) to view the document and complete the preceding tasks. The following describes the distributed startup mode and provides an example. +MindSpore Transformers supports distributed pre-training, fine-tuning, and inference tasks for large models with one click. You can click the link of each model in [Model List](#models-list) to see the corresponding documentation, and you can also refer to [Start Tasks](https://www.mindspore.cn/mindformers/docs/en/dev/function/start_tasks.html) to learn how to start the above tasks. -It is recommended that MindFormers launch model training and inference in distributed mode. Currently, the `scripts/msrun_launcher.sh` distributed launch script is provided as the main way to launch models. For details about the `msrun` feature, see [msrun Launching](https://www.mindspore.cn/tutorials/zh-CN/master/parallel/msrun_launcher.html). -The input parameters of the script are described as follows. - - | **Parameter** | **Required on Single-Node** | **Required on Multi-Node** | **Default Value** | **Description** | - |------------------|:---------------------------:|:--------------------------:|:-----------------:|---------------------------------------------------------------------| - | WORKER_NUM | ✓ | ✓ | 8 | Total number of compute devices used on all nodes | - | LOCAL_WORKER | - | ✓ | 8 | Number of compute devices used on the current node | - | MASTER_ADDR | - | ✓ | 127.0.0.1 | IP address of the primary node to be started in distributed mode | - | MASTER_PORT | - | ✓ | 8118 | Port number bound for distributed startup | - | NODE_RANK | - | ✓ | 0 | Rank ID of the current node | - | LOG_DIR | - | ✓ | output/msrun_log | Log output path. If the path does not exist, create it recursively. | - | JOIN | - | ✓ | False | Specifies whether to wait for all distributed processes to exit. | - | CLUSTER_TIME_OUT | - | ✓ | 7200 | Waiting time for distributed startup, in seconds. | - -> Note: If you need to specify `device_id` for launching, you can set the environment variable `ASCEND_RT_VISIBLE_DEVICES`. For example, to use devices 2 and 3, input `export ASCEND_RT_VISIBLE_DEVICES=2,3`. - -### Single-Node Multi-Device - -```shell -# 1. Single-node multi-device quick launch mode. Eight devices are launched by default. -bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config {CONFIG_PATH} \ - --run_mode {train/finetune/eval/predict}" - -# 2. Single-node multi-device quick launch mode. You only need to set the number of devices to be used. -bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config {CONFIG_PATH} \ - --run_mode {train/finetune/eval/predict}" WORKER_NUM - -# 3. Single-node multi-device custom launch mode. -bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config {CONFIG_PATH} \ - --run_mode {train/finetune/eval/predict}" \ - WORKER_NUM MASTER_PORT LOG_DIR JOIN CLUSTER_TIME_OUT - ``` - -- Examples - - ```shell - # Single-node multi-device quick launch mode. Eight devices are launched by default. - bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config path/to/xxx.yaml \ - --run_mode finetune" - - # Single-node multi-device quick launch mode. - bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config path/to/xxx.yaml \ - --run_mode finetune" 8 - - # Single-node multi-device custom launch mode. - bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config path/to/xxx.yaml \ - --run_mode finetune" \ - 8 8118 output/msrun_log False 300 - ``` - -### Multi-Node Multi-Device - -To execute the multi-node multi-device script for distributed training, you need to run the script on different nodes and set `MASTER_ADDR` to the IP address of the primary node. -The IP address should be the same across all nodes, and only the `NODE_RANK` parameter varies across nodes. - - ```shell - # Multi-node multi-device custom launch mode. - bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config {CONFIG_PATH} \ - --run_mode {train/finetune/eval/predict}" \ - WORKER_NUM LOCAL_WORKER MASTER_ADDR MASTER_PORT NODE_RANK LOG_DIR JOIN CLUSTER_TIME_OUT - ``` - -- Examples - - ```shell - # Node 0, with IP address 192.168.1.1, serves as the primary node. There are a total of 8 devices, with 4 devices allocated per node. - bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config {CONFIG_PATH} \ - --run_mode {train/finetune/eval/predict}" \ - 8 4 192.168.1.1 8118 0 output/msrun_log False 300 - - # Node 1, with IP address 192.168.1.2, has the same launch command as node 0, with the only difference being the NODE_RANK parameter. - bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config {CONFIG_PATH} \ - --run_mode {train/finetune/eval/predict}" \ - 8 4 192.168.1.1 8118 1 output/msrun_log False 300 - ``` - -### Single-Device Launch - -MindFormers provides the `run_mindformer.py` script as the single-device launch method. This script can be used to complete the single-device training, fine-tuning, evaluation, and inference of a model based on the model configuration file. - -```shell -# The input parameters for running run_mindformer.py will override the parameters in the model configuration file. -python run_mindformer.py --config {CONFIG_PATH} --run_mode {train/finetune/eval/predict} -``` +For more information about the functions of MindSpore Transformers, please refer to [MindSpore Transformers Documentation](https://www.mindspore.cn/mindformers/docs/en/dev/index.html). ## 4. Life Cycle And Version Matching Strategy -MindFormers version has the following five maintenance phases: +MindSpore Transformers version has the following five maintenance phases: -| **Status** | **Duration** | **Description** | -|-------------------|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Plan | 1-3 months | Planning function. | -| Develop | 3 months | Build function. | -| Preserve | 6-12 months | Incorporate all solved problems and release new versions. For MindFormers of different versions, implement a differentiated preservation plan: the preservation period of the general version is 6 months, while that of the long-term support version is 12 months. | -| No Preserve | 0—3 months | Incorporate all the solved problems, there is no full-time maintenance team, and there is no plan to release a new version. | -| End of Life (EOL) | N/A | The branch is closed and no longer accepts any modifications. | +| **Status** | **Duration** | **Description** | +|-------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------| +| Plan | 1-3 months | Planning function. | +| Develop | 3 months | Build function. | +| Preserve | 6 months | Incorporate all solved problems and release new versions. | +| No Preserve | 0—3 months | Incorporate all the solved problems, there is no full-time maintenance team, and there is no plan to release a new version. | +| End of Life (EOL) | N/A | The branch is closed and no longer accepts any modifications. | -MindFormers released version preservation policy: +MindSpore Transformers released version preservation policy: -| **MindFormers Version** | **Corresponding Label** | **Preservation Policy** | **Current Status** | **Release Time** | **Subsequent Status** | **EOL Date** | -|-------------------------|-------------------------|-------------------------|--------------------|------------------|-----------------------------------------|--------------| -| 1.3.2 | v1.3.2 | General Version | No Preserve | 2024/12/20 | No preserve expected from 2025/06/20 | | -| 1.2.0 | v1.2.0 | General Version | No Preserve | 2024/07/12 | No preserve expected from 2025/01/12 | | -| 1.1.0 | v1.1.0 | General Version | No Preserve | 2024/04/15 | End of life is expected from 2025/01/15 | 2025/01/15 | +| **MindSpore Transformers Version** | **Corresponding Label** | **Current Status** | **Release Time** | **Subsequent Status** | **EOL Date** | +|:----------------------------------:|:-----------------------:|:------------------:|:----------------:|:------------------------------------:|:------------:| +| 1.5.0 | v1.5.0 | Preserve | 2025/04/29 | No preserve expected from 2025/10/29 | 2026/01/29 | +| 1.3.2 | v1.3.2 | Preserve | 2024/12/20 | No preserve expected from 2025/06/20 | 2025/09/20 | +| 1.2.0 | v1.2.0 | End of Life | 2024/07/12 | - | 2025/04/12 | +| 1.1.0 | v1.1.0 | End of Life | 2024/04/15 | - | 2025/01/15 | ## 5. Disclaimer @@ -235,7 +152,7 @@ MindFormers released version preservation policy: ## 6. Contribution -We welcome contributions to the community. For details, see [MindFormers Contribution Guidelines](https://www.mindspore.cn/mindformers/docs/en/dev/faq/mindformers_contribution.html). +We welcome contributions to the community. For details, see [MindSpore Transformers Contribution Guidelines](https://www.mindspore.cn/mindformers/docs/en/dev/faq/mindformers_contribution.html). ## 7. License diff --git a/README_CN.md b/README_CN.md index f23c33930..66e13edef 100644 --- a/README_CN.md +++ b/README_CN.md @@ -6,25 +6,23 @@ ## 一、介绍 -MindSpore Transformers套件的目标是构建一个大模型训练、微调、评估、推理、部署的全流程开发套件，提供业内主流的Transformer类预训练模型和SOTA下游任务应用，涵盖丰富的并行特性。期望帮助用户轻松的实现大模型训练和创新研发。 +MindSpore Transformers套件的目标是构建一个大模型预训练、微调、评测、推理、部署的全流程开发套件，提供业内主流的Transformer类大语言模型（Large Language Models, LLMs）和多模态理解模型（Multimodal Models, MMs）。期望帮助用户轻松地实现大模型全流程开发。 -MindSpore Transformers套件基于MindSpore内置的并行技术和组件化设计，具备如下特点： +MindSpore Transformers套件基于MindSpore内置的多维混合并行技术和组件化设计，具备如下特点： -- 一行代码实现从单卡到大规模集群训练的无缝切换； -- 提供灵活易用的个性化并行配置； -- 能够自动进行拓扑感知，高效地融合数据并行和模型并行策略； -- 一键启动任意任务的单卡/多卡训练、微调、评估、推理流程； -- 支持用户进行组件化配置任意模块，如优化器、学习策略、网络组装等； -- 提供Trainer、pipeline、AutoClass等高阶易用性接口； -- 提供预置SOTA权重自动下载及加载功能； -- 支持人工智能计算中心无缝迁移部署； +- 一键启动模型单卡或多卡预训练、微调、评测、推理、部署流程； +- 提供丰富的多维混合并行能力可供灵活易用地进行个性化配置； +- 大模型训推系统级深度优化，原生支持超大规模集群高效训推，故障快速恢复； +- 支持任务组件配置化开发。任意模块可通过统一配置进行使能，包括模型网络、优化器、学习率策略等； +- 提供训练精度/性能监控指标实时可视化能力等。 -欲获取MindFormers相关使用教程以及API文档，请参阅[**MindFormers文档**](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/index.html)，以下提供部分内容的快速跳转链接： +欲获取MindSpore Transformers相关使用教程以及API文档，请参阅[**MindSpore Transformers文档**](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/index.html)，以下提供部分内容的快速跳转链接： -- 📝 [快速启动](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/quick_start/source_code_start.html) +- 📝 [启动任务](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/function/start_tasks.html) - 📝 [大模型预训练](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/usage/pre_training.html) - 📝 [大模型微调](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/usage/sft_tuning.html) -- 📝 [MindIE服务化部署](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/usage/mindie_deployment.html) +- 📝 [大模型评测](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/usage/evaluation.html) +- 📝 [服务化部署](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/usage/mindie_deployment.html) 如果您对MindSpore Transformers有任何建议，请通过issue与我们联系，我们将及时处理。 @@ -32,53 +30,61 @@ MindSpore Transformers套件基于MindSpore内置的并行技术和组件化设 当前MindSpore Transformers全量的模型列表如下： -| 模型名 | 支持规格 | 模型类型 | 最新支持版本 | -|:--------------------------------------------------------------------------------------------------------|:------------------------------|:------------:|:------:| -| [CodeLlama](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/codellama.md) | 34B | 稠密LLM | 在研版本 | -| [CogVLM2-Image](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/cogvlm2_image.md) | 19B | MM | 在研版本 | -| [CogVLM2-Video](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/cogvlm2_video.md) | 13B | MM | 在研版本 | -| [DeepSeek-V3](https://gitee.com/mindspore/mindformers/blob/dev/research/deepseek3) | 671B | 稀疏LLM | 在研版本 | -| [DeepSeek-V2](https://gitee.com/mindspore/mindformers/blob/dev/research/deepseek2) | 236B | 稀疏LLM | 在研版本 | -| [DeepSeek-Coder-V1.5](https://gitee.com/mindspore/mindformers/blob/dev/research/deepseek1_5) | 7B | 稠密LLM | 在研版本 | -| [DeepSeek-Coder](https://gitee.com/mindspore/mindformers/blob/dev/research/deepseek) | 33B | 稠密LLM | 在研版本 | -| [GLM4](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/glm4.md) | 9B | 稠密LLM | 在研版本 | -| [GLM3-32K](https://gitee.com/mindspore/mindformers/blob/dev/research/glm32k) | 6B | 稠密LLM | 在研版本 | -| [GLM3](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/glm3.md) | 6B | 稠密LLM | 在研版本 | -| [InternLM2](https://gitee.com/mindspore/mindformers/blob/dev/research/internlm2) | 7B/20B | 稠密LLM | 在研版本 | -| [Llama3.1](https://gitee.com/mindspore/mindformers/blob/dev/research/llama3_1) | 8B/70B | 稠密LLM | 在研版本 | -| [Llama3](https://gitee.com/mindspore/mindformers/blob/dev/research/llama3) | 8B/70B | 稠密LLM | 在研版本 | -| [Llama2](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/llama2.md) | 7B/13B/70B | 稠密LLM | 在研版本 | -| [Mixtral](https://gitee.com/mindspore/mindformers/blob/dev/research/mixtral) | 8x7B | 稀疏LLM | 在研版本 | -| [Qwen2](https://gitee.com/mindspore/mindformers/blob/dev/research/qwen2) | 0.5B/1.5B/7B/57B/57B-A14B/72B | 稠密/稀疏LLM | 在研版本 | -| [Qwen1.5](https://gitee.com/mindspore/mindformers/blob/dev/research/qwen1_5) | 7B/14B/72B | 稠密LLM | 在研版本 | -| [Qwen-VL](https://gitee.com/mindspore/mindformers/blob/dev/research/qwenvl) | 9.6B | MM | 在研版本 | -| [Whisper](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/whisper.md) | 1.5B | MM | 在研版本 | -| [Yi](https://gitee.com/mindspore/mindformers/blob/dev/research/yi) | 6B/34B | 稠密LLM | 在研版本 | -| [Baichuan2](https://gitee.com/mindspore/mindformers/blob/r1.3.0/research/baichuan2/baichuan2.md) | 7B/13B | 稠密LLM | 1.3.2 | -| [GLM2](https://gitee.com/mindspore/mindformers/blob/r1.3.0/docs/model_cards/glm2.md) | 6B | 稠密LLM | 1.3.2 | -| [GPT2](https://gitee.com/mindspore/mindformers/blob/r1.3.0/docs/model_cards/gpt2.md) | 124M/13B | 稠密LLM | 1.3.2 | -| [InternLM](https://gitee.com/mindspore/mindformers/blob/r1.3.0/research/internlm/internlm.md) | 7B/20B | 稠密LLM | 1.3.2 | -| [Qwen](https://gitee.com/mindspore/mindformers/blob/r1.3.0/research/qwen/qwen.md) | 7B/14B | 稠密LLM | 1.3.2 | -| [CodeGeex2](https://gitee.com/mindspore/mindformers/blob/r1.1.0/docs/model_cards/codegeex2.md) | 6B | 稠密LLM | 1.1.0 | -| [WizardCoder](https://gitee.com/mindspore/mindformers/blob/r1.1.0/research/wizardcoder/wizardcoder.md) | 15B | 稠密LLM | 1.1.0 | -| [Baichuan](https://gitee.com/mindspore/mindformers/blob/r1.0/research/baichuan/baichuan.md) | 7B/13B | 稠密LLM | 1.0 | -| [Blip2](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/blip2.md) | 8.1B | MM | 1.0 | -| [Bloom](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/bloom.md) | 560M/7.1B/65B/176B | 稠密LLM | 1.0 | -| [Clip](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/clip.md) | 149M/428M | MM | 1.0 | -| [CodeGeex](https://gitee.com/mindspore/mindformers/blob/r1.0/research/codegeex/codegeex.md) | 13B | 稠密LLM | 1.0 | -| [GLM](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/glm.md) | 6B | 稠密LLM | 1.0 | -| [iFlytekSpark](https://gitee.com/mindspore/mindformers/blob/r1.0/research/iflytekspark/iflytekspark.md) | 13B | 稠密LLM | 1.0 | -| [Llama](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/llama.md) | 7B/13B | 稠密LLM | 1.0 | -| [MAE](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/mae.md) | 86M | MM | 1.0 | -| [Mengzi3](https://gitee.com/mindspore/mindformers/blob/r1.0/research/mengzi3/mengzi3.md) | 13B | 稠密LLM | 1.0 | -| [PanguAlpha](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/pangualpha.md) | 2.6B/13B | 稠密LLM | 1.0 | -| [SAM](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/sam.md) | 91M/308M/636M | MM | 1.0 | -| [Skywork](https://gitee.com/mindspore/mindformers/blob/r1.0/research/skywork/skywork.md) | 13B | 稠密LLM | 1.0 | -| [Swin](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/swin.md) | 88M | MM | 1.0 | -| [T5](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/t5.md) | 14M/60M | 稠密LLM | 1.0 | -| [VisualGLM](https://gitee.com/mindspore/mindformers/blob/r1.0/research/visualglm/visualglm.md) | 6B | MM | 1.0 | -| [Ziya](https://gitee.com/mindspore/mindformers/blob/r1.0/research/ziya/ziya.md) | 13B | 稠密LLM | 1.0 | -| [Bert](https://gitee.com/mindspore/mindformers/blob/r0.8/docs/model_cards/bert.md) | 4M/110M | 稠密LLM | 0.8 | +| 模型名 | 支持规格 | 模型类型 | 最新支持版本 | +|:--------------------------------------------------------------------------------------------------------|:------------------------------|:--------:|:----------:| +| [DeepSeek-V3](https://gitee.com/mindspore/mindformers/blob/dev/research/deepseek3) | 671B | 稀疏LLM | 在研版本、1.5.0 | +| [GLM4](https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/glm4.md) | 9B | 稠密LLM | 在研版本、1.5.0 | +| [Llama3.1](https://gitee.com/mindspore/mindformers/blob/dev/research/llama3_1) | 8B/70B | 稠密LLM | 在研版本、1.5.0 | +| [Qwen2.5](https://gitee.com/mindspore/mindformers/blob/dev/research/qwen2_5) | 0.5B/1.5B/7B/14B/32B/72B | 稠密LLM | 在研版本、1.5.0 | +| [TeleChat2](https://gitee.com/mindspore/mindformers/blob/dev/research/telechat2) | 7B/35B/115B | 稠密LLM | 在研版本、1.5.0 | +| [CodeLlama](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/codellama.md) | 34B | 稠密LLM | 1.5.0 | +| [CogVLM2-Image](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/cogvlm2_image.md) | 19B | MM | 1.5.0 | +| [CogVLM2-Video](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/cogvlm2_video.md) | 13B | MM | 1.5.0 | +| [DeepSeek-V2](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/deepseek2) | 236B | 稀疏LLM | 1.5.0 | +| [DeepSeek-Coder-V1.5](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/deepseek1_5) | 7B | 稠密LLM | 1.5.0 | +| [DeepSeek-Coder](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/deepseek) | 33B | 稠密LLM | 1.5.0 | +| [GLM3-32K](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/glm32k) | 6B | 稠密LLM | 1.5.0 | +| [GLM3](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/glm3.md) | 6B | 稠密LLM | 1.5.0 | +| [InternLM2](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/internlm2) | 7B/20B | 稠密LLM | 1.5.0 | +| [Llama3.2](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/llama3_2.md) | 3B | 稠密LLM | 1.5.0 | +| [Llama3.2-Vision](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/mllama.md) | 11B | MM | 1.5.0 | +| [Llama3](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/llama3) | 8B/70B | 稠密LLM | 1.5.0 | +| [Llama2](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/llama2.md) | 7B/13B/70B | 稠密LLM | 1.5.0 | +| [Mixtral](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/mixtral) | 8x7B | 稀疏LLM | 1.5.0 | +| [Qwen2](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/qwen2) | 0.5B/1.5B/7B/57B/57B-A14B/72B | 稠密/稀疏LLM | 1.5.0 | +| [Qwen1.5](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/qwen1_5) | 7B/14B/72B | 稠密LLM | 1.5.0 | +| [Qwen-VL](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/qwenvl) | 9.6B | MM | 1.5.0 | +| [TeleChat](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/telechat) | 7B/12B/52B | 稠密LLM | 1.5.0 | +| [Whisper](https://gitee.com/mindspore/mindformers/blob/r1.5.0/docs/model_cards/whisper.md) | 1.5B | MM | 1.5.0 | +| [Yi](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/yi) | 6B/34B | 稠密LLM | 1.5.0 | +| [YiZhao](https://gitee.com/mindspore/mindformers/blob/r1.5.0/research/yizhao) | 12B | 稠密LLM | 1.5.0 | +| [Baichuan2](https://gitee.com/mindspore/mindformers/blob/r1.3.0/research/baichuan2/baichuan2.md) | 7B/13B | 稠密LLM | 1.3.2 | +| [GLM2](https://gitee.com/mindspore/mindformers/blob/r1.3.0/docs/model_cards/glm2.md) | 6B | 稠密LLM | 1.3.2 | +| [GPT2](https://gitee.com/mindspore/mindformers/blob/r1.3.0/docs/model_cards/gpt2.md) | 124M/13B | 稠密LLM | 1.3.2 | +| [InternLM](https://gitee.com/mindspore/mindformers/blob/r1.3.0/research/internlm/internlm.md) | 7B/20B | 稠密LLM | 1.3.2 | +| [Qwen](https://gitee.com/mindspore/mindformers/blob/r1.3.0/research/qwen/qwen.md) | 7B/14B | 稠密LLM | 1.3.2 | +| [CodeGeex2](https://gitee.com/mindspore/mindformers/blob/r1.1.0/docs/model_cards/codegeex2.md) | 6B | 稠密LLM | 1.1.0 | +| [WizardCoder](https://gitee.com/mindspore/mindformers/blob/r1.1.0/research/wizardcoder/wizardcoder.md) | 15B | 稠密LLM | 1.1.0 | +| [Baichuan](https://gitee.com/mindspore/mindformers/blob/r1.0/research/baichuan/baichuan.md) | 7B/13B | 稠密LLM | 1.0 | +| [Blip2](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/blip2.md) | 8.1B | MM | 1.0 | +| [Bloom](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/bloom.md) | 560M/7.1B/65B/176B | 稠密LLM | 1.0 | +| [Clip](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/clip.md) | 149M/428M | MM | 1.0 | +| [CodeGeex](https://gitee.com/mindspore/mindformers/blob/r1.0/research/codegeex/codegeex.md) | 13B | 稠密LLM | 1.0 | +| [GLM](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/glm.md) | 6B | 稠密LLM | 1.0 | +| [iFlytekSpark](https://gitee.com/mindspore/mindformers/blob/r1.0/research/iflytekspark/iflytekspark.md) | 13B | 稠密LLM | 1.0 | +| [Llama](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/llama.md) | 7B/13B | 稠密LLM | 1.0 | +| [MAE](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/mae.md) | 86M | MM | 1.0 | +| [Mengzi3](https://gitee.com/mindspore/mindformers/blob/r1.0/research/mengzi3/mengzi3.md) | 13B | 稠密LLM | 1.0 | +| [PanguAlpha](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/pangualpha.md) | 2.6B/13B | 稠密LLM | 1.0 | +| [SAM](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/sam.md) | 91M/308M/636M | MM | 1.0 | +| [Skywork](https://gitee.com/mindspore/mindformers/blob/r1.0/research/skywork/skywork.md) | 13B | 稠密LLM | 1.0 | +| [Swin](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/swin.md) | 88M | MM | 1.0 | +| [T5](https://gitee.com/mindspore/mindformers/blob/r1.0/docs/model_cards/t5.md) | 14M/60M | 稠密LLM | 1.0 | +| [VisualGLM](https://gitee.com/mindspore/mindformers/blob/r1.0/research/visualglm/visualglm.md) | 6B | MM | 1.0 | +| [Ziya](https://gitee.com/mindspore/mindformers/blob/r1.0/research/ziya/ziya.md) | 13B | 稠密LLM | 1.0 | +| [Bert](https://gitee.com/mindspore/mindformers/blob/r0.8/docs/model_cards/bert.md) | 4M/110M | 稠密LLM | 0.8 | + +模型维护策略跟随最新支持版本的[生命周期及版本配套策略](#四生命周期及版本配套策略)。 ## 二、安装 @@ -88,20 +94,22 @@ MindSpore Transformers套件基于MindSpore内置的并行技术和组件化设 当前套件建议使用的Python版本为3.11.4。 -| MindFormers | MindSpore | CANN | 固件与驱动 | 镜像链接 | -|:-----------:|:---------:|:----:|:-----:|:----:| -| 在研版本 | 在研版本 | 在研版本 | 在研版本 | 不涉及 | +| MindSpore Transformers | MindSpore | CANN | 固件与驱动 | 镜像链接 | +|:----------------------:|:---------:|:----:|:-----:|:----:| +| 在研版本 | 在研版本 | 在研版本 | 在研版本 | 不涉及 | 历史版本配套关系： -| MindFormers | MindSpore | CANN | 固件与驱动 | 镜像链接 | -|:-----------:|:------------------------------------------:|:------------------------------------------------------------------------------------------------------------:|:------------------------------------------------------------------------:|:--------------------------------------------------------------------:| -| r1.3.0 | [2.4.0](https://www.mindspore.cn/install/) | [8.0.RC3.beta1](https://www.hiascend.com/developer/download/community/result?module=cann&cann=8.0.RC3.beta1) | [24.1.RC3](https://www.hiascend.com/hardware/firmware-drivers/community) | [Link](http://mirrors.cn-central-221.ovaijisuan.com/detail/154.html) | -| r1.2.0 | [2.3.0](https://www.mindspore.cn/install/) | [8.0.RC2.beta1](https://www.hiascend.com/developer/download/community/result?module=cann&cann=8.0.RC2.beta1) | [24.1.RC2](https://www.hiascend.com/hardware/firmware-drivers/community) | [Link](http://mirrors.cn-central-221.ovaijisuan.com/detail/138.html) | +| MindSpore Transformers | MindSpore | CANN | 固件与驱动 | 镜像链接 | +|:----------------------:|:---------------------------------------------:|:--------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------:|:--------------------------------------------------------------------:| +| 1.5.0 | [2.6.0-rc1](https://www.mindspore.cn/install) | [8.1.RC1](https://www.hiascend.com/document/detail/zh/canncommercial/81RC1/softwareinst/instg/instg_0000.html) | [25.0.RC1](https://www.hiascend.com/document/detail/zh/canncommercial/81RC1/softwareinst/instg/instg_0000.html) | 即将发布 | +| 1.3.2 | [2.4.10](https://www.mindspore.cn/versions) | [8.0.0](https://www.hiascend.com/document/detail/zh/canncommercial/800/softwareinst/instg/instg_0000.html) | [24.1.0](https://www.hiascend.com/document/detail/zh/canncommercial/800/softwareinst/instg/instg_0000.html) | [Link](http://mirrors.cn-central-221.ovaijisuan.com/detail/168.html) | +| 1.3.0 | [2.4.0](https://www.mindspore.cn/versions) | [8.0.RC3](https://www.hiascend.com/document/detail/zh/canncommercial/80RC3/softwareinst/instg/instg_0000.html) | [24.1.RC3](https://www.hiascend.com/document/detail/zh/canncommercial/80RC3/softwareinst/instg/instg_0000.html) | [Link](http://mirrors.cn-central-221.ovaijisuan.com/detail/154.html) | +| 1.2.0 | [2.3.0](https://www.mindspore.cn/versions) | [8.0.RC2](https://www.hiascend.com/document/detail/zh/canncommercial/80RC2/softwareinst/instg/instg_0000.html) | [24.1.RC2](https://www.hiascend.com/document/detail/zh/canncommercial/80RC2/softwareinst/instg/instg_0000.html) | [Link](http://mirrors.cn-central-221.ovaijisuan.com/detail/138.html) | ### 源码编译安装 -MindFormers目前支持源码编译安装，用户可以执行如下命令进行安装。 +MindSpore Transformers目前支持源码编译安装，用户可以执行如下命令进行安装。 ```shell git clone -b dev https://gitee.com/mindspore/mindformers.git @@ -111,121 +119,30 @@ bash build.sh ## 三、使用指南 -MindFormers支持模型启动预训练、微调、推理、评测等功能，可点击[支持模型](#支持模型)中模型名称查看文档完成上述任务，以下为模型分布式启动方式的说明与示例。 +MindSpore Transformers支持一键启动大模型的分布式预训练、微调、推理任务，可点击[模型列表](#模型列表)中各模型的链接查看对应使用文档，同时可以参考[启动任务](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/function/start_tasks.html)以了解上述任务的启动方法。 -MindFormers推荐使用分布式方式拉起模型训练、推理等功能，目前提供`scripts/msrun_launcher.sh`分布式启动脚本作为模型的主要启动方式，`msrun`特性说明可以参考[msrun启动](https://www.mindspore.cn/tutorials/zh-CN/master/parallel/msrun_launcher.html)。 -该脚本主要输入参数说明如下： - - | **参数** | **单机是否必选** | **多机是否必选** | **默认值** | **说明** | - |------------------|:----------:|:----------:|:----------------:|------------------| - | WORKER_NUM | ✓ | ✓ | 8 | 所有节点中使用计算卡的总数 | - | LOCAL_WORKER | - | ✓ | 8 | 当前节点中使用计算卡的数量 | - | MASTER_ADDR | - | ✓ | 127.0.0.1 | 指定分布式启动主节点的ip | - | MASTER_PORT | - | ✓ | 8118 | 指定分布式启动绑定的端口号 | - | NODE_RANK | - | ✓ | 0 | 指定当前节点的rank id | - | LOG_DIR | - | ✓ | output/msrun_log | 日志输出路径，若不存在则递归创建 | - | JOIN | - | ✓ | False | 是否等待所有分布式进程退出 | - | CLUSTER_TIME_OUT | - | ✓ | 7200 | 分布式启动的等待时间，单位为秒 | - -> 注：如果需要指定`device_id`启动，可以设置环境变量`ASCEND_RT_VISIBLE_DEVICES`，如要配置使用2、3卡则输入`export ASCEND_RT_VISIBLE_DEVICES=2,3`。 - -### 单机多卡 - -```shell -# 1. 单机多卡快速启动方式，默认8卡启动 -bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config {CONFIG_PATH} \ - --run_mode {train/finetune/eval/predict}" - -# 2. 单机多卡快速启动方式，仅设置使用卡数即可 -bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config {CONFIG_PATH} \ - --run_mode {train/finetune/eval/predict}" WORKER_NUM - -# 3. 单机多卡自定义启动方式 -bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config {CONFIG_PATH} \ - --run_mode {train/finetune/eval/predict}" \ - WORKER_NUM MASTER_PORT LOG_DIR JOIN CLUSTER_TIME_OUT - ``` - -- 使用示例 - - ```shell - # 单机多卡快速启动方式，默认8卡启动 - bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config path/to/xxx.yaml \ - --run_mode finetune" - - # 单机多卡快速启动方式 - bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config path/to/xxx.yaml \ - --run_mode finetune" 8 - - # 单机多卡自定义启动方式 - bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config path/to/xxx.yaml \ - --run_mode finetune" \ - 8 8118 output/msrun_log False 300 - ``` - -### 多机多卡 - -多机多卡执行脚本进行分布式训练需要分别在不同节点运行脚本，并将参数MASTER_ADDR设置为主节点的ip地址， -所有节点设置的ip地址相同，不同节点之间仅参数NODE_RANK不同。 - - ```shell - # 多机多卡自定义启动方式 - bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config {CONFIG_PATH} \ - --run_mode {train/finetune/eval/predict}" \ - WORKER_NUM LOCAL_WORKER MASTER_ADDR MASTER_PORT NODE_RANK LOG_DIR JOIN CLUSTER_TIME_OUT - ``` - -- 使用示例 - - ```shell - # 节点0，节点ip为192.168.1.1，作为主节点，总共8卡且每个节点4卡 - bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config {CONFIG_PATH} \ - --run_mode {train/finetune/eval/predict}" \ - 8 4 192.168.1.1 8118 0 output/msrun_log False 300 - - # 节点1，节点ip为192.168.1.2，节点0与节点1启动命令仅参数NODE_RANK不同 - bash scripts/msrun_launcher.sh "run_mindformer.py \ - --config {CONFIG_PATH} \ - --run_mode {train/finetune/eval/predict}" \ - 8 4 192.168.1.1 8118 1 output/msrun_log False 300 - ``` - -### 单卡启动 - -MindFormers提供`run_mindformer.py`脚本作为单卡启动方法，该脚本可以根据模型配置文件，完成支持模型的单卡训练、微调、评估、推理流程。 - -```shell -# 运行run_mindformer.py的入参会覆盖模型配置文件中的参数 -python run_mindformer.py --config {CONFIG_PATH} --run_mode {train/finetune/eval/predict} -``` +关于MindSpore Transformers的更多功能说明可参阅[MindSpore Transformers文档](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/index.html)。 ## 四、生命周期及版本配套策略 -MindFormers版本有以下五个维护阶段： +MindSpore Transformers版本有以下五个维护阶段： -| **状态** | **期限** | **说明** | -|---------------|---------|-------------------------------------------------------------------------| -| 计划 | 1-3 个月 | 规划功能。 | -| 开发 | 3 个月 | 构建功能。 | -| 维护 | 6-12 个月 | 合入所有已解决的问题并发布新版本，对于不同版本的MindFormers，实施差异化的维护计划：标准版维护期为6个月，而长期支持版则为12个月。 | -| 无维护 | 0-3 个月 | 合入所有已解决的问题，没有专职维护团队，且不计划发布新版本。 | -| 生命周期终止（EOL） | N/A | 分支进行封闭，不再接受任何修改。 | +| **状态** | **期限** | **说明** | +|:-----------:|:------:|:-------------------------------| +| 计划 | 1-3 个月 | 规划功能。 | +| 开发 | 3 个月 | 构建功能。 | +| 维护 | 6 个月 | 合入所有已解决的问题并发布新版本。 | +| 无维护 | 0-3 个月 | 合入所有已解决的问题，没有专职维护团队，且不计划发布新版本。 | +| 生命周期终止（EOL） | N/A | 分支进行封闭，不再接受任何修改。 | -MindFormers已发布版本维护策略： +MindSpore Transformers已发布版本维护策略： -| **MindFormers版本** | **对应标签** | **维护策略** | **当前状态** | **发布时间** | **后续状态** | **EOL日期** | -|-------------------|----------|----------|----------|------------|--------------------|------------| -| 1.3.2 | v1.3.2 | 常规版本 | 维护 | 2024/12/20 | 预计2025/06/20起无维护 | | -| 1.2.0 | v1.2.0 | 常规版本 | 维护 | 2024/07/12 | 预计2025/01/12起无维护 | | -| 1.1.0 | v1.1.0 | 常规版本 | 无维护 | 2024/04/15 | 预计2025/01/15生命周期终止 | 2025/01/15 | +| **MindSpore Transformers版本** | **对应标签** | **当前状态** | **发布时间** | **后续状态** | **EOL日期** | +|:----------------------------:|:--------:|:--------:|:----------:|:----------------:|:----------:| +| 1.5.0 | v1.5.0 | 维护 | 2025/04/29 | 预计2025/10/29起无维护 | 2026/01/29 | +| 1.3.2 | v1.3.2 | 维护 | 2024/12/20 | 预计2025/06/20起无维护 | 2025/09/20 | +| 1.2.0 | v1.2.0 | 生命周期终止 | 2024/07/12 | - | 2025/04/12 | +| 1.1.0 | v1.1.0 | 生命周期终止 | 2024/04/15 | - | 2025/01/15 | ## 五、免责声明 @@ -235,7 +152,7 @@ MindFormers已发布版本维护策略： ## 六、贡献 -欢迎参与社区贡献，可参考[MindFormers贡献指南](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/faq/mindformers_contribution.html)。 +欢迎参与社区贡献，可参考[MindSpore Transformers贡献指南](https://www.mindspore.cn/mindformers/docs/zh-CN/dev/faq/mindformers_contribution.html)。 ## 七、许可证 -- Gitee