{"release":{"tag":{"name":"v1.2.0","path":"/mindspore/mindformers/tags/v1.2.0","tree_path":"/mindspore/mindformers/tree/v1.2.0","message":"# MindSpore Transformers 1.2.0 RELEASE NOTE\r\n\r\n以下为MindSpore Transformers (以下称为MindFormers) 套件 1.2.0 版本的变更日志，相较于1.1.0版本有以下关键新特性和bug\r\nfix。\r\n\r\n## 新特性\r\n\r\n- **新增模型支持带框架推理**：新增支持模型包含Qwen1.5_7b、Qwen1.5_14b、Qwen1.5_72b、Llama3_70b、Yi_34b等。\r\n- **新增模型支持bfloat16训练**：新增支持模型包含Qwen1.5_7b、Qwen1.5_14b、Qwen1.5_72b、Llama3_70b、Yi_34b等。\r\n- [AdamW优化器](https://gitee.com/mindspore/mindformers/pulls/3310)：新增AdamW优化器，对齐Megatron AdamW。\r\n- **支持MindIE进行服务化部署**：[MindIE](https://www.hiascend.com/software/mindie)，全称Mind Inference\r\n  Engine，是华为昇腾针对AI全场景业务的推理加速套件。MindFormers新增对MindIE的对接，承载在模型应用层MindIE-LLM，通过MindIE-Service对MindFormers中LLM模型进行部署。\r\n- [长序列训练](https://gitee.com/mindspore/mindformers/tree/r1.2.0/docs/feature_cards/Long_Sequence_Training.md)：新增支持长序列训练特性，通过在配置yaml文件中设置`parallel_config.context_parallel`开启序列并行，当前支持32k至256k。\r\n- [断点续训权重加载2.0](https://gitee.com/mindspore/mindformers/tree/r1.2.0/docs/feature_cards/Resume_Training.md)：断点续训场景下，新增指定续训权重功能，新增故障恢复下进行权重完整性校验并自动加载最新完整权重。\r\n- [权重自动转换2.0](https://gitee.com/mindspore/mindformers/tree/r1.2.0/docs/feature_cards/Transform_Ckpt.md)：自动权重转换新增多进程转换。\r\n\r\n## 新模型\r\n\r\n以下为新支持模型：\r\n\r\n| 模型                                                                                            | 规格                                                            |\r\n|-----------------------------------------------------------------------------------------------|---------------------------------------------------------------|\r\n| [Mixtral](https://gitee.com/mindspore/mindformers/tree/r1.2.0/research/mixtral/mixtral.md)    | Mixtral_8x7b（32k预训练、推理）                                       |\r\n| [Qwen1.5](https://gitee.com/mindspore/mindformers/tree/r1.2.0/research/qwen1_5/qwen1_5.md)    | Qwen1.5_7b（预训练、微调、推理）、Qwen1.5_14b（预训练、微调、推理）、Qwen1.5_72b（预训练） |\r\n| [Llama3](https://gitee.com/mindspore/mindformers/tree/r1.2.0/research/llama3/llama3.md)       | Llama3_70b（预训练、微调）                                            |\r\n| [Deepseek](https://gitee.com/mindspore/mindformers/tree/r1.2.0/research/deepseek/deepseek.md) | Deepseek_33b（微调）                                              |\r\n| [Yi](https://gitee.com/mindspore/mindformers/tree/r1.2.0/research/yi/yi.md)                   | Yi_6b（微调）、Yi_34b（微调）                                          |\r\n| [QwenVL](https://gitee.com/mindspore/mindformers/tree/r1.2.0/research/qwenvl/qwenvl.md)       | QwenVL_9.6b（微调、推理）                                            |\r\n\r\n## Bugfix\r\n\r\n在当前版本发布周期内，我们进行了模型/功能/易用性/文档等诸多方面的Bugfix，在此仅列举部分修复内容：\r\n\r\n- [!3345](https://gitee.com/mindspore/mindformers/pulls/3345)：修复`Linear`在`transpose_b=False`时传入正确`weight`仍报错问题。\r\n- [!3277](https://gitee.com/mindspore/mindformers/pulls/3277)：修复使用梯度累积时，`warpper`传入了错误的`micro_batch_num`问题。\r\n\r\n## 贡献者\r\n\r\n感谢以下人员做出的贡献：\r\n\r\nChenhua Geng，heqinglin，koukairui，renyujin，shuchi，陈心锐，陈子恒，冯浩，胡思超，黄磊，黄生帅，黄勇，黄子灵，倪钰鑫，苏海波，李子垠，杨星宇，牛君豪，张森镇，张又文，谭纬城，吴致远，杨星宇，刘群，曹宇麟，方泽华，金仁操，刘群，李永文，钱驾宏，吴昊天，杨璇，汪家傲，范益，陈昱坤，李洋\r\n\r\n欢迎以任何形式对项目提供贡献！","commit":{"id":"980171aaa24cf8392050045c68342fbd58105652","short_id":"980171a","title":"!3597 baichuan2_7b推理精度问题修复","title_markdown":"\u003Ca title=\"Pull Request: baichuan2_7b推理精度问题修复\" class=\"gfm gfm-pull_request\" href=\"/mindspore/mindformers/pulls/3597\"\u003E!3597\u003C/a\u003Ebaichuan2_7b推理精度问题修复","description":"Merge pull request !3597 from Yule100/cherry-pick-1721801961","description_markdown":"Merge pull request \u003Ca title=\"Pull Request: baichuan2_7b推理精度问题修复\" class=\"gfm gfm-pull_request\" href=\"/mindspore/mindformers/pulls/3597\"\u003E!3597\u003C/a\u003Efrom Yule100/cherry-pick-1721801961","message":"!3597 baichuan2_7b推理精度问题修复\nMerge pull request !3597 from Yule100/cherry-pick-1721801961","message_markdown":"\u003Ca title=\"Pull Request: baichuan2_7b推理精度问题修复\" class=\"gfm gfm-pull_request\" href=\"/mindspore/mindformers/pulls/3597\"\u003E!3597\u003C/a\u003Ebaichuan2_7b推理精度问题修复\nMerge pull request \u003Ca title=\"Pull Request: baichuan2_7b推理精度问题修复\" class=\"gfm gfm-pull_request\" href=\"/mindspore/mindformers/pulls/3597\"\u003E!3597\u003C/a\u003Efrom Yule100/cherry-pick-1721801961","detail_path":"/mindspore/mindformers/commit/980171aaa24cf8392050045c68342fbd58105652","commits_path":"/mindspore/mindformers/commits/980171aaa24cf8392050045c68342fbd58105652","tree_path":"/mindspore/mindformers/tree/980171aaa24cf8392050045c68342fbd58105652","author":{"name":"i-robot","email":"huawei_ci_bot@163.com","username":"I-am-a-robot","user_path":"/I-am-a-robot","enterprise_user_path":"/mind_spore/dashboard/members/I-am-a-robot","image_path":"no_portrait.png#i-robot-I-am-a-robot","is_gitee_user":true,"is_enterprise_user":true,"widget_url":"https://gitee.com/widgets/gitee_double_eleven.png"},"committer":{"name":"Gitee GPG Bot","email":"noreply@gitee.com","username":"gitee-bot","user_path":"/gitee-bot","enterprise_user_path":null,"image_path":"https://foruda.gitee.com/avatar/1677201213385506226/10186697_gitee-bot_1639518846.png!avatar30","is_gitee_user":true,"is_enterprise_user":false,"widget_url":""},"authored_date":"2024-07-24T08:34:15+00:00","committed_date":"2024-07-24T08:34:15+00:00","signature":null,"build_state":null},"archive_path":"/mindspore/mindformers/repository/archive/v1.2.0","signature":null},"operating":{"edit":false,"download":true,"destroy":false,"enterprise_forbid_zip":false},"release":{"title":"MindSpore Transformers v1.2.0","path":"/mindspore/mindformers/releases/tag/v1.2.0","tag_path":"/mindspore/mindformers/tree/v1.2.0","project_id":20740755,"created_at":"2024-07-24T19:00:51+08:00","is_prerelease":false,"description":"# MindSpore Transformers 1.2.0 RELEASE NOTE\r\n\r\n以下为MindSpore Transformers (以下称为MindFormers) 套件 1.2.0 版本的变更日志，相较于[1.1.0](https://gitee.com/mindspore/mindformers/blob/r1.1.0/docs/RELEASE_NOTE.md)版本有以下关键新特性和bug\r\nfix。\r\n\r\n## 新特性\r\n\r\n- **新增模型支持带框架推理**：新增支持模型包含Qwen1.5_7b、Qwen1.5_14b、Qwen1.5_72b、Llama3_70b、Yi_34b等。\r\n- **新增模型支持bfloat16训练**：新增支持模型包含Qwen1.5_7b、Qwen1.5_14b、Qwen1.5_72b、Llama3_70b、Yi_34b等。\r\n- [AdamW优化器](https://gitee.com/mindspore/mindformers/pulls/3310)：新增AdamW优化器，对齐Megatron AdamW。\r\n- **支持MindIE进行服务化部署**：[MindIE](https://www.hiascend.com/software/mindie)，全称Mind Inference\r\n  Engine，是华为昇腾针对AI全场景业务的推理加速套件。MindFormers新增对MindIE的对接，承载在模型应用层MindIE-LLM，通过MindIE-Service对MindFormers中LLM模型进行部署。\r\n- [长序列训练](https://gitee.com/mindspore/mindformers/tree/r1.2.0/docs/feature_cards/Long_Sequence_Training.md)：新增支持长序列训练特性，通过在配置yaml文件中设置`parallel_config.context_parallel`开启序列并行，当前支持32k至256k。\r\n- [断点续训权重加载2.0](https://gitee.com/mindspore/mindformers/tree/r1.2.0/docs/feature_cards/Resume_Training.md)：断点续训场景下，新增指定续训权重功能，新增故障恢复下进行权重完整性校验并自动加载最新完整权重。\r\n- [权重自动转换2.0](https://gitee.com/mindspore/mindformers/tree/r1.2.0/docs/feature_cards/Transform_Ckpt.md)：自动权重转换新增多进程转换。\r\n\r\n## 新模型\r\n\r\n以下为新支持模型：\r\n\r\n| 模型                                                                                            | 规格                                                            |\r\n|-----------------------------------------------------------------------------------------------|---------------------------------------------------------------|\r\n| [Mixtral](https://gitee.com/mindspore/mindformers/tree/r1.2.0/research/mixtral/mixtral.md)    | Mixtral_8x7b（新增32k预训练、推理）                                       \r\n| [Qwen1.5](https://gitee.com/mindspore/mindformers/tree/r1.2.0/research/qwen1_5/qwen1_5.md)    | Qwen1.5_7b（预训练、微调、推理）、Qwen1.5_14b（预训练、微调、推理）、Qwen1.5_72b（新增预训练） |\r\n| [Llama3](https://gitee.com/mindspore/mindformers/tree/r1.2.0/research/llama3/llama3.md)       | Llama3_70b（新增预训练、微调）                                            |\r\n| [Deepseek](https://gitee.com/mindspore/mindformers/tree/r1.2.0/research/deepseek/deepseek.md) | Deepseek_Coder_33b（新增微调）                                              |\r\n| [Yi](https://gitee.com/mindspore/mindformers/tree/r1.2.0/research/yi/yi.md)                   | Yi_6b（新增微调）、Yi_34b（新增预训练、微调）                                          |\r\n| [QwenVL](https://gitee.com/mindspore/mindformers/tree/r1.2.0/research/qwenvl/qwenvl.md)       | QwenVL_9.6b（微调、推理）                                            |\r\n\r\n## Bugfix\r\n\r\n在当前版本发布周期内，我们进行了模型/功能/易用性/文档等诸多方面的Bugfix，在此仅列举部分修复内容：\r\n\r\n- [!3345](https://gitee.com/mindspore/mindformers/pulls/3345)：修复`Linear`在`transpose_b=False`时传入正确`weight`仍报错问题。\r\n- [!3277](https://gitee.com/mindspore/mindformers/pulls/3277)：修复使用梯度累积时，`warpper`传入了错误的`micro_batch_num`问题。\r\n\r\n## 贡献者\r\n\r\n感谢以下人员做出的贡献：\r\n\r\nChenhua Geng，heqinglin，koukairui，renyujin，shuchi，陈心锐，陈子恒，冯浩，胡思超，黄磊，黄生帅，黄勇，黄子灵，倪钰鑫，苏海波，李子垠，杨星宇，牛君豪，张森镇，张又文，谭纬城，吴致远，杨星宇，刘群，曹宇麟，方泽华，金仁操，刘群，李永文，钱驾宏，吴昊天，杨璇，汪家傲，范益，陈昱坤，李洋\r\n\r\n欢迎以任何形式对项目提供贡献！","author":{"name":"hsshuai","username":"hss-shuai","path":"/hss-shuai","avatar_url":"no_portrait.png#hsshuai-hss-shuai"},"attach_files":[],"zip_download_url":"/mindspore/mindformers/releases/tag/v1.2.0.zip","tar_download_url":"/mindspore/mindformers/releases/tag/v1.2.0.tar.gz"}}}