Appearance & Root Cause

name	about	labels
Bug Report	Use this template for reporting a bug	kind/bug

Describe the current behavior / 问题描述 (Mandatory / 必填)

qwen/baichuan2/internlm/skywork/iflytekspark/wizardcoder等网络在910B环境上训练失败
模型仓地址：https://gitee.com/mindspore/mindformers/tree/r1.0/research/baichuan2

Environment / 环境信息 (Mandatory / 必填)

Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端:
/device ascend/

Software Environment / 软件环境 (Mandatory / 必填):
-- MindSpore version (e.g., 1.7.0.Bxxx) :
-- Python version (e.g., Python 3.7.5) :
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):
-- GCC/Compiler version (if compiled from source):

失败版本：
CANN：Milan_C15/20240302
MS：r2.2_20240411041520_c4eddc9d731
MF：r1.0_20240411161523_64bb772e6b78

Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式:
/mode pynative
/mode graph

Related testcase / 关联用例 (Mandatory / 必填)

用例仓地址：MindFormers_Test/cases/baichuan2/7b/train/
用例
test_mf_baichuan2_7b_train_belle_8p_0001

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

get code from mindformers
cd mindformers/research
bash run_singlenode.sh '''python ./baichuan2/run_baichuan2.py --config ./research/baichuan2/run_baichuan2_7b_4096_910b.yaml --load_checkpoint /home/workspace/large_model_ckpt//baichuan2/7b/train/ --auto_trans_ckpt True --use_parallel True --run_mode finetune --train_data /home/workspace/large_model_dataset//baichuan2/belle_4096.mindrecord''' /home/workspace/config/hccl_8p.json [0,8] 8
验证网络是否训练成功

Describe the expected behavior / 预期结果 (Mandatory / 必填)

网络训练成功

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

INFO 2024-04-11 23:16:30 - test_mf_baichuan2_7b_train_belle_8p_0001 - train.py:print_err_info_in_log:321 - 
[ERROR] 2024-04-11 23:15:51,763 [mindformers/tools/cloud_adapter/cloud_monitor.py:43] wrapper: Traceback (most recent call last):
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
   result = run_func(*args, **kwargs)
 File "./baichuan2/run_baichuan2.py", line 169, in main
   trainer.finetune(finetune_checkpoint=ckpt, auto_trans_ckpt=config.auto_trans_ckpt, resume_training=resume)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/_checkparam.py", line 1313, in wrapper
   return func(*args, **kwargs)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/trainer.py", line 491, in finetune
   is_full_config=True)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 104, in train
   **kwargs)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/base_trainer.py", line 739, in training_process
   transform_and_load_checkpoint(config, model, network, dataset)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/utils.py", line 348, in transform_and_load_checkpoint
   dst_ckpt_strategy=dst_ckpt_strategy)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/utils.py", line 604, in transform_ckpt
   wait_transform(config, ckpt_dir, world_size)
 File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/utils.py", line 654, in wait_transform
   raise ValueError(f"Transform failed, find {transform_failed_txts}.")
ValueError: Transform failed, find ['./output/transformed_checkpoint/train/transform_failed_rank_0.txt'].
INFO 2024-04-11 23:16:30 - test_mf_baichuan2_7b_train_belle_8p_0001 - train.py:print_err_info_in_log:323 -

Special notes for this issue/备注 (Optional / 选填)

走给寇凯睿

Please assign maintainer to check this issue.
请为此issue分配处理人。
@zhangjie18

感谢您的提问，您可以评论//mindspore-assistant更快获取帮助：

如果您刚刚接触MindSpore，或许您可以在教程找到答案
如果您是资深Pytorch用户，您或许需要：

如果您遇到动态图问题，可以设置set_context(pynative_synchronize=True)查看报错栈协助定位
模型精度调优问题可参考官网调优指南
如果您反馈的是框架BUG，请确认您在ISSUE中提供了MindSpore版本、使用的后端类型（CPU、GPU、Ascend）、环境、训练的代码官方链接以及可以复现报错的代码的启动方式等必要的定位信息
如果您已经定位出问题根因，欢迎提交PR参与MindSpore开源社区，我们会尽快review

Appearance & Root Cause

修复问题单引入，取元素前未对src_strategy_file和dst_strategy_file判空，2024/04/11修复PR已合入。

Fix Solution

添加判空。

Fix Description & Test Suggestion

!68166:【r2.2】修复ckpt转换src或dst策略文件为空时bug

Self-test Report & DT Review

ckpe转换不在ut/st中添加用例。
ms 2024/04/12 daily验证已通过。

Suggested regression version

ms 2024/04/12 daily及之后版本。

回归版本：CANN：Milan_C15/20240302
MS：r2.2_20240412041523_5fe9f78bde8ac3b7
MF：r1.0_20240411161523_64bb772e6b78
回归步骤：参考issue步骤
基本问题：已解决
输入图片说明

测试结论：回归通过
回归时间：2024.4.12

GVP MindSpore / mindspore

内容风险标识

[ST][MF][qwen/baichuan2/internlm/skywork/iflytekspark/wizardcoder/][910B]网络训练失败，ValueError: Transform failed, find ['./output/transformed_checkpoint/train/transform_failed_rank_0.txt']

Describe the current behavior / 问题描述 (Mandatory / 必填)

Environment / 环境信息 (Mandatory / 必填)

Related testcase / 关联用例 (Mandatory / 必填)

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

Describe the expected behavior / 预期结果 (Mandatory / 必填)

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

Special notes for this issue/备注 (Optional / 选填)

评论 (4)

Appearance & Root Cause

Fix Solution

Fix Description & Test Suggestion

Self-test Report & DT Review

Suggested regression version

GVPMindSpore / mindspore

内容风险标识

[ST][MF][qwen/baichuan2/internlm/skywork/iflytekspark/wizardcoder/][910B]网络训练失败，ValueError: Transform failed, find ['./output/transformed_checkpoint/train/transform_failed_rank_0.txt']

Describe the current behavior / 问题描述 (Mandatory / 必填)

Environment / 环境信息 (Mandatory / 必填)

Related testcase / 关联用例 (Mandatory / 必填)

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

Describe the expected behavior / 预期结果 (Mandatory / 必填)

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

Special notes for this issue/备注 (Optional / 选填)

评论 (4)

Appearance & Root Cause

Fix Solution

Fix Description & Test Suggestion

Self-test Report & DT Review

Suggested regression version

搜索帮助

GVP MindSpore / mindspore