[ST][MS][MF][baichuan2_7b][910B1 8p]网络训练失败，卡6加载权重时报找不到权重目录，其他几张卡是加载成功得

| name       | about	                             | labels   |
| ---------- | ------------------------------------- | -------- |
| Bug Report | Use this template for reporting a bug | kind/bug |

### Describe the current behavior / 问题描述 (Mandatory / 必填)
 baichuan2_7b网络在910B1环境上微调时，卡6加载权重时报找不到权重目录，其他几张卡是加载成功得
模型仓地址：https://gitee.com/mindspore/mindformers/tree/dev/research/baichuan2
 
### Environment / 环境信息 (Mandatory / 必填)
- **Hardware Environment(`Ascend`/`GPU`/`CPU`)  / 硬件环境**: 
> Please delete the backend not involved / 请删除不涉及的后端:
> /device ascend

- **Software Environment / 软件环境 (Mandatory / 必填)**:
-- MindSpore version (e.g., 1.7.0.Bxxx) :
-- Python version (e.g., Python 3.7.5) :
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):
-- GCC/Compiler version (if compiled from source):

run：Milan_C15/20231227
mindspore版本：r2.2_53ae1681df396021214a7414ad320cd1250968d2_20240103041522
mindformers版本：dev_008f37f85227bcd3f421cd6110ddee1ad56f0cc5_20240103121534

- **Excute Mode / 执行模式 (Mandatory / 必填)(`PyNative`/`Graph`)**: 
> Please delete the mode not involved / 请删除不涉及的模式:
> /mode pynative
> /mode graph

### Related testcase / 关联用例 (Mandatory / 必填)

模型仓地址：MindFormers_Test/cases/baichuan2/7b/train/
用例：
test_mf_baichuan2_7b_train_belle_8p_0001
 
### Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)
1. get code from mindformers
2.  cd mindformers/research
3. bash run_singlenode.sh '''python ./baichuan2/run_baichuan2.py --config ./research/baichuan2/run_baichuan2_7b_910b.yaml --load_checkpoint /home/workspace/large_model_ckpt//baichuan2/7b/train/ --auto_trans_ckpt True --use_parallel True --run_mode finetune --train_data /home/workspace/large_model_dataset//baichuan2/belle512.mindrecord''' /home/workspace/config/hccl_8p.json [0,8] 8
4. 验证网络是否训练成功

### Describe the expected behavior / 预期结果 (Mandatory / 必填)
 网络训练成功
### Related log / screenshot / 日志 / 截图 (Mandatory / 必填)
 ```
2024-01-03 21:26:03,168 - mindformers[mindformers/trainer/utils.py:352] - INFO - Find exist softlink dir /home/jenkins/workspace/TDT_deployment/MindFormers_Test/cases/baichuan2/7b/train/test_mf_baichuan2_7b_train_belle_8p_0001/research/softlink_ckpt/ and delete it.
2024-01-03 21:26:03,177 - mindformers[mindformers/trainer/utils.py:362] - INFO - Make soft link of checkpoint file from /home/workspace/large_model_ckpt//baichuan2/7b/train/ to softlink_ckpt/train
2024-01-03 21:26:03,208 - mindformers[mindformers/tools/cloud_adapter/cloud_monitor.py:43] - ERROR - Traceback (most recent call last):
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
    result = run_func(*args, **kwargs)
  File "./baichuan2/run_baichuan2.py", line 169, in main
    trainer.finetune(finetune_checkpoint=ckpt, auto_trans_ckpt=config.auto_trans_ckpt, resume_training=resume)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/_checkparam.py", line 1313, in wrapper
    return func(*args, **kwargs)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/trainer.py", line 504, in finetune
    is_full_config=True)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 104, in train
    **kwargs)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/base_trainer.py", line 696, in training_process
    transform_and_load_checkpoint(config, model, network, dataset)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/utils.py", line 331, in transform_and_load_checkpoint
    softlink_dir = check_ckpt_for_transform(config.load_checkpoint)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/utils.py", line 364, in check_ckpt_for_transform
    os.symlink(ckpt_dir, soft_link)
FileNotFoundError: [Errno 2] No such file or directory: '/home/workspace/large_model_ckpt//baichuan2/7b/train/' -> 'softlink_ckpt/train'

[WARNING] MD(154874,ffff97bffaa0,python):2024-01-03-21:26:03.529.236 [mindspore/ccsrc/minddata/dataset/engine/datasetops/data_queue_op.cc:163] ~DataQueueOp]
preprocess_batch: 100;
batch_queue: 1, 1, 1, 1, 1, 1, 1, 1, 1, 1;
            push_start_time -> push_end_time
2024-01-03-21:22:22.450.836 -> 2024-01-03-21:22:22.451.042
2024-01-03-21:22:22.451.151 -> 2024-01-03-21:22:22.451.354
2024-01-03-21:22:22.451.456 -> 2024-01-03-21:22:22.451.655
2024-01-03-21:22:22.451.756 -> 2024-01-03-21:22:22.451.968
2024-01-03-21:22:22.452.073 -> 2024-01-03-21:22:22.452.273
2024-01-03-21:22:22.452.380 -> 2024-01-03-21:22:22.452.584
2024-01-03-21:22:22.452.686 -> 2024-01-03-21:22:22.452.885
2024-01-03-21:22:22.453.000 -> 2024-01-03-21:22:22.453.201
2024-01-03-21:22:22.453.305 -> 2024-01-03-21:22:22.453.500
2024-01-03-21:22:22.453.598 -> 2024-01-03-21:22:22.453.800
For more details, please refer to the FAQ at https://www.mindspore.cn/docs/en/master/faq/data_processing.html.
Traceback (most recent call last):
  File "./baichuan2/run_baichuan2.py", line 255, in <module>
    micro_batch_num=args.micro_batch_num)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/tools/cloud_adapter/cloud_monitor.py", line 44, in wrapper
    raise exc
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
    result = run_func(*args, **kwargs)
  File "./baichuan2/run_baichuan2.py", line 169, in main
    trainer.finetune(finetune_checkpoint=ckpt, auto_trans_ckpt=config.auto_trans_ckpt, resume_training=resume)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/_checkparam.py", line 1313, in wrapper
    return func(*args, **kwargs)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/trainer.py", line 504, in finetune
    is_full_config=True)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 104, in train
    **kwargs)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/base_trainer.py", line 696, in training_process
    transform_and_load_checkpoint(config, model, network, dataset)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/utils.py", line 331, in transform_and_load_checkpoint
    softlink_dir = check_ckpt_for_transform(config.load_checkpoint)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindformers/trainer/utils.py", line 364, in check_ckpt_for_transform
    os.symlink(ckpt_dir, soft_link)
FileNotFoundError: [Errno 2] No such file or directory: '/home/workspace/large_model_ckpt//baichuan2/7b/train/' -> 'softlink_ckpt/train'

```
### Special notes for this issue/备注 (Optional / 选填)
走给黄欣靓

GVP MindSpore/mindformers
关闭

内容风险标识

评论 (4)

GVPMindSpore/mindformers关闭

内容风险标识