【910b】qwen1.5 预训练报错

| name       | about	                             | labels   |
| ---------- | ------------------------------------- | -------- |
| Bug Report | Use this template for reporting a bug | kind/bug |

### Describe the current behavior / 问题描述 (Mandatory / 必填)
 
 
### Environment / 环境信息 (Mandatory / 必填)
910b

- **Software Environment / 软件环境 (Mandatory / 必填)**:
-- MindSpore version (2.3.0) :
-- Python version (Python 3.9.516) :

- **Excute Mode / 执行模式 (Mandatory / 必填)(`PyNative`/`Graph`)**: 
> Please delete the mode not involved / 请删除不涉及的模式:
> /mode graph

### Related testcase / 关联用例 (Mandatory / 必填)
 
### Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)
bash scripts/msrun_launcher.sh "run_mindformer.py \
  --config ./mindformers/research/qwen1_5/pretrain_qwen1_5_7b.yaml \
  --run_mode train" 2

### Describe the expected behavior / 预期结果 (Mandatory / 必填)
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.886.077 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/Sqrt-op0 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.886.763 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op0 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.887.108 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op1 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.887.325 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op2 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.887.630 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op3 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.887.830 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op4 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.888.143 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op5 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.888.604 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op6 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.888.999 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op7 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.889.231 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op8 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.889.619 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op9 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.889.873 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op10 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.890.235 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op11 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.890.492 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op12 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.890.806 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op13 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.891.047 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op14 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.891.462 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op15 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.891.673 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op16 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.892.060 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op17 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.892.338 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op18 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.892.732 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op19 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.893.050 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op20 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.893.445 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op21 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.893.702 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op22 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.894.143 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op23 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.894.372 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op24 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.894.680 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op25 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.894.889 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op26 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.895.257 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op27 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.895.516 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op28 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.895.896 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op29 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.896.092 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op30 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.896.437 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op31 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.896.622 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op32 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.896.978 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op33 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.897.182 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op34 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.897.539 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op35 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.897.866 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op36 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.898.198 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op37 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.898.456 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op38 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.898.784 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op39 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.898.976 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op40 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.899.310 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op41 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.899.534 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op42 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.899.876 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op43 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.900.117 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op44 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.900.488 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op45 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.900.706 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op46 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.901.099 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op47 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.901.323 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op48 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.901.686 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op49 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.901.974 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op50 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.902.416 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op51 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.902.650 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op52 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.902.993 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op53 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.903.221 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op54 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.903.577 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op55 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:40:58.903.805 [mindspore/ccsrc/frontend/parallel/graph_util/graph_utils.cc:68] GetTensorRedistributionFromCNode] Default/network-MFTrainOneStepCell/clip_grad_norm-ClipGradNorm/ExpandDims-op56 has no OperatorInfo.
[WARNING] PARALLEL(4650,7fa2f53cc740,python):2024-07-31-19:41:13.648.749 [mindspore/ccsrc/frontend/parallel/pass/overlap_recompute_allgather_and_flashattention_grad.cc:192] OverlapRecomputeAllGatherAndFlashAttentionGrad] Currently, duplicated allgather overlap with flashattention grad only support in lazy_line mode.
2024-07-31 19:41:31,986 - mindformers[mindformers/tools/cloud_adapter/cloud_monitor.py:43] - ERROR - Traceback (most recent call last):
  File "./mindformers/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
    result = run_func(*args, **kwargs)
  File "./mindformers/run_mindformer.py", line 41, in main
    trainer.train()
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/mindspore/_checkparam.py", line 1352, in wrapper
    return func(*args, **kwargs)
  File "./mindformers/mindformers/trainer/trainer.py", line 421, in train
    self.trainer.train(
  File "./mindformers/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 113, in train
    self.training_process(
  File "./mindformers/mindformers/trainer/base_trainer.py", line 750, in training_process
    transform_and_load_checkpoint(config, model, network, dataset)
  File "./mindformers/mindformers/trainer/utils.py", line 366, in transform_and_load_checkpoint
    build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict)
  File "./mindformers/mindformers/trainer/utils.py", line 434, in build_model
    model.build(train_dataset=dataset, epoch=config.runner_config.epochs,
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/mindspore/train/model.py", line 1496, in build
    self._init(train_dataset, valid_dataset, sink_size, epoch)
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/mindspore/train/model.py", line 717, in _init
    train_network.compile(*inputs)
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/mindspore/nn/cell.py", line 1054, in compile
    _cell_graph_executor.compile(self, *self._compile_args, phase=self.phase,
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/mindspore/common/api.py", line 1819, in compile
    result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode())
RuntimeError: aclnnAllFiniteGetWorkspaceSize call failed, please check!

----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EZ1001: 2024-07-31-19:41:30.340.803 Get regInfo failed, The binary_info_config.json of socVersion [ascend910b] does not support opType [AllFinite].[THREAD:4650]
        TraceBack (most recent call last):
        Check nnopExecutor != nullptr failed[THREAD:4650]

(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/kernel/opapi/aclnn/all_finite_aclnn_kernel.h:36 operator()

Traceback (most recent call last):
  File "./mindformers/run_mindformer.py", line 271, in <module>
    main(config_)
  File "./mindformers/mindformers/tools/cloud_adapter/cloud_monitor.py", line 44, in wrapper
    raise exc
  File "./mindformers/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
    result = run_func(*args, **kwargs)
  File "./mindformers/run_mindformer.py", line 41, in main
    trainer.train()
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/mindspore/_checkparam.py", line 1352, in wrapper
    return func(*args, **kwargs)
  File "./mindformers/mindformers/trainer/trainer.py", line 421, in train
    self.trainer.train(
  File "./mindformers/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 113, in train
    self.training_process(
  File "./mindformers/mindformers/trainer/base_trainer.py", line 750, in training_process
    transform_and_load_checkpoint(config, model, network, dataset)
  File "./mindformers/mindformers/trainer/utils.py", line 366, in transform_and_load_checkpoint
    build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict)
  File "./mindformers/mindformers/trainer/utils.py", line 434, in build_model
    model.build(train_dataset=dataset, epoch=config.runner_config.epochs,
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/mindspore/train/model.py", line 1496, in build
    self._init(train_dataset, valid_dataset, sink_size, epoch)
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/mindspore/train/model.py", line 717, in _init
    train_network.compile(*inputs)
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/mindspore/nn/cell.py", line 1054, in compile
    _cell_graph_executor.compile(self, *self._compile_args, phase=self.phase,
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/mindspore/common/api.py", line 1819, in compile
    result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode())
RuntimeError: aclnnAllFiniteGetWorkspaceSize call failed, please check!

----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EZ1001: 2024-07-31-19:41:30.340.803 Get regInfo failed, The binary_info_config.json of socVersion [ascend910b] does not support opType [AllFinite].[THREAD:4650]
        TraceBack (most recent call last):
        Check nnopExecutor != nullptr failed[THREAD:4650]

(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/kernel/opapi/aclnn/all_finite_aclnn_kernel.h:36 operator()

[WARNING] MD(4650,7fa2f53cc740,python):2024-07-31-19:41:32.154.178 [mindspore/ccsrc/minddata/dataset/engine/datasetops/data_queue_op.cc:163] ~DataQueueOp] 
channel_name: b81504c6-4f31-11ef-ad95-525400f5ae11;
have_sent: 100;
host_queue: 1, 1, 1, 1, 1, 1, 1, 1, 1, 1;
device_queue: 90, 91, 92, 93, 94, 95, 96, 97, 98, 99;
      push_first_start_time -> push_first_end_time
2024-07-31-19:40:43.535.659 -> 2024-07-31-19:40:43.536.645
            push_start_time -> push_end_time
2024-07-31-19:40:43.567.777 -> 2024-07-31-19:40:43.568.008
2024-07-31-19:40:43.568.100 -> 2024-07-31-19:40:43.568.328
2024-07-31-19:40:43.568.420 -> 2024-07-31-19:40:43.568.644
2024-07-31-19:40:43.568.735 -> 2024-07-31-19:40:43.568.967
2024-07-31-19:40:43.569.061 -> 2024-07-31-19:40:43.569.293
2024-07-31-19:40:43.569.382 -> 2024-07-31-19:40:43.569.609
2024-07-31-19:40:43.569.701 -> 2024-07-31-19:40:43.569.937
2024-07-31-19:40:43.570.032 -> 2024-07-31-19:40:43.570.273
2024-07-31-19:40:43.570.367 -> 2024-07-31-19:40:43.570.606
2024-07-31-19:40:43.570.699 -> 2024-07-31-19:40:43.570.936
For more details, please refer to the FAQ at https://www.mindspore.cn/docs/en/master/faq/data_processing.html.

### Related log / screenshot / 日志 / 截图 (Mandatory / 必填)
 
### Special notes for this issue/备注 (Optional / 选填)

GVP MindSpore/mindformers
关闭

内容风险标识

评论 (5)

GVPMindSpore/mindformers关闭

内容风险标识