[Bug]: [master][models-vgg19分布式训练报错E19999]

问题描述

master分支，分布式训练vgg19报错，训练数据集为CIFAR-10
模型地址：https://gitee.com/mindspore/models/tree/master/official/cv/VGG/vgg19

环境信息

Hardware Environment(Ascend/GPU/CPU) / 硬件环境:

Please delete the backend not involved / 请删除不涉及的后端:
/device Ascend

Software Environment / 软件环境 (Mandatory / 必填):
-- MindSpore version (e.g., 2.0.0) :mindspore2.2.0
-- Python version (e.g., Python 3.7.5) :python3.9.18
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):aarch64
-- GCC/Compiler version (if compiled from source): 7.3.0
Execute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

Please delete the mode not involved / 请删除不涉及的模式:
/mode pynative

关联用例

/

重现步骤

1.CIFAR-10数据集下载并解压，按照指导要求生成数据集文件夹结构。 cifar-10-batches-bin训练数据集和cifar-10-verify-bin推理数据集
2.开始训练，分布式训练指令：sh scripts/run_distribute_train.sh /user/config/jobstart_hccl.json /home/ma-user/work/datasets/

预期结果

#grep "result: " train_parallel*/log
train_parallel0/log:epoch: 1 step: 97, loss is 1.9060308
train_parallel0/log:epcoh: 2 step: 97, loss is 1.6003821
...
train_parallel1/log:epoch: 1 step: 97, loss is 1.7095519
train_parallel1/log:epcoh: 2 step: 97, loss is 1.7133579
...
...

日志/截图

[ERROR] GE_ADPT(1336439,ffffbee4d010,python):2024-06-12-14:30:44.987.954 [mindspore/ccsrc/transform/graph_ir/graph_runner.cc:396] CompileGraph] Call GE CompileGraph Failed, ret is: 1343225857
Traceback (most recent call last):
  File "/home/ma-user/work/models/official/cv/VGG/vgg19/train_parallel0/train.py", line 241, in <module>
    run_train()
  File "/home/ma-user/work/models/official/cv/VGG/vgg19/train_parallel0/model_utils/moxing_adapter.py", line 107, in wrapped_func
    run_func(*args, **kwargs)
  File "/home/ma-user/work/models/official/cv/VGG/vgg19/train_parallel0/train.py", line 237, in run_train
    model.train(config.max_epoch, dataset, callbacks=callbacks, dataset_sink_mode=True)
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/train/model.py", line 1068, in train
    self._train(epoch,
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/train/model.py", line 114, in wrapper
    func(self, *args, **kwargs)
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/train/model.py", line 623, in _train
    self._train_dataset_sink_process(epoch, train_dataset, list_callback,
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/train/model.py", line 708, in _train_dataset_sink_process
    outputs = train_network(*inputs)
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 680, in __call__
    out = self.compile_and_run(*args, **kwargs)
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 1020, in compile_and_run
    self.compile(*args, **kwargs)
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/nn/cell.py", line 997, in compile
    _cell_graph_executor.compile(self, phase=self.phase,
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/mindspore/common/api.py", line 1547, in compile
    result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode())
RuntimeError: Compile graph kernel_graph_1 failed.

----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
E19999: Inner Error!
E19999  Call OptimizeOriginalGraph failed, ret:1343225860, engine_name:hccl_graph_optimizer, graph_name:kernel_graph_1[FUNC:OptimizeOriginalGraph][FILE:graph_optimize.cc][LINE:178]
        TraceBack (most recent call last):
        [Call][PreRun] Failed, graph_id:2, session_id:0.[FUNC:CompileGraph][FILE:graph_manager.cc][LINE:4113]
        [Compile][Graph]Compile graph failed, error code:1343225857, session_id:0, graph_id:2.[FUNC:CompileGraph][FILE:ge_api.cc][LINE:1153]

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ge_graph_executor.cc:813 CompileGraph

[WARNING] MD(1336439,ffffbee4d010,python):2024-06-12-14:30:47.285.355 [mindspore/ccsrc/minddata/dataset/engine/datasetops/data_queue_op.cc:163] ~DataQueueOp] 
preprocess_batch: 55;
batch_queue: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0;
            push_start_time -> push_end_time
2024-06-12-14:30:42.400.589 -> 2024-06-12-14:30:42.405.504
2024-06-12-14:30:42.415.994 -> 2024-06-12-14:30:42.420.820
2024-06-12-14:30:42.431.977 -> 2024-06-12-14:30:42.437.217
2024-06-12-14:30:42.448.047 -> 2024-06-12-14:30:42.452.266
2024-06-12-14:30:42.462.149 -> 2024-06-12-14:30:42.466.970
2024-06-12-14:30:42.477.327 -> 2024-06-12-14:30:42.482.330
2024-06-12-14:30:42.492.919 -> 2024-06-12-14:30:42.498.127
2024-06-12-14:30:42.508.943 -> 2024-06-12-14:30:42.513.136
2024-06-12-14:30:42.523.855 -> 2024-06-12-14:30:42.529.007
2024-06-12-14:30:42.539.776 -> 2024-06-12-14:30:42.544.727
For more details, please refer to the FAQ at https://www.mindspore.cn/docs/en/master/faq/data_processing.html.

备注

Please assign maintainer to check this issue.
请为此issue分配处理人。
@ivss @fangwenyi @lujiale @tuoyuanyuan @chengxiaoli

感谢您的提问，您可以评论//mindspore-assistant更快获取帮助：

如果您刚刚接触MindSpore，或许您可以在教程找到答案
如果您是资深Pytorch用户，您或许需要：

如果您遇到动态图问题，可以设置set_context(pynative_synchronize=True)查看报错栈协助定位
模型精度调优问题可参考官网调优指南
如果您反馈的是框架BUG，请确认您在ISSUE中提供了MindSpore版本、使用的后端类型（CPU、GPU、Ascend）、环境、训练的代码官方链接以及可以复现报错的代码的启动方式等必要的定位信息
如果您已经定位出问题根因，欢迎提交PR参与MindSpore开源社区，我们会尽快review

MindSpore/models

内容风险标识

问题描述

环境信息

关联用例

重现步骤

预期结果

日志/截图

备注

评论 (2)

MindSpore/models .gitee-modal { width: 500px !important; }

内容风险标识

[Bug]: [master][models-vgg19分布式训练报错E19999]

问题描述

环境信息

关联用例

重现步骤

预期结果

日志/截图

备注

评论 (2)

搜索帮助

MindSpore/models