【Atlas800型号9000】【模型训练】在MindSpore 1.10.0.B060版本进行ASR_dynamic_train_ascend模型测试，训练失败。

Environment

• Hardware Environment(Ascend/GPU/CPU):
  Ascend Data Center Solution 22.0.RC3.B120
  驱动和固件:A800-9000 1.0.18.B030
  https://cmc-rnd.tools.huawei.com/cmcversion/index/releaseView?deltaId=4715917168345088&isSelect=Software

Uncomment only one /device <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:
/device ascend

• Software Environment:
  -- MindSpore version (source or binary):mindspore1.10.0.B060
  -- Python version (e.g., Python 3.7.5):Python 3.7.5
  -- OS platform and distribution 4.19.36-vhulk1907.1.0.h475.eulerosv2r8.aarch64
  -- GCC/Compiler version (if compiled from source):
  https://cmc-szv.clouddragon.huawei.com/cmcversion/index/releaseView?deltaId=4721339983200384&isSelect=Software

Related testcase

在MindSpore 1.10.0.B060版本在虚拟机进行ASR_dynamic_train_ascend模型测试，训练失败。

Steps to reproduce the issue

1. https://codehub-y.huawei.com/mindspore/modelinternal/files?ref=master&filePath=media_net_for_smoke%2FASR_dynamic_train_ascend
2. 环境10.174.216.229 root/Ascend@123
3. 路径：/home/CI_daily/5326391/modelinternal-r1.10/media_net_for_smoke/ASR_dynamic_train_ascend
4. 执行命令： bash ./scripts/run_1p.sh /home/data/aishell/ config/default_config.yaml 0

Describe the current behavior

在MindSpore 1.10.0.B060版本进行ASR_dynamic_train_ascend模型测试，训练失败

Describe the expected behavior

报错如下：
[WARNING] MD(65692,fffe397fa1e0,python3):2022-12-29-12:06:50.088.829 [mindspore/ccsrc/minddata/dataset/engine/datasetops/source/generator_op.cc:195] operator()] Bad performance attention, it takes more than 25 seconds to generator.next new row, which might cause GetNext timeout problem when sink_mode=True. You can increase the parameter num_parallel_workers in GeneratorDataset / optimize the efficiency of obtaining samples in the user-defined generator function.
Traceback (most recent call last):
File "e2e_sink_dev.py", line 214, in
run()
File "e2e_sink_dev.py", line 208, in run
model.train(sink_epochs, dataset, callbacks=callback_list, sink_size=1, dataset_sink_mode=True)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 1051, in train
initial_epoch=initial_epoch)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 98, in wrapper
func(self, *args, **kwargs)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 625, in _train
cb_params, sink_size, initial_epoch, valid_infos)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/train/model.py", line 703, in _train_dataset_sink_process
outputs = train_network(*inputs)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py", line 596, in call
out = self.compile_and_run(*args)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py", line 985, in compile_and_run
self.compile(*inputs)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/nn/cell.py", line 957, in compile
jit_config_dict=self._jit_config_dict)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/mindspore/common/api.py", line 1131, in compile
result = self._graph_executor.compile(obj, args_list, phase, self._use_vm_mode())
RuntimeError: Single op compile failed, op: mat_mul_4547133356613339742_3

Related log / screenshot

Special notes for this issue

在MindSpore 1.10.0.B060版本进行ASR_dynamic_train_ascend模型测试，训练失败。