模型训练过程中报错：ACL stream synchronize failed, error code:507018

E39999: Inner Error! [00:00<?, ?it/s]/
E39999  An exception occurred during AICPU execution, stream_id:2, task_id:5390, errcode:21008, msg:inner error[FUNC:ProcessAicpuErrorInfo][FILE:device_error_proc.cc][LINE:716]
        TraceBack (most recent call last):
        Kernel task happen error, retCode=0x2a, [aicpu exception].[FUNC:PreCheckTaskErr][FILE:task_info.cc][LINE:1518]
        Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=5390, errorCode=2a.[FUNC:PrintAicpuErrorInfo][FILE:task_info.cc][LINE:1364]
        Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=5390, fault op_name=[FUNC:GetError][FILE:stream.cc][LINE:1418]
        rtStreamSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
        synchronize stream failed, runtime result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

DEVICE[0] PID[24746]:
EXCEPTION TASK:
  Exception info:TGID=975760, model id=65535, stream id=2, stream phase=3, task id=5390, task type=aicpu kernel, recently received task id=5391, recently send task id=5389, task phase=RUN
  Message info[0]:aicpu=0,slot_id=0,report_mailbox_flag=0x5a5a5a5a,state=0x5210
    Other info[0]:time=2024-06-05-02:21:35.210.371, function=proc_aicpu_task_done, line=970, error code=0x2a
  0%|          | 0/8 [06:00<?, ?it/s]
Traceback (most recent call last):
  File "train.py", line 862, in <module>
    main(opt)
  File "train.py", line 637, in main
    train(opt.hyp, opt, device, callbacks)
  File "train.py", line 397, in train
    loss, loss_items = compute_loss(pred, targets.to(device))  # loss scaled by batch_size
  File "/home/yolov5/yolov5-master/utils/loss.py", line 170, in __call__
    t[range(n), tcls[i]] = self.cp
RuntimeError: ACL stream synchronize failed, error code:507018
[W NPUStream.cpp:393] Warning: NPU warning, error code is 507018[Error]:
[Error]: The aicpu execution is abnormal.
        Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
EH9999  wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)
[W NPUStream.cpp:393] Warning: NPU warning, error code is 507018[Error]:
[Error]: The aicpu execution is abnormal.
        Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
EH9999  wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)

Ascend/modelzoo

内容风险标识

评论 (4)

Ascend/modelzoo .gitee-modal { width: 500px !important; }

内容风险标识