EH9999: Inner Error!

环境：torch 2.1.0 torch_npu 2.1.0+git4ab2402 CANN 8.0.RC1

使用torch_npu迁移IDEA的DINO模型https://github.com/IDEA-Research/DINO?tab=readme-ov-file时报错。
且设置了1、4、16的batch_size，只有设置bs为4时跑了150个step报错。
报错信息如下
EH9999: Inner Error!
EH9999  reserve memory address failed, runtime result = 207000[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
        The error from device(chipId:1, dieId:0), serial number is 4, there is an fftsplus aivector error exception, core id is 12, error code = 0x800000, dump info: pc start: 0x1240c11ce27c, current: 0x1240c11cfad0, vec error info: 0xd201759f40, mte error info: 0x3a030028fa, ifu error info: 0x641fc7c61c5c0, ccu error info: 0x3c63ce317c6faf70, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x124100340080.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1169]
        The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x30028fa, fixp_error1 info: 0x3a fsmId:1, tslot:0, thread:0, ctxid:0, blk:1, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1181]
        Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:task_info.cc][LINE:1754]
        AICORE Kernel task happen error, retCode=0x26.[FUNC:GetError][FILE:stream.cc][LINE:1512]
        Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=18803, flip_num=40, fault kernel_name=DiagV2_19805c5e58e043224cccf51a06108010ae13i32_high_performance_2103_mix_aiv, fault kernel info ext=none, program id=105, hash=9907267370218063543.[FUNC:GetError][FILE:stream.cc][LINE:1512]
        [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1512]
        rtStreamSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
        synchronize stream failed, runtime result = 507015[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

Traceback (most recent call last):
  File "/home/ma-user/work/evaluation-sh/Ocr_transform_torch_npu/DINO/main-4bs.py", line 392, in <module>
    main(args)
  File "/home/ma-user/work/evaluation-sh/Ocr_transform_torch_npu/DINO/main-4bs.py", line 276, in main
    train_stats = train_one_epoch(
  File "/home/ma-user/work/evaluation-sh/Ocr_transform_torch_npu/DINO/engine.py", line 53, in train_one_epoch
    loss_dict = criterion(outputs, targets)
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/ma-user/work/evaluation-sh/Ocr_transform_torch_npu/DINO/models/dino/dino.py", line 569, in forward
    l_dict = self.get_loss(loss, aux_outputs, targets, indices, num_boxes, **kwargs)
  File "/home/ma-user/work/evaluation-sh/Ocr_transform_torch_npu/DINO/models/dino/dino.py", line 481, in get_loss
    return loss_map[loss](outputs, targets, indices, num_boxes, **kwargs)
  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ma-user/work/evaluation-sh/Ocr_transform_torch_npu/DINO/models/dino/dino.py", line 397, in loss_cardinality
    tgt_lengths = torch.as_tensor([len(v["labels"]) for v in targets], device=device)
RuntimeError: ACL stream synchronize failed, error code:507015
[W compiler_depend.ts:365] Warning: NPU warning, error code is 507015[Error]: 
[Error]: The aicore execution is abnormal. 
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)
[W compiler_depend.ts:365] Warning: NPU warning, error code is 507015[Error]: 
[Error]: The aicore execution is abnormal. 
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)

Ascend/pytorch
暂停

内容风险标识

评论 (12)

Ascend/pytorch暂停 .gitee-modal { width: 500px !important; }

内容风险标识