AddCustom样例

一、问题现象（附报错日志上下文）：
样例里面的Add_custom怎么不支持三维矩阵的相加？
样例里面的Add_custom如果输入两个形状为[32, 1024]的矩阵相加没问题，但是如果输入两个形状为[32, 1024*1024]的矩阵，就会报如下错误。

二、报错日志
[E compiler_depend.ts:270] call aclnnMaskedSelect failed, detail:EZ9999: Inner Error!
EZ9999: 2024-12-11-16:17:20.221.176  The error from device(chipId:4, dieId:0), serial number is 17, there is an aivec error exception, core id is 9, error code = 0x200000000, dump info: pc start: 0x1241001a3000, current: 0x1241001a350c, vec error info: 0x4f1c75520c, mte error info: 0x8d0000003c, ifu error info: 0x5bdbe119de040, ccu error info: 0x6bf20269544c38c1, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12414040dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        TraceBack (most recent call last):
        The extend info: errcode:(0x200000000, 0, 0) errorStr: The write address of the MTE instruction is out of range. fixp_error0 info: 0x3c, fixp_error1 info: 0x8d fsmId:0, tslot:5, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:4, dieId:0), serial number is 17, there is an aivec error exception, core id is 11, error code = 0x200000000, dump info: pc start: 0x1241001a3000, current: 0x1241001a3518, vec error info: 0x6a0715b839, mte error info: 0x8d0000003c, ifu error info: 0x392415cfc0200, ccu error info: 0x193bb81646800040, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12414040dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x200000000, 0x4000, 0) errorStr: The write address of the MTE instruction is out of range.CCU instruction address check error. fixp_error0 info: 0x3c, fixp_error1 info: 0x8d fsmId:0, tslot:5, thread:0, ctxid:0, blk:2, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:4, dieId:0), serial number is 17, there is an aivec error exception, core id is 13, error code = 0x200000000, dump info: pc start: 0x1241001a3000, current: 0x1241001a350c, vec error info: 0xa312358b68, mte error info: 0x8d0000003c, ifu error info: 0x11016720180, ccu error info: 0xa52464050500883, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12414040dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x200000000, 0, 0) errorStr: The write address of the MTE instruction is out of range. fixp_error0 info: 0x3c, fixp_error1 info: 0x8d fsmId:0, tslot:5, thread:0, ctxid:0, blk:4, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:4, dieId:0), serial number is 17, there is an aivec error exception, core id is 14, error code = 0x200000000, dump info: pc start: 0x1241001a3000, current: 0x1241001a3518, vec error info: 0xaa0c770e21, mte error info: 0x8d0000003c, ifu error info: 0x64d89d5461c00, ccu error info: 0xcc32308946800040, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12414040dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x200000000, 0x4000, 0) errorStr: The write address of the MTE instruction is out of range.CCU instruction address check error. fixp_error0 info: 0x3c, fixp_error1 info: 0x8d fsmId:0, tslot:5, thread:0, ctxid:0, blk:5, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:4, dieId:0), serial number is 17, there is an aivec error exception, core id is 15, error code = 0x200000000, dump info: pc start: 0x1241001a3000, current: 0x1241001a350c, vec error info: 0x6f04dc18a3, mte error info: 0x8d0000003c, ifu error info: 0x79560f8442000, ccu error info: 0x5608e1f068800040, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12414040dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x200000000, 0, 0) errorStr: The write address of the MTE instruction is out of range. fixp_error0 info: 0x3c, fixp_error1 info: 0x8d fsmId:0, tslot:5, thread:0, ctxid:0, blk:6, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:4, dieId:0), serial number is 17, there is an aivec error exception, core id is 20, error code = 0x200000000, dump info: pc start: 0x1241001a3000, current: 0x1241001a3518, vec error info: 0x4310c43729, mte error info: 0x8d0000003c, ifu error info: 0x9cf9e94024c0, ccu error info: 0xc0bf19c846800040, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12414040dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x200000000, 0x4000, 0) errorStr: The write address of the MTE instruction is out of range.CCU instruction address check error. fixp_error0 info: 0x3c, fixp_error1 info: 0x8d fsmId:0, tslot:5, thread:0, ctxid:0, blk:7, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:4, dieId:0), serial number is 17, there is an aivec error exception, core id is 10, error code = 0x200000000, dump info: pc start: 0x1241001a3000, current: 0x1241001a3500, vec error info: 0x7b14b48580, mte error info: 0x8d0000003c, ifu error info: 0x44ef91d8cdac0, ccu error info: 0xf6ca0f9e023f083c, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12414040dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x200000000, 0, 0) errorStr: The write address of the MTE instruction is out of range. fixp_error0 info: 0x3c, fixp_error1 info: 0x8d fsmId:1, tslot:5, thread:0, ctxid:0, blk:1, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:4, dieId:0), serial number is 17, there is an aivec error exception, core id is 12, error code = 0x200000000, dump info: pc start: 0x1241001a3000, current: 0x1241001a3500, vec error info: 0x511d332228, mte error info: 0x8d0000003c, ifu error info: 0x10837ff884f80, ccu error info: 0x90e60f0d315ba60c, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12414040dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x200000000, 0, 0) errorStr: The write address of the MTE instruction is out of range. fixp_error0 info: 0x3c, fixp_error1 info: 0x8d fsmId:1, tslot:5, thread:0, ctxid:0, blk:3, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        Kernel task happen error, retCode=0x31, [vector core exception].[FUNC:PreCheckTaskErr][FILE:task_info.cc][LINE:1780]
        AIV Kernel happen error, retCode=0x31.[FUNC:GetError][FILE:stream.cc][LINE:1512]
        Aicore kernel execute failed, device_id=0, stream_id=38, report_stream_id=38, task_id=55, flip_num=0, fault kernel_name=add_custom_0, fault kernel info ext=none, program id=0, hash=6183514293749665014.[FUNC:GetError][FILE:stream.cc][LINE:1512]
        [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1512]
        rtStreamSynchronize execute failed, reason=[vector core exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
        Assert ((rtStreamSynchronize(stream)) == 0) failed
        Assert ((extInfoHandle_->UpdateOutputShapeFromExtInfo(outputs_, stream)) == OK) failed
        launch failed for MaskedSelect, errno:361001.

[ERROR] 2024-12-11-16:17:20 (PID:2487787, Device:0, RankID:-1) ERR01005 OPS internal error
Exception raised from operator() at build/CMakeFiles/torch_npu.dir/compiler_depend.ts:49 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xffff84855898 in /home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xffff8480e2a8 in /home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xb049c8 (0xfffd9ebdf9c8 in /home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe82110 (0xfffd9ef5d110 in /home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x5d1160 (0xfffd9e6ac160 in /home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x5d15f8 (0xfffd9e6ac5f8 in /home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x5ced1c (0xfffd9e6a9d1c in /home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0xd181c (0xffff846d381c in /usr/local/gcc10.2.0/lib64/libstdc++.so.6)
frame #8: <unknown function> + 0x87b0 (0xffff9ac9c7b0 in /usr/lib64/libpthread.so.0)
frame #9: <unknown function> + 0xd522c (0xffff9aae022c in /usr/lib64/libc.so.6)

Traceback (most recent call last):
  File "/home/ma-user/work/wxl/operator_RC1/AddCustomSample/KernelLaunch/CppExtensions/add_custom_test.py", line 18, in <module>
    print(z)
  File "/home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch/_tensor.py", line 431, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch/_tensor_str.py", line 664, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch/_tensor_str.py", line 595, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch/_tensor_str.py", line 347, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch/_tensor_str.py", line 137, in __init__
    nonzero_finite_vals = torch.masked_select(
RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is aclnnMaskedSelect.
Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-12-11-16:17:20 (PID:2487787, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[W compiler_depend.ts:409] Warning: NPU warning, error code is 507035[Error]:
[Error]: The vector core execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[vector core exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2024-12-11-16:17:20.229.177  wait for compute device to finish failed, runtime result = 507035.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeUsedDevices)
[W compiler_depend.ts:392] Warning: NPU warning, error code is 507035[Error]:
[Error]: The vector core execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[vector core exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2024-12-11-16:17:20.231.527  wait for compute device to finish failed, runtime result = 507035.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)

三、软件版本:
-- CANN 版本 : 8.0.RC1
--Pytorch 版本: 2.1.0
--Python 版本 :3.9.10

Ascend/samples

内容风险标识

评论 (1)

Ascend/samples .gitee-modal { width: 500px !important; }

内容风险标识