samples/operator/AddCustomSample/KernelLaunch/CppExtensions 在910B机器上有问题

一、问题现象（附报错日志上下文）：
xxxx

二、软件版本:
-- CANN 版本 (e.g., CANN 3.0.x，5.x.x):  8.0.RC1.1
--Tensorflow/Pytorch/MindSpore 版本: torch_npu 2.1.0.post3
--Python 版本 (e.g., Python 3.7.5): 3.9
-- MindStudio版本 (e.g., MindStudio 2.0.0 (beta3)):
--操作系统版本 (e.g., Ubuntu 18.04): Linux app-hcnas0k2-75c5457849-g7vl9 5.10.0-153.12.0.92.oe2203sp2.x86_64 #1 SMP Wed Jun 28 23:04:48 CST 2023 x86_64 x86_64 x86_64 GNU/Linux

三、测试步骤：
 python add_custom_test.py 
Warning: Device do not support double dtype now, dtype cast repalce with float.
tensor([[ 0.7549, -0.8813,  0.6733,  ...,  0.2185, -1.4590,  0.7178],
        [-1.3076,  0.8994,  0.7734,  ...,  1.0225,  0.9565,  0.4407],
        [ 0.2208, -0.6519, -1.3740,  ...,  0.4512,  0.6094, -0.4568],
        ...,
        [-0.0869, -0.3525, -0.1232,  ..., -0.0702,  1.0088,  0.3511],
        [ 2.2246, -0.4773, -0.5415,  ...,  0.1980,  1.1260, -1.0967],
        [ 0.0035, -1.2334, -1.1846,  ...,  0.2976, -0.1313,  0.6099]],
       device='npu:0', dtype=torch.float16)
tensor([[-0.9849,  0.2142,  1.1143,  ..., -1.1436,  1.3008, -0.5962],
        [-0.1976,  0.2510, -0.4382,  ...,  1.2305, -0.2874, -1.3965],
        [-0.0023, -1.6982,  1.3125,  ..., -0.0640, -0.6235, -0.2659],
        ...,
        [-0.9048, -0.3789,  1.7227,  ...,  0.1132, -0.2573,  0.1935],
        [ 1.1855,  0.4023,  0.8042,  ...,  0.6938,  1.0342, -1.0498],
        [-0.4692, -0.6587,  0.0566,  ..., -0.8091,  1.3174, -0.3730]],
       device='npu:0', dtype=torch.float16)
Traceback (most recent call last):
  File "/samples/operator/AddCustomSample/KernelLaunch/CppExtensions/add_custom_test.py", line 12, in <module>
    print(z)
  File "/opt/miniconda3/envs/torch_npu_py39/lib/python3.9/site-packages/torch/_tensor.py", line 431, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/opt/miniconda3/envs/torch_npu_py39/lib/python3.9/site-packages/torch/_tensor_str.py", line 664, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/opt/miniconda3/envs/torch_npu_py39/lib/python3.9/site-packages/torch/_tensor_str.py", line 595, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/opt/miniconda3/envs/torch_npu_py39/lib/python3.9/site-packages/torch/_tensor_str.py", line 329, in _tensor_str
    self = self.float()
RuntimeError: Sync:torch_npu/csrc/framework/OpCommand.cpp:190 NPU error, error code is 507015
[ERROR] 2024-07-29-10:29:46 (PID:1538992, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal. 
        Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: 2024-07-29-10:29:46.886.086  The error from device(chipId:6, dieId:0), serial number is 8, there is an aicore error exception, core id is 6, error code = 0x10, dump info: pc start: 0x1240c01a3000, current: 0x1240c01a3038, vec error info: 0, mte error info: 0x1b5d2208a9, ifu error info: 0x52aea7b4d8300, ccu error info: 0x8040000046000002, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12418000dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        TraceBack (most recent call last):
        The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0xd2208a9, fixp_error1 info: 0x1b fsmId:0, tslot:4, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:6, dieId:0), serial number is 8, there is an aicore error exception, core id is 7, error code = 0x10, dump info: pc start: 0x1240c01a3000, current: 0x1240c01a3038, vec error info: 0, mte error info: 0xcb3ada6caa, ifu error info: 0x6562325b83d00, ccu error info: 0x8040000046000002, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12418000dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0xada6caa, fixp_error1 info: 0xcb fsmId:0, tslot:4, thread:0, ctxid:0, blk:1, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:6, dieId:0), serial number is 8, there is an aicore error exception, core id is 8, error code = 0x10, dump info: pc start: 0x1240c01a3000, current: 0x1240c01a3038, vec error info: 0, mte error info: 0x1d5739b8a8, ifu error info: 0x1159fb1eedec0, ccu error info: 0x8040000046000002, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12418000dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0x739b8a8, fixp_error1 info: 0x1d fsmId:0, tslot:4, thread:0, ctxid:0, blk:2, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:6, dieId:0), serial number is 8, there is an aicore error exception, core id is 9, error code = 0x10, dump info: pc start: 0x1240c01a3000, current: 0x1240c01a3038, vec error info: 0, mte error info: 0x4426c1773, ifu error info: 0x212de06ba6c00, ccu error info: 0x8040000046000002, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12418000dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0x26c1773, fixp_error1 info: 0x4 fsmId:0, tslot:4, thread:0, ctxid:0, blk:3, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:6, dieId:0), serial number is 8, there is an aicore error exception, core id is 10, error code = 0x10, dump info: pc start: 0x1240c01a3000, current: 0x1240c01a3038, vec error info: 0, mte error info: 0x219e046150, ifu error info: 0x76e587ebdda00, ccu error info: 0x8040000046000002, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12418000dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0xe046150, fixp_error1 info: 0x21 fsmId:0, tslot:4, thread:0, ctxid:0, blk:4, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:6, dieId:0), serial number is 8, there is an aicore error exception, core id is 11, error code = 0x10, dump info: pc start: 0x1240c01a3000, current: 0x1240c01a3038, vec error info: 0, mte error info: 0x900afca24, ifu error info: 0x19600565e1f80, ccu error info: 0x8040000046000002, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12418000dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0xafca24, fixp_error1 info: 0x9 fsmId:0, tslot:4, thread:0, ctxid:0, blk:5, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:6, dieId:0), serial number is 8, there is an aicore error exception, core id is 12, error code = 0x10, dump info: pc start: 0x1240c01a3000, current: 0x1240c01a3038, vec error info: 0, mte error info: 0xa38f522159, ifu error info: 0x76ff78e023c0, ccu error info: 0x8040000046000002, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12418000dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0xf522159, fixp_error1 info: 0xa3 fsmId:0, tslot:4, thread:0, ctxid:0, blk:6, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        The error from device(chipId:6, dieId:0), serial number is 8, there is an aicore error exception, core id is 13, error code = 0x10, dump info: pc start: 0x1240c01a3000, current: 0x1240c01a3038, vec error info: 0, mte error info: 0xbc00324901, ifu error info: 0x19fb61e086c00, ccu error info: 0x8040000046000002, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x12418000dc00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1170]
        The extend info: errcode:(0x10, 0, 0) errorStr: Illegal instruction, which is usually caused by unaligned UUB addresses. fixp_error0 info: 0x324901, fixp_error1 info: 0xbc fsmId:0, tslot:4, thread:0, ctxid:0, blk:7, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1182]
        Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:task_info.cc][LINE:1780]
        AICORE Kernel task happen error, retCode=0x26.[FUNC:GetError][FILE:stream.cc][LINE:1512]
        Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=55, flip_num=0, fault kernel_name=add_custom_0, fault kernel info ext=none, program id=0, hash=13875772728910118696.[FUNC:GetError][FILE:stream.cc][LINE:1512]
        [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1512]
        rtStreamSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
        synchronize stream failed, runtime result = 507015[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

[W NPUStream.cpp:382] Warning: NPU warning, error code is 507015[Error]: 
[Error]: The aicore execution is abnormal. 
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2024-07-29-10:29:46.890.642  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeUsedDevices)
[W NPUStream.cpp:365] Warning: NPU warning, error code is 507015[Error]: 
[Error]: The aicore execution is abnormal. 
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: 2024-07-29-10:29:46.891.303  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)

四、日志信息:
AddCustomSample/KernelLaunch/CppExtensions# env 
SHELL=/bin/bash
COLORTERM=truecolor
TERM_PROGRAM_VERSION=1.88.0
CONDA_EXE=/opt/miniconda3/bin/conda
_CE_M=
PWD=/samples/operator/AddCustomSample/KernelLaunch/CppExtensions
LOGNAME=root
CONDA_PREFIX=/opt/miniconda3/envs/torch_npu_py39
TZ=UTC-8
VSCODE_GIT_ASKPASS_NODE=/root/.vscode-server/cli/servers/Stable-5c3e652f63e798a5ac2f31ffd0d863669328dc4c/server/node
MOTD_SHOWN=pam
HOME=/root
LANG=en_US.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
SSL_CERT_DIR=/usr/lib/ssl/certs
CONDA_PROMPT_MODIFIER=(torch_npu_py39) 
GIT_ASKPASS=/root/.vscode-server/cli/servers/Stable-5c3e652f63e798a5ac2f31ffd0d863669328dc4c/server/extensions/git/dist/askpass.sh
SSH_CONNECTION=101.230.164.50 50044 10.119.37.152 22
TOOLCHAIN_HOME=/usr/local/Ascend/ascend-toolkit/latest/toolkit
ASCEND_TOOLKIT_HOME=/usr/local/Ascend/ascend-toolkit/latest
VSCODE_GIT_ASKPASS_EXTRA_ARGS=
LESSCLOSE=/usr/bin/lesspipe %s %s
PYTHONPATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe:/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe:
TERM=xterm-256color
_CE_CONDA=
LESSOPEN=| /usr/bin/lesspipe %s
USER=root
VSCODE_GIT_IPC_HANDLE=/tmp/vscode-git-a66982a2d4.sock
CONDA_SHLVL=2
SHLVL=2
ASCEND_OPP_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp
CONDA_PYTHON_EXE=/opt/miniconda3/bin/python
LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64:/usr/local/Ascend/ascend-toolkit/latest/tools/aml/lib64/plugin:/usr/local/Ascend/ascend-toolkit/latest/lib64:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/opskernel:/usr/local/Ascend/ascend-toolkit/latest/lib64/plugin/nnengine:/usr/local/Ascend/ascend-toolkit/latest/opp/built-in/op_impl/ai_core/tbe/op_tiling/lib/linux/x86_64:/usr/local/Ascend/ascend-toolkit/latest/fwkacllib/lib64/:/usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/common/:/usr/local/Ascend/driver/lib64/driver/:/usr/local/Ascend/add-ons/:/usr/lib/aarch64_64-linux-gnu
ASCEND_AICPU_PATH=/usr/local/Ascend/ascend-toolkit/latest
SSL_CERT_FILE=/usr/lib/ssl/certs/ca-certificates.crt
SSH_CLIENT=101.230.164.50 50044 22
CONDA_DEFAULT_ENV=torch_npu_py39
VSCODE_GIT_ASKPASS_MAIN=/root/.vscode-server/cli/servers/Stable-5c3e652f63e798a5ac2f31ffd0d863669328dc4c/server/extensions/git/dist/askpass-main.js
BROWSER=/root/.vscode-server/cli/servers/Stable-5c3e652f63e798a5ac2f31ffd0d863669328dc4c/server/bin/helpers/browser.sh
PATH=/usr/local/Ascend/ascend-toolkit/latest/bin:/usr/local/Ascend/ascend-toolkit/latest/compiler/ccec_compiler/bin:/usr/local/Ascend/ascend-toolkit/latest/tools/ccec_compiler/bin:/usr/local/Ascend/ascend-toolkit/latest/bin:/usr/local/Ascend/ascend-toolkit/latest/compiler/ccec_compiler/bin:/usr/local/Ascend/ascend-toolkit/latest/tools/ccec_compiler/bin:/opt/miniconda3/envs/torch_npu_py39/bin:/opt/miniconda3/condabin:/root/.vscode-server/cli/servers/Stable-5c3e652f63e798a5ac2f31ffd0d863669328dc4c/server/bin/remote-cli:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
ASCEND_HOME_PATH=/usr/local/Ascend/ascend-toolkit/latest
CONDA_PREFIX_1=/opt/miniconda3
OLDPWD=/MOE/samples/operator/AddCustomSample/KernelLaunch/CppExtensions/build
ASCEND_LAUNCH_BLOCKING=1
TERM_PROGRAM=vscode
VSCODE_IPC_HOOK_CLI=/tmp/vscode-ipc-6eb269c3-f410-4800-bebc-90fdfa19d38a.sock
_=/usr/bin/env
请根据自己的运行环境参考以下方式搜集日志信息，如果涉及到算子开发相关的问题，建议也提供UT/ST测试和单算子集成测试相关的日志。

日志提供方式:
将日志打包后作为附件上传。若日志大小超出附件限制，则可上传至外部网盘后提供链接。

获取方法请参考wiki：
https://gitee.com/ascend/modelzoo/wikis/%E5%A6%82%E4%BD%95%E8%8E%B7%E5%8F%96%E6%97%A5%E5%BF%97%E5%92%8C%E8%AE%A1%E7%AE%97%E5%9B%BE?sort_id=4097825

Ascend/samples

内容风险标识

评论 (2)

Ascend/samples .gitee-modal { width: 500px !important; }

内容风险标识