AvgPoolV2GradD算子报错507015

一、问题现象（附报错日志上下文）：
AvgPoolV2GradD算子
![输入图片说明](https://images.gitee.com/uploads/images/2021/0714/170341_314dd540_1709862.png "屏幕截图.png")
![输入图片说明](https://images.gitee.com/uploads/images/2021/0714/170250_bc4a669c_1709862.png "屏幕截图.png")
错误代码507015，如何排查问题

二、软件版本: 5.0.2
-- CANN 版本 (e.g., CANN 3.0.x，5.x.x):  
--Tensorflow/Pytorch/MindSpore 版本:
--Python 版本 (e.g., Python 3.7.5):
-- MindStudio版本 (e.g., MindStudio 2.0.0 (beta3)):
--操作系统版本 (e.g., Ubuntu 18.04): Red Hat 4.8.5-36

三、测试步骤：
xxxx

四、日志信息:
```
[ERROR] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.318 [engine.cc:922]4011 ReportExceptProc:Task exception! stream_id=552, task_id=12, type=0, retCode=0x26.
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.327 [device_error_proc.cc:557] 4011 ProcErrorInfo: Begin to process device error info.
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.459 [device_error_proc.cc:503] 4011 ProcessOneElementInRingBuffer: it needs process 1 error messages.
[ERROR] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.476 [device_error_proc.cc:407]4011 ProcessCoreErrorInfo:The error from device(0), serial number is 229, there is a aicore error, core id is 0, error code = 0x800000, error string = The DDR address of the MTE instruction is out of range. dump info: pc start: 0x1000108040026000, current: 0x108040026344, vec error info: 0xffc7b7d, mte error info: 0x60000d0, ifu error info: 0x3fff55f97fe80, ccu error info: 0x0, cube error info: 0x5f, biu error info: 0x0, aic error mask: 0x65000200d000288, para base: 0x108000ad2800.
[ERROR] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.489 [device_error_proc.cc:407]4011 ProcessCoreErrorInfo:The error from device(0), serial number is 229, there is a aicore error, core id is 1, error code = 0x800000, error string = The DDR address of the MTE instruction is out of range. dump info: pc start: 0x1000108040026000, current: 0x108040026344, vec error info: 0xceffaff, mte error info: 0x60000d0, ifu error info: 0x3f733ef402280, ccu error info: 0x0, cube error info: 0x5f, biu error info: 0x0, aic error mask: 0x65000200d000288, para base: 0x108000ad2800.
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.573 [device_error_proc.cc:598] 4011 ProcErrorInfo: finished to process device error info, retCode=0.
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.585 [engine.cc:933] 4011 ReportExceptProc: excptCallBack_ is null.
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.591 [stream.cc:901] 4011 TryDelRecordedTask: del public task from stream, stream_id=552, tailTaskId=12, delTaskId=10, head=2, tail=7
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.600 [logger.cc:1300] 4011 TaskFinished: device_id=0, stream_id=552, task_id=10, task_type=0,task_finish_num=44
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.609 [task.cc:73] 4011 TaskFailCallBack: task ok, stream_id=552, task_id=10, retCode=0
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.620 [stream.cc:901] 4011 TryDelRecordedTask: del public task from stream, stream_id=552, tailTaskId=12, delTaskId=11, head=3, tail=7
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.627 [logger.cc:1300] 4011 TaskFinished: device_id=0, stream_id=552, task_id=11, task_type=0,task_finish_num=45
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.632 [task.cc:73] 4011 TaskFailCallBack: task ok, stream_id=552, task_id=11, retCode=0
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.638 [stream.cc:901] 4011 TryDelRecordedTask: del public task from stream, stream_id=552, tailTaskId=12, delTaskId=12, head=4, tail=7
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.643 [logger.cc:1300] 4011 TaskFinished: device_id=0, stream_id=552, task_id=12, task_type=0,task_finish_num=46
[ERROR] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.716.649 [task.cc:730]4011 PreCheckTaskErr:Kernel task happen error, retCode=0x26, [aicore exception].
[ERROR] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.470 [task.cc:712]4011 PrintErrorInfo:Aicore kernel execute failed, device_id=0, stream_id=552, report_stream_id=552, task_id=12, fault kernel_name=1/AvgPoolV2GradD_tvmbin, func_name=te_avgpoolv2gradd_2f3e32dac9352c44c510558ee4e2abbbd11759f096460e5253bba90a88323249_30b599a383a17a69_0__kernel0, program id=20, hash=3207780278308406204.
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.486 [task.cc:88] 4011 TaskFailCallBack: rtCode=0x7150026,[aicore exception], errorTaskId=12, errorStreamId=552
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.506 [engine.cc:610] 4011 ProcessTaskReport: RTS_DRIVER: report receive, stream_id=552, sq_id=297, task_id=15, sq_head=7.
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.516 [engine.cc:610] 4011 ProcessTaskReport: RTS_DRIVER: report receive, stream_id=552, sq_id=297, task_id=15, sq_head=7.
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.524 [stream.cc:901] 4011 TryDelRecordedTask: del public task from stream, stream_id=552, tailTaskId=15, delTaskId=13, head=5, tail=7
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.531 [logger.cc:1300] 4011 TaskFinished: device_id=0, stream_id=552, task_id=13, task_type=0,task_finish_num=47
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.536 [task.cc:73] 4011 TaskFailCallBack: task ok, stream_id=552, task_id=13, retCode=0
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.542 [stream.cc:901] 4011 TryDelRecordedTask: del public task from stream, stream_id=552, tailTaskId=15, delTaskId=14, head=6, tail=7
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.548 [logger.cc:1300] 4011 TaskFinished: device_id=0, stream_id=552, task_id=14, task_type=0,task_finish_num=48
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.553 [task.cc:73] 4011 TaskFailCallBack: task ok, stream_id=552, task_id=14, retCode=0
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.558 [stream.cc:901] 4011 TryDelRecordedTask: del public task from stream, stream_id=552, tailTaskId=15, delTaskId=15, head=7, tail=7
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.564 [logger.cc:1300] 4011 TaskFinished: device_id=0, stream_id=552, task_id=15, task_type=2,task_finish_num=49
[ERROR] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.593 [stream.cc:690]3786 GetError:[FINAL][FINAL]Stream Synchronize failed, stream=552 retCode=0x26, [aicore exception].
[ERROR] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.655 [logger.cc:271]3786 StreamSynchronize:[FINAL][FINAL]Stream synchronize failed, stream = 0x46231030
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.661 [npu_driver.cc:380] 4010 CommandOccupy: sqId=286, deviceId=0, tsId=0, command=0xf00011e0140, cmdCount=1.
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.713 [engine.cc:802] 4010 SendingRun: CommandOccupy. sqId=286, cqId=179, deviceId=0, retCode=0.
[ERROR] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.720 [api_c.cc:504]3786 rtStreamSynchronize:[FINAL][FINAL]ErrCode=507015, desc=[aicore exception], InnerCode=0x7150026
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.720 [stream.cc:869] 4010 AddTaskToStream: recorded public task to stream, stream_id=541, task_id=5, task_type=6, head=5, tail=6
[ERROR] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.756 [error_message_manage.cc:26]3786 ReportFuncErrorReason:[FINAL][FINAL]rtStreamSynchronize execute failed, reason=[aicore exception]
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.760 [logger.cc:1289] 4010 TaskLaunchedEx: device_id=0, stream_id=541, task_id=5, task_type=6,task_launched_num=50
[ERROR] ASCENDCL(3786,avgpool2d):2021-07-14-17:06:03.719.799 [/home/jenkins/agent/workspace/Compile_GraphEngine_Centos_ARM/acl/runtime/stream.cpp:69]3786 aclrtSynchronizeStream: synchronize stream failed, runtime result = 507015
[INFO] RUNTIME(3786,avgpool2d):2021-07-14-17:06:03.719.806 [npu_driver.cc:407] 4010 CommandSend: Command send success, device_id=0, ts_id=0, sq_id=286, reportCount=1, command=0xf00011e0140.
```
请根据自己的运行环境参考以下方式搜集日志信息，如果涉及到算子开发相关的问题，建议也提供UT/ST测试和单算子集成测试相关的日志。

日志提供方式:
将日志打包后作为附件上传。若日志大小超出附件限制，则可上传至外部网盘后提供链接。

获取方法请参考wiki：
https://gitee.com/ascend/modelzoo/wikis/%E5%A6%82%E4%BD%95%E8%8E%B7%E5%8F%96%E6%97%A5%E5%BF%97%E5%92%8C%E8%AE%A1%E7%AE%97%E5%9B%BE?sort_id=4097825

Ascend/modelzoo
暂停

内容风险标识

评论 (42)

Ascend/modelzoo暂停 .gitee-modal { width: 500px !important; }

内容风险标识