在线推理报错【aicpu exception】

pytorch在线推理运行时报错
日志如下：
[EVENT] PROFILING(3303,python):2023-02-17-15:18:00.405.119 [msprof_callback_impl.cpp:199] >>> (tid:3303) Started to register profiling ctrl callback.
[EVENT] PROFILING(3303,python):2023-02-17-15:18:02.031.651 [msprof_callback_impl.cpp:78] >>> (tid:3303) MsprofCtrlCallback called, type: 255
[EVENT] PROFILING(3303,python):2023-02-17-15:18:02.031.738 [prof_acl_mgr.cpp:1190] >>> (tid:3303) Init profiling for dynamic profiling
[TRACE] GE(3303,python):2023-02-17-15:18:05.927.227 [status:INIT] [ge_api.cc:194]3303 GEInitializeImpl:GEInitialize start
[EVENT] PROFILING(3303,python):2023-02-17-15:18:05.936.153 [msprof_callback_impl.cpp:78] >>> (tid:3303) MsprofCtrlCallback called, type: 255
[EVENT] PROFILING(3303,python):2023-02-17-15:18:05.936.183 [prof_acl_mgr.cpp:1190] >>> (tid:3303) Init profiling for dynamic profiling
[TRACE] GE(3303,python):2023-02-17-15:18:06.261.072 [status:RUNNING] [ge_api.cc:248]3303 GEInitializeImpl:Initializing environment
[TRACE] GE(3303,python):2023-02-17-15:18:14.896.545 [status:STOP] [ge_api.cc:295]3303 GEInitializeImpl:GEInitialize finished
[TRACE] GE(3303,python):2023-02-17-15:18:14.897.817 [status:INIT] [ge_api.cc:194]3303 GEInitializeImpl:GEInitialize start
[ERROR] RUNTIME(3303,python):2023-02-17-15:18:42.416.521 [engine.cc:1101]3303 ReportExceptProc:[FINAL][FINAL]Task exception! device_id=0, stream_id=2, task_id=3810, type=1, retCode=0x2a, [aicpu exception].
[ERROR] RUNTIME(3303,python):2023-02-17-15:18:42.417.656 [task.cc:1064]3303 PreCheckTaskErr:[FINAL][FINAL]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(3303,python):2023-02-17-15:18:42.417.664 [task.cc:1064]3303 PreCheckTaskErr:[FINAL][FINAL]Kernel task happen error, retCode=0x2a, [aicpu exception].
[ERROR] RUNTIME(3303,python):2023-02-17-15:18:42.417.691 [task.cc:773]3303 PrintAicpuErrorInfo:[FINAL][FINAL]report error module_type=0, module_name=E39999
[ERROR] RUNTIME(3303,python):2023-02-17-15:18:42.417.694 [task.cc:773]3303 PrintAicpuErrorInfo:[FINAL][FINAL]Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=3810.
[ERROR] RUNTIME(3303,python):2023-02-17-15:18:42.417.703 [task.cc:781]3303 PrintAicpuErrorInfo:[FINAL][FINAL]Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=3810, fault op_name=Index.
[ERROR] RUNTIME(3303,python):2023-02-17-15:18:42.418.243 [stream.cc:918]3303 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x2a, [aicpu exception].
[ERROR] RUNTIME(3303,python):2023-02-17-15:18:42.418.247 [stream.cc:921]3303 GetError:[FINAL][FINAL]report error module_type=0, module_name=E39999
[ERROR] RUNTIME(3303,python):2023-02-17-15:18:42.418.249 [stream.cc:921]3303 GetError:[FINAL][FINAL]Aicpu kernel execute failed, device_id=0, stream_id=2, task_id=3810, fault op_name=Index
[ERROR] RUNTIME(3303,python):2023-02-17-15:18:42.418.256 [logger.cc:305]3303 StreamSynchronize:[FINAL][FINAL]Stream synchronize failed, stream = 0x558b930e2e80
[ERROR] RUNTIME(3303,python):2023-02-17-15:18:42.418.266 [api_c.cc:661]3303 rtStreamSynchronize:[FINAL][FINAL]ErrCode=507018, desc=[aicpu exception], InnerCode=0x715002a
[ERROR] RUNTIME(3303,python):2023-02-17-15:18:42.418.270 [error_message_manage.cc:49]3303 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3303,python):2023-02-17-15:18:42.418.274 [error_message_manage.cc:49]3303 FuncErrorReason:[FINAL][FINAL]rtStreamSynchronize execute failed, reason=[aicpu exception]
[ERROR] GE(3303,python):2023-02-17-15:18:42.418.304 [stream.cc:22]3303 SyncStream: ErrorNo: 4294967295(failed) [FINAL][FINAL]Assert ((rtStreamSynchronize(stream)) == 0) failed
[ERROR] ASCENDCL(3303,python):2023-02-17-15:18:42.418.310 [op_executor.cpp:366]3303 DoExecuteAsync: [FINAL][FINAL][Exec][Op]Execute op failed, ge result = 4294967295
[ERROR] OP(3303,python):2023-02-17-15:18:42.520.894 [split_combination_ops.cc:777][OP_PROTO] ConcatInferShapeCommon:777 OpName:[ConcatD] "the input shape dims should be equal except merge axis,shapes:[[15, 1, ], [76725, 1, ], ]axis:1"
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.180 [op_desc.cc:1295]3303 CallInferFunc: ErrorNo: 4294967295(failed) [COMP][PRE_OPT][Call][InferFunc] for ConcatD failed. ret:4294967295
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.320 [infershape_pass.cc:120]3303 Infer: ErrorNo: 1343242270(Prepare Graph infershape failed) [COMP][PRE_OPT][Call][InferShapeAndType] for node:ConcatD(ConcatD) failed
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.325 [infer_base_pass.cc:114]3303 InferAndUpdate: ErrorNo: 1343242270(Prepare Graph infershape failed) [COMP][PRE_OPT][Call][Infer] failed for node ConcatD, ret: 1343242270
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.328 [infer_base_pass.cc:71]3303 Run: ErrorNo: 1343242270(Prepare Graph infershape failed) [COMP][PRE_OPT][Call][InferAndUpdate] for node ConcatD failed! ret: 1343242270
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.348 [base_pass.cc:532]3303 RunPassesOnNode: ErrorNo: 1343225860(Internal errors) [COMP][PRE_OPT][Process][Pass] InferShapePass on node ConcatD failed, result 4294967295, the passes will be terminated immediately.
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.352 [base_pass.cc:483]3303 RunPassesNodeOnce: ErrorNo: 4294967295(failed) [COMP][PRE_OPT][Process][Passes] on node ConcatD type ConcatD failed, error code:4294967295
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.356 [base_pass.cc:439]3303 RunPassesGraphRepass: ErrorNo: 4294967295(failed) [COMP][PRE_OPT][Process][Passes] on node ConcatD type ConcatD failed, error code:4294967295
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.360 [graph_prepare.cc:2197]3303 InferShapeForPreprocess: ErrorNo: 4294967295(failed) [COMP][PRE_OPT][Run][GePasses] infershape for preprocess failed, ret:4294967295.
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.369 [graph_prepare.cc:1745]3303 FormatAndShapeProcess: ErrorNo: 1343242270(Prepare Graph infershape failed) [COMP][PRE_OPT][Call][InferShapeForPreprocess] Prepare Graph infershape failed
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.372 [graph_prepare.cc:1887]3303 PrepareDynShape: ErrorNo: 1343242270(Prepare Graph infershape failed) [COMP][PRE_OPT][Process][Prepare_FormatAndShapeProcess] failed
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.512 [graph_manager.cc:772]3303 PreRunOptimizeOriginalGraph: ErrorNo: 1343242270(Prepare Graph infershape failed) [COMP][PRE_OPT][Process][GraphManager_stages.preparer.PrepareDynShape] failed
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.582 [graph_manager.cc:3299]3303 OptimizeGraph: ErrorNo: 1343242270(Prepare Graph infershape failed) [COMP][PRE_OPT][Run][PreRunOptimizeOriginalGraph] failed for graph:20230217151842_online_NPU, session_id:417
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.587 [pne_model_builder.cc:112]3303 OptimizeGraph: ErrorNo: 4294967295(failed) [COMP][PRE_OPT][Optimize][Graph] failed, graph = 20230217151842_online_NPU, engine = NPU
[ERROR] GE(3303,python):2023-02-17-15:18:42.521.592 [graph_manager.cc:918]3303 PreRun: ErrorNo: 4294967295(failed) [COMP][PRE_OPT][Build][Model] failed, session_id:417, graph_id:417.
[ERROR] GE(3303,python):2023-02-17-15:18:42.522.011 [graph_manager.cc:961]3303 StartForRunGraph: ErrorNo: 1343242270(Prepare Graph infershape failed) [COMP][PRE_OPT][Call][PreRun] Failed, graph_id:417, session_id:417.
[ERROR] GE(3303,python):2023-02-17-15:18:42.522.014 [graph_manager.cc:1360]3303 BuildGraph: ErrorNo: 1343242270(Prepare Graph infershape failed) [COMP][PRE_OPT][Call][StartForRunGraph] failed! graph_id:417.
[ERROR] GE(3303,python):2023-02-17-15:18:42.522.047 [ge_generator.cc:1497]3303 BuildModel: ErrorNo: 1343266819(Graph manager build graph failed.) [COMP][DEFAULT][Build][Graph] fail, graph id: 417
[ERROR] ASCENDCL(3303,python):2023-02-17-15:18:42.522.071 [local_compiler.cpp:93]3303 OnlineCompileAndDump: [COMP][DEFAULT][Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 1343242270
[ERROR] ASCENDCL(3303,python):2023-02-17-15:18:42.522.082 [local_compiler.cpp:106]3303 OnlineCompileAndDump: [COMP][DEFAULT]OnlineCompileAndDump failed

Ascend/pytorch
暂停

内容风险标识

评论 (5)

Ascend/pytorch暂停 .gitee-modal { width: 500px !important; }

内容风险标识