[南航][Flownet2][pycharm调试modelarts-NPU时报错 GeOp7_0GEOP::::DoRunAsync Failed]

[南航][Flownet2][pycharm调试modelarts-NPU时报错 GeOp7_0GEOP::::DoRunAsync Failed]
一、问题现象：
pycharm调试modelarts-NPU时报错
(https://images.gitee.com/uploads/images/2021/0602/210129_a8d0d243_9131327.png)
[ERROR] TEFUSION(153,python):2021-06-02-18:50:58.279.546  ===========END OF DETAILED OP INFO OF NODE gradients/FlowNet2/fuse_conv0/Conv2D_grad/Conv2DBackpropFilter=============
[ERROR] FE(153,python):2021-06-02-18:50:58.281.649 [../../../../../../fusion_engine/optimizer/adapter/tbe_adapter/tbe_op_store_adapter.cc:581]1430 ParallelCompileOp:"Thread[281439610335712] recompile single op[gradients/FlowNet2/fuse_conv1/Conv2D_grad/Conv2DBackpropFilter] failed"
[ERROR] FE(153,python):2021-06-02-18:50:58.281.669 [../../../../../../fusion_engine/optimizer/adapter/tbe_adapter/tbe_op_store_adapter.cc:581]1430 ParallelCompileOp:"Thread[281439610335712] recompile single op[gradients/FlowNet2/fuse_conv1_1/Conv2D_grad/Conv2DBackpropFilter] failed"
[ERROR] FE(153,python):2021-06-02-18:50:58.281.676 [../../../../../../fusion_engine/optimizer/adapter/tbe_adapter/tbe_op_store_adapter.cc:581]1430 ParallelCompileOp:"Thread[281439610335712] recompile single op[gradients/FlowNet2/fuse_conv0/Conv2D_grad/Conv2DBackpropFilter] failed"
[ERROR] FE(153,python):2021-06-02-18:50:58.281.758 [../../../../../../fusion_engine/optimizer/graph_optimizer/op_compiler/op_compiler.cc:532]1430 CompileOp:"CompileOp failed, graph[partition0_rank74_new_sub_graph341]."
[ERROR] FE(153,python):2021-06-02-18:50:58.281.829 [../../../../../../fusion_engine/optimizer/graph_optimizer/fe_graph_optimizer.cc:672]1430 OptimizeFusedCompileOpAndCalcTensorSize:"CompileOp failed, graph name = partition0_rank74_new_sub_graph341."
[ERROR] FE(153,python):2021-06-02-18:50:58.281.834 [../../../../../../fusion_engine/optimizer/graph_optimizer/fe_graph_optimizer.cc:844]1430 OptimizeFusedGraph:"Failed to do optimize fused graph after UB match for graph[partition0_rank74_new_sub_graph341]."
[ERROR] GE(153,python):2021-06-02-18:50:58.282.063 [../../../../../../graphengine/ge/graph/optimize/graph_optimize.cc:118]1430 OptimizeSubGraph: ErrorNo: -1(failed) [OptimizeSubGraph][OptimizeFusedGraph]: graph optimize failed, ret:-1
[ERROR] GE(153,python):2021-06-02-18:50:58.282.078 [../../../../../../graphengine/ge/graph/manager/graph_manager.cc:2519]1430 ProcessSubGraphWithMultiThreads: ErrorNo: -1(failed) SubGraph optimize Failed AIcoreEngine
[ERROR] GE(153,python):2021-06-02-18:50:58.282.661 [../../../../../../graphengine/ge/graph/manager/graph_manager.cc:674]763 SetSubgraph: ErrorNo: -1(failed) Multiply optimize subgraph failed
[ERROR] GE(153,python):2021-06-02-18:50:58.282.712 [../../../../../../graphengine/ge/graph/manager/graph_manager.cc:3014]763 OptimizeSubgraph: ErrorNo: -1(failed) Graph set subgraph Failed
[EVENT] GE(153,python):2021-06-02-18:50:58.282.745 [../../../../../../graphengine/ge/graph/manager/graph_manager.cc:723]763 PreRunOptimizeSubGraph:[GEPERFTRACE] The time cost of GraphManager::OptimizeSubgraph is [84278749] micro second.
[ERROR] GE(153,python):2021-06-02-18:50:58.282.752 [../../../../../../graphengine/ge/graph/manager/graph_manager.cc:723]763 PreRunOptimizeSubGraph: ErrorNo: -1(failed) Failed to process GraphManager_OptimizeSubgraph
[ERROR] GE(153,python):2021-06-02-18:50:58.282.812 [../../../../../../graphengine/ge/graph/manager/graph_manager.cc:816]763 PreRun: ErrorNo: -1(failed) Run PreRunOptimizeSubGraph failed for graph:ge_default_20210602184920.
[ERROR] GE(153,python):2021-06-02-18:50:58.305.897 [../../../../../../graphengine/ge/graph/manager/graph_manager.cc:2879]763 ReturnError: ErrorNo: -1(failed) PreRun Failed, thread exit...
[EVENT] TDT(153,python):2021-06-02-18:50:58.617.156 [../../../../../../tdt/common/src/log.cpp:148]"HostSendPool: {blockSize: 3072K, totalNum: 4, freeNum: 4}" "HostRecvPool: {blockSize: 3072K, totalNum: 1, freeNum: 1}" "DeviceRecvPool: " "HostCtrlPool: {SendPool: 4, FreePool: 4}, {RecvPool: 1, FreePool: 1}",[../../../../../../tdt/common/src/memory_pool.cpp:689:GetHostPoolStatus]747
[EVENT] TDT(153,python):2021-06-02-18:50:59.617.311 [../../../../../../tdt/common/src/log.cpp:148]"HostSendPool: {blockSize: 3072K, totalNum: 4, freeNum: 4}" "HostRecvPool: {blockSize: 3072K, totalNum: 1, freeNum: 1}" "DeviceRecvPool: " "HostCtrlPool: {SendPool: 4, FreePool: 4}, {RecvPool: 1, FreePool: 1}",[../../../../../../tdt/common/src/memory_pool.cpp:689:GetHostPoolStatus]747
[EVENT] TDT(153,python):2021-06-02-18:51:00.617.455 [../../../../../../tdt/common/src/log.cpp:148]"HostSendPool: {blockSize: 3072K, totalNum: 4, freeNum: 4}" "HostRecvPool: {blockSize: 3072K, totalNum: 1, freeNum: 1}" "DeviceRecvPool: " "HostCtrlPool: {SendPool: 4, FreePool: 4}, {RecvPool: 1, FreePool: 1}",[../../../../../../tdt/common/src/memory_pool.cpp:689:GetHostPoolStatus]747
[ERROR] FMK(153,python):2021-06-02-18:51:01.306.100 [tf_adapter/common/adp_logger.cc:32][TF_ADAPTER] [tf_adapter/kernels/geop_npu.cc:701] GeOp7_0GEOP::::DoRunAsync Failed
2021-06-02 18:51:01.306160: F tf_adapter/kernels/geop_npu.cc:702] GeOp7_0GEOP::::DoRunAsync Failed
初步分析：
类似问题issue: [Link description](https://gitee.com/ascend/modelzoo/issues/I278GS?from=project-issue)
https://gitee.com/ascend/modelzoo/issues/I278GS?from=project-issue
光流部分使用了相同的warp和correlation算子，应该是类似问题。

二、软件版本:
 pycharm toolkit:
    frequently-used
    Ascend-Powered-Engine TF1.15-python3.7-aarch64

三、日志信息
Atlas服务器:
完整日志：
[Link description](https://pan.baidu.com/s/1zFDDsczUfruv1XgGyrMhjQ)
https://pan.baidu.com/s/1zFDDsczUfruv1XgGyrMhjQ
提取码: idhw

ModelArts: 
提供 jobid deviceid
({
"pod_name": "job9fb41a83-job-ma-flownet2-npu-2021-0",
"server_id": "192.168.0.5",
"devices": [
{
"device_id": "2",
"device_ip": "192.3.163.238"
})

Ascend/modelzoo
暂停

内容风险标识

评论 (13)

Ascend/modelzoo暂停 .gitee-modal { width: 500px !important; }

内容风险标识