Failed to compile Op [Conv3DBackpropInput]

内部员工，可联系工号：821855

一、问题现象（附报错日志上下文）：
使用torch_npu进行训练时，计算loss时对conv3d算子出现相关问题

```
Traceback (most recent call last):
  File "/data/v14_less_sharpen_v2/././basicsr/train.py", line 250, in <module>
    train_pipeline(root_path)
  File "/data/v14_less_sharpen_v2/././basicsr/train.py", line 203, in train_pipeline
    model.optimize_parameters(current_iter)
  File "/data/v14_less_sharpen_v2/./basicsr/models/femasr_model.py", line 204, in optimize_parameters
    l_g_total.mean().backward()
  File "/home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch/_tensor.py", line 492, in backward
    torch.autograd.backward(
  File "/home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: InnerRun:/usr1/02/workspace/j_yxiCvvHE/pytorch/torch_npu/csrc/framework/OpParamMaker.cpp:203 NPU error, error code is 500002
[Error]: A GE error occurs in the system.
        Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue
E19014: Value [input __input0 shape] for Op [online_Node_Output] is invalid. Reason: contains negative or zero dimension.
        TraceBack (most recent call last):
        online_Node_Output(NetOutput) Verify failed.[FUNC:Verify][FILE:node_utils_ex.cc][LINE:127]
        Verifying online_Node_Output failed.[FUNC:InferShapeAndType][FILE:infershape_pass.cc][LINE:137]
        Sessin_id 0 does not exist, graph_id 27[FUNC:GetJsonObject][FILE:analyzer.cc][LINE:162]
        Param:graph_info is nullptr, check invalid[FUNC:DoAnalyze][FILE:analyzer.cc][LINE:260]
        Param:graph_info is nullptr, check invalid[FUNC:SaveAnalyzerDataToFile][FILE:analyzer.cc][LINE:217]
        Call InferShapeAndType for node:online_Node_Output(NetOutput) failed[FUNC:Infer][FILE:infershape_pass.cc][LINE:119]
        process pass InferShapePass on node:online_Node_Output failed, ret:4294967295[FUNC:RunPassesOnNode][FILE:base_pass.cc][LINE:571]
        Failed to compile Op [Conv3DBackpropInput]. (oppath: [Compile /usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv3d_backprop_input.py failed with errormsg/stack: File "/home/ma-user/anaconda3/envs/PyTorch-2.1.0/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'impl.conv3d_backprop_input'
], optype: [Conv3DBackpropInput])
        Compile op[Conv3DBackpropInput] failed, oppath[/usr/local/Ascend/ascend-toolkit/7.0.RC1/opp/built-in/op_impl/ai_core/tbe/impl/conv3d_backprop_input.py], optype[Conv3DBackpropInput], taskID[231]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_manager.cc][LINE:763]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281470193529312] recompile single op[Conv3DBackpropInput] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:973]
        [SubGraphOpt][Compile][ParalCompOp] Thread[281470193529312] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:1019]
        [SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:1129]
        [GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1291]
        Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph2[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:131]
        subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:991]
        build graph failed, graph id:27, ret:-1[FUNC:BuildModelWithGraphId][FILE:ge_generator.cc][LINE:1615]
        [Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        [Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
        build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
```
![image-20240525162617026](http://image.huawei.com/tiny-lts/v1/images/cbbf03dfa19d793731f646fddd910513_1901x787.png)

二、软件版本:
host驱动：
```
cat /usr/local/Ascend/driver/version.info 
Version=23.0.3
ascendhal_version=7.35.19
aicpu_version=1.0
tdt_version=1.0
log_version=1.0
prof_version=2.0
dvppkernels_version=1.1
tsfw_version=1.0
Innerversion=V100R001C15SPC005B220
compatible_version=[V100R001C30],[V100R001C13],[V100R001C15]
compatible_version_fw=[7.0.0,7.1.99]
package_version=23.0.3
```
-- CANN 版本 : 7.0.0.5.242:7.0.RC1
-- Pytorch版本: 2.1.0
-- torch-npu: 2.1.0rc1.post20231013
--Python 版本: 3.9.10
-- MindStudio版本:不涉及
--操作系统版本: eulerosv2r10（Linux vhulk2211.3.0.h1543.eulerosv2r10.aarch6）

**三。疑问：** 
1. 是否可能是环境问题，比如要额外安装Ascend910B-opp_kernel-XXXXX.run,但是不知道如何找到对应7.0.RC1的opp_kernel
2. 是否是算子conv3d就不支持，是否有开发计划

Ascend/pytorch
暂停

内容风险标识

评论 (1)

Ascend/pytorch暂停 .gitee-modal { width: 500px !important; }

内容风险标识