编译安装后无法通过测试

一、问题现象（附报错日志上下文）：
在安装 pytorch 后，运行 test/test_network_ops/test_div.py 报错，NPU error code 500002

二、软件版本:
-- CANN 版本 (e.g., CANN 3.0.x，5.x.x):  6.0.RC1.alpha003
--Tensorflow/Pytorch/MindSpore 版本: pytorch 1.11.0
--Python 版本 (e.g., Python 3.7.5): Python 3.9
-- MindStudio版本 (e.g., MindStudio 2.0.0 (beta3)): -
--操作系统版本 (e.g., Ubuntu 18.04): CentOS 7.9

三、测试步骤：
```
$ conda create -n pytorch python=3.9
$ conda install pytorch=1.11.0 torchvision torchaudio cpuonly -c pytorch
$ cd pytorch/
$ bash ci/build.sh --python=3.9
$ pip install --upgrade dist/torch_npu-1.11rc1-cp39-cp39-linux_x86_64.whl
$ source env.sh
$ python3 ./test_div.py
```

四、日志信息:
```
[W OperatorEntry.cpp:133] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::is_pinned(Tensor self, Device? device=None) -> (bool)
    registered at /opt/conda/conda-bld/pytorch_1646755883846/work/build/aten/src/ATen/RegisterSchema.cpp:6
  dispatch key: BackendSelect
  previous kernel: registered at /opt/conda/conda-bld/pytorch_1646755883846/work/build/aten/src/ATen/RegisterBackendSelect.cpp:606
       new kernel: registered at /home/zzz/ascend-toolkits/pytorch_1_11/pytorch/torch_npu/csrc/aten/PinMemory.cpp:43 (function registerKernel)
[W OperatorEntry.cpp:133] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::_pin_memory(Tensor self, Device? device=None) -> (Tensor)
    registered at /opt/conda/conda-bld/pytorch_1646755883846/work/build/aten/src/ATen/RegisterSchema.cpp:6
  dispatch key: BackendSelect
  previous kernel: registered at /opt/conda/conda-bld/pytorch_1646755883846/work/build/aten/src/ATen/RegisterBackendSelect.cpp:606
       new kernel: registered at /home/zzz/ascend-toolkits/pytorch_1_11/pytorch/torch_npu/csrc/aten/PinMemory.cpp:43 (function registerKernel)
[W OperatorEntry.cpp:133] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::_has_compatible_shallow_copy_type(Tensor self, Tensor from) -> (bool)
    registered at /opt/conda/conda-bld/pytorch_1646755883846/work/build/aten/src/ATen/RegisterSchema.cpp:6
  dispatch key: (catch all)
  previous kernel: registered at /opt/conda/conda-bld/pytorch_1646755883846/work/build/aten/src/ATen/RegisterCompositeImplicitAutograd.cpp:10303
       new kernel: registered at /home/zzz/ascend-toolkits/pytorch_1_11/pytorch/torch_npu/csrc/aten/ops/HasCompatibleShallowCopyType.cpp:37 (function registerKernel)
.........EEE.
======================================================================
ERROR: test_div_shape_format_fp16 (__main__.TestDiv)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/zzz/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch_npu/testing/decorator.py", line 91, in wrapper
    func(*args, **kw)
  File "/home/zzz/ascend-toolkits/pytorch_1_11/pytorch/test/test_network_ops/./test_div.py", line 67, in test_div_shape_format_fp16
    cpu_input1, npu_input1 = create_common_tensor(item, 1, 100)
  File "/home/zzz/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch_npu/testing/common_utils.py", line 107, in create_common_tensor
    npu_input = torch_npu.npu_format_cast(npu_input, npu_format)
RuntimeError: Run:/home/zzz/ascend-toolkits/pytorch_1_11/pytorch/torch_npu/csrc/framework/OpParamMaker.cpp:136 NPU error,NPU error code is:500002
E49999: Inner Error!
E49999  Compile op[trans_TransData_1,] failed, oppath[/home/zzz/Ascend/ascend-toolkit/6.0.RC1.alpha003/opp/op_impl/built-in/ai_core/tbe/impl/dynamic/trans_data.py], optype[TransData], taskID[64]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:864]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140448425297664] recompile single op[trans_TransData_1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:831]
        [SubGraphOpt][Compile][ParalCompOp] Thread[140448425297664] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:871]
        [SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:890]
        [GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1460]
        Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank2_new_sub_graph2[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:130]
        subgraph 1 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:714]
        build graph failed, graph id:31, ret:-1[FUNC:BuildModel][FILE:ge_generator.cc][LINE:1496]
        [Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]
        [Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:146]
        build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:146]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140448291079936] recompile single op[trans_TransData_3] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:831]
        [SubGraphOpt][Compile][ParalCompOp] Thread[140448291079936] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:871]
        build graph failed, graph id:32, ret:-1[FUNC:BuildModel][FILE:ge_generator.cc][LINE:1496]

======================================================================
ERROR: test_div_shape_format_fp32 (__main__.TestDiv)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/zzz/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch_npu/testing/decorator.py", line 91, in wrapper
    func(*args, **kw)
  File "/home/zzz/ascend-toolkits/pytorch_1_11/pytorch/test/test_network_ops/./test_div.py", line 82, in test_div_shape_format_fp32
    cpu_input1, npu_input1 = create_common_tensor(item, 1, 100)
  File "/home/zzz/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch_npu/testing/common_utils.py", line 107, in create_common_tensor
    npu_input = torch_npu.npu_format_cast(npu_input, npu_format)
RuntimeError: Run:/home/zzz/ascend-toolkits/pytorch_1_11/pytorch/torch_npu/csrc/framework/OpParamMaker.cpp:136 NPU error,NPU error code is:500002
E29999: Inner Error!
E29999  [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140448400119552] recompile single op[trans_TransData_5] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:831]
        [SubGraphOpt][Compile][ParalCompOp] Thread[140448400119552] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:871]
        [SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:890]
        [GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1460]
        Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank2_new_sub_graph2[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:130]
        subgraph 1 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:714]
        build graph failed, graph id:33, ret:-1[FUNC:BuildModel][FILE:ge_generator.cc][LINE:1496]
        [Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]
        [Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:146]
        build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:146]
        Compile op[trans_TransData_7,] failed, oppath[/home/zzz/Ascend/ascend-toolkit/6.0.RC1.alpha003/opp/op_impl/built-in/ai_core/tbe/impl/dynamic/trans_data.py], optype[TransData], taskID[81]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:864]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140448769234688] recompile single op[trans_TransData_7] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:831]
        [SubGraphOpt][Compile][ParalCompOp] Thread[140448769234688] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:871]
        build graph failed, graph id:38, ret:-1[FUNC:BuildModel][FILE:ge_generator.cc][LINE:1496]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140449037604608] recompile single op[trans_TransData_9] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:831]
        [SubGraphOpt][Compile][ParalCompOp] Thread[140449037604608] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:871]
        build graph failed, graph id:39, ret:-1[FUNC:BuildModel][FILE:ge_generator.cc][LINE:1496]

======================================================================
ERROR: test_div_shape_format_fp32_1 (__main__.TestDiv)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/zzz/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch_npu/testing/decorator.py", line 91, in wrapper
    func(*args, **kw)
  File "/home/zzz/ascend-toolkits/pytorch_1_11/pytorch/test/test_network_ops/./test_div.py", line 123, in test_div_shape_format_fp32_1
    cpu_input1, npu_input1 = create_common_tensor(item, 1, 100)
  File "/home/zzz/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch_npu/testing/common_utils.py", line 107, in create_common_tensor
    npu_input = torch_npu.npu_format_cast(npu_input, npu_format)
RuntimeError: Run:/home/zzz/ascend-toolkits/pytorch_1_11/pytorch/torch_npu/csrc/framework/OpParamMaker.cpp:136 NPU error,NPU error code is:500002
E29999: Inner Error!
E29999  [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140448760841984] recompile single op[trans_TransData_11] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:831]
        [SubGraphOpt][Compile][ParalCompOp] Thread[140448760841984] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:871]
        [SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:890]
        [GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1460]
        Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank2_new_sub_graph2[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:130]
        subgraph 1 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:714]
        build graph failed, graph id:40, ret:-1[FUNC:BuildModel][FILE:ge_generator.cc][LINE:1496]
        [Build][SingleOpModel]call ge interface generator.BuildSingleOpModel failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:162]
        [Build][Op]Fail to build op model[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:146]
        build op model failed, result = 500002[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:146]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140448760841984] recompile single op[trans_TransData_13] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:831]
        build graph failed, graph id:44, ret:-1[FUNC:BuildModel][FILE:ge_generator.cc][LINE:1496]
        [SubGraphOpt][Compile][ProcFailedCompTask] Thread[140448786020096] recompile single op[trans_TransData_15] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:831]
        [SubGraphOpt][Compile][ParalCompOp] Thread[140448786020096] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:871]
        build graph failed, graph id:45, ret:-1[FUNC:BuildModel][FILE:ge_generator.cc][LINE:1496]

----------------------------------------------------------------------
Ran 13 tests in 27.964s

FAILED (errors=3)
```

Ascend/pytorch

内容风险标识

评论 (4)

Ascend/pytorch .gitee-modal { width: 500px !important; }

内容风险标识