pytorch训练代码自动迁移报错

一、问题现象（附报错日志上下文）：

我有一段自己写的nlp的基线模型训练代码，拿到华为服务器上用cpu跑是没问题的，然后放到npu上跑就出现> RuntimeError: Initialize:/usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/core/npu/sys_ctrl/npu_sys_ctrl.cpp:177 NPU error, error code is 4294967295
E40020: Failed to import Python module [ModuleNotFoundError: No module named 'decorator'.].

我采用自动前移的方法 在头文件里使用import torch_npu; import import transfer_to_npu

二、软件版本:
-- CANN 版本 (e.g., CANN 3.0.x，5.x.x):  
CANN 6.3.RC1
--Tensorflow/Pytorch/MindSpore 版本:
Pytorch 1.11.0
--Python 版本 (e.g., Python 3.7.11):
Python 3.7.11
-- MindStudio版本 (e.g., MindStudio 2.0.0 (beta3)):
MindX 5.0.RC1
--操作系统版本 (e.g., Ubuntu 20.04):
Ubuntu 20.04

三、测试步骤：
在ubuntu的terminal命令行里输入 python train_expert_net_via_embedding.py

四、日志信息:
报错日志：
(huawei) HwHiAiUser@administrator:~/projects/AI_matching/tools$ python train_expert_net_via_embedding.py
[W OperatorEntry.cpp:133] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::is_pinned(Tensor self, Device? device=None) -> (bool)
    registered at aten/src/ATen/RegisterSchema.cpp:6
  dispatch key: BackendSelect
  previous kernel: registered at aten/src/ATen/RegisterBackendSelect.cpp:606
       new kernel: registered at /usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/aten/PinMemory.cpp:42 (function registerKernel)
[W OperatorEntry.cpp:133] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::_pin_memory(Tensor self, Device? device=None) -> (Tensor)
    registered at aten/src/ATen/RegisterSchema.cpp:6
  dispatch key: BackendSelect
  previous kernel: registered at aten/src/ATen/RegisterBackendSelect.cpp:606
       new kernel: registered at /usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/aten/PinMemory.cpp:42 (function registerKernel)
[W OperatorEntry.cpp:133] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::argmax(Tensor self, int? dim=None, bool keepdim=False) -> (Tensor)
    registered at aten/src/ATen/RegisterSchema.cpp:6
  dispatch key: XLA
  previous kernel: registered at aten/src/ATen/RegisterCPU.cpp:21063
       new kernel: registered at /usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/aten/RegisterNPU.cpp:15958 (function registerKernel)
[W OperatorEntry.cpp:133] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::argmin(Tensor self, int? dim=None, bool keepdim=False) -> (Tensor)
    registered at aten/src/ATen/RegisterSchema.cpp:6
  dispatch key: XLA
  previous kernel: registered at aten/src/ATen/RegisterCPU.cpp:21063
       new kernel: registered at /usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/aten/RegisterNPU.cpp:15958 (function registerKernel)
[W OperatorEntry.cpp:133] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::_embedding_bag_dense_backward(Tensor grad, Tensor indices, Tensor offset2bag, Tensor bag_size, Tensor maximum_indices, int num_weights, bool scale_grad_by_freq, int mode, Tensor? per_sample_weights, int padding_idx=-1) -> (Tensor)
    registered at aten/src/ATen/RegisterSchema.cpp:6
  dispatch key: XLA
  previous kernel: registered at aten/src/ATen/RegisterCPU.cpp:21063
       new kernel: registered at /usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/aten/RegisterNPU.cpp:15958 (function registerKernel)
[W OperatorEntry.cpp:133] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::empty.memory_format(int[] size, *, int? dtype=None, int? layout=None, Device? device=None, bool? pin_memory=None, int? memory_format=None) -> (Tensor)
    registered at aten/src/ATen/RegisterSchema.cpp:6
  dispatch key: CPU
  previous kernel: registered at aten/src/ATen/RegisterBackendSelect.cpp:606
       new kernel: registered at /usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/aten/common/EmptyTensor.cpp:115 (function registerKernel)
[W OperatorEntry.cpp:133] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::empty_strided(int[] size, int[] stride, *, int? dtype=None, int? layout=None, Device? device=None, bool? pin_memory=None) -> (Tensor)
    registered at aten/src/ATen/RegisterSchema.cpp:6
  dispatch key: CPU
  previous kernel: registered at aten/src/ATen/RegisterBackendSelect.cpp:606
       new kernel: registered at /usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/aten/common/EmptyTensor.cpp:115 (function registerKernel)
[W OperatorEntry.cpp:133] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::_to_copy(Tensor self, *, int? dtype=None, int? layout=None, Device? device=None, bool? pin_memory=None, bool non_blocking=False, int? memory_format=None) -> (Tensor)
    registered at aten/src/ATen/RegisterSchema.cpp:6
  dispatch key: BackendSelect
  previous kernel: registered at aten/src/ATen/RegisterBackendSelect.cpp:606
       new kernel: registered at /usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/aten/common/ToKernelNpu.cpp:215 (function registerKernel)
[W OperatorEntry.cpp:133] Warning: Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: aten::_has_compatible_shallow_copy_type(Tensor self, Tensor from) -> (bool)
    registered at aten/src/ATen/RegisterSchema.cpp:6
  dispatch key: (catch all)
  previous kernel: registered at aten/src/ATen/RegisterCompositeImplicitAutograd.cpp:10303
       new kernel: registered at /usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/aten/ops/HasCompatibleShallowCopyType.cpp:37 (function registerKernel)
/usr/local/Ascend/ascend-toolkit/latest/tools/ms_fmk_transplt/torch_npu_bridge/transfer_to_npu.py:97: ImportWarning: 
    *************************************************************************************************************
    The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
    The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
    The backend in torch.distributed.init_process_group set to hccl now..
    The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
    The device parameters have been replaced with npu in the function below:
    torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
    *************************************************************************************************************
    
  warnings.warn(msg, ImportWarning)
/usr/local/Ascend/ascend-toolkit/6.3.RC1/python/site-packages/tbe/tvm/contrib/ccec.py:705: DeprecationWarning: invalid escape sequence \L
  if not dirpath.find("AppData\Local\Temp"):
Traceback (most recent call last):
  File "train_expert_net_via_embedding.py", line 589, in <module>
    inputs,label1,label2 = inputs.cuda(),label1.cuda(),label2.cuda()
  File "/home/HwHiAiUser/anaconda3/envs/huawei/lib/python3.7/site-packages/torch_npu/utils/device_guard.py", line 38, in wrapper
    return func(*args, **kwargs)
  File "/home/HwHiAiUser/anaconda3/envs/huawei/lib/python3.7/site-packages/torch_npu/utils/tensor_methods.py", line 66, in _npu
    return torch_npu._C.npu(self, *args, **kwargs)
RuntimeError: Initialize:/usr1/workspace/FPTA_Daily_open_pytorchv1.11.0-5.0.rc1/CODE/torch_npu/csrc/core/npu/sys_ctrl/npu_sys_ctrl.cpp:177 NPU error, error code is 4294967295
E40020: Failed to import Python module [ModuleNotFoundError: No module named 'decorator'.].
        Solution: Check that all required components are properly installed and the specified Python path matches the Python installation directory. (If the path does not match the directory, run set_env.sh in the installation package.)
        TraceBack (most recent call last):
        [GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1533]
        [SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:67]
        [SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:115]
        [FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:357]
        PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:100]
        OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:188]
        GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:121]

THPModule_npu_shutdown success.
请根据自己的运行环境参考以下方式搜集日志信息，如果涉及到算子开发相关的问题，建议也提供UT/ST测试和单算子集成测试相关的日志。

日志提供方式:
将日志打包后作为附件上传。若日志大小超出附件限制，则可上传至外部网盘后提供链接。

获取方法请参考wiki：
https://gitee.com/ascend/modelzoo/wikis/%E5%A6%82%E4%BD%95%E8%8E%B7%E5%8F%96%E6%97%A5%E5%BF%97%E5%92%8C%E8%AE%A1%E7%AE%97%E5%9B%BE?sort_id=4097825

Ascend/modelzoo
暂停

内容风险标识

评论 (3)

Ascend/modelzoo暂停 .gitee-modal { width: 500px !important; }

内容风险标识