TorchNpu initialize failed

一、问题现象（附报错日志上下文）：
(Pdb) c
Traceback (most recent call last):
  File "/usr/local/python3.10/lib/python3.10/pdb.py", line 1723, in main
    pdb._runscript(mainpyfile)
  File "/usr/local/python3.10/lib/python3.10/pdb.py", line 1583, in _runscript
    self.run(statement)
  File "/usr/local/python3.10/lib/python3.10/bdb.py", line 598, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/mnj/models/code/trl/trl/GRPOTrainer_test.py", line 11, in <module>
    training_args = GRPOConfig(output_dir="Qwen2.5-0.5B-Instruct-GRPO", logging_steps=10, use_vllm=True)
  File "<string>", line 167, in __init__
  File "/home/mnj/models/code/trl/trl/trl/trainer/grpo_config.py", line 383, in __post_init__
    super().__post_init__()
  File "/usr/local/python3.10/lib/python3.10/site-packages/transformers/training_args.py", line 1791, in __post_init__
    self.device
  File "/usr/local/python3.10/lib/python3.10/site-packages/transformers/training_args.py", line 2313, in device
    return self._setup_devices
  File "/usr/local/python3.10/lib/python3.10/site-packages/transformers/utils/generic.py", line 62, in __get__
    cached = self.fget(obj)
  File "/usr/local/python3.10/lib/python3.10/site-packages/transformers/training_args.py", line 2289, in _setup_devices
    torch.npu.set_device(device)
  File "/usr/local/python3.10/lib/python3.10/site-packages/torch_npu/npu/utils.py", line 83, in set_device
    torch_npu._C._npu_setDevice(device_id)
RuntimeError: SetPrecisionMode:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:155 NPU function error: at_npu::native::AclSetCompileopt(aclCompileOpt::ACL_PRECISION_MODE, precision_mode), error code is 500001
[ERROR] 2025-04-07-04:00:56 (PID:43964, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The internal ACL of the system is incorrect.
        Rectify the fault based on the error information in the ascend log.
E90000: [PID: 43964] 2025-04-07-04:00:56.778.425 Compile operator failed, cause: module '__main__' has no attribute '__spec__'   File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/interface.py", line 33, in cann_kb_init
    return RouteServer.initialize(**locals())
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 54, in wrapper
    return func(cls, *args, **kwargs)
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 169, in initialize
    main_mod, main_path = config_main_info()
  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/utils/common.py", line 37, in config_main_info
    main_module_name = getattr(main_module.__spec__, "name", None)

TraceBack (most recent call last):
        AOE Failed to call InitCannKB[FUNC:Initialize][FILE:python_adapter_manager.cc][LINE:47]
        Failed to initialize TeConfigInfo.
        [GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeTeFusion][FILE:tbe_op_store_adapter.cc][LINE:1816]
        [GraphOpt][InitializeInner][InitTeFusion]: Failed to initialize TeFusion.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1783]
        [SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79]
        [SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120]
        [FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:115]
        PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:82]
        OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:249]
        GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:177]
        GEInitialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:337]
        [Initialize][Ge]GEInitialize failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        [Init][Compiler]Init compiler failed[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
        [Set][Options]OpCompileProcessor init failed![FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]

Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /usr/local/python3.10/lib/python3.10/site-packages/torch_npu/npu/utils.py(83)set_device()
-> torch_npu._C._npu_setDevice(device_id)

二、软件版本:
-- CANN 版本 (CANN 8.0.0):  
--Pytorch版本: 2.5.1
--Python 版本 (Python 3.10.15):
--操作系统版本 (e.g., Ubuntu 22.04):

三、测试步骤：
启动

四、日志信息:
run/plog/下的日志（Debug下的plog见评论）：
  1 [EVENT] PROFILING(43964,python):2025-04-07-04:00:39.872.070 [msprof_callback_impl.cpp:336] >>> (tid:43964) Started to register profiling ctrl callback.
  2 [EVENT] PROFILING(43964,python):2025-04-07-04:00:39.872.385 [msprof_callback_impl.cpp:343] >>> (tid:43964) Started to register profiling hash id callback.
  3 [INFO] PROFILING(43964,python):2025-04-07-04:00:39.872.394 [prof_atls_plugin.cpp:117] (tid:43964) RegisterProfileCallback, callback type is 7
  4 [EVENT] PROFILING(43964,python):2025-04-07-04:00:39.872.398 [msprof_callback_impl.cpp:350] >>> (tid:43964) Started to register profiling enable host freq callback.
  5 [INFO] PROFILING(43964,python):2025-04-07-04:00:39.872.401 [prof_atls_plugin.cpp:117] (tid:43964) RegisterProfileCallback, callback type is 8
  6 [INFO] ATRACE(43964,python):2025-04-07-04:00:39.873.280 [trace_attr.c:188](tid:43964) attr init success, timeout=0ms.
  7 [INFO] ATRACE(43964,python):2025-04-07-04:00:39.873.302 [trace_recorder.c:121](tid:43964) use root path: /root/ascend
  8 [INFO] RUNTIME(43964,python):2025-04-07-04:00:39.879.144 [task_fail_callback_manager.cc:52] 43964 TaskFailCallBackManager: Constructor.
  9 [INFO] HCCL(43964,python):2025-04-07-04:00:39.965.933 [adapter_rts.cc:2646][43964][adapter_rts.cc][CallBackInitRts] g_deviceType [6] g_deviceLogicId [-1] g_devicePhyId [-1]
 10 [INFO] RUNTIME(43964,python):2025-04-07-04:00:40.111.025 [runtime.cc:5549] 43964 GetVisibleDevices: real deviceCnt:2 userDeviceCnt:2 isSetVisibleDev:1 ASCEND_RT_VISIBLE_DEVICES:[0,1] available visible devices:[0,1]
 11 [INFO] PROFILING(43964,python):2025-04-07-04:00:40.113.695 [prof_atls_plugin.cpp:210] (tid:43964) Module[7] register callback of ctrl handle.
 12 [INFO] IDEDD(43964,python):2025-04-07-04:00:52.875.840 [adx_dump_record.cpp:623][tid:44100] start dump thread, remote dump record temp path : /home/mnj/models/code/trl/trl/.
 13 [INFO] IDEDD(43964,python):2025-04-07-04:00:52.875.928 [adx_server_manager.cpp:203][tid:44101] Run Server(0) Process
 14 [INFO] IDEDD(43964,python):2025-04-07-04:00:52.875.982 [hdc_comm_opt.cpp:59][tid:44101] open device[0] server[13]
 15 [INFO] IDEDD(43964,python):2025-04-07-04:00:52.876.136 [hdc_api.cpp:119][tid:44101] logDevId 0 create HDC server successfully
 16 [INFO] IDEDD(43964,python):2025-04-07-04:00:52.876.167 [hdc_comm_opt.cpp:59][tid:44101] open device[1] server[13]
 17 [INFO] IDEDD(43964,python):2025-04-07-04:00:52.876.280 [hdc_api.cpp:119][tid:44101] logDevId 1 create HDC server successfully
 18 [INFO] PROFILING(43964,python):2025-04-07-04:00:52.878.414 [prof_atls_plugin.cpp:210] (tid:43964) Module[48] register callback of ctrl handle.
 19 [INFO] PROFILING(43964,python):2025-04-07-04:00:52.878.432 [prof_atls_plugin.cpp:210] (tid:43964) Module[45] register callback of ctrl handle.
 20 [INFO] GE(43964,python):2025-04-07-04:00:53.057.789 [op_tiling_manager.cc:109]43964 ~FuncPerfScope:[GEPERFTRACE] The time cost of OpTilingManager::LoadSo is [179345] micro second.
 21 [INFO] PROFILING(43964,python):2025-04-07-04:00:53.072.429 [prof_atls_plugin.cpp:210] (tid:43964) Module[6] register callback of ctrl handle.
 22 [EVENT] PROFILING(43964,python):2025-04-07-04:00:53.578.682 [msprof_callback_impl.cpp:89] >>> (tid:43964) MsprofCtrlCallback called, type: 255
 23 [EVENT] PROFILING(43964,python):2025-04-07-04:00:53.578.986 [ai_drv_dev_api.cpp:333] >>> (tid:43964) Succeeded to DrvGetApiVersion version: 0x72313
 24 [INFO] TDT(43964,python):2025-04-07-04:00:53.579.955 [client_manager.cpp:433][IsHostEnvironment][tid:43964] [TsdClient] logicDeviceId is [0], hostaicpunum[0]
 25 [INFO] TDT(43964,python):2025-04-07-04:00:53.579.964 [client_manager.cpp:462][GetClientRunMode][tid:43964] runningMode:0
 26 [INFO] TDT(43964,python):2025-04-07-04:00:53.579.968 [client_manager.cpp:126][GetInstance][tid:43964] [ClientManager] Current mode:1
 27 [INFO] TDT(43964,python):2025-04-07-04:00:53.579.988 [process_mode_manager.cpp:415][GetLogLevel][tid:43964] [TsdClient] get ASCEND_GLOBAL_LOG_LEVEL [] ASCEND_GLOBAL_EVENT_ENABLE [] ASCEND_MODULE_LOG_LEVEL []
 28 [INFO] TDT(43964,python):2025-04-07-04:00:53.580.070 [process_mode_manager.cpp:134][OpenProcess][tid:43964] [ProcessModeManager] enter into open process deviceId[0] rankSize[0]
 29 [INFO] TDT(43964,python):2025-04-07-04:00:53.580.617 [process_mode_manager.cpp:545][InitTsdClient][tid:43964] [TsdClient] deviceId[0] begin to init hdc client
 30 [INFO] TDT(43964,python):2025-04-07-04:00:53.580.714 [version_verify.cpp:36][SetVersionInfo][tid:43964] VersionVerify: send client version to server
 31 [INFO] TDT(43964,python):2025-04-07-04:00:53.580.747 [version_verify.cpp:52][SetVersionInfo][tid:43964] send feature_info:{msg_type:35, features:{check before send aicpu package}}
 32 [INFO] TDT(43964,python):2025-04-07-04:00:53.580.752 [version_verify.cpp:52][SetVersionInfo][tid:43964] send feature_info:{msg_type:37, features:{check before send open qs message}}
 33 [INFO] TDT(43964,python):2025-04-07-04:00:53.580.765 [version_verify.cpp:52][SetVersionInfo][tid:43964] send feature_info:{msg_type:64, features:{get check code retry}}
 34 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.053 [version_verify.cpp:68][PeerVersionCheck][tid:43964] VersionVerify: Check client version info, server[1230], client[1230]
 35 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.062 [version_verify.cpp:89][ParseVersionInfo][tid:43964] VersionVerify: pass client version info success
 36 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.065 [hdc_client.cpp:275][CheckHdcConnection][tid:43964] Service[2] create hdc success
 37 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.075 [version_verify.cpp:122][SpecialFeatureCheck][tid:43964] VersionVerify: new type[35], supported
 38 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.102 [process_mode_manager.cpp:922][GetDeviceCheckCodeOnce][tid:43964] [TsdClient][deviceId=0] [sessionId=1] wait package info respond
 39 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.193 [version_verify.cpp:128][SpecialFeatureCheck][tid:43964] VersionVerify: msgType[64] not supported, please check and update you software
 40 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.235 [process_mode_manager.cpp:2161][SendExtendPackage][tid:43964] device not support extend pkg skip send
 41 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.243 [process_mode_manager.cpp:545][InitTsdClient][tid:43964] [TsdClient] deviceId[0] begin to init hdc client
 42 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.316 [version_verify.cpp:36][SetVersionInfo][tid:43964] VersionVerify: send client version to server
 43 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.321 [version_verify.cpp:52][SetVersionInfo][tid:43964] send feature_info:{msg_type:35, features:{check before send aicpu package}}
 44 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.325 [version_verify.cpp:52][SetVersionInfo][tid:43964] send feature_info:{msg_type:37, features:{check before send open qs message}}
 45 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.328 [version_verify.cpp:52][SetVersionInfo][tid:43964] send feature_info:{msg_type:64, features:{get check code retry}}
 46 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.591 [version_verify.cpp:68][PeerVersionCheck][tid:43964] VersionVerify: Check client version info, server[1230], client[1230]
 47 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.596 [version_verify.cpp:89][ParseVersionInfo][tid:43964] VersionVerify: pass client version info success
 48 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.599 [hdc_client.cpp:275][CheckHdcConnection][tid:43964] Service[2] create hdc success
 49 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.605 [process_mode_manager.cpp:600][ConstructOpenMsg][tid:43964] [TsdClient] tsd get process sign successfully, procpid[2542256] signSize[48]
 50 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.611 [version_verify.cpp:114][SpecialFeatureCheck][tid:43964] VersionVerify: previous type[6], supported
 51 [INFO] TDT(43964,python):2025-04-07-04:00:53.581.627 [process_mode_manager.cpp:151][OpenProcess][tid:43964] [ProcessModeManager] deviceId[0] sessionId[1] rankSize[0], wait sub process start respond
 52 [INFO] TDT(43964,python):2025-04-07-04:00:53.737.044 [stub_process_mode_nowin.cpp:60][ProcessQueueForMdc][tid:43964] [TsdClient] it is unnecessary of current mode[0] to grant queue auth to aicpusd
 53 [INFO] TDT(43964,python):2025-04-07-04:00:53.737.055 [stub_process_mode_nowin.cpp:98][OpenInHost][tid:43964] enter into OpenInHost deviceid[0]
 54 [INFO] TDT(43964,python):2025-04-07-04:00:53.737.074 [client_manager.cpp:433][IsHostEnvironment][tid:43964] [TsdClient] logicDeviceId is [0], hostaicpunum[0]
 55 [INFO] TDT(43964,python):2025-04-07-04:00:53.737.078 [stub_process_mode_nowin.cpp:102][OpenInHost][tid:43964] host cpu not support
 56 [INFO] TDT(43964,python):2025-04-07-04:00:53.737.081 [process_mode_manager.cpp:185][OpenProcess][tid:43964] [TsdClient][deviceId=0] [sessionId=1] start hccp and computer process success
 57 [INFO] RUNTIME(43964,python):2025-04-07-04:00:53.741.420 [raw_device.cc:237] 43964 Init: isAddrFlat:0
 58 [INFO] TDT(43964,python):2025-04-07-04:00:53.742.276 [tsd_client.cpp:172][TsdCapabilityGet][tid:43964] TsdCapabilityGet Begin.
 59 [INFO] TDT(43964,python):2025-04-07-04:00:53.742.285 [process_mode_manager.cpp:1351][CapabilityGet][tid:43964] [ProcessModeManager] enter into CapabilityGet process deviceId[0] type[0]
 60 [INFO] TDT(43964,python):2025-04-07-04:00:53.742.295 [version_verify.cpp:114][SpecialFeatureCheck][tid:43964] VersionVerify: previous type[38], supported
 61 [INFO] TDT(43964,python):2025-04-07-04:00:53.742.322 [process_mode_manager.cpp:1374][CapabilityGet][tid:43964] [TsdClient][deviceId=0] [sessionId=1] [type=0]send capability success
 62 [INFO] TDT(43964,python):2025-04-07-04:00:53.742.466 [process_mode_manager.cpp:829][PidQosMsgProc][tid:43964] [TsdClient] PidQosMsgProc recvMsg realDeviceId[0] msgType[39] localDevId[0] rspCode[0],pidqos[1]
 63 [INFO] TDT(43964,python):2025-04-07-04:00:53.742.471 [process_mode_manager.cpp:1379][CapabilityGet][tid:43964] [TsdClient][logicDeviceId_=0][type=0][pidQos=1][ret=0]recv capability respond finish
 64 [INFO] RUNTIME(43964,python):2025-04-07-04:00:53.742.479 [engine.cc:76] 43964 Engine: Constructor.
 65 [INFO] RUNTIME(43964,python):2025-04-07-04:00:53.742.482 [stars_engine.cc:41] 43964 StarsEngine: Constructor.
 66 [INFO] RUNTIME(43964,python):2025-04-07-04:00:53.755.271 [npu_driver.cc:5784] 44103 GetDeviceStatus: GetDeviceStatus status=1.
 67 [INFO] RUNTIME(43964,python):2025-04-07-04:00:53.756.916 [device_error_proc.cc:446] 43964 GetTschCapability: Tsch not support capability feature, use old solution.
 68 [INFO] ATRACE(43964,python):2025-04-07-04:00:53.758.190 [tracer_mgr_operate.c:64](tid:43964) create object RUNTIME_ATRACE_DEV0_TS0 success, exitSave(false).
 69 [INFO] TDT(43964,python):2025-04-07-04:00:53.758.238 [client_manager.cpp:195][SetProfilingCallback][tid:43964] [TsdClient] set profiling callback success
 70 [INFO] GE(43964,python):2025-04-07-04:00:53.763.653 [ge_api.cc:174][status:INIT]43964 GEInitializeImpl:GEInitialize start
 71 [INFO] GE(43964,python):2025-04-07-04:00:53.763.735 [ge_api.cc:188]43964 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::GEAPICheckSupportedGlobalOptions is [72] micro second.
 72 [INFO] GE(43964,python):2025-04-07-04:00:53.772.784 [ge_api.cc:203]43964 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::CheckOptionsValid is [325] micro second.
 73 [INFO] GE(43964,python):2025-04-07-04:00:53.773.450 [ge_api.cc:217]43964 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::ManagerInitialize is [2] micro second.
 74 [EVENT] PROFILING(43964,python):2025-04-07-04:00:53.946.960 [msprof_callback_impl.cpp:89] >>> (tid:43964) MsprofCtrlCallback called, type: 255
 75 [EVENT] PROFILING(43964,python):2025-04-07-04:00:53.946.997 [ai_drv_dev_api.cpp:333] >>> (tid:43964) Succeeded to DrvGetApiVersion version: 0x72313
 76 [INFO] GE(43964,python):2025-04-07-04:00:53.947.089 [ge_api.cc:236]43964 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::InitProfiling is [200] micro second.
 77 [INFO] GE(43964,python):2025-04-07-04:00:54.009.587 [ge_api.cc:246]43964 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::InitPreparation is [62379] micro second.
 78 [INFO] GE(43964,python):2025-04-07-04:00:54.009.611 [gelib.cc:91]43964 Initialize:[GEPERFTRACE] GE Init Start
 79 [INFO] GE(43964,python):2025-04-07-04:00:54.184.001 [gelib.cc:172]43964 Initialize:[GEPERFTRACE] The time cost of GELib::InitializePrepare is [174379] micro second.
 80 [INFO] GE(43964,python):2025-04-07-04:00:54.184.556 [gelib.cc:327]43964 SystemInitialize:Infer init GELib success, device id :-1
 81 [INFO] GE(43964,python):2025-04-07-04:00:54.184.565 [gelib.cc:199]43964 InnerInitialize:[GEPERFTRACE] The time cost of InnerInitialize::SystemInitialize is [13] micro second.
 82 [INFO] GE(43964,python):2025-04-07-04:00:54.189.582 [process_node_engine_manager.cc:63]43964 Initialize:[GEPERFTRACE] The time cost of ProcessNodeEngineManager::Initialize[HOST_CPU] is [1] micro second.
 83 [INFO] GE(43964,python):2025-04-07-04:00:54.189.613 [process_node_engine_manager.cc:63]43964 Initialize:[GEPERFTRACE] The time cost of ProcessNodeEngineManager::Initialize[NPU] is [12] micro second.
 84 [INFO] GE(43964,python):2025-04-07-04:00:54.189.618 [process_node_engine_manager.cc:63]43964 Initialize:[GEPERFTRACE] The time cost of ProcessNodeEngineManager::Initialize[PS] is [1] micro second.
 85 [INFO] GE(43964,python):2025-04-07-04:00:54.189.622 [process_node_engine_manager.cc:63]43964 Initialize:[GEPERFTRACE] The time cost of ProcessNodeEngineManager::Initialize[UDF] is [0] micro second.
 86 [INFO] GE(43964,python):2025-04-07-04:00:54.189.628 [gelib.cc:223]43964 InnerInitialize:[GEPERFTRACE] The time cost of InnerInitialize::ProcessNodeEngine is [4144] micro second.
 87 [INFO] GE(43964,python):2025-04-07-04:00:54.191.155 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [GetDNNEngineObjs] in [libengine.so] is [91] micro second.
 88 [INFO] GE(43964,python):2025-04-07-04:00:54.191.167 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[AIcoreEngine] is [0] micro second.
 89 [INFO] GE(43964,python):2025-04-07-04:00:54.191.173 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_HCCL] is [0] micro second.
 90 [INFO] GE(43964,python):2025-04-07-04:00:54.191.177 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_AICPU] is [0] micro second.
 91 [INFO] GE(43964,python):2025-04-07-04:00:54.191.181 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_AICPU_ASCEND] is [0] micro second.
 92 [INFO] GE(43964,python):2025-04-07-04:00:54.191.184 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_AICPU_ASCEND_FFTS_PLUS] is [0] micro second.
 93 [INFO] GE(43964,python):2025-04-07-04:00:54.191.188 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_AICPU_FFTS_PLUS] is [0] micro second.
 94 [INFO] GE(43964,python):2025-04-07-04:00:54.191.191 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_DVPP] is [0] micro second.
 95 [INFO] GE(43964,python):2025-04-07-04:00:54.191.194 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_GE_LOCAL] is [0] micro second.
 96 [INFO] GE(43964,python):2025-04-07-04:00:54.191.198 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_HOST_CPU] is [0] micro second.
 97 [INFO] GE(43964,python):2025-04-07-04:00:54.191.201 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_RTS] is [0] micro second.
 98 [INFO] GE(43964,python):2025-04-07-04:00:54.191.205 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_RTS_FFTS_PLUS] is [0] micro second.
 99 [INFO] GE(43964,python):2025-04-07-04:00:54.191.208 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DSAEngine] is [1] micro second.
100 [INFO] GE(43964,python):2025-04-07-04:00:54.191.211 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[VectorEngine] is [0] micro second.
101 [INFO] GE(43964,python):2025-04-07-04:00:54.191.215 [dnnengine_manager.cc:128]43964 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[ffts_plus] is [0] micro second.
102 [INFO] GE(43964,python):2025-04-07-04:00:54.191.742 [gelib.cc:234]43964 InnerInitialize:[GEPERFTRACE] The time cost of InnerInitialize::EngineInitialize is [2106] micro second.
103 [INFO] GE(43964,python):2025-04-07-04:00:54.193.150 [op_tiling_manager.cc:109]43964 ~FuncPerfScope:[GEPERFTRACE] The time cost of OpTilingManager::LoadSo is [1167] micro second.
104 [INFO] GE(43964,python):2025-04-07-04:00:54.479.072 [plugin_manager.cc:705]43964 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libfe.so] is [285876] micro second.
105 [INFO] GE(43964,python):2025-04-07-04:00:54.481.512 [plugin_manager.cc:705]43964 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libge_local_engine.so] is [2347] micro second.
106 [INFO] GE(43964,python):2025-04-07-04:00:54.495.936 [plugin_manager.cc:705]43964 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/librts_engine.so] is [14377] micro second.
107 [INFO] GE(43964,python):2025-04-07-04:00:54.520.632 [plugin_manager.cc:705]43964 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libaicpu_ascend_engine.so] is [24649] micro second.
108 [INFO] GE(43964,python):2025-04-07-04:00:54.524.955 [plugin_manager.cc:705]43964 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libhost_cpu_engine.so] is [4268] micro second.
109 [INFO] GE(43964,python):2025-04-07-04:00:54.543.313 [plugin_manager.cc:705]43964 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libaicpu_tf_engine.so] is [18314] micro second.
110 [INFO] GE(43964,python):2025-04-07-04:00:54.574.302 [plugin_manager.cc:705]43964 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libffts.so] is [30940] micro second.
111 [INFO] GE(43964,python):2025-04-07-04:00:54.578.175 [plugin_manager.cc:705]43964 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libdvpp_engine.so] is [3811] micro second.
112 [INFO] GE(43964,python):2025-04-07-04:00:54.618.260 [plugin_manager.cc:705]43964 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/libhcom_graph_adaptor.so] is [40038] micro second.
113 [INFO] GE(43964,python):2025-04-07-04:00:54.618.292 [ops_kernel_manager.cc:78]43964 Initialize:[GEPERFTRACE] The time cost of OpsKernelManager::LoadPluginManagerSo is [425129] micro second.
114 [INFO] GE(43964,python):2025-04-07-04:00:54.618.655 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Initialize] in [libaicpu_ascend_engine.so] is [347] micro second.
115 [INFO] GE(43964,python):2025-04-07-04:00:54.619.061 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Initialize] in [libaicpu_tf_engine.so] is [395] micro second.
116 [INFO] GE(43964,python):2025-04-07-04:00:54.619.093 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Initialize] in [libdvpp_engine.so] is [22] micro second.
117 [INFO] ATRACE(43964,python):2025-04-07-04:00:54.619.224 [tracer_mgr_operate.c:64](tid:43964) create object FE_Global_Trace success, exitSave(false).
118 [INFO] ATRACE(43964,python):2025-04-07-04:00:54.619.282 [tracer_mgr_operate.c:64](tid:43964) create object FE_Statistics_Trace success, exitSave(false).
119 [INFO] TUNE(43964,python):2025-04-07-04:00:54.623.610 [cann_kb_pyfunc_mgr.cpp:72][CANNKB][Tid:43964]"CannKbPyfuncMgr: Enter PyObjectInit, reference_ is 0!"
120 [INFO] TUNE(43964,python):2025-04-07-04:00:54.623.633 [handle_manager.cpp:115][CANNKB][Tid:43964]"Start to run init functions to load dynamic python lib!"
121 [INFO] TUNE(43964,python):2025-04-07-04:00:54.623.800 [handle_manager.cpp:407][CANNKB][Tid:43964]"Init functions of loading dynamic python lib end!"
122 [INFO] TUNE(43964,python):2025-04-07-04:00:54.623.805 [cann_kb_pyfunc_mgr.cpp:24][CANNKB][Tid:43964]"CANN_KB_Py has already been initialized."
123 [INFO] TUNE(43964,python):2025-04-07-04:00:56.698.237 [cann_kb_pyfunc_mgr.cpp:117][CANNKB][Tid:43964]"CannKbPyfuncMgr: Run PyObjectInit successfully!"
124 [INFO] GE(43964,python):2025-04-07-04:00:56.779.246 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Initialize] in [libfe.so] is [2160142] micro second.
125 [INFO] GE(43964,python):2025-04-07-04:00:56.779.357 [gelib.cc:246]43964 InnerInitialize:[GEPERFTRACE] The time cost of InnerInitialize::OpsManagerInitialize is [2587588] micro second.
126 [INFO] GE(43964,python):2025-04-07-04:00:56.799.166 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libaicpu_ascend_engine.so] is [19615] micro second.
127 [INFO] GE(43964,python):2025-04-07-04:00:56.799.219 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libaicpu_tf_engine.so] is [34] micro second.
128 [INFO] GE(43964,python):2025-04-07-04:00:56.799.238 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libdvpp_engine.so] is [3] micro second.
129 [INFO] ATRACE(43964,python):2025-04-07-04:00:56.806.884 [tracer_mgr_operate.c:246](tid:43964) destroy object FE_Global_Trace, exitSave(false).
130 [INFO] ATRACE(43964,python):2025-04-07-04:00:56.806.902 [tracer_mgr_operate.c:246](tid:43964) destroy object FE_Statistics_Trace, exitSave(false).
131 [INFO] GE(43964,python):2025-04-07-04:00:56.806.918 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libfe.so] is [7675] micro second.
132 [INFO] GE(43964,python):2025-04-07-04:00:56.806.930 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libffts.so] is [6] micro second.
133 [INFO] GE(43964,python):2025-04-07-04:00:56.806.951 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libge_local_engine.so] is [16] micro second.
134 [INFO] GE(43964,python):2025-04-07-04:00:56.807.005 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libhcom_graph_adaptor.so] is [49] micro second.
135 [INFO] GE(43964,python):2025-04-07-04:00:56.807.063 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libhost_cpu_engine.so] is [52] micro second.
136 [INFO] GE(43964,python):2025-04-07-04:00:56.807.088 [dnnengine_manager.cc:41]43964 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [librts_engine.so] is [20] micro second.
137 [INFO] GE(43964,python):2025-04-07-04:00:56.807.297 [ge_api.cc:252]43964 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::GELibInitialize is [2797687] micro second.
138 [INFO] RUNTIME(43964,python):2025-04-07-04:00:56.807.968 [api_impl.cc:6836] 43964 PeekLastErr: level=0 err=0.

Ascend/pytorch
暂停

内容风险标识

评论 (2)

Ascend/pytorch暂停 .gitee-modal { width: 500px !important; }

内容风险标识