登录
注册
开源
企业版
高校版
搜索
帮助中心
使用条款
关于我们
开源
企业版
高校版
私有云
模力方舟
AI 队友
登录
注册
代码拉取完成,页面将自动刷新
仓库状态说明
开源项目
>
人工智能
>
机器学习/深度学习
&&
捐赠
捐赠前请先登录
取消
前往登录
扫描微信二维码支付
取消
支付完成
支付提示
将跳转至支付宝完成支付
确定
取消
Watch
不关注
关注所有动态
仅关注版本发行动态
关注但不提醒动态
91
Star
661
Fork
1.5K
Ascend
/
pytorch
暂停
代码
Issues
38
Pull Requests
350
Wiki
统计
流水线
服务
质量分析
Jenkins for Gitee
腾讯云托管
腾讯云 Serverless
悬镜安全
阿里云 SAE
Codeblitz
SBOM
开发画像分析
我知道了,不再自动展开
更新失败,请稍后重试!
移除标识
内容风险标识
本任务被
标识为内容中包含有代码安全 Bug 、隐私泄露等敏感信息,仓库外成员不可访问
ubuntu安装torch_npu检测报错
DONE
#IC3JL6
缺陷
xiulian
创建于
2025-04-24 10:08
一、问题现象(附报错日志上下文): 测试torch_npu是否安装成功的时候报错 二、软件版本: -- CANN 版本 (e.g., CANN 3.0.x,5.x.x): 8.0 --Tensorflow/Pytorch/MindSpore 版本:torch==2.4.0 torch-npu==2.4.0.post2 --Python 版本 (e.g., Python 3.7.5): 3.10.12 -- MindStudio版本 (e.g., MindStudio 2.0.0 (beta3)): --操作系统版本 (e.g., Ubuntu 18.04):22.04.5 LTS 三、测试步骤: >>> import torch >>> import torch_npu [EVENT] PROFILING(489597,python):2025-04-24-09:46:03.953.629 [msprof_callback_impl.cpp:336] >>> (tid:489597) Started to register profiling ctrl callback. [EVENT] PROFILING(489597,python):2025-04-24-09:46:03.953.901 [msprof_callback_impl.cpp:343] >>> (tid:489597) Started to register profiling hash id callback. [INFO] PROFILING(489597,python):2025-04-24-09:46:03.953.917 [prof_atls_plugin.cpp:117] (tid:489597) RegisterProfileCallback, callback type is 7 [EVENT] PROFILING(489597,python):2025-04-24-09:46:03.953.923 [msprof_callback_impl.cpp:350] >>> (tid:489597) Started to register profiling enable host freq callback. [INFO] PROFILING(489597,python):2025-04-24-09:46:03.953.930 [prof_atls_plugin.cpp:117] (tid:489597) RegisterProfileCallback, callback type is 8 [INFO] ATRACE(489597,python):2025-04-24-09:46:03.955.706 [trace_attr.c:188](tid:489597) attr init success, timeout=0ms. [INFO] ATRACE(489597,python):2025-04-24-09:46:03.955.747 [trace_recorder.c:121](tid:489597) use root path: /root/ascend [INFO] RUNTIME(489597,python):2025-04-24-09:46:03.963.853 [task_fail_callback_manager.cc:52] 489597 TaskFailCallBackManager: Constructor. [INFO] HCCL(489597,python):2025-04-24-09:46:04.114.775 [adapter_rts.cc:2646][489597][adapter_rts.cc][CallBackInitRts] g_deviceType [6] g_deviceLogicId [-1] g_devicePhyId [-1] [INFO] RUNTIME(489597,python):2025-04-24-09:46:04.329.275 [runtime.cc:5471] 489597 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set [INFO] PROFILING(489597,python):2025-04-24-09:46:04.332.664 [prof_atls_plugin.cpp:210] (tid:489597) Module[7] register callback of ctrl handle. /root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch_npu/__init__.py:248: UserWarning: On the interactive interface, the value of TASK_QUEUE_ENABLE is set to 0 by default. Do not set it to 1 to prevent some unknown errors warnings.warn("On the interactive interface, the value of TASK_QUEUE_ENABLE is set to 0 by default. \ >>> x = torch.randn(2, 2).npu() 四、日志信息: [INFO] IDEDD(489597,python):2025-04-24-09:46:09.348.314 [adx_dump_record.cpp:623][tid:489747] start dump thread, remote dump record temp path : /root/. [INFO] IDEDD(489597,python):2025-04-24-09:46:09.348.455 [adx_server_manager.cpp:203][tid:489748] Run Server(0) Process [INFO] IDEDD(489597,python):2025-04-24-09:46:09.348.504 [hdc_comm_opt.cpp:59][tid:489748] open device[0] server[13] [INFO] DRV(489597,python):2025-04-24-09:46:09.348.620 [drv_log_user.c:623][ascend][curpid:489597,489748][drv][hdc][share_log_read_in_single_module]Server create success.(dev_id=0; fid=0; service_type="service_dump"; pid=386088) [INFO] IDEDD(489597,python):2025-04-24-09:46:09.348.635 [hdc_api.cpp:119][tid:489748] logDevId 0 create HDC server successfully [INFO] IDEDD(489597,python):2025-04-24-09:46:09.348.679 [hdc_comm_opt.cpp:59][tid:489748] open device[1] server[13] [INFO] DRV(489597,python):2025-04-24-09:46:09.348.820 [drv_log_user.c:623][ascend][curpid:489597,489748][drv][hdc][share_log_read_in_single_module]Server create success.(dev_id=1; fid=0; service_type="service_dump"; pid=386088) [INFO] IDEDD(489597,python):2025-04-24-09:46:09.348.828 [hdc_api.cpp:119][tid:489748] logDevId 1 create HDC server successfully [INFO] PROFILING(489597,python):2025-04-24-09:46:09.351.651 [prof_atls_plugin.cpp:210] (tid:489597) Module[48] register callback of ctrl handle. [INFO] PROFILING(489597,python):2025-04-24-09:46:09.351.675 [prof_atls_plugin.cpp:210] (tid:489597) Module[45] register callback of ctrl handle. [INFO] GE(489597,python):2025-04-24-09:46:09.616.680 [op_tiling_manager.cc:109]489597 ~FuncPerfScope:[GEPERFTRACE] The time cost of OpTilingManager::LoadSo is [264987] micro second. [INFO] PROFILING(489597,python):2025-04-24-09:46:09.634.604 [prof_atls_plugin.cpp:210] (tid:489597) Module[6] register callback of ctrl handle. [EVENT] PROFILING(489597,python):2025-04-24-09:46:10.314.474 [msprof_callback_impl.cpp:89] >>> (tid:489597) MsprofCtrlCallback called, type: 255 [EVENT] PROFILING(489597,python):2025-04-24-09:46:10.314.865 [ai_drv_dev_api.cpp:333] >>> (tid:489597) Succeeded to DrvGetApiVersion version: 0x72317 [INFO] ATRACE(489597,python):2025-04-24-09:46:10.317.241 [atrace_client_core.c:160](tid:489597) start to receive device[0] trace log. [INFO] TDT(489597,python):2025-04-24-09:46:10.317.326 [client_manager.cpp:433][IsHostEnvironment][tid:489597] [TsdClient] logicDeviceId is [0], hostaicpunum[0] [INFO] TDT(489597,python):2025-04-24-09:46:10.317.338 [client_manager.cpp:462][GetClientRunMode][tid:489597] runningMode:0 [INFO] TDT(489597,python):2025-04-24-09:46:10.317.344 [client_manager.cpp:126][GetInstance][tid:489597] [ClientManager] Current mode:1 [INFO] TDT(489597,python):2025-04-24-09:46:10.317.366 [process_mode_manager.cpp:415][GetLogLevel][tid:489597] [TsdClient] get ASCEND_GLOBAL_LOG_LEVEL [] ASCEND_GLOBAL_EVENT_ENABLE [] ASCEND_MODULE_LOG_LEVEL [] [INFO] TDT(489597,python):2025-04-24-09:46:10.317.461 [process_mode_manager.cpp:134][OpenProcess][tid:489597] [ProcessModeManager] enter into open process deviceId[0] rankSize[0] [INFO] TDT(489597,python):2025-04-24-09:46:10.317.962 [process_mode_manager.cpp:545][InitTsdClient][tid:489597] [TsdClient] deviceId[0] begin to init hdc client [INFO] TDT(489597,python):2025-04-24-09:46:10.318.060 [version_verify.cpp:36][SetVersionInfo][tid:489597] VersionVerify: send client version to server [INFO] TDT(489597,python):2025-04-24-09:46:10.318.077 [version_verify.cpp:52][SetVersionInfo][tid:489597] send feature_info:{msg_type:35, features:{check before send aicpu package}} [INFO] TDT(489597,python):2025-04-24-09:46:10.318.085 [version_verify.cpp:52][SetVersionInfo][tid:489597] send feature_info:{msg_type:37, features:{check before send open qs message}} [INFO] TDT(489597,python):2025-04-24-09:46:10.318.093 [version_verify.cpp:52][SetVersionInfo][tid:489597] send feature_info:{msg_type:64, features:{get check code retry}} [INFO] TDT(489597,python):2025-04-24-09:46:10.318.451 [version_verify.cpp:68][PeerVersionCheck][tid:489597] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(489597,python):2025-04-24-09:46:10.318.466 [version_verify.cpp:89][ParseVersionInfo][tid:489597] VersionVerify: pass client version info success [INFO] TDT(489597,python):2025-04-24-09:46:10.318.472 [hdc_client.cpp:275][CheckHdcConnection][tid:489597] Service[2] create hdc success [INFO] TDT(489597,python):2025-04-24-09:46:10.318.488 [version_verify.cpp:122][SpecialFeatureCheck][tid:489597] VersionVerify: new type[35], supported [INFO] TDT(489597,python):2025-04-24-09:46:10.318.529 [process_mode_manager.cpp:922][GetDeviceCheckCodeOnce][tid:489597] [TsdClient][deviceId=0] [sessionId=1] wait package info respond [INFO] TDT(489597,python):2025-04-24-09:46:10.319.529 [version_verify.cpp:122][SpecialFeatureCheck][tid:489597] VersionVerify: new type[64], supported [INFO] TDT(489597,python):2025-04-24-09:46:10.319.597 [process_mode_manager.cpp:545][InitTsdClient][tid:489597] [TsdClient] deviceId[0] begin to init hdc client [INFO] TDT(489597,python):2025-04-24-09:46:10.319.676 [version_verify.cpp:36][SetVersionInfo][tid:489597] VersionVerify: send client version to server [INFO] TDT(489597,python):2025-04-24-09:46:10.319.687 [version_verify.cpp:52][SetVersionInfo][tid:489597] send feature_info:{msg_type:35, features:{check before send aicpu package}} [INFO] TDT(489597,python):2025-04-24-09:46:10.319.694 [version_verify.cpp:52][SetVersionInfo][tid:489597] send feature_info:{msg_type:37, features:{check before send open qs message}} [INFO] TDT(489597,python):2025-04-24-09:46:10.319.702 [version_verify.cpp:52][SetVersionInfo][tid:489597] send feature_info:{msg_type:64, features:{get check code retry}} [INFO] TDT(489597,python):2025-04-24-09:46:10.319.892 [version_verify.cpp:68][PeerVersionCheck][tid:489597] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(489597,python):2025-04-24-09:46:10.319.903 [version_verify.cpp:89][ParseVersionInfo][tid:489597] VersionVerify: pass client version info success [INFO] TDT(489597,python):2025-04-24-09:46:10.319.908 [hdc_client.cpp:275][CheckHdcConnection][tid:489597] Service[2] create hdc success [INFO] TDT(489597,python):2025-04-24-09:46:10.319.919 [process_mode_manager.cpp:600][ConstructOpenMsg][tid:489597] [TsdClient] tsd get process sign successfully, procpid[386088] signSize[48] [INFO] TDT(489597,python):2025-04-24-09:46:10.319.930 [version_verify.cpp:114][SpecialFeatureCheck][tid:489597] VersionVerify: previous type[6], supported [INFO] TDT(489597,python):2025-04-24-09:46:10.319.948 [process_mode_manager.cpp:151][OpenProcess][tid:489597] [ProcessModeManager] deviceId[0] sessionId[1] rankSize[0], wait sub process start respond [INFO] DRV(489597,python):2025-04-24-09:46:10.489.123 [devmm_svm.c:99][ascend][curpid:489597,489597][drv][devmm][devmm_setup_device]DrvMemDeviceOpen. (devid=0) [INFO] DRV(489597,python):2025-04-24-09:46:10.491.273 [devmm_virt_interface.c:394][ascend][curpid:489597,489597][drv][devmm][devmm_ioctl_init_process]Init_process details. (hostpid=489597) [INFO] DRV(489597,python):2025-04-24-09:46:10.491.590 [drv_log_user.c:623][ascend][curpid:489597,489597][drv][devmm][share_log_read_in_single_module]Setup device succeeded. (logical_devid=0; devid=0; vfid=0; hostpid=386088; devpid=2615) [INFO] DRV(489597,python):2025-04-24-09:46:10.491.606 [devmm_svm.c:84][ascend][curpid:489597,489597][drv][devmm][devmm_set_device_info]Device info. (devid=0; dvpp_size=17179869184; support_bar_mem=1; support_dev_read_only=1; support_dev_mem_map_host=1; support_bar_huge_mem=1; is_alloced_by_malloc=0; host_rw_dev_ro=1; double_pgtable_offset=228698418577408) [INFO] TDT(489597,python):2025-04-24-09:46:10.491.873 [stub_process_mode_nowin.cpp:60][ProcessQueueForMdc][tid:489597] [TsdClient] it is unnecessary of current mode[0] to grant queue auth to aicpusd [INFO] TDT(489597,python):2025-04-24-09:46:10.491.885 [stub_process_mode_nowin.cpp:98][OpenInHost][tid:489597] enter into OpenInHost deviceid[0] [INFO] TDT(489597,python):2025-04-24-09:46:10.491.908 [client_manager.cpp:433][IsHostEnvironment][tid:489597] [TsdClient] logicDeviceId is [0], hostaicpunum[0] [INFO] TDT(489597,python):2025-04-24-09:46:10.491.915 [stub_process_mode_nowin.cpp:102][OpenInHost][tid:489597] host cpu not support [INFO] TDT(489597,python):2025-04-24-09:46:10.491.920 [process_mode_manager.cpp:185][OpenProcess][tid:489597] [TsdClient][deviceId=0] [sessionId=1] start hccp and computer process success [INFO] DRV(489597,python):2025-04-24-09:46:10.496.002 [devmm_svm.c:99][ascend][curpid:489597,489597][drv][devmm][devmm_setup_device]DrvMemDeviceOpen. (devid=0) [INFO] DRV(489597,python):2025-04-24-09:46:10.496.024 [devmm_svm.c:84][ascend][curpid:489597,489597][drv][devmm][devmm_set_device_info]Device info. (devid=0; dvpp_size=17179869184; support_bar_mem=1; support_dev_read_only=1; support_dev_mem_map_host=1; support_bar_huge_mem=1; is_alloced_by_malloc=0; host_rw_dev_ro=1; double_pgtable_offset=228698418577408) [INFO] RUNTIME(489597,python):2025-04-24-09:46:10.496.100 [raw_device.cc:237] 489597 Init: isAddrFlat:0 [INFO] TDT(489597,python):2025-04-24-09:46:10.497.130 [tsd_client.cpp:172][TsdCapabilityGet][tid:489597] TsdCapabilityGet Begin. [INFO] TDT(489597,python):2025-04-24-09:46:10.497.143 [process_mode_manager.cpp:1351][CapabilityGet][tid:489597] [ProcessModeManager] enter into CapabilityGet process deviceId[0] type[0] [INFO] TDT(489597,python):2025-04-24-09:46:10.497.155 [version_verify.cpp:114][SpecialFeatureCheck][tid:489597] VersionVerify: previous type[38], supported [INFO] TDT(489597,python):2025-04-24-09:46:10.497.183 [process_mode_manager.cpp:1374][CapabilityGet][tid:489597] [TsdClient][deviceId=0] [sessionId=1] [type=0]send capability success [INFO] TDT(489597,python):2025-04-24-09:46:10.498.312 [process_mode_manager.cpp:829][PidQosMsgProc][tid:489597] [TsdClient] PidQosMsgProc recvMsg realDeviceId[0] msgType[39] localDevId[0] rspCode[0],pidqos[1] [INFO] TDT(489597,python):2025-04-24-09:46:10.498.322 [process_mode_manager.cpp:1379][CapabilityGet][tid:489597] [TsdClient][logicDeviceId_=0][type=0][pidQos=1][ret=0]recv capability respond finish [INFO] RUNTIME(489597,python):2025-04-24-09:46:10.498.334 [engine.cc:76] 489597 Engine: Constructor. [INFO] RUNTIME(489597,python):2025-04-24-09:46:10.498.340 [stars_engine.cc:41] 489597 StarsEngine: Constructor. [INFO] RUNTIME(489597,python):2025-04-24-09:46:10.518.655 [npu_driver.cc:5784] 489773 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(489597,python):2025-04-24-09:46:10.522.164 [tracer_mgr_operate.c:64](tid:489597) create object RUNTIME_ATRACE_DEV0_TS0 success, exitSave(false). [INFO] TDT(489597,python):2025-04-24-09:46:10.522.197 [client_manager.cpp:195][SetProfilingCallback][tid:489597] [TsdClient] set profiling callback success [INFO] GE(489597,python):2025-04-24-09:46:10.529.705 [ge_api.cc:174][status:INIT]489597 GEInitializeImpl:GEInitialize start [INFO] GE(489597,python):2025-04-24-09:46:10.529.726 [ge_api.cc:188]489597 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::GEAPICheckSupportedGlobalOptions is [4] micro second. [INFO] GE(489597,python):2025-04-24-09:46:10.556.068 [ge_api.cc:203]489597 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::CheckOptionsValid is [75] micro second. [INFO] GE(489597,python):2025-04-24-09:46:10.556.581 [ge_api.cc:217]489597 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::ManagerInitialize is [2] micro second. [EVENT] PROFILING(489597,python):2025-04-24-09:46:10.699.533 [msprof_callback_impl.cpp:89] >>> (tid:489597) MsprofCtrlCallback called, type: 255 [EVENT] PROFILING(489597,python):2025-04-24-09:46:10.699.576 [ai_drv_dev_api.cpp:333] >>> (tid:489597) Succeeded to DrvGetApiVersion version: 0x72317 [INFO] GE(489597,python):2025-04-24-09:46:10.699.655 [ge_api.cc:236]489597 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::InitProfiling is [144] micro second. [INFO] GE(489597,python):2025-04-24-09:46:10.799.231 [ge_api.cc:246]489597 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::InitPreparation is [99452] micro second. [INFO] GE(489597,python):2025-04-24-09:46:10.799.282 [gelib.cc:91]489597 Initialize:[GEPERFTRACE] GE Init Start [INFO] GE(489597,python):2025-04-24-09:46:11.258.433 [gelib.cc:172]489597 Initialize:[GEPERFTRACE] The time cost of GELib::InitializePrepare is [459139] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.258.716 [gelib.cc:327]489597 SystemInitialize:Infer init GELib success, device id :-1 [INFO] GE(489597,python):2025-04-24-09:46:11.258.728 [gelib.cc:199]489597 InnerInitialize:[GEPERFTRACE] The time cost of InnerInitialize::SystemInitialize is [17] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.262.878 [process_node_engine_manager.cc:63]489597 Initialize:[GEPERFTRACE] The time cost of ProcessNodeEngineManager::Initialize[HOST_CPU] is [1] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.262.897 [process_node_engine_manager.cc:63]489597 Initialize:[GEPERFTRACE] The time cost of ProcessNodeEngineManager::Initialize[NPU] is [1] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.262.905 [process_node_engine_manager.cc:63]489597 Initialize:[GEPERFTRACE] The time cost of ProcessNodeEngineManager::Initialize[PS] is [1] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.262.911 [process_node_engine_manager.cc:63]489597 Initialize:[GEPERFTRACE] The time cost of ProcessNodeEngineManager::Initialize[UDF] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.262.920 [gelib.cc:223]489597 InnerInitialize:[GEPERFTRACE] The time cost of InnerInitialize::ProcessNodeEngine is [3589] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.091 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [GetDNNEngineObjs] in [libengine.so] is [103] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.107 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[AIcoreEngine] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.116 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_HCCL] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.123 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_AICPU] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.130 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_AICPU_ASCEND] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.136 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_AICPU_ASCEND_FFTS_PLUS] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.142 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_AICPU_FFTS_PLUS] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.148 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_DVPP] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.154 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_GE_LOCAL] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.160 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_HOST_CPU] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.166 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_RTS] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.172 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_RTS_FFTS_PLUS] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.177 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DSAEngine] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.183 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[VectorEngine] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.188 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[ffts_plus] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.900 [gelib.cc:234]489597 InnerInitialize:[GEPERFTRACE] The time cost of InnerInitialize::EngineInitialize is [1972] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.265.921 [op_tiling_manager.cc:109]489597 ~FuncPerfScope:[GEPERFTRACE] The time cost of OpTilingManager::LoadSo is [836] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.595.124 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libfe.so] is [329144] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.597.902 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libge_local_engine.so] is [2659] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.612.642 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/librts_engine.so] is [14676] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.635.677 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libaicpu_ascend_engine.so] is [22954] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.639.663 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libhost_cpu_engine.so] is [3898] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.656.661 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libaicpu_tf_engine.so] is [16933] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.689.378 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libffts.so] is [32630] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.694.137 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libdvpp_engine.so] is [4671] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.736.822 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/libhcom_graph_adaptor.so] is [42622] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.736.864 [ops_kernel_manager.cc:78]489597 Initialize:[GEPERFTRACE] The time cost of OpsKernelManager::LoadPluginManagerSo is [470926] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.737.234 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Initialize] in [libaicpu_ascend_engine.so] is [348] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.737.421 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Initialize] in [libaicpu_tf_engine.so] is [172] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.737.452 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Initialize] in [libdvpp_engine.so] is [18] micro second. [INFO] ATRACE(489597,python):2025-04-24-09:46:11.737.565 [tracer_mgr_operate.c:64](tid:489597) create object FE_Global_Trace success, exitSave(false). [INFO] ATRACE(489597,python):2025-04-24-09:46:11.737.642 [tracer_mgr_operate.c:64](tid:489597) create object FE_Statistics_Trace success, exitSave(false). [INFO] TUNE(489597,python):2025-04-24-09:46:11.742.640 [cann_kb_pyfunc_mgr.cpp:72][CANNKB][Tid:489597]"CannKbPyfuncMgr: Enter PyObjectInit, reference_ is 0!" [INFO] TUNE(489597,python):2025-04-24-09:46:11.742.671 [handle_manager.cpp:115][CANNKB][Tid:489597]"Start to run init functions to load dynamic python lib!" [INFO] TUNE(489597,python):2025-04-24-09:46:11.742.732 [handle_manager.cpp:407][CANNKB][Tid:489597]"Init functions of loading dynamic python lib end!" [INFO] TUNE(489597,python):2025-04-24-09:46:11.742.740 [cann_kb_pyfunc_mgr.cpp:24][CANNKB][Tid:489597]"CANN_KB_Py has already been initialized." [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.155 [pywrapper.cpp:94][CANNKB][Tid:489597]"Traceback (most recent call last): " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.186 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/__init__.py", line 98, in <module> from . import dsl " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.194 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/__init__.py", line 20, in <module> from .api import ceil " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.200 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/api.py", line 27, in <module> from .compute import cast " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.206 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/compute/cast.py", line 34, in <module> from tbe.dsl.unify_schedule.util import is_v220 " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.213 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/unify_schedule/__init__.py", line 35, in <module> from . import vector " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.218 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/unify_schedule/vector/__init__.py", line 30, in <module> from . import sparse_apply " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.225 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/unify_schedule/vector/sparse_apply/__init__.py", line 20, in <module> from . import sparse_apply_schedule " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.231 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/unify_schedule/vector/sparse_apply/sparse_apply_schedule.py", line 23, in <module> import te.platform as tbe_platform " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.236 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/te/platform/__init__.py", line 125, in <module> from .cce_util import get_type_bits " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.242 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/te/platform/cce_util.py", line 34, in <module> from tvm.topi.utils import equal_const_int " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.247 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/8.0.0/python/site-packages/tbe/tvm/topi/__init__.py", line 50, in <module> from . import nn " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.252 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/8.0.0/python/site-packages/tbe/tvm/topi/nn/__init__.py", line 35, in <module> from .conv3d_transpose import * " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.257 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/8.0.0/python/site-packages/tbe/tvm/topi/nn/conv3d_transpose.py", line 21, in <module> from tvm import relay " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.263 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/8.0.0/python/site-packages/tbe/tvm/relay/__init__.py", line 64, in <module> from .transform import memory_plan " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.268 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/8.0.0/python/site-packages/tbe/tvm/relay/transform/memory_plan.py", line 23, in <module> import attr " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.274 [pywrapper.cpp:94][CANNKB][Tid:489597]"ModuleNotFoundError: No module named 'attr' " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.513 [cann_kb_pyfunc_mgr.cpp:98][CANNKB][Tid:489597]"PyObjectInit: Import repository_manager_log error" [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.528 [py_interface.cpp:28][CANNKB][Tid:489597]"PyObjectInit of CannKbPyfuncMgr Error!" [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.535 [cann_kb_api.cpp:24][CANNKB][Tid:489597]"Run PyInterfaceInit Error!" [ERROR] TEFUSION(489597,python):2025-04-24-09:46:13.434.547 [python_adapter_manager.cc:94]489597 InitCannKB call CannKbInit failed. res = [3]. init params: [Ascend910B2, 24, ]. [ERROR] TEFUSION(489597,python):2025-04-24-09:46:13.434.593 [python_adapter_manager.cc:47]489597 Initialize AOE Failed to call InitCannKB [ERROR] FE(489597,python):2025-04-24-09:46:13.434.630 [tbe_op_store_adapter.cc:1816]489597 InitializeTeFusion:"[GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe." [ERROR] FE(489597,python):2025-04-24-09:46:13.434.679 [tbe_op_store_adapter.cc:1783]489597 InitializeInner:"[GraphOpt][InitializeInner][InitTeFusion]: Failed to initialize TeFusion." [ERROR] FE(489597,python):2025-04-24-09:46:13.434.702 [op_store_adapter_manager.cc:79]489597 InitializeAdapter:"[SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295]" [ERROR] FE(489597,python):2025-04-24-09:46:13.434.787 [op_store_adapter_manager.cc:120]489597 Initialize:"[SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom]." [ERROR] FE(489597,python):2025-04-24-09:46:13.434.808 [fusion_manager.cc:115]489597 Initialize:"[FusionMngr][Init] Op store adapter manager init failed." [INFO] GE(489597,python):2025-04-24-09:46:13.434.896 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Initialize] in [libfe.so] is [1697432] micro second. [ERROR] GE(489597,python):2025-04-24-09:46:13.434.990 [ops_kernel_manager.cc:82]489597 Initialize: ErrorNo: 1343250441(There is no valid so about OpsKernelInfoStore or GraphOptimizer.) [INIT][OPS_KER]PluginManager InvokeAll failed. [INFO] GE(489597,python):2025-04-24-09:46:13.435.005 [gelib.cc:246]489597 InnerInitialize:[GEPERFTRACE] The time cost of InnerInitialize::OpsManagerInitialize is [2170091] micro second. [ERROR] GE(489597,python):2025-04-24-09:46:13.435.016 [gelib.cc:248]489597 InnerInitialize: ErrorNo: 1343250441(There is no valid so about OpsKernelInfoStore or GraphOptimizer.) [INIT][OPS_KER][Init][OpsManager]GE ops manager initial failed. [INFO] GE(489597,python):2025-04-24-09:46:13.452.771 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libaicpu_ascend_engine.so] is [17707] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.452.811 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libaicpu_tf_engine.so] is [17] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.452.825 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libdvpp_engine.so] is [2] micro second. [INFO] ATRACE(489597,python):2025-04-24-09:46:13.458.637 [tracer_mgr_operate.c:246](tid:489597) destroy object FE_Global_Trace, exitSave(false). [INFO] ATRACE(489597,python):2025-04-24-09:46:13.458.653 [tracer_mgr_operate.c:246](tid:489597) destroy object FE_Statistics_Trace, exitSave(false). [INFO] GE(489597,python):2025-04-24-09:46:13.458.668 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libfe.so] is [5835] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.458.682 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libffts.so] is [4] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.458.696 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libge_local_engine.so] is [6] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.458.719 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libhcom_graph_adaptor.so] is [14] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.458.748 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libhost_cpu_engine.so] is [21] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.458.767 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [librts_engine.so] is [11] micro second. [ERROR] GE(489597,python):2025-04-24-09:46:13.458.839 [gelib.cc:176]489597 Initialize: ErrorNo: 1343250441(There is no valid so about OpsKernelInfoStore or GraphOptimizer.) [INIT][OPS_KER][Init][GeLib]GeLib initial failed. [INFO] GE(489597,python):2025-04-24-09:46:13.458.887 [ge_api.cc:252]489597 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::GELibInitialize is [2659607] micro second. [ERROR] GE(489597,python):2025-04-24-09:46:13.458.910 [ge_api.cc:254]489597 GEInitializeImpl: ErrorNo: 1343229953(GEInitialize Failed.) [INIT][OPS_KER][Init][GELib]Failed, error code = 1343250441 [ERROR] GE(489597,python):2025-04-24-09:46:13.458.955 [ge_api.cc:336]489597 GEInitialize: ErrorNo: 4294967295(failed) [INIT][OPS_KER][GEInit] GEInitialize failed, error code:4294967295. [ERROR] ASCENDCL(489597,python):2025-04-24-09:46:13.458.987 [local_compiler.cpp:76]489597 Init: [INIT][OPS_KER][Initialize][Ge]GEInitialize failed. ge result = 4294967295 [ERROR] ASCENDCL(489597,python):2025-04-24-09:46:13.459.021 [op_compile_service.cpp:73]489597 SetCompileStrategy: [INIT][OPS_KER][Init][Compiler]Init compiler failed [ERROR] ASCENDCL(489597,python):2025-04-24-09:46:13.459.059 [op_compile_processor.cpp:67]489597 Init: [INIT][OPS_KER][Set][Options]OpCompileProcessor init failed! [ERROR] ASCENDCL(489597,python):2025-04-24-09:46:13.459.082 [op_compile_processor.cpp:111]489597 SetCompileOpt: [INIT][OPS_KER][Init][Env]init env failed! [INFO] RUNTIME(489597,python):2025-04-24-09:46:13.459.241 [api_impl.cc:6836] 489597 PeekLastErr: level=0 err=0. [INFO] RUNTIME(489597,python):2025-04-24-09:46:13.459.251 [api_impl.cc:6836] 489597 PeekLastErr: level=0 err=0. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/utils/backend_registration.py", line 147, in wrap_tensor_to device_idx = _normalization_device(custom_backend_name, device) File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/utils/backend_registration.py", line 97, in _normalization_device return _get_current_device_index() File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/utils/backend_registration.py", line 91, in _get_current_device_index return getattr(getattr(torch, custom_backend_name), _get_device_index)() File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch_npu/npu/utils.py", line 62, in current_device torch_npu.npu._lazy_init() File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch_npu/npu/__init__.py", line 215, in _lazy_init torch_npu._C._npu_init() RuntimeError: Initialize:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:247 NPU function error: at_npu::native::AclSetCompileopt(aclCompileOpt::ACL_PRECISION_MODE, precision_mode), error code is 500001 [ERROR] 2025-04-24-09:46:13 (PID:489597, Device:0, RankID:-1) ERR00100 PTA call acl api failed [Error]: The internal ACL of the system is incorrect. Rectify the fault based on the error information in the ascend log. EC0010: [PID: 489597] 2025-04-24-09:46:13.434.298 Failed to import Python module [ModuleNotFoundError: No module named 'attr'.]. Solution: Check that all required components are properly installed and the specified Python path matches the Python installation directory. (If the path does not match the directory, run set_env.sh in the installation package.) TraceBack (most recent call last): AOE Failed to call InitCannKB[FUNC:Initialize][FILE:python_adapter_manager.cc][LINE:47] Failed to initialize TeConfigInfo. [GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeTeFusion][FILE:tbe_op_store_adapter.cc][LINE:1816] [GraphOpt][InitializeInner][InitTeFusion]: Failed to initialize TeFusion.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1783] [SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79] [SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120] [FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:115] PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:82] OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:249] GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:177] GEInitialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:337] [Initialize][Ge]GEInitialize failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] [Init][Compiler]Init compiler failed[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145] [Set][Options]OpCompileProcessor init failed![FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145] 请根据自己的运行环境参考以下方式搜集日志信息,如果涉及到算子开发相关的问题,建议也提供UT/ST测试和单算子集成测试相关的日志。 日志提供方式: 将日志打包后作为附件上传。若日志大小超出附件限制,则可上传至外部网盘后提供链接。 获取方法请参考wiki: https://gitee.com/ascend/modelzoo/wikis/%E5%A6%82%E4%BD%95%E8%8E%B7%E5%8F%96%E6%97%A5%E5%BF%97%E5%92%8C%E8%AE%A1%E7%AE%97%E5%9B%BE?sort_id=4097825
一、问题现象(附报错日志上下文): 测试torch_npu是否安装成功的时候报错 二、软件版本: -- CANN 版本 (e.g., CANN 3.0.x,5.x.x): 8.0 --Tensorflow/Pytorch/MindSpore 版本:torch==2.4.0 torch-npu==2.4.0.post2 --Python 版本 (e.g., Python 3.7.5): 3.10.12 -- MindStudio版本 (e.g., MindStudio 2.0.0 (beta3)): --操作系统版本 (e.g., Ubuntu 18.04):22.04.5 LTS 三、测试步骤: >>> import torch >>> import torch_npu [EVENT] PROFILING(489597,python):2025-04-24-09:46:03.953.629 [msprof_callback_impl.cpp:336] >>> (tid:489597) Started to register profiling ctrl callback. [EVENT] PROFILING(489597,python):2025-04-24-09:46:03.953.901 [msprof_callback_impl.cpp:343] >>> (tid:489597) Started to register profiling hash id callback. [INFO] PROFILING(489597,python):2025-04-24-09:46:03.953.917 [prof_atls_plugin.cpp:117] (tid:489597) RegisterProfileCallback, callback type is 7 [EVENT] PROFILING(489597,python):2025-04-24-09:46:03.953.923 [msprof_callback_impl.cpp:350] >>> (tid:489597) Started to register profiling enable host freq callback. [INFO] PROFILING(489597,python):2025-04-24-09:46:03.953.930 [prof_atls_plugin.cpp:117] (tid:489597) RegisterProfileCallback, callback type is 8 [INFO] ATRACE(489597,python):2025-04-24-09:46:03.955.706 [trace_attr.c:188](tid:489597) attr init success, timeout=0ms. [INFO] ATRACE(489597,python):2025-04-24-09:46:03.955.747 [trace_recorder.c:121](tid:489597) use root path: /root/ascend [INFO] RUNTIME(489597,python):2025-04-24-09:46:03.963.853 [task_fail_callback_manager.cc:52] 489597 TaskFailCallBackManager: Constructor. [INFO] HCCL(489597,python):2025-04-24-09:46:04.114.775 [adapter_rts.cc:2646][489597][adapter_rts.cc][CallBackInitRts] g_deviceType [6] g_deviceLogicId [-1] g_devicePhyId [-1] [INFO] RUNTIME(489597,python):2025-04-24-09:46:04.329.275 [runtime.cc:5471] 489597 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set [INFO] PROFILING(489597,python):2025-04-24-09:46:04.332.664 [prof_atls_plugin.cpp:210] (tid:489597) Module[7] register callback of ctrl handle. /root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch_npu/__init__.py:248: UserWarning: On the interactive interface, the value of TASK_QUEUE_ENABLE is set to 0 by default. Do not set it to 1 to prevent some unknown errors warnings.warn("On the interactive interface, the value of TASK_QUEUE_ENABLE is set to 0 by default. \ >>> x = torch.randn(2, 2).npu() 四、日志信息: [INFO] IDEDD(489597,python):2025-04-24-09:46:09.348.314 [adx_dump_record.cpp:623][tid:489747] start dump thread, remote dump record temp path : /root/. [INFO] IDEDD(489597,python):2025-04-24-09:46:09.348.455 [adx_server_manager.cpp:203][tid:489748] Run Server(0) Process [INFO] IDEDD(489597,python):2025-04-24-09:46:09.348.504 [hdc_comm_opt.cpp:59][tid:489748] open device[0] server[13] [INFO] DRV(489597,python):2025-04-24-09:46:09.348.620 [drv_log_user.c:623][ascend][curpid:489597,489748][drv][hdc][share_log_read_in_single_module]Server create success.(dev_id=0; fid=0; service_type="service_dump"; pid=386088) [INFO] IDEDD(489597,python):2025-04-24-09:46:09.348.635 [hdc_api.cpp:119][tid:489748] logDevId 0 create HDC server successfully [INFO] IDEDD(489597,python):2025-04-24-09:46:09.348.679 [hdc_comm_opt.cpp:59][tid:489748] open device[1] server[13] [INFO] DRV(489597,python):2025-04-24-09:46:09.348.820 [drv_log_user.c:623][ascend][curpid:489597,489748][drv][hdc][share_log_read_in_single_module]Server create success.(dev_id=1; fid=0; service_type="service_dump"; pid=386088) [INFO] IDEDD(489597,python):2025-04-24-09:46:09.348.828 [hdc_api.cpp:119][tid:489748] logDevId 1 create HDC server successfully [INFO] PROFILING(489597,python):2025-04-24-09:46:09.351.651 [prof_atls_plugin.cpp:210] (tid:489597) Module[48] register callback of ctrl handle. [INFO] PROFILING(489597,python):2025-04-24-09:46:09.351.675 [prof_atls_plugin.cpp:210] (tid:489597) Module[45] register callback of ctrl handle. [INFO] GE(489597,python):2025-04-24-09:46:09.616.680 [op_tiling_manager.cc:109]489597 ~FuncPerfScope:[GEPERFTRACE] The time cost of OpTilingManager::LoadSo is [264987] micro second. [INFO] PROFILING(489597,python):2025-04-24-09:46:09.634.604 [prof_atls_plugin.cpp:210] (tid:489597) Module[6] register callback of ctrl handle. [EVENT] PROFILING(489597,python):2025-04-24-09:46:10.314.474 [msprof_callback_impl.cpp:89] >>> (tid:489597) MsprofCtrlCallback called, type: 255 [EVENT] PROFILING(489597,python):2025-04-24-09:46:10.314.865 [ai_drv_dev_api.cpp:333] >>> (tid:489597) Succeeded to DrvGetApiVersion version: 0x72317 [INFO] ATRACE(489597,python):2025-04-24-09:46:10.317.241 [atrace_client_core.c:160](tid:489597) start to receive device[0] trace log. [INFO] TDT(489597,python):2025-04-24-09:46:10.317.326 [client_manager.cpp:433][IsHostEnvironment][tid:489597] [TsdClient] logicDeviceId is [0], hostaicpunum[0] [INFO] TDT(489597,python):2025-04-24-09:46:10.317.338 [client_manager.cpp:462][GetClientRunMode][tid:489597] runningMode:0 [INFO] TDT(489597,python):2025-04-24-09:46:10.317.344 [client_manager.cpp:126][GetInstance][tid:489597] [ClientManager] Current mode:1 [INFO] TDT(489597,python):2025-04-24-09:46:10.317.366 [process_mode_manager.cpp:415][GetLogLevel][tid:489597] [TsdClient] get ASCEND_GLOBAL_LOG_LEVEL [] ASCEND_GLOBAL_EVENT_ENABLE [] ASCEND_MODULE_LOG_LEVEL [] [INFO] TDT(489597,python):2025-04-24-09:46:10.317.461 [process_mode_manager.cpp:134][OpenProcess][tid:489597] [ProcessModeManager] enter into open process deviceId[0] rankSize[0] [INFO] TDT(489597,python):2025-04-24-09:46:10.317.962 [process_mode_manager.cpp:545][InitTsdClient][tid:489597] [TsdClient] deviceId[0] begin to init hdc client [INFO] TDT(489597,python):2025-04-24-09:46:10.318.060 [version_verify.cpp:36][SetVersionInfo][tid:489597] VersionVerify: send client version to server [INFO] TDT(489597,python):2025-04-24-09:46:10.318.077 [version_verify.cpp:52][SetVersionInfo][tid:489597] send feature_info:{msg_type:35, features:{check before send aicpu package}} [INFO] TDT(489597,python):2025-04-24-09:46:10.318.085 [version_verify.cpp:52][SetVersionInfo][tid:489597] send feature_info:{msg_type:37, features:{check before send open qs message}} [INFO] TDT(489597,python):2025-04-24-09:46:10.318.093 [version_verify.cpp:52][SetVersionInfo][tid:489597] send feature_info:{msg_type:64, features:{get check code retry}} [INFO] TDT(489597,python):2025-04-24-09:46:10.318.451 [version_verify.cpp:68][PeerVersionCheck][tid:489597] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(489597,python):2025-04-24-09:46:10.318.466 [version_verify.cpp:89][ParseVersionInfo][tid:489597] VersionVerify: pass client version info success [INFO] TDT(489597,python):2025-04-24-09:46:10.318.472 [hdc_client.cpp:275][CheckHdcConnection][tid:489597] Service[2] create hdc success [INFO] TDT(489597,python):2025-04-24-09:46:10.318.488 [version_verify.cpp:122][SpecialFeatureCheck][tid:489597] VersionVerify: new type[35], supported [INFO] TDT(489597,python):2025-04-24-09:46:10.318.529 [process_mode_manager.cpp:922][GetDeviceCheckCodeOnce][tid:489597] [TsdClient][deviceId=0] [sessionId=1] wait package info respond [INFO] TDT(489597,python):2025-04-24-09:46:10.319.529 [version_verify.cpp:122][SpecialFeatureCheck][tid:489597] VersionVerify: new type[64], supported [INFO] TDT(489597,python):2025-04-24-09:46:10.319.597 [process_mode_manager.cpp:545][InitTsdClient][tid:489597] [TsdClient] deviceId[0] begin to init hdc client [INFO] TDT(489597,python):2025-04-24-09:46:10.319.676 [version_verify.cpp:36][SetVersionInfo][tid:489597] VersionVerify: send client version to server [INFO] TDT(489597,python):2025-04-24-09:46:10.319.687 [version_verify.cpp:52][SetVersionInfo][tid:489597] send feature_info:{msg_type:35, features:{check before send aicpu package}} [INFO] TDT(489597,python):2025-04-24-09:46:10.319.694 [version_verify.cpp:52][SetVersionInfo][tid:489597] send feature_info:{msg_type:37, features:{check before send open qs message}} [INFO] TDT(489597,python):2025-04-24-09:46:10.319.702 [version_verify.cpp:52][SetVersionInfo][tid:489597] send feature_info:{msg_type:64, features:{get check code retry}} [INFO] TDT(489597,python):2025-04-24-09:46:10.319.892 [version_verify.cpp:68][PeerVersionCheck][tid:489597] VersionVerify: Check client version info, server[1230], client[1230] [INFO] TDT(489597,python):2025-04-24-09:46:10.319.903 [version_verify.cpp:89][ParseVersionInfo][tid:489597] VersionVerify: pass client version info success [INFO] TDT(489597,python):2025-04-24-09:46:10.319.908 [hdc_client.cpp:275][CheckHdcConnection][tid:489597] Service[2] create hdc success [INFO] TDT(489597,python):2025-04-24-09:46:10.319.919 [process_mode_manager.cpp:600][ConstructOpenMsg][tid:489597] [TsdClient] tsd get process sign successfully, procpid[386088] signSize[48] [INFO] TDT(489597,python):2025-04-24-09:46:10.319.930 [version_verify.cpp:114][SpecialFeatureCheck][tid:489597] VersionVerify: previous type[6], supported [INFO] TDT(489597,python):2025-04-24-09:46:10.319.948 [process_mode_manager.cpp:151][OpenProcess][tid:489597] [ProcessModeManager] deviceId[0] sessionId[1] rankSize[0], wait sub process start respond [INFO] DRV(489597,python):2025-04-24-09:46:10.489.123 [devmm_svm.c:99][ascend][curpid:489597,489597][drv][devmm][devmm_setup_device]DrvMemDeviceOpen. (devid=0) [INFO] DRV(489597,python):2025-04-24-09:46:10.491.273 [devmm_virt_interface.c:394][ascend][curpid:489597,489597][drv][devmm][devmm_ioctl_init_process]Init_process details. (hostpid=489597) [INFO] DRV(489597,python):2025-04-24-09:46:10.491.590 [drv_log_user.c:623][ascend][curpid:489597,489597][drv][devmm][share_log_read_in_single_module]Setup device succeeded. (logical_devid=0; devid=0; vfid=0; hostpid=386088; devpid=2615) [INFO] DRV(489597,python):2025-04-24-09:46:10.491.606 [devmm_svm.c:84][ascend][curpid:489597,489597][drv][devmm][devmm_set_device_info]Device info. (devid=0; dvpp_size=17179869184; support_bar_mem=1; support_dev_read_only=1; support_dev_mem_map_host=1; support_bar_huge_mem=1; is_alloced_by_malloc=0; host_rw_dev_ro=1; double_pgtable_offset=228698418577408) [INFO] TDT(489597,python):2025-04-24-09:46:10.491.873 [stub_process_mode_nowin.cpp:60][ProcessQueueForMdc][tid:489597] [TsdClient] it is unnecessary of current mode[0] to grant queue auth to aicpusd [INFO] TDT(489597,python):2025-04-24-09:46:10.491.885 [stub_process_mode_nowin.cpp:98][OpenInHost][tid:489597] enter into OpenInHost deviceid[0] [INFO] TDT(489597,python):2025-04-24-09:46:10.491.908 [client_manager.cpp:433][IsHostEnvironment][tid:489597] [TsdClient] logicDeviceId is [0], hostaicpunum[0] [INFO] TDT(489597,python):2025-04-24-09:46:10.491.915 [stub_process_mode_nowin.cpp:102][OpenInHost][tid:489597] host cpu not support [INFO] TDT(489597,python):2025-04-24-09:46:10.491.920 [process_mode_manager.cpp:185][OpenProcess][tid:489597] [TsdClient][deviceId=0] [sessionId=1] start hccp and computer process success [INFO] DRV(489597,python):2025-04-24-09:46:10.496.002 [devmm_svm.c:99][ascend][curpid:489597,489597][drv][devmm][devmm_setup_device]DrvMemDeviceOpen. (devid=0) [INFO] DRV(489597,python):2025-04-24-09:46:10.496.024 [devmm_svm.c:84][ascend][curpid:489597,489597][drv][devmm][devmm_set_device_info]Device info. (devid=0; dvpp_size=17179869184; support_bar_mem=1; support_dev_read_only=1; support_dev_mem_map_host=1; support_bar_huge_mem=1; is_alloced_by_malloc=0; host_rw_dev_ro=1; double_pgtable_offset=228698418577408) [INFO] RUNTIME(489597,python):2025-04-24-09:46:10.496.100 [raw_device.cc:237] 489597 Init: isAddrFlat:0 [INFO] TDT(489597,python):2025-04-24-09:46:10.497.130 [tsd_client.cpp:172][TsdCapabilityGet][tid:489597] TsdCapabilityGet Begin. [INFO] TDT(489597,python):2025-04-24-09:46:10.497.143 [process_mode_manager.cpp:1351][CapabilityGet][tid:489597] [ProcessModeManager] enter into CapabilityGet process deviceId[0] type[0] [INFO] TDT(489597,python):2025-04-24-09:46:10.497.155 [version_verify.cpp:114][SpecialFeatureCheck][tid:489597] VersionVerify: previous type[38], supported [INFO] TDT(489597,python):2025-04-24-09:46:10.497.183 [process_mode_manager.cpp:1374][CapabilityGet][tid:489597] [TsdClient][deviceId=0] [sessionId=1] [type=0]send capability success [INFO] TDT(489597,python):2025-04-24-09:46:10.498.312 [process_mode_manager.cpp:829][PidQosMsgProc][tid:489597] [TsdClient] PidQosMsgProc recvMsg realDeviceId[0] msgType[39] localDevId[0] rspCode[0],pidqos[1] [INFO] TDT(489597,python):2025-04-24-09:46:10.498.322 [process_mode_manager.cpp:1379][CapabilityGet][tid:489597] [TsdClient][logicDeviceId_=0][type=0][pidQos=1][ret=0]recv capability respond finish [INFO] RUNTIME(489597,python):2025-04-24-09:46:10.498.334 [engine.cc:76] 489597 Engine: Constructor. [INFO] RUNTIME(489597,python):2025-04-24-09:46:10.498.340 [stars_engine.cc:41] 489597 StarsEngine: Constructor. [INFO] RUNTIME(489597,python):2025-04-24-09:46:10.518.655 [npu_driver.cc:5784] 489773 GetDeviceStatus: GetDeviceStatus status=1. [INFO] ATRACE(489597,python):2025-04-24-09:46:10.522.164 [tracer_mgr_operate.c:64](tid:489597) create object RUNTIME_ATRACE_DEV0_TS0 success, exitSave(false). [INFO] TDT(489597,python):2025-04-24-09:46:10.522.197 [client_manager.cpp:195][SetProfilingCallback][tid:489597] [TsdClient] set profiling callback success [INFO] GE(489597,python):2025-04-24-09:46:10.529.705 [ge_api.cc:174][status:INIT]489597 GEInitializeImpl:GEInitialize start [INFO] GE(489597,python):2025-04-24-09:46:10.529.726 [ge_api.cc:188]489597 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::GEAPICheckSupportedGlobalOptions is [4] micro second. [INFO] GE(489597,python):2025-04-24-09:46:10.556.068 [ge_api.cc:203]489597 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::CheckOptionsValid is [75] micro second. [INFO] GE(489597,python):2025-04-24-09:46:10.556.581 [ge_api.cc:217]489597 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::ManagerInitialize is [2] micro second. [EVENT] PROFILING(489597,python):2025-04-24-09:46:10.699.533 [msprof_callback_impl.cpp:89] >>> (tid:489597) MsprofCtrlCallback called, type: 255 [EVENT] PROFILING(489597,python):2025-04-24-09:46:10.699.576 [ai_drv_dev_api.cpp:333] >>> (tid:489597) Succeeded to DrvGetApiVersion version: 0x72317 [INFO] GE(489597,python):2025-04-24-09:46:10.699.655 [ge_api.cc:236]489597 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::InitProfiling is [144] micro second. [INFO] GE(489597,python):2025-04-24-09:46:10.799.231 [ge_api.cc:246]489597 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::InitPreparation is [99452] micro second. [INFO] GE(489597,python):2025-04-24-09:46:10.799.282 [gelib.cc:91]489597 Initialize:[GEPERFTRACE] GE Init Start [INFO] GE(489597,python):2025-04-24-09:46:11.258.433 [gelib.cc:172]489597 Initialize:[GEPERFTRACE] The time cost of GELib::InitializePrepare is [459139] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.258.716 [gelib.cc:327]489597 SystemInitialize:Infer init GELib success, device id :-1 [INFO] GE(489597,python):2025-04-24-09:46:11.258.728 [gelib.cc:199]489597 InnerInitialize:[GEPERFTRACE] The time cost of InnerInitialize::SystemInitialize is [17] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.262.878 [process_node_engine_manager.cc:63]489597 Initialize:[GEPERFTRACE] The time cost of ProcessNodeEngineManager::Initialize[HOST_CPU] is [1] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.262.897 [process_node_engine_manager.cc:63]489597 Initialize:[GEPERFTRACE] The time cost of ProcessNodeEngineManager::Initialize[NPU] is [1] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.262.905 [process_node_engine_manager.cc:63]489597 Initialize:[GEPERFTRACE] The time cost of ProcessNodeEngineManager::Initialize[PS] is [1] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.262.911 [process_node_engine_manager.cc:63]489597 Initialize:[GEPERFTRACE] The time cost of ProcessNodeEngineManager::Initialize[UDF] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.262.920 [gelib.cc:223]489597 InnerInitialize:[GEPERFTRACE] The time cost of InnerInitialize::ProcessNodeEngine is [3589] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.091 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [GetDNNEngineObjs] in [libengine.so] is [103] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.107 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[AIcoreEngine] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.116 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_HCCL] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.123 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_AICPU] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.130 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_AICPU_ASCEND] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.136 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_AICPU_ASCEND_FFTS_PLUS] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.142 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_AICPU_FFTS_PLUS] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.148 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_DVPP] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.154 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_GE_LOCAL] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.160 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_HOST_CPU] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.166 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_RTS] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.172 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DNN_VM_RTS_FFTS_PLUS] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.177 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[DSAEngine] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.183 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[VectorEngine] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.188 [dnnengine_manager.cc:128]489597 Initialize:[GEPERFTRACE] The time cost of DNNEngineManager::Initialize[ffts_plus] is [0] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.264.900 [gelib.cc:234]489597 InnerInitialize:[GEPERFTRACE] The time cost of InnerInitialize::EngineInitialize is [1972] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.265.921 [op_tiling_manager.cc:109]489597 ~FuncPerfScope:[GEPERFTRACE] The time cost of OpTilingManager::LoadSo is [836] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.595.124 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libfe.so] is [329144] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.597.902 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libge_local_engine.so] is [2659] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.612.642 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/librts_engine.so] is [14676] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.635.677 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libaicpu_ascend_engine.so] is [22954] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.639.663 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libhost_cpu_engine.so] is [3898] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.656.661 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libaicpu_tf_engine.so] is [16933] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.689.378 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libffts.so] is [32630] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.694.137 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/plugin/opskernel/libdvpp_engine.so] is [4671] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.736.822 [plugin_manager.cc:705]489597 LoadSoWithFlags:[GEPERFTRACE] The time cost of PluginManager::Dlopen[/usr/local/Ascend/ascend-toolkit/8.0.0/aarch64-linux/lib64/libhcom_graph_adaptor.so] is [42622] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.736.864 [ops_kernel_manager.cc:78]489597 Initialize:[GEPERFTRACE] The time cost of OpsKernelManager::LoadPluginManagerSo is [470926] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.737.234 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Initialize] in [libaicpu_ascend_engine.so] is [348] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.737.421 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Initialize] in [libaicpu_tf_engine.so] is [172] micro second. [INFO] GE(489597,python):2025-04-24-09:46:11.737.452 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Initialize] in [libdvpp_engine.so] is [18] micro second. [INFO] ATRACE(489597,python):2025-04-24-09:46:11.737.565 [tracer_mgr_operate.c:64](tid:489597) create object FE_Global_Trace success, exitSave(false). [INFO] ATRACE(489597,python):2025-04-24-09:46:11.737.642 [tracer_mgr_operate.c:64](tid:489597) create object FE_Statistics_Trace success, exitSave(false). [INFO] TUNE(489597,python):2025-04-24-09:46:11.742.640 [cann_kb_pyfunc_mgr.cpp:72][CANNKB][Tid:489597]"CannKbPyfuncMgr: Enter PyObjectInit, reference_ is 0!" [INFO] TUNE(489597,python):2025-04-24-09:46:11.742.671 [handle_manager.cpp:115][CANNKB][Tid:489597]"Start to run init functions to load dynamic python lib!" [INFO] TUNE(489597,python):2025-04-24-09:46:11.742.732 [handle_manager.cpp:407][CANNKB][Tid:489597]"Init functions of loading dynamic python lib end!" [INFO] TUNE(489597,python):2025-04-24-09:46:11.742.740 [cann_kb_pyfunc_mgr.cpp:24][CANNKB][Tid:489597]"CANN_KB_Py has already been initialized." [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.155 [pywrapper.cpp:94][CANNKB][Tid:489597]"Traceback (most recent call last): " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.186 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/__init__.py", line 98, in <module> from . import dsl " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.194 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/__init__.py", line 20, in <module> from .api import ceil " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.200 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/api.py", line 27, in <module> from .compute import cast " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.206 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/compute/cast.py", line 34, in <module> from tbe.dsl.unify_schedule.util import is_v220 " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.213 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/unify_schedule/__init__.py", line 35, in <module> from . import vector " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.218 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/unify_schedule/vector/__init__.py", line 30, in <module> from . import sparse_apply " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.225 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/unify_schedule/vector/sparse_apply/__init__.py", line 20, in <module> from . import sparse_apply_schedule " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.231 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/dsl/unify_schedule/vector/sparse_apply/sparse_apply_schedule.py", line 23, in <module> import te.platform as tbe_platform " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.236 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/te/platform/__init__.py", line 125, in <module> from .cce_util import get_type_bits " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.242 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/te/platform/cce_util.py", line 34, in <module> from tvm.topi.utils import equal_const_int " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.247 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/8.0.0/python/site-packages/tbe/tvm/topi/__init__.py", line 50, in <module> from . import nn " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.252 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/8.0.0/python/site-packages/tbe/tvm/topi/nn/__init__.py", line 35, in <module> from .conv3d_transpose import * " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.257 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/8.0.0/python/site-packages/tbe/tvm/topi/nn/conv3d_transpose.py", line 21, in <module> from tvm import relay " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.263 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/8.0.0/python/site-packages/tbe/tvm/relay/__init__.py", line 64, in <module> from .transform import memory_plan " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.268 [pywrapper.cpp:94][CANNKB][Tid:489597]" File "/usr/local/Ascend/ascend-toolkit/8.0.0/python/site-packages/tbe/tvm/relay/transform/memory_plan.py", line 23, in <module> import attr " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.274 [pywrapper.cpp:94][CANNKB][Tid:489597]"ModuleNotFoundError: No module named 'attr' " [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.513 [cann_kb_pyfunc_mgr.cpp:98][CANNKB][Tid:489597]"PyObjectInit: Import repository_manager_log error" [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.528 [py_interface.cpp:28][CANNKB][Tid:489597]"PyObjectInit of CannKbPyfuncMgr Error!" [ERROR] TUNE(489597,python):2025-04-24-09:46:13.434.535 [cann_kb_api.cpp:24][CANNKB][Tid:489597]"Run PyInterfaceInit Error!" [ERROR] TEFUSION(489597,python):2025-04-24-09:46:13.434.547 [python_adapter_manager.cc:94]489597 InitCannKB call CannKbInit failed. res = [3]. init params: [Ascend910B2, 24, ]. [ERROR] TEFUSION(489597,python):2025-04-24-09:46:13.434.593 [python_adapter_manager.cc:47]489597 Initialize AOE Failed to call InitCannKB [ERROR] FE(489597,python):2025-04-24-09:46:13.434.630 [tbe_op_store_adapter.cc:1816]489597 InitializeTeFusion:"[GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe." [ERROR] FE(489597,python):2025-04-24-09:46:13.434.679 [tbe_op_store_adapter.cc:1783]489597 InitializeInner:"[GraphOpt][InitializeInner][InitTeFusion]: Failed to initialize TeFusion." [ERROR] FE(489597,python):2025-04-24-09:46:13.434.702 [op_store_adapter_manager.cc:79]489597 InitializeAdapter:"[SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295]" [ERROR] FE(489597,python):2025-04-24-09:46:13.434.787 [op_store_adapter_manager.cc:120]489597 Initialize:"[SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom]." [ERROR] FE(489597,python):2025-04-24-09:46:13.434.808 [fusion_manager.cc:115]489597 Initialize:"[FusionMngr][Init] Op store adapter manager init failed." [INFO] GE(489597,python):2025-04-24-09:46:13.434.896 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Initialize] in [libfe.so] is [1697432] micro second. [ERROR] GE(489597,python):2025-04-24-09:46:13.434.990 [ops_kernel_manager.cc:82]489597 Initialize: ErrorNo: 1343250441(There is no valid so about OpsKernelInfoStore or GraphOptimizer.) [INIT][OPS_KER]PluginManager InvokeAll failed. [INFO] GE(489597,python):2025-04-24-09:46:13.435.005 [gelib.cc:246]489597 InnerInitialize:[GEPERFTRACE] The time cost of InnerInitialize::OpsManagerInitialize is [2170091] micro second. [ERROR] GE(489597,python):2025-04-24-09:46:13.435.016 [gelib.cc:248]489597 InnerInitialize: ErrorNo: 1343250441(There is no valid so about OpsKernelInfoStore or GraphOptimizer.) [INIT][OPS_KER][Init][OpsManager]GE ops manager initial failed. [INFO] GE(489597,python):2025-04-24-09:46:13.452.771 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libaicpu_ascend_engine.so] is [17707] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.452.811 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libaicpu_tf_engine.so] is [17] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.452.825 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libdvpp_engine.so] is [2] micro second. [INFO] ATRACE(489597,python):2025-04-24-09:46:13.458.637 [tracer_mgr_operate.c:246](tid:489597) destroy object FE_Global_Trace, exitSave(false). [INFO] ATRACE(489597,python):2025-04-24-09:46:13.458.653 [tracer_mgr_operate.c:246](tid:489597) destroy object FE_Statistics_Trace, exitSave(false). [INFO] GE(489597,python):2025-04-24-09:46:13.458.668 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libfe.so] is [5835] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.458.682 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libffts.so] is [4] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.458.696 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libge_local_engine.so] is [6] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.458.719 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libhcom_graph_adaptor.so] is [14] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.458.748 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [libhost_cpu_engine.so] is [21] micro second. [INFO] GE(489597,python):2025-04-24-09:46:13.458.767 [dnnengine_manager.cc:41]489597 ~InvokeFuncPerfRecorder:[GEPERFTRACE] The time cost of InvokeAll [Finalize] in [librts_engine.so] is [11] micro second. [ERROR] GE(489597,python):2025-04-24-09:46:13.458.839 [gelib.cc:176]489597 Initialize: ErrorNo: 1343250441(There is no valid so about OpsKernelInfoStore or GraphOptimizer.) [INIT][OPS_KER][Init][GeLib]GeLib initial failed. [INFO] GE(489597,python):2025-04-24-09:46:13.458.887 [ge_api.cc:252]489597 GEInitializeImpl:[GEPERFTRACE] The time cost of GEInitialize::GELibInitialize is [2659607] micro second. [ERROR] GE(489597,python):2025-04-24-09:46:13.458.910 [ge_api.cc:254]489597 GEInitializeImpl: ErrorNo: 1343229953(GEInitialize Failed.) [INIT][OPS_KER][Init][GELib]Failed, error code = 1343250441 [ERROR] GE(489597,python):2025-04-24-09:46:13.458.955 [ge_api.cc:336]489597 GEInitialize: ErrorNo: 4294967295(failed) [INIT][OPS_KER][GEInit] GEInitialize failed, error code:4294967295. [ERROR] ASCENDCL(489597,python):2025-04-24-09:46:13.458.987 [local_compiler.cpp:76]489597 Init: [INIT][OPS_KER][Initialize][Ge]GEInitialize failed. ge result = 4294967295 [ERROR] ASCENDCL(489597,python):2025-04-24-09:46:13.459.021 [op_compile_service.cpp:73]489597 SetCompileStrategy: [INIT][OPS_KER][Init][Compiler]Init compiler failed [ERROR] ASCENDCL(489597,python):2025-04-24-09:46:13.459.059 [op_compile_processor.cpp:67]489597 Init: [INIT][OPS_KER][Set][Options]OpCompileProcessor init failed! [ERROR] ASCENDCL(489597,python):2025-04-24-09:46:13.459.082 [op_compile_processor.cpp:111]489597 SetCompileOpt: [INIT][OPS_KER][Init][Env]init env failed! [INFO] RUNTIME(489597,python):2025-04-24-09:46:13.459.241 [api_impl.cc:6836] 489597 PeekLastErr: level=0 err=0. [INFO] RUNTIME(489597,python):2025-04-24-09:46:13.459.251 [api_impl.cc:6836] 489597 PeekLastErr: level=0 err=0. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/utils/backend_registration.py", line 147, in wrap_tensor_to device_idx = _normalization_device(custom_backend_name, device) File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/utils/backend_registration.py", line 97, in _normalization_device return _get_current_device_index() File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch/utils/backend_registration.py", line 91, in _get_current_device_index return getattr(getattr(torch, custom_backend_name), _get_device_index)() File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch_npu/npu/utils.py", line 62, in current_device torch_npu.npu._lazy_init() File "/root/miniconda3/envs/vllm/lib/python3.10/site-packages/torch_npu/npu/__init__.py", line 215, in _lazy_init torch_npu._C._npu_init() RuntimeError: Initialize:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:247 NPU function error: at_npu::native::AclSetCompileopt(aclCompileOpt::ACL_PRECISION_MODE, precision_mode), error code is 500001 [ERROR] 2025-04-24-09:46:13 (PID:489597, Device:0, RankID:-1) ERR00100 PTA call acl api failed [Error]: The internal ACL of the system is incorrect. Rectify the fault based on the error information in the ascend log. EC0010: [PID: 489597] 2025-04-24-09:46:13.434.298 Failed to import Python module [ModuleNotFoundError: No module named 'attr'.]. Solution: Check that all required components are properly installed and the specified Python path matches the Python installation directory. (If the path does not match the directory, run set_env.sh in the installation package.) TraceBack (most recent call last): AOE Failed to call InitCannKB[FUNC:Initialize][FILE:python_adapter_manager.cc][LINE:47] Failed to initialize TeConfigInfo. [GraphOpt][InitializeInner][InitTbeFunc] Failed to init tbe.[FUNC:InitializeTeFusion][FILE:tbe_op_store_adapter.cc][LINE:1816] [GraphOpt][InitializeInner][InitTeFusion]: Failed to initialize TeFusion.[FUNC:InitializeInner][FILE:tbe_op_store_adapter.cc][LINE:1783] [SubGraphOpt][PreCompileOp][InitAdapter] InitializeAdapter adapter [tbe_op_adapter] failed! Ret [4294967295][FUNC:InitializeAdapter][FILE:op_store_adapter_manager.cc][LINE:79] [SubGraphOpt][PreCompileOp][Init] Initialize op store adapter failed, OpsStoreName[tbe-custom].[FUNC:Initialize][FILE:op_store_adapter_manager.cc][LINE:120] [FusionMngr][Init] Op store adapter manager init failed.[FUNC:Initialize][FILE:fusion_manager.cc][LINE:115] PluginManager InvokeAll failed.[FUNC:Initialize][FILE:ops_kernel_manager.cc][LINE:82] OpsManager initialize failed.[FUNC:InnerInitialize][FILE:gelib.cc][LINE:249] GELib::InnerInitialize failed.[FUNC:Initialize][FILE:gelib.cc][LINE:177] GEInitialize failed.[FUNC:GEInitialize][FILE:ge_api.cc][LINE:337] [Initialize][Ge]GEInitialize failed. ge result = 4294967295[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] [Init][Compiler]Init compiler failed[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145] [Set][Options]OpCompileProcessor init failed![FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145] 请根据自己的运行环境参考以下方式搜集日志信息,如果涉及到算子开发相关的问题,建议也提供UT/ST测试和单算子集成测试相关的日志。 日志提供方式: 将日志打包后作为附件上传。若日志大小超出附件限制,则可上传至外部网盘后提供链接。 获取方法请参考wiki: https://gitee.com/ascend/modelzoo/wikis/%E5%A6%82%E4%BD%95%E8%8E%B7%E5%8F%96%E6%97%A5%E5%BF%97%E5%92%8C%E8%AE%A1%E7%AE%97%E5%9B%BE?sort_id=4097825
评论 (
2
)
登录
后才可以发表评论
状态
DONE
TODO
WIP
DONE
CLOSED
REJECTED
负责人
未设置
标签
未设置
项目
未立项任务
未立项任务
里程碑
未关联里程碑
未关联里程碑
Pull Requests
未关联
未关联
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
未关联
分支 (
-
)
标签 (
-
)
开始日期   -   截止日期
-
置顶选项
不置顶
置顶等级:高
置顶等级:中
置顶等级:低
优先级
不指定
严重
主要
次要
不重要
预计工期
(小时)
参与者(2)
Python
1
https://gitee.com/ascend/pytorch.git
git@gitee.com:ascend/pytorch.git
ascend
pytorch
pytorch
点此查找更多帮助
搜索帮助
Git 命令在线学习
如何在 Gitee 导入 GitHub 仓库
Git 仓库基础操作
企业版和社区版功能对比
SSH 公钥设置
如何处理代码冲突
仓库体积过大,如何减小?
如何找回被删除的仓库数据
Gitee 产品配额说明
GitHub仓库快速导入Gitee及同步更新
什么是 Release(发行版)
将 PHP 项目自动发布到 packagist.org
仓库举报
回到顶部
登录提示
该操作需登录 Gitee 帐号,请先登录后再操作。
立即登录
没有帐号,去注册