登录
注册
开源
企业版
高校版
搜索
帮助中心
使用条款
关于我们
开源
企业版
高校版
私有云
模力方舟
AI 队友
登录
注册
Gitee 2025年度个人数据报告已发布，快来看看你的成长👉
代码拉取完成，页面将自动刷新
仓库状态说明
开源项目
>
人工智能
>
机器学习/深度学习
&&
捐赠
捐赠前请先登录
取消
前往登录
扫描微信二维码支付
取消
支付完成
支付提示
将跳转至支付宝完成支付
确定
取消
Watch
不关注
关注所有动态
仅关注版本发行动态
关注但不提醒动态
90
Star
659
Fork
1.5K
Ascend
/
pytorch
暂停
代码
Issues
41
Pull Requests
350
Wiki
统计
流水线
服务
质量分析
Jenkins for Gitee
腾讯云托管
腾讯云 Serverless
悬镜安全
阿里云 SAE
Codeblitz
SBOM
我知道了，不再自动展开
更新失败，请稍后重试！
移除标识
内容风险标识
本任务被
标识为内容中包含有代码安全 Bug 、隐私泄露等敏感信息，仓库外成员不可访问
单机多卡训练出现RuntimeError，error code is 107020.
DONE
#IAVXDS
训练问题
Rs Lee
创建于
2024-10-10 10:57
一、问题现象（附报错日志上下文）： 使用四卡【4,5,6,7】进行分布式训练时报错，详见训练日志报错信息。在训练第一个epo完成并生成checkpoint.0后报错.麻烦看一些，谢谢，下述是训练日志信息 Inner error, see details in Ascend logs.Inner error, see details in Ascend logs.Inner error, see details in Ascend logs.Traceback (most recent call last): File "bin/train_ddp.py", line 118, in <module> main() File "bin/train_ddp.py", line 110, in main dist.barrier() File "/usr/local/python3.8.10/lib/python3.8/site-packages/torch_npu/distributed/distributed_c10d.py", line 2001, in barrier work.wait() RuntimeError: npuSynchronizeDevice:/usr1/01/workspace/j_ctViaEnN/pytorch/torch_npu/csrc/core/npu/NPUStream.cpp:387 NPU error, error code is 107020. 二、软件版本: -- CANN 版本 (7.0.RC1): --Pytorch 版本：'1.11.0'[输入链接说明](http://) --Python 版本 (Python 3.8.10): --操作系统版本 (CentOS Linux 7 (AltArch)): --芯片是910B 三、测试步骤： xxxx 四、日志信息: plog 信息， [EVENT] PROFILING(559979,python3):2024-10-11-18:54:48.134.248 [msprof_callback_impl.cpp:324] >>> (tid:559979) Started to register profiling ctrl callback. [EVENT] PROFILING(559979,python3):2024-10-11-18:54:48.134.401 [msprof_callback_impl.cpp:331] >>> (tid:559979) Started to register profiling hash id callback. [EVENT] PROFILING(559979,python3):2024-10-11-18:54:48.134.408 [prof_atls_plugin.cpp:83] >>> (tid:559979) RegisterProfileCallback, callback type is 7 [EVENT] PROFILING(559979,python3):2024-10-11-18:54:48.134.412 [msprof_callback_impl.cpp:338] >>> (tid:559979) Started to register profiling enable host freq callback. [EVENT] PROFILING(559979,python3):2024-10-11-18:54:48.134.416 [prof_atls_plugin.cpp:83] >>> (tid:559979) RegisterProfileCallback, callback type is 8 [EVENT] RUNTIME(559979,python3):2024-10-11-18:54:49.716.827 [runtime.cc:1010] 559979 GetAicoreNumByLevel: workingDev_=0 [EVENT] RUNTIME(559979,python3):2024-10-11-18:54:49.716.876 [runtime.cc:4125] 559979 GetVisibleDevices: real deviceCnt:8 userDeviceCnt:4 isSetVisibleDev:1 ASCEND_RT_VISIBLE_DEVICES:4,5,6,7 [EVENT] PROFILING(559979,python3):2024-10-11-18:54:49.717.312 [prof_atls_plugin.cpp:154] >>> (tid:559979) Module[7] register callback of ctrl handle. [EVENT] PROFILING(559979,python3):2024-10-11-18:54:51.402.611 [prof_atls_plugin.cpp:154] >>> (tid:559979) Module[48] register callback of ctrl handle. [EVENT] PROFILING(559979,python3):2024-10-11-18:54:51.403.170 [prof_atls_plugin.cpp:154] >>> (tid:559979) Module[45] register callback of ctrl handle. [EVENT] PROFILING(559979,python3):2024-10-11-18:54:51.755.924 [msprof_callback_impl.cpp:79] >>> (tid:559979) MsprofCtrlCallback called, type: 255 [EVENT] PROFILING(559979,python3):2024-10-11-18:54:51.756.223 [ai_drv_dev_api.cpp:333] >>> (tid:559979) Succeeded to DrvGetApiVersion version: 0x71f0d [EVENT] RUNTIME(559979,python3):2024-10-11-18:54:52.716.196 [npu_driver.cc:5134] 561246 GetDeviceStatus: GetDeviceStatus status=1. [TRACE] GE(559979,python3):2024-10-11-18:54:52.720.621 [status:INIT] [ge_api.cc:206]559979 GEInitializeImpl:GEInitialize start [EVENT] PROFILING(559979,python3):2024-10-11-18:54:52.797.506 [msprof_callback_impl.cpp:79] >>> (tid:559979) MsprofCtrlCallback called, type: 255 [EVENT] PROFILING(559979,python3):2024-10-11-18:54:52.797.516 [ai_drv_dev_api.cpp:333] >>> (tid:559979) Succeeded to DrvGetApiVersion version: 0x71f0d [TRACE] GE(559979,python3):2024-10-11-18:54:52.855.381 [status:RUNNING] [ge_api.cc:270]559979 GEInitializeImpl:Initializing environment [EVENT] TUNE(559979,python3):2024-10-11-18:54:53.537.528 [cann_kb_pyfunc_mgr.cpp:72][CANNKB][Tid:559979]"CannKbPyfuncMgr: Enter PyObjectInit, reference_ is 0!" [EVENT] TUNE(559979,python3):2024-10-11-18:54:53.537.890 [handle_manager.cpp:115][CANNKB][Tid:559979]"Start to run init functions to load dynamic python lib!" [EVENT] TUNE(559979,python3):2024-10-11-18:54:53.537.943 [handle_manager.cpp:407][CANNKB][Tid:559979]"Init functions of loading dynamic python lib end!" [EVENT] TUNE(559979,python3):2024-10-11-18:54:53.537.947 [cann_kb_pyfunc_mgr.cpp:24][CANNKB][Tid:559979]"CANN_KB_Py has already been initialized." [EVENT] TUNE(559979,python3):2024-10-11-18:54:56.127.126 [cann_kb_pyfunc_mgr.cpp:117][CANNKB][Tid:559979]"CannKbPyfuncMgr: Run PyObjectInit successfully!" [EVENT] TBE(559979,python3):2024-10-11-18:54:56.365.127 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:312][repository_manager] get_compiler_core_num core_num = [8]. [EVENT] TBE(559979,python3):2024-10-11-18:55:00.326.901 [../../../../../../latest/python/site-packages/te_fusion/parallel_compilation.py:552][init] The time cost of random buffer compile is [743720] micro second. [EVENT] TBE(559979,python3):2024-10-11-18:55:00.863.813 [../../../../../../latest/python/site-packages/te_fusion/parallel_compilation.py:552][init] The time cost of random buffer compile is [510928] micro second. [TRACE] GE(559979,python3):2024-10-11-18:55:03.154.031 [status:STOP] [ge_api.cc:313]559979 GEInitializeImpl:GEInitialize finished [TRACE] GE(559979,python3):2024-10-11-18:55:03.155.305 [status:INIT] [ge_api.cc:206]559979 GEInitializeImpl:GEInitialize start [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.276.265 [ra_host.c:1792]tid:559979,ra_get_ifnum(1792) : Input parameters: phy_id[0], nic_position:[0] [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.286.156 [ra_host.c:1840]tid:559979,ra_get_ifaddrs(1840) : Input parameters: phy_id[0], nic_position:[0], interface num[25] [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.292.754 [ra_host.c:1632]tid:559979,ra_socket_set_white_list_status(1632) : Input parameters: enable[0] [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.292.938 [ra_host.c:248]tid:559979,ra_init(248) : Input parameters: phy_id[4], nic_position:[0] [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.293.221 [rs_ssl.c:1062]tid:559979,rs_ssl_init(1062) : TLS SWITCH (0) [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.293.387 [rs_epoll.c:470]tid:563748,rs_epoll_handle(470) : pthread[epoll_pthread] is alive! [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.293.694 [rs_epoll.c:596]tid:563749,rs_connect_handle(596) : pthread[connect_pthread] is alive! [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.293.956 [rs.c:397]tid:559979,rs_init(397) : rs init success, chip_id[4] [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.293.979 [ra_host.c:409]tid:559979,ra_socket_init_v1(409) : socket init:mode=0 phy_id=4 family=2 ip=172.18.0.1 [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.293.989 [ra_host.c:813]tid:559979,ra_socket_listen_start(813) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.294.052 [rs_socket.c:646]tid:559979,rs_socket_listen_bind_listen(646) : socket bind: family 2, addr 172.18.0.1, port 60004 [TRACE] HCCL(559979,python3):2024-10-11-18:55:09.294.144 [status:init] [op_base.cc:305][559979]HcclGetRootInfo success, take time [23043]us [EVENT] HCCP(559979,python3):2024-10-11-18:55:10.278.970 [ra_host.c:248]tid:559979,ra_init(248) : Input parameters: phy_id[4], nic_position:[1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:10.279.494 [ra_hdc.c:1255]tid:559979,ra_hdc_init(1255) : hdc init start! logic id is 4, phy id is 4 [EVENT] HCCP(559979,python3):2024-10-11-18:55:10.281.131 [ra_hdc.c:1290]tid:559979,ra_hdc_init(1290) : hdc init OK! phy_id[4] [EVENT] HCCP(559979,python3):2024-10-11-18:55:10.288.469 [ra_host.c:1840]tid:559979,ra_get_ifaddrs(1840) : Input parameters: phy_id[4], nic_position:[1], interface num[2] [EVENT] HCCP(559979,python3):2024-10-11-18:55:10.293.414 [ra_host.c:735]tid:559979,ra_socket_batch_connect(735) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1], remote_ip[172.18.0.1], tag:[topo_detect_default_tag_60004] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.495.010 [ra_host.c:773]tid:563750,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.495.084 [ra_host.c:773]tid:559979,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.539.710 [ra_host.c:773]tid:563750,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.539.957 [ra_host.c:773]tid:563750,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.539.999 [ra_host.c:773]tid:563750,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.540.068 [ra_host.c:851]tid:563750,ra_socket_listen_stop(851) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.540.128 [ra_host.c:480]tid:559979,ra_socket_deinit(480) : Input parameters: phy_id[4] family[2] local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.540.147 [rs.c:1263]tid:559979,rs_socket_deinit(1263) : socket deinit success, phy_id:4, local_ip:172.18.0.1 [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.540.164 [ra_host.c:304]tid:559979,ra_deinit(304) : Input parameters: phy_id[4], nic_position:[0] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.680.595 [rs.c:1415]tid:559979,rs_deinit(1415) : rs_deinit chip_id[4] ok [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.680.610 [ra_host.c:304]tid:559979,ra_deinit(304) : Input parameters: phy_id[4], nic_position:[1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.680.616 [ra_hdc.c:1325]tid:559979,ra_hdc_deinit(1325) : hdc deinit start! phy_id[4] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.680.790 [ra_hdc.c:1360]tid:559979,ra_hdc_deinit(1360) : hdc deinit OK! phy_id[4] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.681.668 [ra_host.c:248]tid:559979,ra_init(248) : Input parameters: phy_id[4], nic_position:[1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.681.994 [ra_hdc.c:1255]tid:559979,ra_hdc_init(1255) : hdc init start! logic id is 4, phy id is 4 [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.682.193 [ra_hdc.c:1290]tid:559979,ra_hdc_init(1290) : hdc init OK! phy_id[4] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.684.207 [ra_host.c:344]tid:559979,ra_socket_init(344) : socket init:mode=1 phy_id=4 family=2 ip=4.0.0.0 [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.684.812 [ra_host.c:813]tid:559979,ra_socket_listen_start(813) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0] [TRACE] HCCL(559979,python3):2024-10-11-18:55:12.689.336 [status:init] [op_base.cc:419][559979]HcclCommInitRootInfo success,take time [3395052]us, rankNum[4], rank[0],rootInfo identifier[172.18.0.1%br-76b90c640974_60004_4_1728644109294135], server[172.18.0.1%br-76b90c640974], device[0] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.693.128 [ra_host.c:1671]tid:565212,ra_socket_white_list_add(1671) : Input parameters: phy_id[4], local_ip[4.0.0.0], num[1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.693.287 [ra_host.c:1671]tid:565212,ra_socket_white_list_add(1671) : Input parameters: phy_id[4], local_ip[4.0.0.0], num[1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.693.374 [ra_host.c:735]tid:565212,ra_socket_batch_connect(735) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0], remote_ip[6.0.0.0], tag:[172.18.0.1%br-76b90c640974_60004_4_1728644109294135] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.905.876 [ra_host.c:735]tid:559979,ra_socket_batch_connect(735) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0], remote_ip[5.0.0.0], tag:[HeartBeat_172.18.0.1/4_to_172.18.0.1/5] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.906.012 [ra_host.c:1671]tid:559979,ra_socket_white_list_add(1671) : Input parameters: phy_id[4], local_ip[4.0.0.0], num[1] [TRACE] HCCL(559979,python3):2024-10-12-02:49:57.584.950 [heartbeat.cc:396][565270]rank [172.18.0.1/4]: rank [172.18.0.1/6] status[1] by rank [172.18.0.1/4] [TRACE] HCCL(559979,python3):2024-10-12-02:49:57.585.024 [heartbeat.cc:396][565270]rank [172.18.0.1/4]: rank [172.18.0.1/5] status[1] by rank [172.18.0.1/4] [EVENT] RUNTIME(559979,python3):2024-10-12-04:06:57.619.693 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:09:59.059.681 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:13:00.499.674 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:16:01.939.683 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:19:03.379.678 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:22:04.819.689 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:25:06.259.678 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:28:07.699.678 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:31:09.139.678 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:34:10.579.687 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.019.738 [engine.cc:1575]559979 ReportExceptProc:[FINAL][FINAL]Task exception! device_id=4, stream_id=2, task_id=51479, type=3(EVENT_WAIT), failuremode =1, retCode=0x105, [task timeout] [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.019.894 [device_error_proc.cc:1022]559979 PrintStreamTimeoutSnapshotInfo:[FINAL][FINAL]stream_id=2, task_id=51479, taskType=3 (EVENT_WAIT), EventId=4. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.019.903 [device_error_proc.cc:1022]559979 PrintStreamTimeoutSnapshotInfo:[FINAL][FINAL]stream_id=3, task_id=40939, taskType=14 (NOTIFY_WAIT), NotifyId=18. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.019.910 [device_error_proc.cc:1022]559979 PrintStreamTimeoutSnapshotInfo:[FINAL][FINAL]stream_id=12, task_id=33226, taskType=14 (NOTIFY_WAIT), NotifyId=14. [EVENT] RUNTIME(559979,python3):2024-10-12-04:35:06.019.914 [engine.cc:1578] 559979 ReportExceptProc: ASCEND_RT_VISIBLE_DEVICES:4,5,6,7 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.020.095 [stream.cc:3116]559979 EnterFailureAbort:[FINAL][FINAL]stream_id=2 enter failure abort. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.020.107 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.020.125 [task_info.cc:3594]559979 DoCompleteSuccessForEventWaitTask:[FINAL][FINAL]event wait error, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.020.131 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.020.142 [task_info.cc:3650]559979 PrintErrorInfoForEventWaitTask:[FINAL][FINAL]Task execute failed, device_id=4, stream_id=2, task_id=51479, flip_num=18, task_type=3(EVENT_WAIT). [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.694 [engine.cc:1931]559979 SyncTask:[FINAL][FINAL]Task Wait:stream_id=2 is ABORT. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.716 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.723 [stream.cc:1483]559979 GetError:[FINAL][FINAL]report error module_type=7, module_name=EE9999 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.728 [stream.cc:1483]559979 GetError:[FINAL][FINAL]Task execute failed, device_id=4, stream_id=2, task_id=51479, flip_num=18, task_type=3(EVENT_WAIT). [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.788 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.793 [context.cc:306]559979 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.816 [engine.cc:1575]559979 ReportExceptProc:[FINAL][FINAL]Task exception! device_id=4, stream_id=3, task_id=40941, type=14(NOTIFY_WAIT), failuremode =1, retCode=0x105, [task timeout] [EVENT] RUNTIME(559979,python3):2024-10-12-04:35:11.059.825 [engine.cc:1578] 559979 ReportExceptProc: ASCEND_RT_VISIBLE_DEVICES:4,5,6,7 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.060.021 [stream.cc:3116]559979 EnterFailureAbort:[FINAL][FINAL]stream_id=3 enter failure abort. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.060.032 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.060.061 [task_info.cc:5083]559979 ReportNotifyErrorForNotifyWaitTask:[FINAL][FINAL]Kernel task happen error, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.060.067 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.060.075 [task_info.cc:5060]559979 PrintErrorInfoForNotifyWaitTask:[FINAL][FINAL]Notify wait execute failed, device_id=4, stream_id=3, task_id=40941, flip_num=0, notify_id=2 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.153 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=3 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.159 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.163 [stream.cc:1483]559979 GetError:[FINAL][FINAL]report error module_type=2, module_name=EI9999 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.167 [stream.cc:1483]559979 GetError:[FINAL][FINAL]Notify wait execute failed, device_id=4, stream_id=3, task_id=40941, flip_num=0, notify_id=2 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.212 [engine.cc:1575]559979 ReportExceptProc:[FINAL][FINAL]Task exception! device_id=4, stream_id=11, task_id=33229, type=14(NOTIFY_WAIT), failuremode =1, retCode=0x105, [task timeout] [EVENT] RUNTIME(559979,python3):2024-10-12-04:35:11.061.218 [engine.cc:1578] 559979 ReportExceptProc: ASCEND_RT_VISIBLE_DEVICES:4,5,6,7 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.366 [stream.cc:3116]559979 EnterFailureAbort:[FINAL][FINAL]stream_id=11 enter failure abort. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.374 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.386 [task_info.cc:5083]559979 ReportNotifyErrorForNotifyWaitTask:[FINAL][FINAL]Kernel task happen error, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.391 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.398 [task_info.cc:5060]559979 PrintErrorInfoForNotifyWaitTask:[FINAL][FINAL]Notify wait execute failed, device_id=4, stream_id=11, task_id=33229, flip_num=0, notify_id=15 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.463 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=11 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.468 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.472 [stream.cc:1483]559979 GetError:[FINAL][FINAL]report error module_type=2, module_name=EI9999 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.476 [stream.cc:1483]559979 GetError:[FINAL][FINAL]Notify wait execute failed, device_id=4, stream_id=11, task_id=33229, flip_num=0, notify_id=15 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.498 [engine.cc:1575]559979 ReportExceptProc:[FINAL][FINAL]Task exception! device_id=4, stream_id=12, task_id=33226, type=14(NOTIFY_WAIT), failuremode =1, retCode=0x105, [task timeout] [EVENT] RUNTIME(559979,python3):2024-10-12-04:35:11.062.503 [engine.cc:1578] 559979 ReportExceptProc: ASCEND_RT_VISIBLE_DEVICES:4,5,6,7 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.646 [stream.cc:3116]559979 EnterFailureAbort:[FINAL][FINAL]stream_id=12 enter failure abort. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.653 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.664 [task_info.cc:5083]559979 ReportNotifyErrorForNotifyWaitTask:[FINAL][FINAL]Kernel task happen error, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.670 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.677 [task_info.cc:5060]559979 PrintErrorInfoForNotifyWaitTask:[FINAL][FINAL]Notify wait execute failed, device_id=4, stream_id=12, task_id=33226, flip_num=0, notify_id=14 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.741 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=12 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.746 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.751 [stream.cc:1483]559979 GetError:[FINAL][FINAL]report error module_type=2, module_name=EI9999 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.755 [stream.cc:1483]559979 GetError:[FINAL][FINAL]Notify wait execute failed, device_id=4, stream_id=12, task_id=33226, flip_num=0, notify_id=14 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.770 [context.cc:311]559979 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.776 [logger.cc:862]559979 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.800 [api_c.cc:1762]559979 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=107020, desc=[task timeout], InnerCode=0x7150059 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.805 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.813 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[task timeout] [ERROR] ASCENDCL(559979,python3):2024-10-12-04:35:11.063.832 [device.cpp:285]559979 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 107020. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.064.921 [device_msg_handler.cc:158]559979 HandleMsgInHostBuf:[FINAL][FINAL] DEVICE[4] PID[559979]: EXCEPTION STREAM: Exception info:TGID=559979, model id=65535, stream id=11, stream phase=3 Message info[0]:RTS_HWTS: slot_id=8, stream_id=11, task_id=33229, result=261 Other info[0]:time=2024-10-12-04:34:50.485.553, function=send_timeout_cq_msg_with_result, line=1383, error code=0x105 EXCEPTION STREAM: Exception info:TGID=559979, model id=65535, stream id=12, stream phase=3 Message info[0]:RTS_HWTS: slot_id=30, stream_id=12, task_id=33226, result=261 Other info[0]:t [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.064.929 [device_msg_handler.cc:158]559979 HandleMsgInHostBuf:[FINAL][FINAL]ime=2024-10-12-04:34:50.485.832, function=send_timeout_cq_msg_with_result, line=1383, error code=0x105 EXCEPTION STREAM: Exception info:TGID=559979, model id=65535, stream id=3, stream phase=3 Message info[0]:RTS_HWTS: slot_id=44, stream_id=3, task_id=40941, result=261 Other info[0]:time=2024-10-12-04:34:50.486.073, function=send_timeout_cq_msg_with_result, line=1383, error code=0x105 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.362 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=2 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.371 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.377 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.382 [context.cc:306]559979 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.390 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=3 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.395 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.414 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=11 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.419 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.425 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=12 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.429 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.433 [context.cc:311]559979 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.437 [logger.cc:862]559979 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.448 [api_c.cc:1762]559979 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=107020, desc=[task timeout], InnerCode=0x7150059 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.452 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.458 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[task timeout] [ERROR] ASCENDCL(559979,python3):2024-10-12-04:35:11.068.484 [device.cpp:285]559979 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 107020. [ERROR] APP(559979,python3):2024-10-12-04:35:11.069.162 [log_inner.cpp:76]559979 $(notdir $(abspath $<)):THPModule_npu_shutdown:78: "[PTA]:"NPU shutdown synchronize device failed."" [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.244 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=2 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.251 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.257 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.261 [context.cc:306]559979 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.269 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=3 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.274 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.292 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=11 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.296 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.301 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=12 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.305 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.309 [context.cc:311]559979 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.313 [logger.cc:862]559979 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.321 [api_c.cc:1762]559979 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=107020, desc=[task timeout], InnerCode=0x7150059 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.325 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.331 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[task timeout] [ERROR] ASCENDCL(559979,python3):2024-10-12-04:35:11.069.355 [device.cpp:285]559979 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 107020. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.034 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=2 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.042 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.048 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.053 [context.cc:306]559979 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.060 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=3 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.065 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.083 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=11 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.087 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.092 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=12 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.096 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.100 [context.cc:311]559979 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.104 [logger.cc:862]559979 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.114 [api_c.cc:1762]559979 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=107020, desc=[task timeout], InnerCode=0x7150059 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.117 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.123 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[task timeout] [ERROR] ASCENDCL(559979,python3):2024-10-12-04:35:11.157.165 [device.cpp:285]559979 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 107020. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.758 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=2 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.767 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.773 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.778 [context.cc:306]559979 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.786 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=3 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.790 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.809 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=11 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.813 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.819 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=12 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.823 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.827 [context.cc:311]559979 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.831 [logger.cc:862]559979 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.839 [api_c.cc:1762]559979 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=107020, desc=[task timeout], InnerCode=0x7150059 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.843 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.849 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[task timeout] [ERROR] ASCENDCL(559979,python3):2024-10-12-04:35:11.157.868 [device.cpp:285]559979 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 107020. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.597 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=2 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.616 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.623 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.628 [context.cc:306]559979 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.639 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=3 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.644 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.668 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=11 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.672 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.679 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=12 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.683 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.688 [context.cc:311]559979 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.692 [logger.cc:862]559979 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.705 [api_c.cc:1762]559979 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=107020, desc=[task timeout], InnerCode=0x7150059 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.710 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.718 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[task timeout] [ERROR] ASCENDCL(559979,python3):2024-10-12-04:35:11.158.763 [device.cpp:285]559979 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 107020. [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.160.468 [ra_host.c:773]tid:559979,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.160.924 [ra_host.c:1705]tid:559979,ra_socket_white_list_del(1705) : Input parameters: phy_id[4], local_ip[4.0.0.0], num[1] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.161.043 [ra_host.c:773]tid:559979,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.186.374 [ra_host.c:773]tid:559979,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.186.392 [ra_host.c:773]tid:559979,ra_socket_batch_close(773) : Input parameters: [1]th, phy_id[4], local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.186.398 [ra_host.c:773]tid:559979,ra_socket_batch_close(773) : Input parameters: [2]th, phy_id[4], local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.117 [ra_host.c:1705]tid:559979,ra_socket_white_list_del(1705) : Input parameters: phy_id[4], local_ip[4.0.0.0], num[1] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.239 [ra_host.c:1705]tid:559979,ra_socket_white_list_del(1705) : Input parameters: phy_id[4], local_ip[4.0.0.0], num[1] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.329 [ra_host.c:851]tid:559979,ra_socket_listen_stop(851) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.423 [ra_host.c:480]tid:559979,ra_socket_deinit(480) : Input parameters: phy_id[4] family[2] local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.540 [ra_host.c:304]tid:559979,ra_deinit(304) : Input parameters: phy_id[4], nic_position:[1] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.544 [ra_hdc.c:1325]tid:559979,ra_hdc_deinit(1325) : hdc deinit start! phy_id[4] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.649 [ra_hdc.c:1360]tid:559979,ra_hdc_deinit(1360) : hdc deinit OK! phy_id[4] [EVENT] PROFILING(559979,python3):2024-10-12-04:35:16.116.564 [msprof_callback_impl.cpp:79] >>> (tid:559979) MsprofCtrlCallback called, type: 3 [EVENT] PROFILING(559979,python3):2024-10-12-04:35:16.116.588 [ai_drv_dev_api.cpp:333] >>> (tid:559979) Succeeded to DrvGetApiVersion version: 0x71f0d [TRACE] GE(559979,python3):2024-10-12-04:35:16.116.689 [status:INIT] [ge_api.cc:363]559979 GEFinalize:GEFinalize start [TRACE] GE(559979,python3):2024-10-12-04:35:16.155.294 [status:RUNNING] [ge_api.cc:374]559979 GEFinalize:Finalizing environment [EVENT] TUNE(559979,python3):2024-10-12-04:35:20.642.423 [cann_kb_pyfunc_mgr.cpp:127][CANNKB][Tid:559979]"CannKbPyfuncMgr: enter PyObjectDeinit function, reference_[1]" [EVENT] TUNE(559979,python3):2024-10-12-04:35:20.642.445 [cann_kb_pyfunc_mgr.cpp:138][CANNKB][Tid:559979]"CannKbPyfuncMgr: PyObjectDeinit function end successfully!" [TRACE] GE(559979,python3):2024-10-12-04:35:22.398.168 [status:STOP] [ge_api.cc:402]559979 GEFinalize:GEFinalize finished [EVENT] PROFILING(559979,python3):2024-10-12-04:35:22.398.901 [msprof_callback_impl.cpp:79] >>> (tid:559979) MsprofCtrlCallback called, type: 3 [EVENT] PROFILING(559979,python3):2024-10-12-04:35:22.399.044 [ai_drv_dev_api.cpp:333] >>> (tid:559979) Succeeded to DrvGetApiVersion version: 0x71f0d [EVENT] RUNTIME(559979,python3):2024-10-12-04:35:30.016.212 [runtime.cc:1441] 559979 ~Runtime: deconstruct runtime. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.443 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]SVM_MEM mem stats(Bytes): module_name=SLOG, module_id=0 current_alloced_size=0, alloced_peak_size=1258291200. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.465 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]DEV_MEM dev4 mem stats(Bytes): module_name=HCCL, module_id=3 current_alloced_size=0, alloced_peak_size=419536896. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.471 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]DEV_MEM dev4 mem stats(Bytes): module_name=RUNTIME, module_id=7 current_alloced_size=0, alloced_peak_size=323792896. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.476 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]DEV_MEM dev4 mem stats(Bytes): module_name=APP, module_id=33 current_alloced_size=2466250752, alloced_peak_size=8298430464. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.481 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]DEV_MEM dev4 mem stats(Bytes): module_name=GE, module_id=45 current_alloced_size=0, alloced_peak_size=833843200. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.487 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 156]DEV_MEM dev4 Cached_size:1652555776Bytes [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.494 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]HOST_MEM mem stats(Bytes): module_name=HCCL, module_id=3 current_alloced_size=0, alloced_peak_size=4096. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.498 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]HOST_MEM mem stats(Bytes): module_name=RUNTIME, module_id=7 current_alloced_size=0, alloced_peak_size=17698816. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.502 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]HOST_MEM mem stats(Bytes): module_name=APP, module_id=33 current_alloced_size=0, alloced_peak_size=4096. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.507 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 156]HOST_MEM Cached_size:20971520Bytes [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.511 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 156]DVPP_MEM dev4 Cached_size:0Bytes
一、问题现象（附报错日志上下文）： 使用四卡【4,5,6,7】进行分布式训练时报错，详见训练日志报错信息。在训练第一个epo完成并生成checkpoint.0后报错.麻烦看一些，谢谢，下述是训练日志信息 Inner error, see details in Ascend logs.Inner error, see details in Ascend logs.Inner error, see details in Ascend logs.Traceback (most recent call last): File "bin/train_ddp.py", line 118, in <module> main() File "bin/train_ddp.py", line 110, in main dist.barrier() File "/usr/local/python3.8.10/lib/python3.8/site-packages/torch_npu/distributed/distributed_c10d.py", line 2001, in barrier work.wait() RuntimeError: npuSynchronizeDevice:/usr1/01/workspace/j_ctViaEnN/pytorch/torch_npu/csrc/core/npu/NPUStream.cpp:387 NPU error, error code is 107020. 二、软件版本: -- CANN 版本 (7.0.RC1): --Pytorch 版本：'1.11.0'[输入链接说明](http://) --Python 版本 (Python 3.8.10): --操作系统版本 (CentOS Linux 7 (AltArch)): --芯片是910B 三、测试步骤： xxxx 四、日志信息: plog 信息， [EVENT] PROFILING(559979,python3):2024-10-11-18:54:48.134.248 [msprof_callback_impl.cpp:324] >>> (tid:559979) Started to register profiling ctrl callback. [EVENT] PROFILING(559979,python3):2024-10-11-18:54:48.134.401 [msprof_callback_impl.cpp:331] >>> (tid:559979) Started to register profiling hash id callback. [EVENT] PROFILING(559979,python3):2024-10-11-18:54:48.134.408 [prof_atls_plugin.cpp:83] >>> (tid:559979) RegisterProfileCallback, callback type is 7 [EVENT] PROFILING(559979,python3):2024-10-11-18:54:48.134.412 [msprof_callback_impl.cpp:338] >>> (tid:559979) Started to register profiling enable host freq callback. [EVENT] PROFILING(559979,python3):2024-10-11-18:54:48.134.416 [prof_atls_plugin.cpp:83] >>> (tid:559979) RegisterProfileCallback, callback type is 8 [EVENT] RUNTIME(559979,python3):2024-10-11-18:54:49.716.827 [runtime.cc:1010] 559979 GetAicoreNumByLevel: workingDev_=0 [EVENT] RUNTIME(559979,python3):2024-10-11-18:54:49.716.876 [runtime.cc:4125] 559979 GetVisibleDevices: real deviceCnt:8 userDeviceCnt:4 isSetVisibleDev:1 ASCEND_RT_VISIBLE_DEVICES:4,5,6,7 [EVENT] PROFILING(559979,python3):2024-10-11-18:54:49.717.312 [prof_atls_plugin.cpp:154] >>> (tid:559979) Module[7] register callback of ctrl handle. [EVENT] PROFILING(559979,python3):2024-10-11-18:54:51.402.611 [prof_atls_plugin.cpp:154] >>> (tid:559979) Module[48] register callback of ctrl handle. [EVENT] PROFILING(559979,python3):2024-10-11-18:54:51.403.170 [prof_atls_plugin.cpp:154] >>> (tid:559979) Module[45] register callback of ctrl handle. [EVENT] PROFILING(559979,python3):2024-10-11-18:54:51.755.924 [msprof_callback_impl.cpp:79] >>> (tid:559979) MsprofCtrlCallback called, type: 255 [EVENT] PROFILING(559979,python3):2024-10-11-18:54:51.756.223 [ai_drv_dev_api.cpp:333] >>> (tid:559979) Succeeded to DrvGetApiVersion version: 0x71f0d [EVENT] RUNTIME(559979,python3):2024-10-11-18:54:52.716.196 [npu_driver.cc:5134] 561246 GetDeviceStatus: GetDeviceStatus status=1. [TRACE] GE(559979,python3):2024-10-11-18:54:52.720.621 [status:INIT] [ge_api.cc:206]559979 GEInitializeImpl:GEInitialize start [EVENT] PROFILING(559979,python3):2024-10-11-18:54:52.797.506 [msprof_callback_impl.cpp:79] >>> (tid:559979) MsprofCtrlCallback called, type: 255 [EVENT] PROFILING(559979,python3):2024-10-11-18:54:52.797.516 [ai_drv_dev_api.cpp:333] >>> (tid:559979) Succeeded to DrvGetApiVersion version: 0x71f0d [TRACE] GE(559979,python3):2024-10-11-18:54:52.855.381 [status:RUNNING] [ge_api.cc:270]559979 GEInitializeImpl:Initializing environment [EVENT] TUNE(559979,python3):2024-10-11-18:54:53.537.528 [cann_kb_pyfunc_mgr.cpp:72][CANNKB][Tid:559979]"CannKbPyfuncMgr: Enter PyObjectInit, reference_ is 0!" [EVENT] TUNE(559979,python3):2024-10-11-18:54:53.537.890 [handle_manager.cpp:115][CANNKB][Tid:559979]"Start to run init functions to load dynamic python lib!" [EVENT] TUNE(559979,python3):2024-10-11-18:54:53.537.943 [handle_manager.cpp:407][CANNKB][Tid:559979]"Init functions of loading dynamic python lib end!" [EVENT] TUNE(559979,python3):2024-10-11-18:54:53.537.947 [cann_kb_pyfunc_mgr.cpp:24][CANNKB][Tid:559979]"CANN_KB_Py has already been initialized." [EVENT] TUNE(559979,python3):2024-10-11-18:54:56.127.126 [cann_kb_pyfunc_mgr.cpp:117][CANNKB][Tid:559979]"CannKbPyfuncMgr: Run PyObjectInit successfully!" [EVENT] TBE(559979,python3):2024-10-11-18:54:56.365.127 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:312][repository_manager] get_compiler_core_num core_num = [8]. [EVENT] TBE(559979,python3):2024-10-11-18:55:00.326.901 [../../../../../../latest/python/site-packages/te_fusion/parallel_compilation.py:552][init] The time cost of random buffer compile is [743720] micro second. [EVENT] TBE(559979,python3):2024-10-11-18:55:00.863.813 [../../../../../../latest/python/site-packages/te_fusion/parallel_compilation.py:552][init] The time cost of random buffer compile is [510928] micro second. [TRACE] GE(559979,python3):2024-10-11-18:55:03.154.031 [status:STOP] [ge_api.cc:313]559979 GEInitializeImpl:GEInitialize finished [TRACE] GE(559979,python3):2024-10-11-18:55:03.155.305 [status:INIT] [ge_api.cc:206]559979 GEInitializeImpl:GEInitialize start [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.276.265 [ra_host.c:1792]tid:559979,ra_get_ifnum(1792) : Input parameters: phy_id[0], nic_position:[0] [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.286.156 [ra_host.c:1840]tid:559979,ra_get_ifaddrs(1840) : Input parameters: phy_id[0], nic_position:[0], interface num[25] [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.292.754 [ra_host.c:1632]tid:559979,ra_socket_set_white_list_status(1632) : Input parameters: enable[0] [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.292.938 [ra_host.c:248]tid:559979,ra_init(248) : Input parameters: phy_id[4], nic_position:[0] [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.293.221 [rs_ssl.c:1062]tid:559979,rs_ssl_init(1062) : TLS SWITCH (0) [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.293.387 [rs_epoll.c:470]tid:563748,rs_epoll_handle(470) : pthread[epoll_pthread] is alive! [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.293.694 [rs_epoll.c:596]tid:563749,rs_connect_handle(596) : pthread[connect_pthread] is alive! [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.293.956 [rs.c:397]tid:559979,rs_init(397) : rs init success, chip_id[4] [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.293.979 [ra_host.c:409]tid:559979,ra_socket_init_v1(409) : socket init:mode=0 phy_id=4 family=2 ip=172.18.0.1 [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.293.989 [ra_host.c:813]tid:559979,ra_socket_listen_start(813) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:09.294.052 [rs_socket.c:646]tid:559979,rs_socket_listen_bind_listen(646) : socket bind: family 2, addr 172.18.0.1, port 60004 [TRACE] HCCL(559979,python3):2024-10-11-18:55:09.294.144 [status:init] [op_base.cc:305][559979]HcclGetRootInfo success, take time [23043]us [EVENT] HCCP(559979,python3):2024-10-11-18:55:10.278.970 [ra_host.c:248]tid:559979,ra_init(248) : Input parameters: phy_id[4], nic_position:[1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:10.279.494 [ra_hdc.c:1255]tid:559979,ra_hdc_init(1255) : hdc init start! logic id is 4, phy id is 4 [EVENT] HCCP(559979,python3):2024-10-11-18:55:10.281.131 [ra_hdc.c:1290]tid:559979,ra_hdc_init(1290) : hdc init OK! phy_id[4] [EVENT] HCCP(559979,python3):2024-10-11-18:55:10.288.469 [ra_host.c:1840]tid:559979,ra_get_ifaddrs(1840) : Input parameters: phy_id[4], nic_position:[1], interface num[2] [EVENT] HCCP(559979,python3):2024-10-11-18:55:10.293.414 [ra_host.c:735]tid:559979,ra_socket_batch_connect(735) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1], remote_ip[172.18.0.1], tag:[topo_detect_default_tag_60004] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.495.010 [ra_host.c:773]tid:563750,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.495.084 [ra_host.c:773]tid:559979,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.539.710 [ra_host.c:773]tid:563750,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.539.957 [ra_host.c:773]tid:563750,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.539.999 [ra_host.c:773]tid:563750,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.540.068 [ra_host.c:851]tid:563750,ra_socket_listen_stop(851) : Input parameters: [0]th, phy_id[4], local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.540.128 [ra_host.c:480]tid:559979,ra_socket_deinit(480) : Input parameters: phy_id[4] family[2] local_ip[172.18.0.1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.540.147 [rs.c:1263]tid:559979,rs_socket_deinit(1263) : socket deinit success, phy_id:4, local_ip:172.18.0.1 [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.540.164 [ra_host.c:304]tid:559979,ra_deinit(304) : Input parameters: phy_id[4], nic_position:[0] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.680.595 [rs.c:1415]tid:559979,rs_deinit(1415) : rs_deinit chip_id[4] ok [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.680.610 [ra_host.c:304]tid:559979,ra_deinit(304) : Input parameters: phy_id[4], nic_position:[1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.680.616 [ra_hdc.c:1325]tid:559979,ra_hdc_deinit(1325) : hdc deinit start! phy_id[4] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.680.790 [ra_hdc.c:1360]tid:559979,ra_hdc_deinit(1360) : hdc deinit OK! phy_id[4] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.681.668 [ra_host.c:248]tid:559979,ra_init(248) : Input parameters: phy_id[4], nic_position:[1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.681.994 [ra_hdc.c:1255]tid:559979,ra_hdc_init(1255) : hdc init start! logic id is 4, phy id is 4 [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.682.193 [ra_hdc.c:1290]tid:559979,ra_hdc_init(1290) : hdc init OK! phy_id[4] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.684.207 [ra_host.c:344]tid:559979,ra_socket_init(344) : socket init:mode=1 phy_id=4 family=2 ip=4.0.0.0 [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.684.812 [ra_host.c:813]tid:559979,ra_socket_listen_start(813) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0] [TRACE] HCCL(559979,python3):2024-10-11-18:55:12.689.336 [status:init] [op_base.cc:419][559979]HcclCommInitRootInfo success,take time [3395052]us, rankNum[4], rank[0],rootInfo identifier[172.18.0.1%br-76b90c640974_60004_4_1728644109294135], server[172.18.0.1%br-76b90c640974], device[0] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.693.128 [ra_host.c:1671]tid:565212,ra_socket_white_list_add(1671) : Input parameters: phy_id[4], local_ip[4.0.0.0], num[1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.693.287 [ra_host.c:1671]tid:565212,ra_socket_white_list_add(1671) : Input parameters: phy_id[4], local_ip[4.0.0.0], num[1] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.693.374 [ra_host.c:735]tid:565212,ra_socket_batch_connect(735) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0], remote_ip[6.0.0.0], tag:[172.18.0.1%br-76b90c640974_60004_4_1728644109294135] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.905.876 [ra_host.c:735]tid:559979,ra_socket_batch_connect(735) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0], remote_ip[5.0.0.0], tag:[HeartBeat_172.18.0.1/4_to_172.18.0.1/5] [EVENT] HCCP(559979,python3):2024-10-11-18:55:12.906.012 [ra_host.c:1671]tid:559979,ra_socket_white_list_add(1671) : Input parameters: phy_id[4], local_ip[4.0.0.0], num[1] [TRACE] HCCL(559979,python3):2024-10-12-02:49:57.584.950 [heartbeat.cc:396][565270]rank [172.18.0.1/4]: rank [172.18.0.1/6] status[1] by rank [172.18.0.1/4] [TRACE] HCCL(559979,python3):2024-10-12-02:49:57.585.024 [heartbeat.cc:396][565270]rank [172.18.0.1/4]: rank [172.18.0.1/5] status[1] by rank [172.18.0.1/4] [EVENT] RUNTIME(559979,python3):2024-10-12-04:06:57.619.693 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:09:59.059.681 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:13:00.499.674 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:16:01.939.683 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:19:03.379.678 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:22:04.819.689 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:25:06.259.678 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:28:07.699.678 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:31:09.139.678 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [EVENT] RUNTIME(559979,python3):2024-10-12-04:34:10.579.687 [engine.cc:1617] 559979 ReportTimeoutProc: report timeout! streamId=2, taskId=51480, pendingTaskNum=125, reportCount=41707, parseTaskCount=41707 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.019.738 [engine.cc:1575]559979 ReportExceptProc:[FINAL][FINAL]Task exception! device_id=4, stream_id=2, task_id=51479, type=3(EVENT_WAIT), failuremode =1, retCode=0x105, [task timeout] [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.019.894 [device_error_proc.cc:1022]559979 PrintStreamTimeoutSnapshotInfo:[FINAL][FINAL]stream_id=2, task_id=51479, taskType=3 (EVENT_WAIT), EventId=4. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.019.903 [device_error_proc.cc:1022]559979 PrintStreamTimeoutSnapshotInfo:[FINAL][FINAL]stream_id=3, task_id=40939, taskType=14 (NOTIFY_WAIT), NotifyId=18. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.019.910 [device_error_proc.cc:1022]559979 PrintStreamTimeoutSnapshotInfo:[FINAL][FINAL]stream_id=12, task_id=33226, taskType=14 (NOTIFY_WAIT), NotifyId=14. [EVENT] RUNTIME(559979,python3):2024-10-12-04:35:06.019.914 [engine.cc:1578] 559979 ReportExceptProc: ASCEND_RT_VISIBLE_DEVICES:4,5,6,7 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.020.095 [stream.cc:3116]559979 EnterFailureAbort:[FINAL][FINAL]stream_id=2 enter failure abort. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.020.107 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.020.125 [task_info.cc:3594]559979 DoCompleteSuccessForEventWaitTask:[FINAL][FINAL]event wait error, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.020.131 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:06.020.142 [task_info.cc:3650]559979 PrintErrorInfoForEventWaitTask:[FINAL][FINAL]Task execute failed, device_id=4, stream_id=2, task_id=51479, flip_num=18, task_type=3(EVENT_WAIT). [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.694 [engine.cc:1931]559979 SyncTask:[FINAL][FINAL]Task Wait:stream_id=2 is ABORT. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.716 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.723 [stream.cc:1483]559979 GetError:[FINAL][FINAL]report error module_type=7, module_name=EE9999 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.728 [stream.cc:1483]559979 GetError:[FINAL][FINAL]Task execute failed, device_id=4, stream_id=2, task_id=51479, flip_num=18, task_type=3(EVENT_WAIT). [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.788 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.793 [context.cc:306]559979 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.059.816 [engine.cc:1575]559979 ReportExceptProc:[FINAL][FINAL]Task exception! device_id=4, stream_id=3, task_id=40941, type=14(NOTIFY_WAIT), failuremode =1, retCode=0x105, [task timeout] [EVENT] RUNTIME(559979,python3):2024-10-12-04:35:11.059.825 [engine.cc:1578] 559979 ReportExceptProc: ASCEND_RT_VISIBLE_DEVICES:4,5,6,7 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.060.021 [stream.cc:3116]559979 EnterFailureAbort:[FINAL][FINAL]stream_id=3 enter failure abort. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.060.032 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.060.061 [task_info.cc:5083]559979 ReportNotifyErrorForNotifyWaitTask:[FINAL][FINAL]Kernel task happen error, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.060.067 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.060.075 [task_info.cc:5060]559979 PrintErrorInfoForNotifyWaitTask:[FINAL][FINAL]Notify wait execute failed, device_id=4, stream_id=3, task_id=40941, flip_num=0, notify_id=2 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.153 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=3 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.159 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.163 [stream.cc:1483]559979 GetError:[FINAL][FINAL]report error module_type=2, module_name=EI9999 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.167 [stream.cc:1483]559979 GetError:[FINAL][FINAL]Notify wait execute failed, device_id=4, stream_id=3, task_id=40941, flip_num=0, notify_id=2 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.212 [engine.cc:1575]559979 ReportExceptProc:[FINAL][FINAL]Task exception! device_id=4, stream_id=11, task_id=33229, type=14(NOTIFY_WAIT), failuremode =1, retCode=0x105, [task timeout] [EVENT] RUNTIME(559979,python3):2024-10-12-04:35:11.061.218 [engine.cc:1578] 559979 ReportExceptProc: ASCEND_RT_VISIBLE_DEVICES:4,5,6,7 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.366 [stream.cc:3116]559979 EnterFailureAbort:[FINAL][FINAL]stream_id=11 enter failure abort. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.374 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.386 [task_info.cc:5083]559979 ReportNotifyErrorForNotifyWaitTask:[FINAL][FINAL]Kernel task happen error, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.391 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.061.398 [task_info.cc:5060]559979 PrintErrorInfoForNotifyWaitTask:[FINAL][FINAL]Notify wait execute failed, device_id=4, stream_id=11, task_id=33229, flip_num=0, notify_id=15 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.463 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=11 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.468 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.472 [stream.cc:1483]559979 GetError:[FINAL][FINAL]report error module_type=2, module_name=EI9999 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.476 [stream.cc:1483]559979 GetError:[FINAL][FINAL]Notify wait execute failed, device_id=4, stream_id=11, task_id=33229, flip_num=0, notify_id=15 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.498 [engine.cc:1575]559979 ReportExceptProc:[FINAL][FINAL]Task exception! device_id=4, stream_id=12, task_id=33226, type=14(NOTIFY_WAIT), failuremode =1, retCode=0x105, [task timeout] [EVENT] RUNTIME(559979,python3):2024-10-12-04:35:11.062.503 [engine.cc:1578] 559979 ReportExceptProc: ASCEND_RT_VISIBLE_DEVICES:4,5,6,7 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.646 [stream.cc:3116]559979 EnterFailureAbort:[FINAL][FINAL]stream_id=12 enter failure abort. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.653 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.664 [task_info.cc:5083]559979 ReportNotifyErrorForNotifyWaitTask:[FINAL][FINAL]Kernel task happen error, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.670 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.062.677 [task_info.cc:5060]559979 PrintErrorInfoForNotifyWaitTask:[FINAL][FINAL]Notify wait execute failed, device_id=4, stream_id=12, task_id=33226, flip_num=0, notify_id=14 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.741 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=12 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.746 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.751 [stream.cc:1483]559979 GetError:[FINAL][FINAL]report error module_type=2, module_name=EI9999 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.755 [stream.cc:1483]559979 GetError:[FINAL][FINAL]Notify wait execute failed, device_id=4, stream_id=12, task_id=33226, flip_num=0, notify_id=14 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.770 [context.cc:311]559979 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.776 [logger.cc:862]559979 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.800 [api_c.cc:1762]559979 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=107020, desc=[task timeout], InnerCode=0x7150059 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.805 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.063.813 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[task timeout] [ERROR] ASCENDCL(559979,python3):2024-10-12-04:35:11.063.832 [device.cpp:285]559979 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 107020. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.064.921 [device_msg_handler.cc:158]559979 HandleMsgInHostBuf:[FINAL][FINAL] DEVICE[4] PID[559979]: EXCEPTION STREAM: Exception info:TGID=559979, model id=65535, stream id=11, stream phase=3 Message info[0]:RTS_HWTS: slot_id=8, stream_id=11, task_id=33229, result=261 Other info[0]:time=2024-10-12-04:34:50.485.553, function=send_timeout_cq_msg_with_result, line=1383, error code=0x105 EXCEPTION STREAM: Exception info:TGID=559979, model id=65535, stream id=12, stream phase=3 Message info[0]:RTS_HWTS: slot_id=30, stream_id=12, task_id=33226, result=261 Other info[0]:t [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.064.929 [device_msg_handler.cc:158]559979 HandleMsgInHostBuf:[FINAL][FINAL]ime=2024-10-12-04:34:50.485.832, function=send_timeout_cq_msg_with_result, line=1383, error code=0x105 EXCEPTION STREAM: Exception info:TGID=559979, model id=65535, stream id=3, stream phase=3 Message info[0]:RTS_HWTS: slot_id=44, stream_id=3, task_id=40941, result=261 Other info[0]:time=2024-10-12-04:34:50.486.073, function=send_timeout_cq_msg_with_result, line=1383, error code=0x105 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.362 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=2 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.371 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.377 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.382 [context.cc:306]559979 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.390 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=3 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.395 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.414 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=11 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.419 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.425 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=12 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.429 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.433 [context.cc:311]559979 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.437 [logger.cc:862]559979 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.448 [api_c.cc:1762]559979 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=107020, desc=[task timeout], InnerCode=0x7150059 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.452 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.068.458 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[task timeout] [ERROR] ASCENDCL(559979,python3):2024-10-12-04:35:11.068.484 [device.cpp:285]559979 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 107020. [ERROR] APP(559979,python3):2024-10-12-04:35:11.069.162 [log_inner.cpp:76]559979 $(notdir $(abspath $<)):THPModule_npu_shutdown:78: "[PTA]:"NPU shutdown synchronize device failed."" [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.244 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=2 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.251 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.257 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.261 [context.cc:306]559979 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.269 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=3 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.274 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.292 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=11 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.296 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.301 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=12 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.305 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.309 [context.cc:311]559979 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.313 [logger.cc:862]559979 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.321 [api_c.cc:1762]559979 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=107020, desc=[task timeout], InnerCode=0x7150059 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.325 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.069.331 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[task timeout] [ERROR] ASCENDCL(559979,python3):2024-10-12-04:35:11.069.355 [device.cpp:285]559979 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 107020. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.034 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=2 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.042 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.048 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.053 [context.cc:306]559979 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.060 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=3 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.065 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.083 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=11 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.087 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.092 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=12 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.096 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.100 [context.cc:311]559979 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.104 [logger.cc:862]559979 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.114 [api_c.cc:1762]559979 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=107020, desc=[task timeout], InnerCode=0x7150059 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.117 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.123 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[task timeout] [ERROR] ASCENDCL(559979,python3):2024-10-12-04:35:11.157.165 [device.cpp:285]559979 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 107020. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.758 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=2 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.767 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.773 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.778 [context.cc:306]559979 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.786 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=3 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.790 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.809 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=11 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.813 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.819 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=12 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.823 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.827 [context.cc:311]559979 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.831 [logger.cc:862]559979 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.839 [api_c.cc:1762]559979 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=107020, desc=[task timeout], InnerCode=0x7150059 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.843 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.157.849 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[task timeout] [ERROR] ASCENDCL(559979,python3):2024-10-12-04:35:11.157.868 [device.cpp:285]559979 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 107020. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.597 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=2 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.616 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.623 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=2, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.628 [context.cc:306]559979 Synchronize:[FINAL][FINAL]sync stream fail, stream_id=2, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.639 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=3 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.644 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=3, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.668 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=11 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.672 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=11, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.679 [stream.cc:2376]559979 WaitForTask:[FINAL][FINAL]Task Wait: device_id=4, stream_id=12 is Abort, RunningState=0 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.683 [stream.cc:1480]559979 GetError:[FINAL][FINAL]Stream Synchronize failed, stream_id=12, retCode=0x105, [task timeout]. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.688 [context.cc:311]559979 Synchronize:[FINAL][FINAL]Synchronize streams, retCode=0x7150059. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.692 [logger.cc:862]559979 DeviceSynchronize:[FINAL][FINAL]Device synchronize failed. [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.705 [api_c.cc:1762]559979 rtDeviceSynchronize:[FINAL][FINAL]ErrCode=107020, desc=[task timeout], InnerCode=0x7150059 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.710 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]report error module_type=3, module_name=EE8888 [ERROR] RUNTIME(559979,python3):2024-10-12-04:35:11.158.718 [error_message_manage.cc:50]559979 FuncErrorReason:[FINAL][FINAL]rtDeviceSynchronize execute failed, reason=[task timeout] [ERROR] ASCENDCL(559979,python3):2024-10-12-04:35:11.158.763 [device.cpp:285]559979 aclrtSynchronizeDevice: [FINAL][FINAL]wait for compute device to finish failed, runtime result = 107020. [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.160.468 [ra_host.c:773]tid:559979,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.160.924 [ra_host.c:1705]tid:559979,ra_socket_white_list_del(1705) : Input parameters: phy_id[4], local_ip[4.0.0.0], num[1] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.161.043 [ra_host.c:773]tid:559979,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.186.374 [ra_host.c:773]tid:559979,ra_socket_batch_close(773) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.186.392 [ra_host.c:773]tid:559979,ra_socket_batch_close(773) : Input parameters: [1]th, phy_id[4], local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.186.398 [ra_host.c:773]tid:559979,ra_socket_batch_close(773) : Input parameters: [2]th, phy_id[4], local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.117 [ra_host.c:1705]tid:559979,ra_socket_white_list_del(1705) : Input parameters: phy_id[4], local_ip[4.0.0.0], num[1] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.239 [ra_host.c:1705]tid:559979,ra_socket_white_list_del(1705) : Input parameters: phy_id[4], local_ip[4.0.0.0], num[1] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.329 [ra_host.c:851]tid:559979,ra_socket_listen_stop(851) : Input parameters: [0]th, phy_id[4], local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.423 [ra_host.c:480]tid:559979,ra_socket_deinit(480) : Input parameters: phy_id[4] family[2] local_ip[4.0.0.0] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.540 [ra_host.c:304]tid:559979,ra_deinit(304) : Input parameters: phy_id[4], nic_position:[1] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.544 [ra_hdc.c:1325]tid:559979,ra_hdc_deinit(1325) : hdc deinit start! phy_id[4] [EVENT] HCCP(559979,python3):2024-10-12-04:35:11.187.649 [ra_hdc.c:1360]tid:559979,ra_hdc_deinit(1360) : hdc deinit OK! phy_id[4] [EVENT] PROFILING(559979,python3):2024-10-12-04:35:16.116.564 [msprof_callback_impl.cpp:79] >>> (tid:559979) MsprofCtrlCallback called, type: 3 [EVENT] PROFILING(559979,python3):2024-10-12-04:35:16.116.588 [ai_drv_dev_api.cpp:333] >>> (tid:559979) Succeeded to DrvGetApiVersion version: 0x71f0d [TRACE] GE(559979,python3):2024-10-12-04:35:16.116.689 [status:INIT] [ge_api.cc:363]559979 GEFinalize:GEFinalize start [TRACE] GE(559979,python3):2024-10-12-04:35:16.155.294 [status:RUNNING] [ge_api.cc:374]559979 GEFinalize:Finalizing environment [EVENT] TUNE(559979,python3):2024-10-12-04:35:20.642.423 [cann_kb_pyfunc_mgr.cpp:127][CANNKB][Tid:559979]"CannKbPyfuncMgr: enter PyObjectDeinit function, reference_[1]" [EVENT] TUNE(559979,python3):2024-10-12-04:35:20.642.445 [cann_kb_pyfunc_mgr.cpp:138][CANNKB][Tid:559979]"CannKbPyfuncMgr: PyObjectDeinit function end successfully!" [TRACE] GE(559979,python3):2024-10-12-04:35:22.398.168 [status:STOP] [ge_api.cc:402]559979 GEFinalize:GEFinalize finished [EVENT] PROFILING(559979,python3):2024-10-12-04:35:22.398.901 [msprof_callback_impl.cpp:79] >>> (tid:559979) MsprofCtrlCallback called, type: 3 [EVENT] PROFILING(559979,python3):2024-10-12-04:35:22.399.044 [ai_drv_dev_api.cpp:333] >>> (tid:559979) Succeeded to DrvGetApiVersion version: 0x71f0d [EVENT] RUNTIME(559979,python3):2024-10-12-04:35:30.016.212 [runtime.cc:1441] 559979 ~Runtime: deconstruct runtime. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.443 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]SVM_MEM mem stats(Bytes): module_name=SLOG, module_id=0 current_alloced_size=0, alloced_peak_size=1258291200. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.465 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]DEV_MEM dev4 mem stats(Bytes): module_name=HCCL, module_id=3 current_alloced_size=0, alloced_peak_size=419536896. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.471 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]DEV_MEM dev4 mem stats(Bytes): module_name=RUNTIME, module_id=7 current_alloced_size=0, alloced_peak_size=323792896. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.476 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]DEV_MEM dev4 mem stats(Bytes): module_name=APP, module_id=33 current_alloced_size=2466250752, alloced_peak_size=8298430464. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.481 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]DEV_MEM dev4 mem stats(Bytes): module_name=GE, module_id=45 current_alloced_size=0, alloced_peak_size=833843200. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.487 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 156]DEV_MEM dev4 Cached_size:1652555776Bytes [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.494 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]HOST_MEM mem stats(Bytes): module_name=HCCL, module_id=3 current_alloced_size=0, alloced_peak_size=4096. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.498 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]HOST_MEM mem stats(Bytes): module_name=RUNTIME, module_id=7 current_alloced_size=0, alloced_peak_size=17698816. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.502 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 150]HOST_MEM mem stats(Bytes): module_name=APP, module_id=33 current_alloced_size=0, alloced_peak_size=4096. [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.507 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 156]HOST_MEM Cached_size:20971520Bytes [EVENT] DRV(559979,python3):2024-10-12-04:35:30.293.511 [ascend][curpid: 559979, 559979][drv][devmm][_devmm_mem_stats_show 156]DVPP_MEM dev4 Cached_size:0Bytes
评论 (
17
)
登录
后才可以发表评论
状态
DONE
TODO
ACCEPTED
Analysing
Feedback
WIP
Replied
CLOSED
DONE
REJECTED
负责人
未设置
标签
未设置
项目
未立项任务
未立项任务
里程碑
未关联里程碑
未关联里程碑
Pull Requests
未关联
未关联
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
未关联
分支 (79)
标签 (186)
master
v2.8.0
v2.1.0
v2.6.0
v2.7.1
v2.5.1
v2.6.0-7.1.0
v2.5.1-7.1.0
v2.1.0-7.1.0
revert-merge-23967-master
revert-merge-23966-v2.8.0
revert-merge-23965-v2.7.1
revert-merge-23964-v2.6.0
revert-merge-23962-v2.5.1
revert-merge-23789-v2.1.0
v2.1.0-7.0.0
v2.4.0-7.0.0
v2.4.0
v2.3.1
v2.3.1-7.0.0
v2.5.1-7.0.0
v2.4.0-6.0.0
v2.3.1-6.0.0
v2.1.0-6.0.0
v2.1.0-6.0.rc3
v2.3.1-6.0.rc3
v2.4.0-6.0.rc3
v2.2.0
v1.11.0-6.0.rc1
v2.1.0-6.0.rc1
v2.2.0-6.0.rc1
v1.11.0-6.0.rc2
v2.1.0-6.0.rc2
v2.2.0-6.0.rc2
v2.3.1-6.0.rc2
v1.11.0
v2.1.0-5.0.0
v2.0.1-5.0.0
v1.11.0-5.0.0
v2.0.1
v2.1.0-5.0.rc3
v1.11.0-5.0.rc3
v2.0.1-5.0.rc3
v1.11.0-5.0.rc3.3
v1.8.1
v1.11.0-x1
v1.8.1-5.0.rc3
v1.11.0-5.0.rc2.2
v1.11.0-zj
v1.11.0-5.0.rc2.1
v2.0.1-5.0.rc2
v1.11.0-5.0.rc2
v1.8.1-5.0.rc2
v2.0.0-5.0.rc2
v1.8.1-5.0.rc1
v1.11.0-5.0.rc1
v1.11.0-yd
v1.11.0-xf
v1.11.0-infer
v1.11.0-bigkernel
v1.11.0-host_api
v1.8.1-3.0.0
v1.11.0-5.0.rc2.t100
v1.8.1-5.0.rc2.t100
v1.8.1-3.0.0-dev
v1.11.0-3.0.0
v2.0-dev
v1.8.1-3.0.rc3
v1.5.0-3.0.0
v1.5.0
v1.8.1-3.0.rc1
v1.11.0-3.0.rc3
v1.8.1-3.0.rc2
v1.5.0-3.0.rc3
v1.5.0-3.0.rc2
2.0.4.tr5
v1.5.0-3.0.rc1
2.0.2.tr5
2.0.3.tr5
v7.2.RC1.alpha002-pytorch2.8.0
v7.2.RC1.alpha002-pytorch2.7.1
v7.2.RC1.alpha002-pytorch2.6.0
v7.2.RC1.alpha002-pytorch2.1.0
v7.1.0.2-pytorch2.5.1
v7.1.0.2-pytorch2.6.0
v7.1.0.2-pytorch2.1.0
v7.0.0.1-pytorch2.4.0
v7.0.0.1-pytorch2.1.0
v7.2.RC1.alpha001-pytorch2.8.0
v7.2.RC1.alpha001-pytorch2.7.1
v7.2.RC1.alpha001-pytorch2.6.0
v7.2.RC1.alpha001-pytorch2.5.1
v7.2.RC1.alpha001-pytorch2.1.0
v7.1.0.1-pytorch2.6.0
v7.1.0.1-pytorch2.5.1
v7.1.0.1-pytorch2.1.0
v7.1.0-pytorch2.6.0
v7.1.0-pytorch2.5.1
v7.1.0-pytorch2.1.0
v7.1.RC1.alpha003-pytorch2.6.0
v7.1.RC1.alpha003-pytorch2.5.1
v7.1.RC1.alpha003-pytorch2.1.0
v7.1.RC1.alpha002-pytorch2.7.1
v7.1.RC1.alpha002-pytorch2.6.0
v7.1.RC1.alpha002-pytorch2.5.1
v7.1.RC1.alpha002-pytorch2.4.0
v7.1.RC1.alpha002-pytorch2.3.1
v7.1.RC1.alpha002-pytorch2.1.0
v6.0.0.1-pytorch2.4.0
v6.0.0.1-pytorch2.3.1
v6.0.0.1-pytorch2.1.0
v7.1.RC1.alpha001-pytorch2.6.0
v7.1.RC1.alpha001-pytorch2.5.1
v7.1.RC1.alpha001-pytorch2.4.0
v7.1.RC1.alpha001-pytorch2.3.1
v7.1.RC1.alpha001-pytorch2.1.0
v7.0.0-pytorch2.5.1
v7.0.0-pytorch2.4.0
v7.0.0-pytorch2.3.1
v7.0.RC1.alpha002-pytorch2.6.0
v7.0.0-pytorch2.1.0
v7.0.RC1.alpha002-pytorch2.5.1
v7.0.RC1.alpha002-pytorch2.4.0
v7.0.RC1.alpha002-pytorch2.3.1
v7.0.RC1.alpha002-pytorch2.1.0
v7.0.RC1.alpha001-pytorch2.5.1
v7.0.RC1.alpha001-pytorch2.1.0
v7.0.RC1.alpha001-pytorch2.4.0
v7.0.RC1.alpha001-pytorch2.3.1
v6.0.0-pytorch2.4.0
v6.0.0-pytorch2.3.1
v6.0.0-pytorch2.1.0
v6.0.0.alpha003-pytorch2.4.0
v6.0.0.alpha003-pytorch2.3.1
v6.0.0.alpha003-pytorch2.1.0
v6.0.0.alpha002-pytorch2.4.0
v6.0.0.alpha002-pytorch2.3.1
v6.0.0.alpha002-pytorch2.1.0
v6.0.0.alpha001-pytorch2.5.1
v6.0.rc3-pytorch2.4.0
v6.0.rc3-pytorch2.3.1
v6.0.rc3-pytorch2.1.0
v6.0.0.alpha001-pytorch2.4.0
v6.0.0.alpha001-pytorch2.3.1
v6.0.0.alpha001-pytorch2.1.0
v6.0.rc2.1-pytorch1.11.0
v6.0.rc2.1-pytorch2.3.1
v6.0.rc2.1-pytorch2.2.0
v6.0.rc2.1-pytorch2.1.0
v6.0.rc3.alpha003-pytorch2.3.1
v6.0.rc3.alpha003-pytorch2.1.0
v6.0.rc3.alpha001-pytorch2.4.0
v6.0.rc3.alpha002-pytorch2.3.1
v6.0.rc3.alpha002-pytorch2.2.0
v6.0.rc3.alpha002-pytorch2.1.0
v6.0.rc3.alpha002-pytorch1.11.0
v6.0.rc2-pytorch2.1.0
v6.0.rc2-pytorch2.3.1
v6.0.rc2-pytorch2.2.0
v6.0.rc2-pytorch1.11.0
v6.0.rc3.alpha001-pytorch2.3.1
v6.0.rc3.alpha001-pytorch2.2.0
v6.0.rc3.alpha001-pytorch2.1.0
v6.0.rc3.alpha001-pytorch1.11.0
v6.0.rc2.alpha002-pytorch2.3.1
v6.0.rc2.alpha003-pytorch1.11.0
v6.0.rc2.alpha003-pytorch2.2.0
v6.0.rc2.alpha003-pytorch2.1.0
v6.0.rc1.1-pytorch2.2.0
v6.0.rc1.1-pytorch2.1.0
v6.0.rc1.1-pytorch1.11.0
v5.0.1.2-pytorch1.11.0
v5.0.1.2-pytorch2.1.0
v5.0.1.2-pytorch2.0.1
v6.0.rc2.alpha002-pytorch2.2.0
v6.0.rc2.alpha002-pytorch2.1.0
v6.0.rc2.alpha002-pytorch1.11.0
v6.0.rc1-pytorch2.2.0
v6.0.rc1-pytorch2.1.0
v6.0.rc1-pytorch1.11.0
v6.0.rc2.alpha001-pytorch2.2.0
v6.0.rc2.alpha001-pytorch2.1.0
v6.0.rc2.alpha001-pytorch1.11.0
v6.0.rc1.alpha003-pytorch2.0.1
v6.0.rc1.alpha003-pytorch2.1.0
v5.0.1.1-pytorch2.0.1
v5.0.1.1-pytorch1.11.0
v5.0.1.1-pytorch2.1.0
v6.0.rc1.alpha003-pytorch1.11.0
v6.0.rc1.alpha002-pytorch2.1.0
v6.0.rc1.alpha002-pytorch1.11.0
v6.0.rc1.alpha002-pytorch2.0.1
v6.0.rc1.alpha001-pytorch2.2.0
v5.0.1-pytorch2.1.0
v5.0.1-pytorch2.0.1
v5.0.1-pytorch1.11.0
v6.0.RC1.alpha001-pytorch2.0.1
v6.0.RC1.alpha001-pytorch2.1.0
v6.0.RC1.alpha001-pytorch1.11.0
v5.0.0-pytorch2.1.0
v5.0.0-pytorch2.0.1
v5.0.0-pytorch1.11.0
v5.0.0.alpha003-pytorch2.1.0
v5.0.0.alpha003-pytorch2.0.1
v5.0.0.alpha003-pytorch1.11.0
v5.0.rc3.3-pytorch1.11.0
v5.0.rc3.2-pytorch1.11.0
v5.0.0.alpha002-pytorch2.1.0
v5.0.0.alpha002-pytorch2.0.1
v5.0.0.alpha002-pytorch1.11.0
v5.0.rc3.1-pytorch1.11.0
v5.0.0.alpha001-pytorch2.1.0
v5.0.0.alpha001-pytorch2.0.1
v5.0.0.alpha001-pytorch1.11.0
v5.0.rc3-pytorch2.1.0
v5.0.rc3-pytorch2.0.1
v5.0.rc3-pytorch1.11.0
v5.0.rc3.alpha003-pytorch2.0.1
v5.0.rc3.alpha003-pytorch1.11.0
v5.0.rc3.alpha003-pytorch1.8.1
v5.0.rc2.2-pytorch1.11.0
v5.0.rc2.1-pytorch1.11.0
v5.0.rc3.alpha002-pytorch2.0.1
v5.0.rc3.alpha002-pytorch1.11.0
v5.0.rc3.alpha002-pytorch1.8.1
v5.0.rc2-pytorch2.0.1
v5.0.rc2-pytorch1.11.0
v5.0.rc2-pytorch1.8.1
v5.0.rc3.alpha001-pytorch1.8.1
v5.0.rc3.alpha001-pytorch1.11.0
v5.0.rc2.alpha003-pytorch1.11.0
v5.0.rc2.alpha003-pytorch1.8.1
v5.0.rc2.alpha002-pytorch1.11.0
v5.0.rc2.alpha002-pytorch1.8.1
v5.0.rc1.alpha003-pytorch1.11.0
v5.0.rc1.alpha003-pytorch1.8.1
v5.0.rc1-pytorch1.11.0
v5.0.rc1-pytorch1.8.1
v5.0.rc1.alpha002-pytorch1.11.0
v5.0.rc1.alpha002-pytorch1.8.1
v5.0.rc1.alpha001-pytorch1.11.0
v5.0.rc1.alpha001-pytorch1.8.1
v3.0.0-pytorch1.11.0
v3.0.0-pytorch1.8.1
v3.0.0-pytorch1.5.0
v3.0.alpha006-pytorch1.8.1
v3.0.alpha005-pytorch1.8.1
v3.0.alpha003-pytorch1.8.1
v3.0.rc3-pytorch1.11.0
v3.0.rc3-pytorch1.8.1
v3.0.rc3-pytorch1.5.0
v3.0.rc2-pytorch1.8.1
v3.0.rc2-pytorch1.5.0
v3.0.rc1-pytorch1.8.1
v3.0.rc1-pytorch1.5.0
v2.0.4
v2.0.4-rc2
v2.0.4-rc1
v2.0.3.1
v2.0.3
v2.0.3-rc4
v2.0.3-rc3
v2.0.3-rc2
v2.0.3-rc1
v2.0.2
开始日期 - 截止日期
-
置顶选项
不置顶
置顶等级：高
置顶等级：中
置顶等级：低
优先级
不指定
严重
主要
次要
不重要
预计工期
（小时）
参与者（2）
Python
1
https://gitee.com/ascend/pytorch.git
git@gitee.com:ascend/pytorch.git
ascend
pytorch
pytorch
点此查找更多帮助
搜索帮助
Git 命令在线学习
如何在 Gitee 导入 GitHub 仓库
Git 仓库基础操作
企业版和社区版功能对比
SSH 公钥设置
如何处理代码冲突
仓库体积过大，如何减小？
如何找回被删除的仓库数据
Gitee 产品配额说明
GitHub仓库快速导入Gitee及同步更新
什么是 Release（发行版）
将 PHP 项目自动发布到 packagist.org
仓库举报
回到顶部
登录提示
该操作需登录 Gitee 帐号，请先登录后再操作。
立即登录
没有帐号，去注册
