transformers+npu+ray训练

一、问题现象（附报错日志上下文）：
继续此issues：[训练问题核查](https://gitee.com/ascend/pytorch/issues/IBI0BR?from=project-issue)
ray华为支持人员建议该问题回到这里：[ray结合](https://github.com/ray-project/ray/issues/49998)
1. 910
训练日志：其实训练成功，但最后报错了
![输入图片说明](https://foruda.gitee.com/images/1739779186945528789/1927b4fe_519029.png "屏幕截图")

2. 910b2
训练日志：其实训练成功，但最后报错了
![输入图片说明](https://foruda.gitee.com/images/1739778964241738317/ac0455ae_519029.png "屏幕截图")

二、软件版本: 
-- CANN 版本 : 8.0.RC3 **（为了匹配上2.1.0.post8，更换的此版本。github也建议可以用8.0.0.alpha3）** 
-- Pytorch 版本:2.1.0
-- Python 版本 :3.9.21
-- transformers :4.48.1
-- torch-npu: 2.1.0.post8 _**（github里面建议的）**_

三、测试步骤：
还是之前那边的issues的训练脚本

四、日志信息:
1. 910-plog日志：
日志1：
[ERROR] RUNTIME(735007,):2025-02-17-16:02:48.606.126 [runtime.cc:5267]735007 GetVisibleDevices:set ASCEND_RT_VISIBLE_DEVICES: error, input data rang[0-1)
[ERROR] RUNTIME(735007,):2025-02-17-16:02:48.607.463 [api_impl.cc:2245]735007 GetDeviceCount:report error module_name=EE1001
[ERROR] RUNTIME(735007,):2025-02-17-16:02:48.607.468 [api_impl.cc:2245]735007 GetDeviceCount:set ASCEND_RT_VISIBLE_DEVICES: error, input data rang[0-1)
[ERROR] RUNTIME(735007,):2025-02-17-16:02:48.616.270 [api_c_device.cc:21]735007 rtGetDeviceCount:ErrCode=207004, desc=[driver error:no valid device], InnerCode=0x7020012
[ERROR] RUNTIME(735007,):2025-02-17-16:02:48.616.280 [error_message_manage.cc:53]735007 FuncErrorReason:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(735007,):2025-02-17-16:02:48.616.287 [error_message_manage.cc:53]735007 FuncErrorReason:rtGetDeviceCount execute failed, reason=[driver error:no valid device]
[ERROR] ASCENDCL(735007,):2025-02-17-16:02:48.616.327 [device.cpp:366]735007 aclrtGetDeviceCount: get device count failed, runtime result = 207004.
[ERROR] APP(735007,):2025-02-17-16:02:48.616.352 [log_inner.cpp:76]735007 compiler_depend.ts:device_count:24: "[PTA]:"get device count of NPU failed""
[ERROR] RUNTIME(735007,):2025-02-17-16:02:49.217.353 [api_impl.cc:2245]735007 GetDeviceCount:report error module_name=EE1001
[ERROR] RUNTIME(735007,):2025-02-17-16:02:49.217.373 [api_impl.cc:2245]735007 GetDeviceCount:set ASCEND_RT_VISIBLE_DEVICES: error, input data rang[0-1)
[ERROR] RUNTIME(735007,):2025-02-17-16:02:49.217.492 [api_c_device.cc:21]735007 rtGetDeviceCount:ErrCode=207004, desc=[driver error:no valid device], InnerCode=0x7020012
[ERROR] RUNTIME(735007,):2025-02-17-16:02:49.217.497 [error_message_manage.cc:53]735007 FuncErrorReason:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(735007,):2025-02-17-16:02:49.217.504 [error_message_manage.cc:53]735007 FuncErrorReason:rtGetDeviceCount execute failed, reason=[driver error:no valid device]
[ERROR] ASCENDCL(735007,):2025-02-17-16:02:49.217.534 [device.cpp:366]735007 aclrtGetDeviceCount: get device count failed, runtime result = 207004.
[ERROR] APP(735007,):2025-02-17-16:02:49.217.556 [log_inner.cpp:76]735007 compiler_depend.ts:device_count:24: "[PTA]:"get device count of NPU failed""

日志2：
[ERROR] DRV(735202,):2025-02-17-16:03:02.078.292 [ascend][curpid: 735202, 735847][drv][hdc][share_log_read_in_single_module 634]Server create success.(dev_id=2; fid=0; service_type="service_dump"; pid=321441)
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.004 [pywrapper.cpp:94][CANNKB][Tid:735202]"Traceback (most recent call last):
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.032 [pywrapper.cpp:94][CANNKB][Tid:735202]"  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/multiprocessing/managers.py", line 802, in _callmethod
    conn = self._tls.connection
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.036 [pywrapper.cpp:94][CANNKB][Tid:735202]"AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.040 [pywrapper.cpp:94][CANNKB][Tid:735202]"
During handling of the above exception, another exception occurred:

"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.043 [pywrapper.cpp:94][CANNKB][Tid:735202]"Traceback (most recent call last):
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.046 [pywrapper.cpp:94][CANNKB][Tid:735202]"  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/interface.py", line 42, in cann_kb_finalize
    RouteServer.finalize()
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.049 [pywrapper.cpp:94][CANNKB][Tid:735202]"  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 53, in wrapper
    return func(cls, *args, **kwargs)
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.052 [pywrapper.cpp:94][CANNKB][Tid:735202]"  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/route.py", line 202, in finalize
    cls.cann_kb_mgr.finalize()
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.054 [pywrapper.cpp:94][CANNKB][Tid:735202]"  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/knowledge_bank_manager.py", line 42, in finalize
    self.__static_mgr.close()
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.057 [pywrapper.cpp:94][CANNKB][Tid:735202]"  File "/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/tbe/common/repository_manager/cann_kb_manager/base_manager.py", line 333, in close
    for name in copy.deepcopy(self.write_kb.keys()):
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.060 [pywrapper.cpp:94][CANNKB][Tid:735202]"  File "<string>", line 2, in keys
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.063 [pywrapper.cpp:94][CANNKB][Tid:735202]"  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/multiprocessing/managers.py", line 806, in _callmethod
    self._connect()
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.065 [pywrapper.cpp:94][CANNKB][Tid:735202]"  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/multiprocessing/managers.py", line 793, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.068 [pywrapper.cpp:94][CANNKB][Tid:735202]"  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/multiprocessing/connection.py", line 502, in Client
    c = SocketClient(address)
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.071 [pywrapper.cpp:94][CANNKB][Tid:735202]"  File "/home/ma-user/anaconda3/envs/MindSpore/lib/python3.9/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.073 [pywrapper.cpp:94][CANNKB][Tid:735202]"ConnectionRefusedError: [Errno 111] Connection refused
"
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.656 [py_interface.cpp:201][CANNKB][Tid:735202]"Failed to call cann_kb_finalize."
[ERROR] TUNE(735202,):2025-02-17-16:06:58.947.664 [cann_kb_api.cpp:40][CANNKB][Tid:735202]"Run CannKbFinalize Error!"
[ERROR] TEFUSION(735202,):2025-02-17-16:06:58.947.681 [fusion_manager.cc:666]735202 TbeDeinit call CannKbFinalize failed. res = [6].
[ERROR] DRV(735202,):2025-02-17-16:07:00.008.146 [ascend][curpid: 735202, 735847][drv][hdc][share_log_read_in_single_module 634]Server destroy success.(dev_id=2; fid=0; service="service_dump"; pid=321441)

2. 910b2-plog日志：
日志1：
[ERROR] RUNTIME(89293,):2025-02-17-08:00:41.797.684 [runtime.cc:5473]89293 GetVisibleDevices:set ASCEND_RT_VISIBLE_DEVICES:[] error, input data rang[0-8)
[ERROR] RUNTIME(89293,):2025-02-17-08:00:41.800.437 [api_impl.cc:2426]89293 GetDeviceCount:report error module_name=EE1001
[ERROR] RUNTIME(89293,):2025-02-17-08:00:41.800.442 [api_impl.cc:2426]89293 GetDeviceCount:set ASCEND_RT_VISIBLE_DEVICES: error, input data rang[0-8)
[ERROR] RUNTIME(89293,):2025-02-17-08:00:41.809.938 [api_c_device.cc:23]89293 rtGetDeviceCount:ErrCode=207004, desc=[driver error:no valid device], InnerCode=0x7020012
[ERROR] RUNTIME(89293,):2025-02-17-08:00:41.809.958 [error_message_manage.cc:53]89293 FuncErrorReason:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(89293,):2025-02-17-08:00:41.809.966 [error_message_manage.cc:53]89293 FuncErrorReason:rtGetDeviceCount execute failed, reason=[driver error:no valid device]
[ERROR] ASCENDCL(89293,):2025-02-17-08:00:41.810.029 [device.cpp:342]89293 aclrtGetDeviceCount: get device count failed, runtime result = 207004.
[ERROR] APP(89293,):2025-02-17-08:00:41.810.075 [log_inner.cpp:76]89293 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:device_count:26: "[PTA]:"get device count of NPU failed""
[ERROR] RUNTIME(89293,):2025-02-17-08:00:43.301.951 [api_impl.cc:2426]89293 GetDeviceCount:report error module_name=EE1001
[ERROR] RUNTIME(89293,):2025-02-17-08:00:43.301.999 [api_impl.cc:2426]89293 GetDeviceCount:set ASCEND_RT_VISIBLE_DEVICES: error, input data rang[0-8)
[ERROR] RUNTIME(89293,):2025-02-17-08:00:43.302.146 [api_c_device.cc:23]89293 rtGetDeviceCount:ErrCode=207004, desc=[driver error:no valid device], InnerCode=0x7020012
[ERROR] RUNTIME(89293,):2025-02-17-08:00:43.302.153 [error_message_manage.cc:53]89293 FuncErrorReason:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(89293,):2025-02-17-08:00:43.302.160 [error_message_manage.cc:53]89293 FuncErrorReason:rtGetDeviceCount execute failed, reason=[driver error:no valid device]
[ERROR] ASCENDCL(89293,):2025-02-17-08:00:43.302.197 [device.cpp:342]89293 aclrtGetDeviceCount: get device count failed, runtime result = 207004.
[ERROR] APP(89293,):2025-02-17-08:00:43.303.030 [log_inner.cpp:76]89293 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:device_count:26: "[PTA]:"get device count of NPU failed""

日志2：
[ERROR] RUNTIME(89293,):2025-02-17-08:00:41.797.684 [runtime.cc:5473]89293 GetVisibleDevices:set ASCEND_RT_VISIBLE_DEVICES:[] error, input data rang[0-8)
[ERROR] RUNTIME(89293,):2025-02-17-08:00:41.800.437 [api_impl.cc:2426]89293 GetDeviceCount:report error module_name=EE1001
[ERROR] RUNTIME(89293,):2025-02-17-08:00:41.800.442 [api_impl.cc:2426]89293 GetDeviceCount:set ASCEND_RT_VISIBLE_DEVICES: error, input data rang[0-8)
[ERROR] RUNTIME(89293,):2025-02-17-08:00:41.809.938 [api_c_device.cc:23]89293 rtGetDeviceCount:ErrCode=207004, desc=[driver error:no valid device], InnerCode=0x7020012
[ERROR] RUNTIME(89293,):2025-02-17-08:00:41.809.958 [error_message_manage.cc:53]89293 FuncErrorReason:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(89293,):2025-02-17-08:00:41.809.966 [error_message_manage.cc:53]89293 FuncErrorReason:rtGetDeviceCount execute failed, reason=[driver error:no valid device]
[ERROR] ASCENDCL(89293,):2025-02-17-08:00:41.810.029 [device.cpp:342]89293 aclrtGetDeviceCount: get device count failed, runtime result = 207004.
[ERROR] APP(89293,):2025-02-17-08:00:41.810.075 [log_inner.cpp:76]89293 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:device_count:26: "[PTA]:"get device count of NPU failed""
[ERROR] RUNTIME(89293,):2025-02-17-08:00:43.301.951 [api_impl.cc:2426]89293 GetDeviceCount:report error module_name=EE1001
[ERROR] RUNTIME(89293,):2025-02-17-08:00:43.301.999 [api_impl.cc:2426]89293 GetDeviceCount:set ASCEND_RT_VISIBLE_DEVICES: error, input data rang[0-8)
[ERROR] RUNTIME(89293,):2025-02-17-08:00:43.302.146 [api_c_device.cc:23]89293 rtGetDeviceCount:ErrCode=207004, desc=[driver error:no valid device], InnerCode=0x7020012
[ERROR] RUNTIME(89293,):2025-02-17-08:00:43.302.153 [error_message_manage.cc:53]89293 FuncErrorReason:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(89293,):2025-02-17-08:00:43.302.160 [error_message_manage.cc:53]89293 FuncErrorReason:rtGetDeviceCount execute failed, reason=[driver error:no valid device]
[ERROR] ASCENDCL(89293,):2025-02-17-08:00:43.302.197 [device.cpp:342]89293 aclrtGetDeviceCount: get device count failed, runtime result = 207004.
[ERROR] APP(89293,):2025-02-17-08:00:43.303.030 [log_inner.cpp:76]89293 build/CMakeFiles/torch_npu.dir/compiler_depend.ts:device_count:26: "[PTA]:"get device count of NPU failed""

Ascend/pytorch
暂停

内容风险标识

评论 (8)

Ascend/pytorch暂停 .gitee-modal { width: 500px !important; }

内容风险标识