[Question|问题咨询]: 多机环境，hccl初始化失败，报错 rtGet device logicid by PhyId failed, return[107001], para: phyId[2], devIndex[0]

### 问题描述

我使用 ray 构建了2机器，4卡的环境：
![输入图片说明](https://foruda.gitee.com/images/1741053951078090780/33968295_9106882.png "屏幕截图")

两台机器情况分别是：
- 机器1：启动的单机8卡docker容器，使用 ASCEND_RT_VISIBLE_DEVICES=2,3，使用其中的 2,3 卡

- 机器2：启动的2卡容器，2卡使用的是机器的6，7卡，容器显示的是 ASCEND_VISIBLE_DEVICES=6,7

torch/torch_npu版本是2.3.1

程序报错：
![输入图片说明](https://foruda.gitee.com/images/1741053980597142052/74e1cae8_9106882.png "屏幕截图")

ascend log 错误日志：
![输入图片说明](https://foruda.gitee.com/images/1741054009855033029/e392e19b_9106882.png "屏幕截图")

文本信息：
/home/tangzhiyi/ascend/log/debug/plog/plog-152624_20250303023245585.log:[ERROR] DRV(152624,python):2025-03-03-02:32:52.300.199 [ascend][curpid: 152624, 152624][drv][hdc][share_log_read_in_single_module 634]Server create success.(dev_id=0; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=1; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=2; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=3; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=4; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=5; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=6; fid=0; service_type="service_dump"; pid=196419) Server create success.(dev_id=7; fid=0; service_type="service_dump"; pid=196419)

/home/tangzhiyi/ascend/log/debug/plog/plog-152624_20250303023302554.log:[ERROR] DRV(152624,python):2025-03-03-02:33:31.579.452 [ascend][curpid: 152624, 152836][drv][hdc][share_log_read_in_single_module 634]Server destroy success.(dev_id=0; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=1; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=2; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=3; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=4; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=5; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=6; fid=0; service="service_dump"; pid=196419) Server destroy success.(dev_id=7; fid=0; service="service_dump"; pid=196419)

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] DRV(46997,rWrapper.warmup_dist):2025-03-03-02:33:19.079.426 [ascend][curpid: 46997, 153334][drv][hdc][share_log_read_in_single_module 634]Server create success.(dev_id=0; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=1; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=2; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=3; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=4; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=5; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=6; fid=0; service_type="service_dump"; pid=27871) Server create success.(dev_id=7; fid=0; service_type="service_dump"; pid=27871)

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.288 [runtime.cc:5606]153273 GetUserDevIdByDeviceId:[INIT][DEFAULT]get userDevId failed, input deviceId:2 real deviceCnt:8 isSetVisibleDev:1 ASCEND_RT_VISIBLE_DEVICES:[3].

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.320 [api_error.cc:1918]153273 GetDeviceIndexByPhyId:[INIT][DEFAULT]report error module_type=0, module_name=EE9999

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.338 [api_error.cc:1918]153273 GetDeviceIndexByPhyId:[INIT][DEFAULT]get error: phyid:2 realDeviceId:2 is err:0x7010003

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.605 [api_c_device.cc:277]153273 rtGetDeviceIndexByPhyId:[INIT][DEFAULT]ErrCode=107001, desc=[device id error], InnerCode=0x7010003

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.633 [error_message_manage.cc:53]153273 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] RUNTIME(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.591.662 [error_message_manage.cc:53]153273 FuncErrorReason:[INIT][DEFAULT]rtGetDeviceIndexByPhyId execute failed, reason=[device id error]

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.566 [adapter_rts.cc:379][153273][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[2], devIndex[0]

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.581 [adapter_rts.cc:845][153273]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.593 [adapter_rts.cc:873][153273]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.605 [hccl_communicator_attrs.cc:219][153273]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.616 [hccl_communicator_attrs.cc:198][153273]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.627 [hccl_communicator_attrs.cc:890][153273]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.637 [hccl_communicator_attrs.cc:29][153273]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.647 [hccl_communicator.cc:233][153273]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.659 [hccl_comm.cc:117][153273][HcclComm][Init]errNo[0x000000000500000f] hccl initialize failed

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.671 [op_base.cc:894][153273][InitCommRootInfo]errNo[0x000000000500000f] hcclComm init error

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCP(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.752 [rs_socket.c:2744]tid:153273,rs_peer_get_ifnum(2744) : param error, g_rs_cb is NULL

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCP(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.763 [ra_peer.c:964]tid:153273,ra_peer_get_ifnum(964) : [get][ra_peer_ifnum]rs_peer_get_ifnum failed(-22)

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCP(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.788 [ra_host.c:1930]tid:153273,ra_get_ifnum(1930) : [get][ra_ifnum]ra_peer_get_ifnum failed, ret(-22)

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.805 [adapter_hccp.cc:1492][153273][Get][IfNum]errNo[0x000000000500000b] ra get if num fail. ret[128303], num[0]

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.816 [adapter_hccp.cc:1355][153273]call trace: hcclRet -> 11

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.827 [sal.cc:346][153273]call trace: hcclRet -> 11

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.843 [op_base.cc:922][153273][InitCommRootInfo]Init failed, return[0x000000000500000f], rankNum[4], rank[1], rootInfo identifier[192.168.215.237%bond0_60002_2_1740969205920414], server[0.0.0.0], logicDevId[0]

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.936 [op_base.cc:1816][153273][HcclCommDestroy] comm is not exist, comm=0xffce937a3320, group=192.168.215.237%bond0_60002_2_1740969205920414, deviceLogicId=0

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.593.232 [op_base.cc:974][153273][Init][CommRootInfoConfig]errNo[0x000000000500000f]HcclCommInitRootInfo failed.

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] DRV(46737,rWrapper.warmup_dist):2025-03-03-02:33:19.079.428 [ascend][curpid: 46737, 153335][drv][hdc][share_log_read_in_single_module 634]Server create success.(dev_id=0; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=1; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=2; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=3; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=4; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=5; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=6; fid=0; service_type="service_dump"; pid=27609) Server create success.(dev_id=7; fid=0; service_type="service_dump"; pid=27609)

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.510.706 [runtime.cc:5606]153274 GetUserDevIdByDeviceId:[INIT][DEFAULT]get userDevId failed, input deviceId:3 real deviceCnt:8 isSetVisibleDev:1 ASCEND_RT_VISIBLE_DEVICES:[2].

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.510.730 [api_error.cc:1918]153274 GetDeviceIndexByPhyId:[INIT][DEFAULT]report error module_type=0, module_name=EE9999

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.510.745 [api_error.cc:1918]153274 GetDeviceIndexByPhyId:[INIT][DEFAULT]get error: phyid:3 realDeviceId:3 is err:0x7010003

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.083 [api_c_device.cc:277]153274 rtGetDeviceIndexByPhyId:[INIT][DEFAULT]ErrCode=107001, desc=[device id error], InnerCode=0x7010003

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.107 [error_message_manage.cc:53]153274 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] RUNTIME(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.129 [error_message_manage.cc:53]153274 FuncErrorReason:[INIT][DEFAULT]rtGetDeviceIndexByPhyId execute failed, reason=[device id error]

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.881 [adapter_rts.cc:379][153274][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[3], devIndex[0]

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.893 [adapter_rts.cc:847][153274]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.902 [adapter_rts.cc:873][153274]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.912 [hccl_communicator_attrs.cc:219][153274]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.920 [hccl_communicator_attrs.cc:198][153274]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.929 [hccl_communicator_attrs.cc:890][153274]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.936 [hccl_communicator_attrs.cc:29][153274]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.944 [hccl_communicator.cc:233][153274]call trace: hcclRet -> 15

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.954 [hccl_comm.cc:117][153274][HcclComm][Init]errNo[0x000000000500000f] hccl initialize failed

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.964 [op_base.cc:894][153274][InitCommRootInfo]errNo[0x000000000500000f] hcclComm init error

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCP(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.038 [rs_socket.c:2744]tid:153274,rs_peer_get_ifnum(2744) : param error, g_rs_cb is NULL

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCP(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.046 [ra_peer.c:964]tid:153274,ra_peer_get_ifnum(964) : [get][ra_peer_ifnum]rs_peer_get_ifnum failed(-22)

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCP(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.056 [ra_host.c:1930]tid:153274,ra_get_ifnum(1930) : [get][ra_ifnum]ra_peer_get_ifnum failed, ret(-22)

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.068 [adapter_hccp.cc:1492][153274][Get][IfNum]errNo[0x000000000500000b] ra get if num fail. ret[128303], num[0]

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.077 [adapter_hccp.cc:1355][153274]call trace: hcclRet -> 11

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.086 [sal.cc:346][153274]call trace: hcclRet -> 11

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.098 [op_base.cc:922][153274][InitCommRootInfo]Init failed, return[0x000000000500000f], rankNum[4], rank[0], rootInfo identifier[192.168.215.237%bond0_60002_2_1740969205920414], server[0.0.0.0], logicDevId[0]

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.177 [op_base.cc:1816][153274][HcclCommDestroy] comm is not exist, comm=0xffceaf820fd0, group=192.168.215.237%bond0_60002_2_1740969205920414, deviceLogicId=0

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.512.414 [op_base.cc:974][153274][Init][CommRootInfoConfig]errNo[0x000000000500000f]HcclCommInitRootInfo failed.

感觉核心问题是：

/home/tangzhiyi/ascend/log/debug/plog/plog-46737_20250303023319108.log:[ERROR] HCCL(46737,rWrapper.warmup_dist):2025-03-03-02:33:29.511.881 [adapter_rts.cc:379][153274][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[3], devIndex[0]

/home/tangzhiyi/ascend/log/debug/plog/plog-46997_20250303023319108.log:[ERROR] HCCL(46997,rWrapper.warmup_dist):2025-03-03-02:33:29.592.566 [adapter_rts.cc:379][153273][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[2], devIndex[0]

/root/ascend/log/debug/plog/plog-664_20250303023250894.log:[ERROR] HCCL(664,rmup_dist):2025-03-03-02:33:01.812.951 [adapter_rts.cc:379][782][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[6], devIndex[0]

/root/ascend/log/debug/plog/plog-665_20250303023250944.log:[ERROR] HCCL(665,rmup_dist):2025-03-03-02:33:01.773.085 [adapter_rts.cc:379][783][Get][DeviceIndex]errNo[0x0000000005000010] rtGet device logicid by PhyId failed, return[107001], para: phyId[7], devIndex[0]

这里我每张卡相当于是一个ray 启动的进程，在当前进程来看，device idx 应该是0，对应的物理卡号是2/3/6/7 应该是没问题的吧？但是报错了，求帮看下是什么原因

初始化代码：

```
def init_process_group(rank: int, world_size: int):
    """init process group."""
    from lmdeploy.pytorch.devices import get_device_manager
    
    # 获取当前设备类型
    device_type = get_device_manager().current_context().device_type
    
    # 根据设备类型选择合适的后端
    if device_type == 'ascend':
        backend = 'hccl'
    else:
        backend = 'nccl'
    
    DIST_TIMEOUT = timedelta(days=35600)
    dist.init_process_group(backend=backend, rank=rank, world_size=world_size, timeout=DIST_TIMEOUT)
    assert dist.is_initialized()
```
cann版本：8.0.0.beta1

cpu架构：aarch64

硬件：Atlas 800T A2

Ascend/cann-hccl
暂停

内容风险标识

评论 (7)

Ascend/cann-hccl暂停 .gitee-modal { width: 500px !important; }

内容风险标识