all_reduce 报错

一、问题现象（附报错日志上下文）：
`dist.all_reduce(start_time_tensor, op=dist.ReduceOp.MIN)` 该操作执行失败

```
[2024-08-19 20:42:09,564] torch.distributed.run: [WARNING] master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.
[2024-08-19 20:42:09,564] torch.distributed.run: [WARNING]
[2024-08-19 20:42:09,564] torch.distributed.run: [WARNING] *****************************************
[2024-08-19 20:42:09,564] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
[2024-08-19 20:42:09,564] torch.distributed.run: [WARNING] *****************************************
/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:299: ImportWarning:
    *************************************************************************************************************
    The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
    The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
    The backend in torch.distributed.init_process_group set to hccl now..
    The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
    The device parameters have been replaced with npu in the function below:
    torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
    *************************************************************************************************************

warnings.warn(msg, ImportWarning)
/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gpytorch/lazy/__init__.py:70: DeprecationWarning: GPyTorch will be replacing all LazyTensor functionality with the linear operator package. Replace all references to gpytorch.lazy.*LazyTensor with linear_operator.operators.*LinearOperator.
  warnings.warn(
/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/gpytorch/lazy/__init__.py:70: DeprecationWarning: GPyTorch will be replacing all LazyTensor functionality with the linear operator package. Replace all references to gpytorch.lazy.*LazyTensor with linear_operator.operators.*LinearOperator.
  warnings.warn(
[W NPUCachingAllocator.cpp:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
[W NPUCachingAllocator.cpp:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
Traceback (most recent call last):
  File "/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 2050, in all_reduce
    work = group.allreduce([tensor], opts)
RuntimeError: create:torch_npu/csrc/distributed/ProcessGroupHCCL.cpp:80 HCCL function error: HcclCommInitRootInfo(numRanks, &rootInfo, rank, &(comm->hcclComm_)), error code is 7
[ERROR] 2024-08-19-20:42:18 (PID:107212, Device:0, RankID:1) ERR02200 DIST call hccl api failed.
EJ0001: [PID: 107212] 2024-08-19-20:42:18.103.684 Failed to initialize the HCCP process. Reason: Maybe the last training process is running.
        Solution: Wait for 10s after killing the last training process and try again.
        TraceBack (most recent call last):
        tsd client wait response fail, hostpid:77509, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:336]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "all_reduce_test.py", line 34, in <module>
    test_all_reduce()
  File "all_reduce_test.py", line 27, in test_all_reduce
    dist.all_reduce(start_time_tensor, op=dist.ReduceOp.MIN)
  File "/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/torch/distributed/c10d_logger.py", line 52, in wrapper
    "args": f"{args}, {kwargs}",
  File "/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/torch/_tensor.py", line 431, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/torch/_tensor_str.py", line 664, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/torch/_tensor_str.py", line 595, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/torch/_tensor_str.py", line 347, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/torch/_tensor_str.py", line 138, in __init__
    tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0)
RuntimeError: Sync:torch_npu/csrc/framework/OpCommand.cpp:183 NPU function error: c10_npu::acl::AclrtSynchronizeStreamWithTimeout(stream), error code is 507018
[ERROR] 2024-08-19-20:42:25 (PID:107212, Device:0, RankID:1) ERR00100 PTA call acl api failed
[Error]: The aicpu execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 107212] 2024-08-19-20:42:25.305.850 Kernel task happen error, retCode=0x2a, [aicpu exception].[FUNC:PreCheckTaskErr][FILE:davinic_kernel_task.cc][LINE:1221]
        TraceBack (most recent call last):
       AICPU Kernel task happen error, retCode=0x2a.[FUNC:GetError][FILE:stream.cc][LINE:1067]
       Aicpu kernel execute failed, device_id=0, stream_id=5, task_id=2, errorCode=2a.[FUNC:PrintAicpuErrorInfo][FILE:davinic_kernel_task.cc][LINE:1024]
       Aicpu kernel execute failed, device_id=0, stream_id=5, task_id=2, fault op_name=[FUNC:GetError][FILE:stream.cc][LINE:1067]
       rtStreamSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
       synchronize stream failed, runtime result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
       Fail to get sq reg virtual addr, deviceId=0, sqId=9.[FUNC:Setup][FILE:stream.cc][LINE:639]
       stream setup failed, retCode=0x7020010.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:5199]
       Sync get device msg failed, retCode=0x7020010.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:5255]
       rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

EJ0001: [PID: 107212] 2024-08-19-20:42:18.103.684 Failed to initialize the HCCP process. Reason: Maybe the last training process is running.
        Solution: Wait for 10s after killing the last training process and try again.
        TraceBack (most recent call last):
        tsd client wait response fail, hostpid:77509, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:336]

[W NPUStream.cpp:409] Warning: NPU warning, error code is 507018[Error]:
[Error]: The aicpu execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 107212] 2024-08-19-20:42:25.321.721 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
       Fail to get sq reg virtual addr, deviceId=0, sqId=10.[FUNC:Setup][FILE:stream.cc][LINE:639]
       stream setup failed, retCode=0x7020010.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:5199]
       Sync get device msg failed, retCode=0x7020010.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:5255]
        rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

EZ9999: Inner Error!
EZ9999: [PID: 107212] 2024-08-19-20:42:25.305.850 Kernel task happen error, retCode=0x2a, [aicpu exception].[FUNC:PreCheckTaskErr][FILE:davinic_kernel_task.cc][LINE:1221]
        TraceBack (most recent call last):
       AICPU Kernel task happen error, retCode=0x2a.[FUNC:GetError][FILE:stream.cc][LINE:1067]
       Aicpu kernel execute failed, device_id=0, stream_id=5, task_id=2, errorCode=2a.[FUNC:PrintAicpuErrorInfo][FILE:davinic_kernel_task.cc][LINE:1024]
       Aicpu kernel execute failed, device_id=0, stream_id=5, task_id=2, fault op_name=[FUNC:GetError][FILE:stream.cc][LINE:1067]
       rtStreamSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
       synchronize stream failed, runtime result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
       Fail to get sq reg virtual addr, deviceId=0, sqId=9.[FUNC:Setup][FILE:stream.cc][LINE:639]
       stream setup failed, retCode=0x7020010.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:5199]
       Sync get device msg failed, retCode=0x7020010.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:5255]
       rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

EJ0001: [PID: 107212] 2024-08-19-20:42:18.103.684 Failed to initialize the HCCP process. Reason: Maybe the last training process is running.
        Solution: Wait for 10s after killing the last training process and try again.
        TraceBack (most recent call last):
        tsd client wait response fail, hostpid:77509, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:336]
 (function npuSynchronizeUsedDevices)
[W NPUStream.cpp:392] Warning: NPU warning, error code is 507018[Error]:
[Error]: The aicpu execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 107212] 2024-08-19-20:42:25.322.195 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
       Fail to get sq reg virtual addr, deviceId=0, sqId=11.[FUNC:Setup][FILE:stream.cc][LINE:639]
       stream setup failed, retCode=0x7020010.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:5199]
       Sync get device msg failed, retCode=0x7020010.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:5255]
        rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 107212] 2024-08-19-20:42:25.321.721 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
       Fail to get sq reg virtual addr, deviceId=0, sqId=10.[FUNC:Setup][FILE:stream.cc][LINE:639]
       stream setup failed, retCode=0x7020010.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:5199]
       Sync get device msg failed, retCode=0x7020010.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:5255]
        rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

EZ9999: Inner Error!
EZ9999: [PID: 107212] 2024-08-19-20:42:25.305.850 Kernel task happen error, retCode=0x2a, [aicpu exception].[FUNC:PreCheckTaskErr][FILE:davinic_kernel_task.cc][LINE:1221]
        TraceBack (most recent call last):
       AICPU Kernel task happen error, retCode=0x2a.[FUNC:GetError][FILE:stream.cc][LINE:1067]
       Aicpu kernel execute failed, device_id=0, stream_id=5, task_id=2, errorCode=2a.[FUNC:PrintAicpuErrorInfo][FILE:davinic_kernel_task.cc][LINE:1024]
       Aicpu kernel execute failed, device_id=0, stream_id=5, task_id=2, fault op_name=[FUNC:GetError][FILE:stream.cc][LINE:1067]
       rtStreamSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
       synchronize stream failed, runtime result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
       Fail to get sq reg virtual addr, deviceId=0, sqId=9.[FUNC:Setup][FILE:stream.cc][LINE:639]
       stream setup failed, retCode=0x7020010.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:5199]
       Sync get device msg failed, retCode=0x7020010.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:5255]
       rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

EJ0001: [PID: 107212] 2024-08-19-20:42:18.103.684 Failed to initialize the HCCP process. Reason: Maybe the last training process is running.
        Solution: Wait for 10s after killing the last training process and try again.
        TraceBack (most recent call last):
        tsd client wait response fail, hostpid:77509, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:336]
 (function npuSynchronizeDevice)
[W NPUStream.cpp:392] Warning: NPU warning, error code is 507018[Error]:
[Error]: The aicpu execution is abnormal.
        Rectify the fault based on the error information in the ascend log.
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 107212] 2024-08-19-20:42:25.322.526 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
       Fail to get sq reg virtual addr, deviceId=0, sqId=12.[FUNC:Setup][FILE:stream.cc][LINE:639]
       stream setup failed, retCode=0x7020010.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:5199]
       Sync get device msg failed, retCode=0x7020010.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:5255]
        rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 107212] 2024-08-19-20:42:25.322.195 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
       Fail to get sq reg virtual addr, deviceId=0, sqId=11.[FUNC:Setup][FILE:stream.cc][LINE:639]
       stream setup failed, retCode=0x7020010.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:5199]
       Sync get device msg failed, retCode=0x7020010.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:5255]
        rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
EH9999: [PID: 107212] 2024-08-19-20:42:25.321.721 wait for compute device to finish failed, runtime result = 507018.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
       Fail to get sq reg virtual addr, deviceId=0, sqId=10.[FUNC:Setup][FILE:stream.cc][LINE:639]
       stream setup failed, retCode=0x7020010.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:5199]
       Sync get device msg failed, retCode=0x7020010.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:5255]
        rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

EZ9999: Inner Error!
EZ9999: [PID: 107212] 2024-08-19-20:42:25.305.850 Kernel task happen error, retCode=0x2a, [aicpu exception].[FUNC:PreCheckTaskErr][FILE:davinic_kernel_task.cc][LINE:1221]
        TraceBack (most recent call last):
       AICPU Kernel task happen error, retCode=0x2a.[FUNC:GetError][FILE:stream.cc][LINE:1067]
       Aicpu kernel execute failed, device_id=0, stream_id=5, task_id=2, errorCode=2a.[FUNC:PrintAicpuErrorInfo][FILE:davinic_kernel_task.cc][LINE:1024]
       Aicpu kernel execute failed, device_id=0, stream_id=5, task_id=2, fault op_name=[FUNC:GetError][FILE:stream.cc][LINE:1067]
       rtStreamSynchronizeWithTimeout execute failed, reason=[aicpu exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
       synchronize stream failed, runtime result = 507018[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
       Fail to get sq reg virtual addr, deviceId=0, sqId=9.[FUNC:Setup][FILE:stream.cc][LINE:639]
       stream setup failed, retCode=0x7020010.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:5199]
       Sync get device msg failed, retCode=0x7020010.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:5255]
       rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
```

二、软件版本:
-- CANN 版本 (e.g., CANN 3.0.x，5.x.x):  8.0.RC3
--Tensorflow/Pytorch/MindSpore 版本: torch-npu 2.1.0.post6  torch 2.1.0+cpu
--Python 版本 (e.g., Python 3.7.5): 3.8
-- MindStudio版本 (e.g., MindStudio 2.0.0 (beta3)):
--操作系统版本 (e.g., Ubuntu 18.04):

三、测试步骤：

```
import torch
import mindspeed.megatron_adaptor
import torch_npu
import torch.distributed as dist

def test_all_reduce():
    dist.init_process_group("nccl")
    rank = dist.get_rank()
    print(f"Start running basic DDP example on rank {rank}.")

import time
    start_time = time.time()

start_time_tensor = torch.tensor([start_time], dtype=torch.double, device='npu')
    dist.all_reduce(start_time_tensor, op=dist.ReduceOp.MIN)
    print(start_time_tensor.item())

dist.destroy_process_group()

if __name__ == "__main__":
    test_all_reduce()
```

保存代码为 `all_reduce_test.py`

```
torchrun --nnodes=1 --nproc_per_node=2 --rdzv_id=100 --rdzv_backend=c10d --rdzv_endpoint=localhost:29411 all_reduce_test.py
```

四、日志信息:
xxxx
请根据自己的运行环境参考以下方式搜集日志信息，如果涉及到算子开发相关的问题，建议也提供UT/ST测试和单算子集成测试相关的日志。

日志提供方式:
将日志打包后作为附件上传。若日志大小超出附件限制，则可上传至外部网盘后提供链接。

获取方法请参考wiki：
https://gitee.com/ascend/modelzoo/wikis/%E5%A6%82%E4%BD%95%E8%8E%B7%E5%8F%96%E6%97%A5%E5%BF%97%E5%92%8C%E8%AE%A1%E7%AE%97%E5%9B%BE?sort_id=4097825

Ascend/MindSpeed
暂停

内容风险标识

评论 (1)

Ascend/MindSpeed暂停 .gitee-modal { width: 500px !important; }

内容风险标识