单机8卡910b环境，qwen1.5-7b进行推理，报错 torch.distributed.elastic.multiprocessing.api

[2024-07-24 14:14:18,921] torch.distributed.run: [WARNING] *****************************************
WARNING:root:Warning: You did not install torch_npu
WARNING:root:Warning: You did not install torch_npu
WARNING:root:Warning: You did not install torch_npu
WARNING:root:Warning: You did not install torch_npu
WARNING:root:Warning: You did not install torch_npu
WARNING:root:Warning: You did not install torch_npu
WARNING:root:Warning: You did not install torch_npu
WARNING:root:Warning: You did not install torch_npu
[2024-07-24 14:14:24,137] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -11) local_rank: 0 (pid: 100589) of binary: /root/miniconda3/envs/qwen1.5/bin/python
Traceback (most recent call last):
  File "/root/miniconda3/envs/qwen1.5/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/root/miniconda3/envs/qwen1.5/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/root/miniconda3/envs/qwen1.5/lib/python3.10/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/root/miniconda3/envs/qwen1.5/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/root/miniconda3/envs/qwen1.5/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/miniconda3/envs/qwen1.5/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError.

当前环境是
python=3.10
torch,torch-npu = 2.1.0

GVP Ascend/MindSpeed-LLM

内容风险标识

评论 (2)

GVPAscend/MindSpeed-LLM

内容风险标识