/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch/distributed/launch.py:181: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use-env is set by default in torchrun.
If your script expects `--local-rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py:18: UserWarning: Register eager implementation for the 'npu' backend of dynamo, as torch_npu was not compiled with torchair.
  warnings.warn(
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py:18: UserWarning: Register eager implementation for the 'npu' backend of dynamo, as torch_npu was not compiled with torchair.
  warnings.warn(
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py:18: UserWarning: Register eager implementation for the 'npu' backend of dynamo, as torch_npu was not compiled with torchair.
  warnings.warn(
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py:18: UserWarning: Register eager implementation for the 'npu' backend of dynamo, as torch_npu was not compiled with torchair.
  warnings.warn(
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py:18: UserWarning: Register eager implementation for the 'npu' backend of dynamo, as torch_npu was not compiled with torchair.
  warnings.warn(
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py:18: UserWarning: Register eager implementation for the 'npu' backend of dynamo, as torch_npu was not compiled with torchair.
  warnings.warn(
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py:18: UserWarning: Register eager implementation for the 'npu' backend of dynamo, as torch_npu was not compiled with torchair.
  warnings.warn(
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/dynamo/__init__.py:18: UserWarning: Register eager implementation for the 'npu' backend of dynamo, as torch_npu was not compiled with torchair.
  warnings.warn(
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:162: ImportWarning: 
    *************************************************************************************************************
    The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
    The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
    The backend in torch.distributed.init_process_group set to hccl now..
    The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
    The device parameters have been replaced with npu in the function below:
    torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
    *************************************************************************************************************
    
  warnings.warn(msg, ImportWarning)
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:124: RuntimeWarning: torch.jit.script will be disabled by transfer_to_npu, which currently does not support it.
  warnings.warn(msg, RuntimeWarning)
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:124: RuntimeWarning: torch.jit.script will be disabled by transfer_to_npu, which currently does not support it.
  warnings.warn(msg, RuntimeWarning)
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:124: RuntimeWarning: torch.jit.script will be disabled by transfer_to_npu, which currently does not support it.
  warnings.warn(msg, RuntimeWarning)
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:124: RuntimeWarning: torch.jit.script will be disabled by transfer_to_npu, which currently does not support it.
  warnings.warn(msg, RuntimeWarning)
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:124: RuntimeWarning: torch.jit.script will be disabled by transfer_to_npu, which currently does not support it.
  warnings.warn(msg, RuntimeWarning)
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:124: RuntimeWarning: torch.jit.script will be disabled by transfer_to_npu, which currently does not support it.
  warnings.warn(msg, RuntimeWarning)
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:124: RuntimeWarning: torch.jit.script will be disabled by transfer_to_npu, which currently does not support it.
  warnings.warn(msg, RuntimeWarning)
/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/contrib/transfer_to_npu.py:124: RuntimeWarning: torch.jit.script will be disabled by transfer_to_npu, which currently does not support it.
  warnings.warn(msg, RuntimeWarning)
Traceback (most recent call last):
  File "pretrain_baichuan.py", line 25, in <module>
Traceback (most recent call last):
Traceback (most recent call last):
  File "pretrain_baichuan.py", line 25, in <module>
  File "pretrain_baichuan.py", line 25, in <module>
Traceback (most recent call last):
  File "pretrain_baichuan.py", line 25, in <module>
Traceback (most recent call last):
  File "pretrain_baichuan.py", line 25, in <module>
Traceback (most recent call last):
  File "pretrain_baichuan.py", line 25, in <module>
        import deepspeed_npuimport deepspeed_npu    

import deepspeed_npu
  File "/root/baichuan2/deepspeed_npu/deepspeed_npu/__init__.py", line 6, in <module>
  File "/root/baichuan2/deepspeed_npu/deepspeed_npu/__init__.py", line 6, in <module>
  File "/root/baichuan2/deepspeed_npu/deepspeed_npu/__init__.py", line 6, in <module>
    import deepspeed_npu
  File "/root/baichuan2/deepspeed_npu/deepspeed_npu/__init__.py", line 6, in <module>
    import deepspeed_npu
  File "/root/baichuan2/deepspeed_npu/deepspeed_npu/__init__.py", line 6, in <module>
    import deepspeed_npu
  File "/root/baichuan2/deepspeed_npu/deepspeed_npu/__init__.py", line 6, in <module>
    FLAG_SUPPORT_INF_NAN = hasattr(torch_npu.npu.utils, 'is_support_inf_nan') and torch_npu.npu.utils.is_support_inf_nan()
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/utils.py", line 313, in is_support_inf_nan
    FLAG_SUPPORT_INF_NAN = hasattr(torch_npu.npu.utils, 'is_support_inf_nan') and torch_npu.npu.utils.is_support_inf_nan()
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/utils.py", line 313, in is_support_inf_nan
    FLAG_SUPPORT_INF_NAN = hasattr(torch_npu.npu.utils, 'is_support_inf_nan') and torch_npu.npu.utils.is_support_inf_nan()
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/utils.py", line 313, in is_support_inf_nan
        FLAG_SUPPORT_INF_NAN = hasattr(torch_npu.npu.utils, 'is_support_inf_nan') and torch_npu.npu.utils.is_support_inf_nan()FLAG_SUPPORT_INF_NAN = hasattr(torch_npu.npu.utils, 'is_support_inf_nan') and torch_npu.npu.utils.is_support_inf_nan()

  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/utils.py", line 313, in is_support_inf_nan
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/utils.py", line 313, in is_support_inf_nan
    FLAG_SUPPORT_INF_NAN = hasattr(torch_npu.npu.utils, 'is_support_inf_nan') and torch_npu.npu.utils.is_support_inf_nan()
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/utils.py", line 313, in is_support_inf_nan
    torch_npu.npu._lazy_init()
      File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/__init__.py", line 203, in _lazy_init
torch_npu.npu._lazy_init()
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/__init__.py", line 203, in _lazy_init
    torch_npu.npu._lazy_init()
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/__init__.py", line 203, in _lazy_init
    torch_npu._C._npu_init()    
torch_npu.npu._lazy_init()    
RuntimeErrortorch_npu._C._npu_init()  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/__init__.py", line 203, in _lazy_init

: Initialize:torch_npu/csrc/core/npu/sys_ctrl/npu_sys_ctrl.cpp:120 NPU error, error code is 507008
[Error]: Failed to obtain the SOC version. 
        Rectify the fault based on the error information in the ascend log.
EE1001: The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        [Init][Version]init soc version failed, ret = 507008[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4541]
        The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
RuntimeError
: Initialize:torch_npu/csrc/core/npu/sys_ctrl/npu_sys_ctrl.cpp:120 NPU error, error code is 507008
[Error]: Failed to obtain the SOC version. 
        Rectify the fault based on the error information in the ascend log.
EE1001: The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        [Init][Version]init soc version failed, ret = 507008[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4541]
        The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
    
torch_npu.npu._lazy_init()
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/__init__.py", line 203, in _lazy_init
    torch_npu._C._npu_init()
RuntimeError: Initialize:torch_npu/csrc/core/npu/sys_ctrl/npu_sys_ctrl.cpp:120 NPU error, error code is 507008
[Error]: Failed to obtain the SOC version. 
        Rectify the fault based on the error information in the ascend log.
EE1001: The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        [Init][Version]init soc version failed, ret = 507008[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4541]
        The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
    
torch_npu.npu._lazy_init()
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/__init__.py", line 203, in _lazy_init
    torch_npu._C._npu_init()
RuntimeError    : torch_npu._C._npu_init()Initialize:torch_npu/csrc/core/npu/sys_ctrl/npu_sys_ctrl.cpp:120 NPU error, error code is 507008
[Error]: Failed to obtain the SOC version. 
        Rectify the fault based on the error information in the ascend log.
EE1001: The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        [Init][Version]init soc version failed, ret = 507008[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4541]
        The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]


RuntimeError: Initialize:torch_npu/csrc/core/npu/sys_ctrl/npu_sys_ctrl.cpp:120 NPU error, error code is 507008
[Error]: Failed to obtain the SOC version. 
        Rectify the fault based on the error information in the ascend log.
EE1001: The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        [Init][Version]init soc version failed, ret = 507008[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4541]
        The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]

    torch_npu._C._npu_init()
RuntimeError: Initialize:torch_npu/csrc/core/npu/sys_ctrl/npu_sys_ctrl.cpp:120 NPU error, error code is 507008
[Error]: Failed to obtain the SOC version. 
        Rectify the fault based on the error information in the ascend log.
EE1001: The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        [Init][Version]init soc version failed, ret = 507008[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4541]
        The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]

Traceback (most recent call last):
Traceback (most recent call last):
  File "pretrain_baichuan.py", line 25, in <module>
  File "pretrain_baichuan.py", line 25, in <module>
        import deepspeed_npuimport deepspeed_npu

  File "/root/baichuan2/deepspeed_npu/deepspeed_npu/__init__.py", line 6, in <module>
  File "/root/baichuan2/deepspeed_npu/deepspeed_npu/__init__.py", line 6, in <module>
        FLAG_SUPPORT_INF_NAN = hasattr(torch_npu.npu.utils, 'is_support_inf_nan') and torch_npu.npu.utils.is_support_inf_nan()FLAG_SUPPORT_INF_NAN = hasattr(torch_npu.npu.utils, 'is_support_inf_nan') and torch_npu.npu.utils.is_support_inf_nan()

  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/utils.py", line 313, in is_support_inf_nan
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/utils.py", line 313, in is_support_inf_nan
    torch_npu.npu._lazy_init()
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/__init__.py", line 203, in _lazy_init
    torch_npu.npu._lazy_init()
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch_npu/npu/__init__.py", line 203, in _lazy_init
    torch_npu._C._npu_init()
RuntimeError: Initialize:torch_npu/csrc/core/npu/sys_ctrl/npu_sys_ctrl.cpp:120 NPU error, error code is 507008
[Error]: Failed to obtain the SOC version. 
        Rectify the fault based on the error information in the ascend log.
EE1001: The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        [Init][Version]init soc version failed, ret = 507008[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4541]
        The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
    
torch_npu._C._npu_init()
RuntimeError: Initialize:torch_npu/csrc/core/npu/sys_ctrl/npu_sys_ctrl.cpp:120 NPU error, error code is 507008
[Error]: Failed to obtain the SOC version. 
        Rectify the fault based on the error information in the ascend log.
EE1001: The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]
        Solution: 1.Check the input parameter range of the function. 2.Check the function invocation relationship.
        TraceBack (most recent call last):
        [Init][Version]init soc version failed, ret = 507008[FUNC:ReportInnerError][FILE:log_inner.cpp][LINE:145]
        ctx is NULL![FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4541]
        The argument is invalid.Reason: rtGetDevMsg execute failed, reason=[context pointer null]

[2024-02-22 14:35:53,097] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 205527) of binary: /root/.local/conda/envs/baichuan2/bin/python
Traceback (most recent call last):
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch/distributed/launch.py", line 196, in <module>
    main()
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch/distributed/launch.py", line 192, in main
    launch(args)
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch/distributed/launch.py", line 177, in launch
    run(args)
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/.local/conda/envs/baichuan2/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
pretrain_baichuan.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-02-22_14:35:53
  host      : dl-231116164921eba-pod-jupyter-b8f66cdd9-knmld
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 205528)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-02-22_14:35:53
  host      : dl-231116164921eba-pod-jupyter-b8f66cdd9-knmld
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 205529)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-02-22_14:35:53
  host      : dl-231116164921eba-pod-jupyter-b8f66cdd9-knmld
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 205530)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time      : 2024-02-22_14:35:53
  host      : dl-231116164921eba-pod-jupyter-b8f66cdd9-knmld
  rank      : 4 (local_rank: 4)
  exitcode  : 1 (pid: 205531)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
  time      : 2024-02-22_14:35:53
  host      : dl-231116164921eba-pod-jupyter-b8f66cdd9-knmld
  rank      : 5 (local_rank: 5)
  exitcode  : 1 (pid: 205532)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
  time      : 2024-02-22_14:35:53
  host      : dl-231116164921eba-pod-jupyter-b8f66cdd9-knmld
  rank      : 6 (local_rank: 6)
  exitcode  : 1 (pid: 205533)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
  time      : 2024-02-22_14:35:53
  host      : dl-231116164921eba-pod-jupyter-b8f66cdd9-knmld
  rank      : 7 (local_rank: 7)
  exitcode  : 1 (pid: 205534)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-02-22_14:35:53
  host      : dl-231116164921eba-pod-jupyter-b8f66cdd9-knmld
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 205527)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================