[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.679 [exchanger_network.cc:280][64810][136514][Wait][AllServerSocketEstab]errNo[0x000000000500000b] server : device[0] rank[0] get socket timeout, total[4] remain[1]
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.735 [exchanger_network.cc:344][64810][136514]call trace: hcclRet -> 4
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.749 [exchanger_network.cc:107][64810][136514][ExchangerNetwork][Init]rank[0] device[0] wait all socket establish 1200 second failed
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.777 [exchanger_network.cc:686][64810][136514]Some NPUs get socket timeout, the details are as follows:
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.791 [exchanger_network.cc:687][64810][136514]   _____________________________________________________
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.802 [exchanger_network.cc:688][64810][136514]   |device[0] userrank[0] exchanger Status: run_step[2]|
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.812 [exchanger_network.cc:689][64810][136514]   |  dest_dev  |  userrank  |    Role    | connStatus |
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.836 [exchanger_network.cc:690][64810][136514]   |------------|------------|------------|------------|
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.855 [exchanger_network.cc:709][64810][136514]   |         0  |         0  |     NA     |     NA     |
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.867 [exchanger_network.cc:709][64810][136514]   |         1  |         1  |   server   |     YES     |
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.877 [exchanger_network.cc:709][64810][136514]   |         2  |         2  |   client   |     YES     |
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.886 [exchanger_network.cc:709][64810][136514]   |         3  |         3  |   server   |     YES     |
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.910 [exchanger_network.cc:709][64810][136514]   |         4  |         4  |   client   |     YES     |
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.919 [exchanger_network.cc:709][64810][136514]   |         5  |         5  |   server   |     NO     |
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.928 [exchanger_network.cc:709][64810][136514]   |         6  |         6  |   client   |     YES     |
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.936 [exchanger_network.cc:709][64810][136514]   |         7  |         7  |   server   |     YES     |
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.945 [exchanger_network.cc:714][64810][136514]   ___________________________________________________________________
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.954 [exchanger_network.cc:715][64810][136514]the connection failure between this device and the target device may be due to the following reasons:
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.963 [exchanger_network.cc:716][64810][136514]1. the connection between this device and the target device is abnormal.
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.972 [exchanger_network.cc:717][64810][136514]2. an exception occurred at the target devices.
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.982 [exchanger_network.cc:719][64810][136514]3. the time difference between the execution of hcom on this device and the target device exceeds the timeout threshold. make sure this by keyworld [Entry-]
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.895.992 [exchanger_network.cc:721][64810][136514]4. the behavior of executing the calculation graph on this device and the target device is inconsistent.
[ERROR] HCCL(64810,python):2023-06-02-13:38:17.896.004 [comm_factory.cc:1470][64810][136514][Get][ExchangerNetwork]exchanger init failed
[ERROR] HCCL(64810,python):2023-06-02-13:38:18.027.621 [comm_factory.cc:223][64810][136514][Create][CommOuter]exchangerNetwork create failed
[ERROR] HCCL(64810,python):2023-06-02-13:38:18.027.652 [hccl_impl.cc:3820][64810][136514][Create][OuterComm]errNo[0x0000000005000006] tag[HcomAllReduce_6629421139219749105_8], created commOuter fail. commOuter[0] is null
[ERROR] HCCL(64810,python):2023-06-02-13:38:18.027.771 [hccl_impl.cc:3675][hccl-64810-2-1685711347-4-7943117906144662061][0][Create][CommByAlg]CreateInnerComm [0] or CreateOuterComm[6] failed. commType[2]
[ERROR] HCCL(64810,python):2023-06-02-13:38:18.027.808 [hccl_impl.cc:3542][hccl-64810-2-1685711347-4-7943117906144662061][0]call trace: hcclRet -> 4
[ERROR] HCCL(64810,python):2023-06-02-13:38:18.029.604 [hccl_impl.cc:1301][hccl-64810-2-1685711347-4-7943117906144662061][0][HcclImpl][AllReduce]errNo[0x0000000005000004]  tag[HcomAllReduce_6629421139219749105_8],all reduce create comm failed
[ERROR] HCCL(64810,python):2023-06-02-13:38:18.029.623 [hccl_comm.cc:290][hccl-64810-2-1685711347-4-7943117906144662061][0]call trace: hcclRet -> 4
[ERROR] HCCL(64810,python):2023-06-02-13:38:18.029.641 [hcom.cc:575][hccl-64810-2-1685711347-4-7943117906144662061][0][AllReduce][Result]errNo[0x0000000005010004] hcclComm all reduce error, tag[HcomAllReduce_6629421139219749105_8], input_ptr[0x1241e6128a00], output_ptr[0x1241e6129000], count[128], data_type[2], op[0]
[ERROR] HCCL(64810,python):2023-06-02-13:38:18.029.659 [hcom_ops_kernel_info_store.cc:766][hccl-64810-2-1685711347-4-7943117906144662061][0]call trace: hcclRet -> 4
[ERROR] HCCL(64810,python):2023-06-02-13:38:18.029.708 [hcom_ops_kernel_info_store.cc:279][hccl-64810-2-1685711347-4-7943117906144662061][0]call trace: hcclRet -> 4
[ERROR] HCCL(64810,python):2023-06-02-13:38:18.029.722 [hcom_ops_kernel_info_store.cc:1791][hccl-64810-2-1685711347-4-7943117906144662061][0][Load][Task]errNo[0x0000000005010004] load task failed. (load op[HcomAllReduce] fail)
[CRITICAL] GE(64810,ffff8df40cf0,python):2023-06-02-13:38:18.030.405 [mindspore/ccsrc/plugin/device/ascend/hal/device/ge_runtime/task/hccl_task.cc:122] Distribute] davinci_model : load task fail, return ret: 1343225860, the task is Default/network-PanguAlphaTrainOneStepWithLossScaleCell/AllReduce-op19888_4645130
EI0006: Getting socket times out. Reason: 1. The remote does not initiate a connect request. some NPUs in the cluster are abnormal.    2. The remote does not initiate a connect request because the collective communication operator is started too late or is not started by some NPU in the cluster.    3. The communication link is disconnected. (For example, the IP addresses are not on the same network segment or the TLS configurations are inconsistent.)
        Solution: 1. Check the rank service processes with other errors or no errors in the cluster.2. If this error is reported for all NPUs, check whether the time difference between the earliest and latest errors is greater than the connect timeout interval (120s by default). If so, adjust the timeout interval by using the HCCL_CONNECT_TIMEOUT environment variable.3. Check the connectivity of the communication link between nodes. (For details, see the TLS command and HCCN connectivity check examples.)

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)
[CRITICAL] DEVICE(64810,ffff8df40cf0,python):2023-06-02-13:40:53.843.711 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:592] LoadTask] Distribute Task Failed,
error msg: davinci_model : load task fail, return ret: 1343225860, the task is Default/network-PanguAlphaTrainOneStepWithLossScaleCell/AllReduce-op19888_4645130

EI0006: Getting socket times out. Reason: 1. The remote does not initiate a connect request. some NPUs in the cluster are abnormal.    2. The remote does not initiate a connect request because the collective communication operator is started too late or is not started by some NPU in the cluster.    3. The communication link is disconnected. (For example, the IP addresses are not on the same network segment or the TLS configurations are inconsistent.)
        Solution: 1. Check the rank service processes with other errors or no errors in the cluster.2. If this error is reported for all NPUs, check whether the time difference between the earliest and latest errors is greater than the connect timeout interval (120s by default). If so, adjust the timeout interval by using the HCCL_CONNECT_TIMEOUT environment variable.3. Check the connectivity of the communication link between nodes. (For details, see the TLS command and HCCN connectivity check examples.)

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)

mindspore/ccsrc/plugin/device/ascend/hal/device/ge_runtime/task/hccl_task.cc:122 Distribute

EI0006: Getting socket times out. Reason: 1. The remote does not initiate a connect request. some NPUs in the cluster are abnormal.    2. The remote does not initiate a connect request because the collective communication operator is started too late or is not started by some NPU in the cluster.    3. The communication link is disconnected. (For example, the IP addresses are not on the same network segment or the TLS configurations are inconsistent.)
        Solution: 1. Check the rank service processes with other errors or no errors in the cluster.2. If this error is reported for all NPUs, check whether the time difference between the earliest and latest errors is greater than the connect timeout interval (120s by default). If so, adjust the timeout interval by using the HCCL_CONNECT_TIMEOUT environment variable.3. Check the connectivity of the communication link between nodes. (For details, see the TLS command and HCCN connectivity check examples.)

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)
[CRITICAL] DEVICE(64810,ffff8df40cf0,python):2023-06-02-13:40:53.844.690 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_kernel_executor.cc:244] PreprocessBeforeRunGraph] Preprocess failed before run graph 1.
Distribute Task Failed,
error msg: davinci_model : load task fail, return ret: 1343225860, the task is Default/network-PanguAlphaTrainOneStepWithLossScaleCell/AllReduce-op19888_4645130

EI0006: Getting socket times out. Reason: 1. The remote does not initiate a connect request. some NPUs in the cluster are abnormal.    2. The remote does not initiate a connect request because the collective communication operator is started too late or is not started by some NPU in the cluster.    3. The communication link is disconnected. (For example, the IP addresses are not on the same network segment or the TLS configurations are inconsistent.)
        Solution: 1. Check the rank service processes with other errors or no errors in the cluster.2. If this error is reported for all NPUs, check whether the time difference between the earliest and latest errors is greater than the connect timeout interval (120s by default). If so, adjust the timeout interval by using the HCCL_CONNECT_TIMEOUT environment variable.3. Check the connectivity of the communication link between nodes. (For details, see the TLS command and HCCN connectivity check examples.)

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)
EI0006: Getting socket times out. Reason: 1. The remote does not initiate a connect request. some NPUs in the cluster are abnormal.    2. The remote does not initiate a connect request because the collective communication operator is started too late or is not started by some NPU in the cluster.    3. The communication link is disconnected. (For example, the IP addresses are not on the same network segment or the TLS configurations are inconsistent.)
        Solution: 1. Check the rank service processes with other errors or no errors in the cluster.2. If this error is reported for all NPUs, check whether the time difference between the earliest and latest errors is greater than the connect timeout interval (120s by default). If so, adjust the timeout interval by using the HCCL_CONNECT_TIMEOUT environment variable.3. Check the connectivity of the communication link between nodes. (For details, see the TLS command and HCCN connectivity check examples.)

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)

mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:592 LoadTask
mindspore/ccsrc/plugin/device/ascend/hal/device/ge_runtime/task/hccl_task.cc:122 Distribute

EI0006: Getting socket times out. Reason: 1. The remote does not initiate a connect request. some NPUs in the cluster are abnormal.    2. The remote does not initiate a connect request because the collective communication operator is started too late or is not started by some NPU in the cluster.    3. The communication link is disconnected. (For example, the IP addresses are not on the same network segment or the TLS configurations are inconsistent.)
        Solution: 1. Check the rank service processes with other errors or no errors in the cluster.2. If this error is reported for all NPUs, check whether the time difference between the earliest and latest errors is greater than the connect timeout interval (120s by default). If so, adjust the timeout interval by using the HCCL_CONNECT_TIMEOUT environment variable.3. Check the connectivity of the communication link between nodes. (For details, see the TLS command and HCCN connectivity check examples.)

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)
[TRACE] HCCL(64810,python):2023-06-02-13:41:02.710.607 [status:stop] [hcom.cc:452][hccl-64810-2-1685711347-4-7943117906144662061][0]hcom destroy complete,take time [678041]us, rankNum[32], rank[0]
Traceback (most recent call last):
  File "/home/jenkins/workspace/TDT_deployment/solution_test/remaining/test_scripts/mindspore/net/pangu_alpha/network/test_ms_pangu_alpha_ascend_train_32p_0001/train.py", line 557, in <module>
    run_train(opt)
  File "/home/jenkins/workspace/TDT_deployment/solution_test/remaining/test_scripts/mindspore/net/pangu_alpha/network/test_ms_pangu_alpha_ascend_train_32p_0001/train.py", line 254, in run_train
    model.train(actual_epoch_num, ds, callbacks=callback, sink_size=args_opt.sink_size, dataset_sink_mode=True)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/train/model.py", line 1048, in train
    initial_epoch=initial_epoch)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/train/model.py", line 102, in wrapper
    func(self, *args, **kwargs)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/train/model.py", line 600, in _train
    cb_params, sink_size, initial_epoch, valid_infos)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/train/model.py", line 684, in _train_dataset_sink_process
    outputs = train_network(*inputs)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/nn/cell.py", line 626, in __call__
    out = self.compile_and_run(*args, **kwargs)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/nn/cell.py", line 945, in compile_and_run
    self.compile(*args, **kwargs)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/nn/cell.py", line 923, in compile
    jit_config_dict=self._jit_config_dict, *args, **kwargs)
  File "/home/miniconda3/envs/ci/lib/python3.7/site-packages/mindspore/common/api.py", line 1401, in compile
    result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode())
RuntimeError: Preprocess failed before run graph 1.

----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EI0006: Getting socket times out. Reason: 1. The remote does not initiate a connect request. some NPUs in the cluster are abnormal.    2. The remote does not initiate a connect request because the collective communication operator is started too late or is not started by some NPU in the cluster.    3. The communication link is disconnected. (For example, the IP addresses are not on the same network segment or the TLS configurations are inconsistent.)
        Solution: 1. Check the rank service processes with other errors or no errors in the cluster.2. If this error is reported for all NPUs, check whether the time difference between the earliest and latest errors is greater than the connect timeout interval (120s by default). If so, adjust the timeout interval by using the HCCL_CONNECT_TIMEOUT environment variable.3. Check the connectivity of the communication link between nodes. (For details, see the TLS command and HCCN connectivity check examples.)

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)

----------------------------------------------------
- Framework Error Message: (For framework developers)
----------------------------------------------------
Distribute Task Failed,
error msg: davinci_model : load task fail, return ret: 1343225860, the task is Default/network-PanguAlphaTrainOneStepWithLossScaleCell/AllReduce-op19888_4645130

----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EI0006: Getting socket times out. Reason: 1. The remote does not initiate a connect request. some NPUs in the cluster are abnormal.    2. The remote does not initiate a connect request because the collective communication operator is started too late or is not started by some NPU in the cluster.    3. The communication link is disconnected. (For example, the IP addresses are not on the same network segment or the TLS configurations are inconsistent.)
        Solution: 1. Check the rank service processes with other errors or no errors in the cluster.2. If this error is reported for all NPUs, check whether the time difference between the earliest and latest errors is greater than the connect timeout interval (120s by default). If so, adjust the timeout interval by using the HCCL_CONNECT_TIMEOUT environment variable.3. Check the connectivity of the communication link between nodes. (For details, see the TLS command and HCCN connectivity check examples.)

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)
EI0006: Getting socket times out. Reason: 1. The remote does not initiate a connect request. some NPUs in the cluster are abnormal.    2. The remote does not initiate a connect request because the collective communication operator is started too late or is not started by some NPU in the cluster.    3. The communication link is disconnected. (For example, the IP addresses are not on the same network segment or the TLS configurations are inconsistent.)
        Solution: 1. Check the rank service processes with other errors or no errors in the cluster.2. If this error is reported for all NPUs, check whether the time difference between the earliest and latest errors is greater than the connect timeout interval (120s by default). If so, adjust the timeout interval by using the HCCL_CONNECT_TIMEOUT environment variable.3. Check the connectivity of the communication link between nodes. (For details, see the TLS command and HCCN connectivity check examples.)

(Please search "Ascend Error Message" at https://www.mindspore.cn for error code description)

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_kernel_executor.cc:244 PreprocessBeforeRunGraph
mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:592 LoadTask
mindspore/ccsrc/plugin/device/ascend/hal/device/ge_runtime/task/hccl_task.cc:122 Distribute