PointPillar运行模型训练脚本报错HCCL function error: HcclCommInitRootInfo(numRanks, &rootInfo, rank, &(comm->hcclComm_)), error code is 7

运行如下的训练脚本：
https://gitee.com/ascend/mxDriving/tree/master/model_examples/OpenPCDet#%E6%A8%A1%E5%9E%8B%E8%AE%AD%E7%BB%83-1

bash train_pointpillar_full_8p.sh报错：

1、重启容器
[axeadmin@pm-799e ~]$ sudo docker restart OpenPCDet
OpenPCDet

2、连接容器
sudo docker container exec -it OpenPCDet /bin/bash

3、检查当前没有python进程在运行
root@pm-799e:/# ps -ef |grep python
root          28      14  0 11:17 pts/1    00:00:00 grep --color=auto python

4、运行训练脚本：
cd /data/lff/autodrive/mxDriving/model_examples/OpenPCDet/tools/test;bash train_pointpillar_full_8p.sh

报错：RuntimeError: create:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:80 HCCL function error: HcclCommInitRootInfo(numRanks, &rootInfo, rank, &(comm->hcclComm_)), error code is 7
root@pm-799e:/# cd /data/lff/autodrive/mxDriving/model_examples/OpenPCDet/tools/test;bash train_pointpillar_full_8p.sh
2024-11-19 11:18:39.649 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=0.
2024-11-19 11:18:39.780 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=1.
2024-11-19 11:18:39.915 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=2.
2024-11-19 11:18:40.043 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=3.
2024-11-19 11:18:40.171 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=4.
2024-11-19 11:18:40.300 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=5.
2024-11-19 11:18:40.428 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=6.
2024-11-19 11:18:40.559 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(0)[ERROR], device id=7.
2024-11-19 11:18:40.573 [MSNPUREPORT][INFO] Set device log level success! log level string=SetLogLevel(2)[DISABLE], device id=0.
48614
/usr/local/lib/python3.8/dist-packages/torch/distributed/launch.py:181: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use-env is set by default in torchrun.
If your script expects `--local-rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

warnings.warn(
[2024-11-19 11:18:42,238] torch.distributed.run: [WARNING] 
[2024-11-19 11:18:42,238] torch.distributed.run: [WARNING] *****************************************
[2024-11-19 11:18:42,238] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-11-19 11:18:42,238] torch.distributed.run: [WARNING] *****************************************
/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py:299: ImportWarning: 
    *************************************************************************************************************
    The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
    The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
    The backend in torch.distributed.init_process_group set to hccl now..
    The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
    The device parameters have been replaced with npu in the function below:
    torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
    *************************************************************************************************************
    
  warnings.warn(msg, ImportWarning)
/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
[W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator())
2024-11-19 11:18:51,568   INFO  **********************Start logging**********************
2024-11-19 11:18:51,570   INFO  CUDA_VISIBLE_DEVICES=ALL
2024-11-19 11:18:51,570   INFO  Training in distributed mode : total_batch_size: 32
2024-11-19 11:18:51,571   INFO  cfg_file         cfgs/kitti_models/pointpillar.yaml
2024-11-19 11:18:51,571   INFO  batch_size       4
2024-11-19 11:18:51,571   INFO  epochs           10
2024-11-19 11:18:51,571   INFO  workers          4
2024-11-19 11:18:51,571   INFO  extra_tag        default
2024-11-19 11:18:51,571   INFO  ckpt             None
2024-11-19 11:18:51,571   INFO  pretrained_model None
2024-11-19 11:18:51,571   INFO  launcher         pytorch
2024-11-19 11:18:51,571   INFO  tcp_port         18888
2024-11-19 11:18:51,571   INFO  sync_bn          False
2024-11-19 11:18:51,571   INFO  fix_random_seed  False
2024-11-19 11:18:51,571   INFO  ckpt_save_interval 1
2024-11-19 11:18:51,571   INFO  local_rank       8
2024-11-19 11:18:51,572   INFO  max_ckpt_save_num 30
2024-11-19 11:18:51,572   INFO  merge_all_iters_to_one_epoch False
2024-11-19 11:18:51,572   INFO  set_cfgs         None
2024-11-19 11:18:51,572   INFO  max_waiting_mins 0
2024-11-19 11:18:51,572   INFO  start_epoch      0
2024-11-19 11:18:51,572   INFO  num_epochs_to_eval 0
2024-11-19 11:18:51,572   INFO  save_to_file     True
2024-11-19 11:18:51,572   INFO  use_tqdm_to_record False
2024-11-19 11:18:51,572   INFO  logger_iter_interval 50
2024-11-19 11:18:51,572   INFO  ckpt_save_time_interval 300
2024-11-19 11:18:51,572   INFO  wo_gpu_stat      False
2024-11-19 11:18:51,572   INFO  use_amp          False
2024-11-19 11:18:51,572   INFO  cfg.ROOT_DIR: /data/lff/autodrive/mxDriving/model_examples/OpenPCDet
2024-11-19 11:18:51,573   INFO  cfg.LOCAL_RANK: 0
2024-11-19 11:18:51,573   INFO  cfg.CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist']
2024-11-19 11:18:51,573   INFO  ----------- DATA_CONFIG -----------
2024-11-19 11:18:51,573   INFO  cfg.DATA_CONFIG.DATASET: KittiDataset
2024-11-19 11:18:51,573   INFO  cfg.DATA_CONFIG.DATA_PATH: ../data/kitti
2024-11-19 11:18:51,573   INFO  cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [0, -39.68, -3, 69.12, 39.68, 1]
2024-11-19 11:18:51,573   INFO  ----------- DATA_SPLIT -----------
2024-11-19 11:18:51,573   INFO  cfg.DATA_CONFIG.DATA_SPLIT.train: train
2024-11-19 11:18:51,573   INFO  cfg.DATA_CONFIG.DATA_SPLIT.test: val
2024-11-19 11:18:51,573   INFO  ----------- INFO_PATH -----------
2024-11-19 11:18:51,573   INFO  cfg.DATA_CONFIG.INFO_PATH.train: ['kitti_infos_train.pkl']
2024-11-19 11:18:51,574   INFO  cfg.DATA_CONFIG.INFO_PATH.test: ['kitti_infos_val.pkl']
2024-11-19 11:18:51,574   INFO  cfg.DATA_CONFIG.GET_ITEM_LIST: ['points']
2024-11-19 11:18:51,574   INFO  cfg.DATA_CONFIG.FOV_POINTS_ONLY: True
2024-11-19 11:18:51,574   INFO  ----------- DATA_AUGMENTOR -----------
2024-11-19 11:18:51,574   INFO  cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder']
2024-11-19 11:18:51,574   INFO  cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'gt_sampling', 'USE_ROAD_PLANE': False, 'DB_INFO_PATH': ['kitti_dbinfos_train.pkl'], 'PREPARE': {'filter_by_min_points': ['Car:5', 'Pedestrian:5', 'Cyclist:5'], 'filter_by_difficulty': [-1]}, 'SAMPLE_GROUPS': ['Car:15', 'Pedestrian:15', 'Cyclist:15'], 'NUM_POINT_FEATURES': 4, 'DATABASE_WITH_FAKELIDAR': False, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': False}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.95, 1.05]}]
2024-11-19 11:18:51,574   INFO  ----------- POINT_FEATURE_ENCODING -----------
2024-11-19 11:18:51,574   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding
2024-11-19 11:18:51,574   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity']
2024-11-19 11:18:51,574   INFO  cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity']
2024-11-19 11:18:51,574   INFO  cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': False}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.16, 0.16, 4], 'MAX_POINTS_PER_VOXEL': 32, 'MAX_NUMBER_OF_VOXELS': {'train': 16000, 'test': 40000}}]
2024-11-19 11:18:51,575   INFO  cfg.DATA_CONFIG._BASE_CONFIG_: cfgs/dataset_configs/kitti_dataset.yaml
2024-11-19 11:18:51,575   INFO  ----------- MODEL -----------
2024-11-19 11:18:51,575   INFO  cfg.MODEL.NAME: PointPillar
2024-11-19 11:18:51,575   INFO  ----------- VFE -----------
2024-11-19 11:18:51,575   INFO  cfg.MODEL.VFE.NAME: PillarVFE
2024-11-19 11:18:51,575   INFO  cfg.MODEL.VFE.WITH_DISTANCE: False
2024-11-19 11:18:51,575   INFO  cfg.MODEL.VFE.USE_ABSLOTE_XYZ: True
2024-11-19 11:18:51,575   INFO  cfg.MODEL.VFE.USE_NORM: True
2024-11-19 11:18:51,575   INFO  cfg.MODEL.VFE.NUM_FILTERS: [64]
2024-11-19 11:18:51,575   INFO  ----------- MAP_TO_BEV -----------
2024-11-19 11:18:51,575   INFO  cfg.MODEL.MAP_TO_BEV.NAME: PointPillarScatter
2024-11-19 11:18:51,575   INFO  cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 64
2024-11-19 11:18:51,575   INFO  ----------- BACKBONE_2D -----------
2024-11-19 11:18:51,576   INFO  cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone
2024-11-19 11:18:51,576   INFO  cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [3, 5, 5]
2024-11-19 11:18:51,576   INFO  cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [2, 2, 2]
2024-11-19 11:18:51,576   INFO  cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [64, 128, 256]
2024-11-19 11:18:51,576   INFO  cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2, 4]
2024-11-19 11:18:51,576   INFO  cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [128, 128, 128]
2024-11-19 11:18:51,576   INFO  ----------- DENSE_HEAD -----------
2024-11-19 11:18:51,576   INFO  cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle
2024-11-19 11:18:51,576   INFO  cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False
2024-11-19 11:18:51,576   INFO  cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True
2024-11-19 11:18:51,576   INFO  cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539
2024-11-19 11:18:51,576   INFO  cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0
2024-11-19 11:18:51,576   INFO  cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2
2024-11-19 11:18:51,577   INFO  cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Car', 'anchor_sizes': [[3.9, 1.6, 1.56]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.6, 'unmatched_threshold': 0.45}, {'class_name': 'Pedestrian', 'anchor_sizes': [[0.8, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}, {'class_name': 'Cyclist', 'anchor_sizes': [[1.76, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}]
2024-11-19 11:18:51,577   INFO  ----------- TARGET_ASSIGNER_CONFIG -----------
2024-11-19 11:18:51,577   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner
2024-11-19 11:18:51,577   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0
2024-11-19 11:18:51,577   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512
2024-11-19 11:18:51,577   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False
2024-11-19 11:18:51,577   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False
2024-11-19 11:18:51,577   INFO  cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder
2024-11-19 11:18:51,577   INFO  ----------- LOSS_CONFIG -----------
2024-11-19 11:18:51,577   INFO  ----------- LOSS_WEIGHTS -----------
2024-11-19 11:18:51,577   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0
2024-11-19 11:18:51,577   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0
2024-11-19 11:18:51,578   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2
2024-11-19 11:18:51,578   INFO  cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
2024-11-19 11:18:51,578   INFO  ----------- POST_PROCESSING -----------
2024-11-19 11:18:51,578   INFO  cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
2024-11-19 11:18:51,578   INFO  cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1
2024-11-19 11:18:51,578   INFO  cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False
2024-11-19 11:18:51,578   INFO  cfg.MODEL.POST_PROCESSING.EVAL_METRIC: kitti
2024-11-19 11:18:51,578   INFO  ----------- NMS_CONFIG -----------
2024-11-19 11:18:51,578   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False
2024-11-19 11:18:51,578   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu
2024-11-19 11:18:51,578   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.01
2024-11-19 11:18:51,578   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096
2024-11-19 11:18:51,579   INFO  cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500
2024-11-19 11:18:51,579   INFO  ----------- OPTIMIZATION -----------
2024-11-19 11:18:51,579   INFO  cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 4
2024-11-19 11:18:51,579   INFO  cfg.OPTIMIZATION.NUM_EPOCHS: 80
2024-11-19 11:18:51,579   INFO  cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle
2024-11-19 11:18:51,579   INFO  cfg.OPTIMIZATION.LR: 0.003
2024-11-19 11:18:51,579   INFO  cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01
2024-11-19 11:18:51,579   INFO  cfg.OPTIMIZATION.MOMENTUM: 0.9
2024-11-19 11:18:51,579   INFO  cfg.OPTIMIZATION.MOMS: [0.95, 0.85]
2024-11-19 11:18:51,579   INFO  cfg.OPTIMIZATION.PCT_START: 0.4
2024-11-19 11:18:51,579   INFO  cfg.OPTIMIZATION.DIV_FACTOR: 10
2024-11-19 11:18:51,579   INFO  cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45]
2024-11-19 11:18:51,579   INFO  cfg.OPTIMIZATION.LR_DECAY: 0.1
2024-11-19 11:18:51,580   INFO  cfg.OPTIMIZATION.LR_CLIP: 1e-07
2024-11-19 11:18:51,580   INFO  cfg.OPTIMIZATION.LR_WARMUP: False
2024-11-19 11:18:51,580   INFO  cfg.OPTIMIZATION.WARMUP_EPOCH: 1
2024-11-19 11:18:51,580   INFO  cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10
2024-11-19 11:18:51,580   INFO  cfg.TAG: pointpillar
2024-11-19 11:18:51,580   INFO  cfg.EXP_GROUP_PATH: kitti_models
2024-11-19 11:18:51,593   INFO  ----------- Create dataloader & network & optimizer -----------
2024-11-19 11:18:52,278   INFO  Database filter by min points Car: 11258 => 10757
2024-11-19 11:18:52,281   INFO  Database filter by min points Pedestrian: 1720 => 1695
2024-11-19 11:18:52,282   INFO  Database filter by min points Cyclist: 635 => 622
2024-11-19 11:18:52,312   INFO  Database filter by difficulty Car: 10757 => 8532
2024-11-19 11:18:52,317   INFO  Database filter by difficulty Pedestrian: 1695 => 1632
2024-11-19 11:18:52,320   INFO  Database filter by difficulty Cyclist: 622 => 470
/data/lff/autodrive/mxDriving/model_examples/OpenPCDet/tools/../pcdet/datasets/kitti/kitti_dataset.py:31: ResourceWarning: unclosed file <_io.TextIOWrapper name='../data/kitti/ImageSets/train.txt' mode='r' encoding='UTF-8'>
  self.sample_id_list = [x.strip() for x in open(split_dir).readlines()] if split_dir.exists() else None
ResourceWarning: Enable tracemalloc to get the object allocation traceback
2024-11-19 11:18:52,338   INFO  Loading KITTI dataset
2024-11-19 11:18:52,535   INFO  Total samples for KITTI dataset: 2879
/data/lff/autodrive/mxDriving/model_examples/OpenPCDet/tools/../pcdet/datasets/kitti/kitti_dataset.py:31: ResourceWarning: unclosed file <_io.TextIOWrapper name='../data/kitti/ImageSets/train.txt' mode='r' encoding='UTF-8'>
  self.sample_id_list = [x.strip() for x in open(split_dir).readlines()] if split_dir.exists() else None
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/usr/local/lib/python3.8/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/data/lff/autodrive/mxDriving/model_examples/OpenPCDet/tools/../pcdet/datasets/kitti/kitti_dataset.py:31: ResourceWarning: unclosed file <_io.TextIOWrapper name='../data/kitti/ImageSets/train.txt' mode='r' encoding='UTF-8'>
  self.sample_id_list = [x.strip() for x in open(split_dir).readlines()] if split_dir.exists() else None
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/data/lff/autodrive/mxDriving/model_examples/OpenPCDet/tools/../pcdet/datasets/kitti/kitti_dataset.py:31: ResourceWarning: unclosed file <_io.TextIOWrapper name='../data/kitti/ImageSets/train.txt' mode='r' encoding='UTF-8'>
  self.sample_id_list = [x.strip() for x in open(split_dir).readlines()] if split_dir.exists() else None
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/data/lff/autodrive/mxDriving/model_examples/OpenPCDet/tools/../pcdet/datasets/kitti/kitti_dataset.py:31: ResourceWarning: unclosed file <_io.TextIOWrapper name='../data/kitti/ImageSets/train.txt' mode='r' encoding='UTF-8'>
  self.sample_id_list = [x.strip() for x in open(split_dir).readlines()] if split_dir.exists() else None
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/usr/local/lib/python3.8/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/usr/local/lib/python3.8/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Traceback (most recent call last):
  File "train.py", line 232, in <module>
    main()
  File "train.py", line 164, in main
    model = nn.parallel.DistributedDataParallel(model, device_ids=[cfg.LOCAL_RANK % torch.cuda.device_count()])
  File "/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py", line 151, in decorated
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 795, in __init__
    _verify_param_shape_across_processes(self.process_group, parameters)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
    return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: create:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:80 HCCL function error: HcclCommInitRootInfo(numRanks, &rootInfo, rank, &(comm->hcclComm_)), error code is 7
[ERROR] 2024-11-19-11:18:55 (PID:254, Device:0, RankID:6) ERR02200 DIST call hccl api failed.
EJ0001: 2024-11-19-11:18:55.180.757 Failed to initialize the HCCP process. Reason: Maybe the last training process is running.
        Solution: Wait for 10s after killing the last training process and try again.
        TraceBack (most recent call last):
        tsd client wait response fail, hostpid:3769297, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:332]
        [drv api] halSqCqAllocate failed: device_id=0, drvRetCode=17![FUNC:NormalSqCqAllocate][FILE:npu_driver.cc][LINE:698]
        [SqCqManage]Alloc sq cq fail, stream_id=26, retCode=0x7020022.[FUNC:AllocStreamSqCq][FILE:stream_sqcq_manage.cc][LINE:45]
        [SqCqManage]Alloc sq cq fail, stream_id=26, retCode=0x7020022.[FUNC:Setup][FILE:stream.cc][LINE:616]
        stream setup failed, retCode=0x7020022.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:4871]
        Sync get device msg failed, retCode=0x7020022.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4928]
        rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

/data/lff/autodrive/mxDriving/model_examples/OpenPCDet/tools/../pcdet/datasets/kitti/kitti_dataset.py:31: ResourceWarning: unclosed file <_io.TextIOWrapper name='../data/kitti/ImageSets/train.txt' mode='r' encoding='UTF-8'>
  self.sample_id_list = [x.strip() for x in open(split_dir).readlines()] if split_dir.exists() else None
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
/data/lff/autodrive/mxDriving/model_examples/OpenPCDet/tools/../pcdet/datasets/kitti/kitti_dataset.py:31: ResourceWarning: unclosed file <_io.TextIOWrapper name='../data/kitti/ImageSets/train.txt' mode='r' encoding='UTF-8'>
  self.sample_id_list = [x.strip() for x in open(split_dir).readlines()] if split_dir.exists() else None
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/data/lff/autodrive/mxDriving/model_examples/OpenPCDet/tools/../pcdet/datasets/kitti/kitti_dataset.py:31: ResourceWarning: unclosed file <_io.TextIOWrapper name='../data/kitti/ImageSets/train.txt' mode='r' encoding='UTF-8'>
  self.sample_id_list = [x.strip() for x in open(split_dir).readlines()] if split_dir.exists() else None
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/local/lib/python3.8/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Traceback (most recent call last):
  File "train.py", line 232, in <module>
    main()
  File "train.py", line 164, in main
    model = nn.parallel.DistributedDataParallel(model, device_ids=[cfg.LOCAL_RANK % torch.cuda.device_count()])
  File "/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py", line 151, in decorated
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 795, in __init__
    _verify_param_shape_across_processes(self.process_group, parameters)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
    return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: create:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:80 HCCL function error: HcclCommInitRootInfo(numRanks, &rootInfo, rank, &(comm->hcclComm_)), error code is 7
[ERROR] 2024-11-19-11:18:56 (PID:253, Device:0, RankID:5) ERR02200 DIST call hccl api failed.
EJ0001: 2024-11-19-11:18:56.673.603 Failed to initialize the HCCP process. Reason: Maybe the last training process is running.
        Solution: Wait for 10s after killing the last training process and try again.
        TraceBack (most recent call last):
        tsd client wait response fail, hostpid:3769296, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:332]
        Fail to get sq reg virtual addr, deviceId=0, sqId=31.[FUNC:Setup][FILE:stream.cc][LINE:647]
        stream setup failed, retCode=0x7020010.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:4871]
        Sync get device msg failed, retCode=0x7020010.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4928]
        rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

Traceback (most recent call last):
  File "train.py", line 232, in <module>
    main()
  File "train.py", line 164, in main
    model = nn.parallel.DistributedDataParallel(model, device_ids=[cfg.LOCAL_RANK % torch.cuda.device_count()])
  File "/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py", line 151, in decorated
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 795, in __init__
    _verify_param_shape_across_processes(self.process_group, parameters)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
    return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: create:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:80 HCCL function error: HcclCommInitRootInfo(numRanks, &rootInfo, rank, &(comm->hcclComm_)), error code is 7
[ERROR] 2024-11-19-11:18:56 (PID:251, Device:0, RankID:3) ERR02200 DIST call hccl api failed.
EJ0001: 2024-11-19-11:18:56.771.559 Failed to initialize the HCCP process. Reason: Maybe the last training process is running.
        Solution: Wait for 10s after killing the last training process and try again.
        TraceBack (most recent call last):
        tsd client wait response fail, hostpid:3769294, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:332]

/usr/local/lib/python3.8/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3526.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Traceback (most recent call last):
  File "train.py", line 232, in <module>
    main()
  File "train.py", line 164, in main
    model = nn.parallel.DistributedDataParallel(model, device_ids=[cfg.LOCAL_RANK % torch.cuda.device_count()])
  File "/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py", line 151, in decorated
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 795, in __init__
    _verify_param_shape_across_processes(self.process_group, parameters)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
    return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: create:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:80 HCCL function error: HcclCommInitRootInfo(numRanks, &rootInfo, rank, &(comm->hcclComm_)), error code is 7
[ERROR] 2024-11-19-11:18:56 (PID:250, Device:0, RankID:2) ERR02200 DIST call hccl api failed.
EJ0001: 2024-11-19-11:18:56.847.974 Failed to initialize the HCCP process. Reason: Maybe the last training process is running.
        Solution: Wait for 10s after killing the last training process and try again.
        TraceBack (most recent call last):
        tsd client wait response fail, hostpid:3769293, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:332]
        Fail to get sq reg virtual addr, deviceId=0, sqId=35.[FUNC:Setup][FILE:stream.cc][LINE:647]
        stream setup failed, retCode=0x7020010.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:4871]
        Sync get device msg failed, retCode=0x7020010.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4928]
        rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

Traceback (most recent call last):
  File "train.py", line 232, in <module>
    main()
  File "train.py", line 164, in main
    model = nn.parallel.DistributedDataParallel(model, device_ids=[cfg.LOCAL_RANK % torch.cuda.device_count()])
  File "/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py", line 151, in decorated
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 795, in __init__
    _verify_param_shape_across_processes(self.process_group, parameters)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
    return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: create:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:80 HCCL function error: HcclCommInitRootInfo(numRanks, &rootInfo, rank, &(comm->hcclComm_)), error code is 7
[ERROR] 2024-11-19-11:18:58 (PID:249, Device:0, RankID:1) ERR02200 DIST call hccl api failed.
EJ0001: 2024-11-19-11:18:58.123.544 Failed to initialize the HCCP process. Reason: Maybe the last training process is running.
        Solution: Wait for 10s after killing the last training process and try again.
        TraceBack (most recent call last):
        tsd client wait response fail, hostpid:3769292, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:332]

Traceback (most recent call last):
  File "train.py", line 232, in <module>
    main()
  File "train.py", line 164, in main
    model = nn.parallel.DistributedDataParallel(model, device_ids=[cfg.LOCAL_RANK % torch.cuda.device_count()])
  File "/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py", line 151, in decorated
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 795, in __init__
    _verify_param_shape_across_processes(self.process_group, parameters)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
    return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: create:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:80 HCCL function error: HcclCommInitRootInfo(numRanks, &rootInfo, rank, &(comm->hcclComm_)), error code is 7
[ERROR] 2024-11-19-11:18:59 (PID:252, Device:0, RankID:4) ERR02200 DIST call hccl api failed.
EJ0001: 2024-11-19-11:18:59.360.320 Failed to initialize the HCCP process. Reason: Maybe the last training process is running.
        Solution: Wait for 10s after killing the last training process and try again.
        TraceBack (most recent call last):
        tsd client wait response fail, hostpid:3769295, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:332]
        Fail to get sq reg virtual addr, deviceId=0, sqId=37.[FUNC:Setup][FILE:stream.cc][LINE:647]
        stream setup failed, retCode=0x7020010.[FUNC:SyncGetDevMsg][FILE:api_impl.cc][LINE:4871]
        Sync get device msg failed, retCode=0x7020010.[FUNC:GetDevErrMsg][FILE:api_impl.cc][LINE:4928]
        rtGetDevMsg execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]

Traceback (most recent call last):
  File "train.py", line 232, in <module>
    main()
  File "train.py", line 164, in main
    model = nn.parallel.DistributedDataParallel(model, device_ids=[cfg.LOCAL_RANK % torch.cuda.device_count()])
  File "/usr/local/lib/python3.8/dist-packages/torch_npu/contrib/transfer_to_npu.py", line 151, in decorated
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/parallel/distributed.py", line 795, in __init__
    _verify_param_shape_across_processes(self.process_group, parameters)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/utils.py", line 265, in _verify_param_shape_across_processes
    return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: create:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:80 HCCL function error: HcclCommInitRootInfo(numRanks, &rootInfo, rank, &(comm->hcclComm_)), error code is 7
[ERROR] 2024-11-19-11:18:59 (PID:255, Device:0, RankID:7) ERR02200 DIST call hccl api failed.
EJ0001: 2024-11-19-11:18:59.645.106 Failed to initialize the HCCP process. Reason: Maybe the last training process is running.
        Solution: Wait for 10s after killing the last training process and try again.
        TraceBack (most recent call last):
        tsd client wait response fail, hostpid:3769298, device response code[1]. unknown device error.[FUNC:WaitRsp][FILE:process_mode_manager.cpp][LINE:332]

[2024-11-19 11:19:02,284] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 248 closing signal SIGTERM
[2024-11-19 11:19:02,284] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 249 closing signal SIGTERM
[2024-11-19 11:19:02,285] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 250 closing signal SIGTERM
[2024-11-19 11:19:02,285] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 252 closing signal SIGTERM
[2024-11-19 11:19:02,285] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 253 closing signal SIGTERM
[2024-11-19 11:19:02,285] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 255 closing signal SIGTERM
[2024-11-19 11:19:04,553] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 3 (pid: 251) of binary: /usr/bin/python

参考https://gitee.com/ascend/ModelLink/issues/IARQOM，关闭python进程，重启容器后重试，仍然有问题

问题已经解决。原因不是端口被占用。

而是代码中初始化分布式的代码写错了

原来的代码中
https://gitee.com/ascend/mxDriving/blob/branch_v6.0.0-RC3/model_examples/OpenPCDet/tools/train.py

输入图片说明

这一行很明显应该是hccl,修改后就可以解决此问题

GVP Ascend/DrivingSDK

Content Risk Flag

Comments (1)

GVPAscend/DrivingSDK

Content Risk Flag