Running Tokenizer on train_dataset报错

一、问题现象（附报错日志上下文）：
在用Qwen2.5 vl跑video数据集时，预处理阶段报错，出现在running tokenizer on train_dataset这一步。

二、软件版本:
-- CANN 版本: CANN 8.0.RC3  
--Tensorflow/Pytorch/MindSpore 版本: torch 2.1
--Python 版本 (e.g., Python 3.7.5): Python 3.9.16

> data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/datasets/table.py:1421: FutureWarning: promote has been superseded by promote_options='default'.
  table = cls._concat_blocks(blocks, axis=0)
Rank 0, running tokenizer on train_dataset (num_proc=16):  65%|██████▍   | 38400/59524 [09:56<02:28, 142.13 examples/s][E ProcessGroupHCCL.cpp:421] [Rank 1] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600133 milliseconds before timing out.
[E ProcessGroupHCCL.cpp:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E ProcessGroupHCCL.cpp:481] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupHCCL.cpp:805] [Rank 1] HCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600133 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 1] HCCL watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600133 milliseconds before timing out.
[E ProcessGroupHCCL.cpp:421] [Rank 6] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600497 milliseconds before timing out.
[E ProcessGroupHCCL.cpp:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E ProcessGroupHCCL.cpp:481] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupHCCL.cpp:805] [Rank 6] HCCL watchdog thread terminated with exception: [Rank 6] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600497 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 6] HCCL watchdog thread terminated with exception: [Rank 6] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600497 milliseconds before timing out.
[E ProcessGroupHCCL.cpp:421] [Rank 4] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600530 milliseconds before timing out.
[E ProcessGroupHCCL.cpp:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E ProcessGroupHCCL.cpp:481] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupHCCL.cpp:805] [Rank 4] HCCL watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600530 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 4] HCCL watchdog thread terminated with exception: [Rank 4] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600530 milliseconds before timing out.
[E ProcessGroupHCCL.cpp:421] [Rank 7] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600590 milliseconds before timing out.
[E ProcessGroupHCCL.cpp:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E ProcessGroupHCCL.cpp:481] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupHCCL.cpp:805] [Rank 7] HCCL watchdog thread terminated with exception: [Rank 7] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600590 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 7] HCCL watchdog thread terminated with exception: [Rank 7] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600590 milliseconds before timing out.
[E ProcessGroupHCCL.cpp:421] [Rank 2] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600570 milliseconds before timing out.
[E ProcessGroupHCCL.cpp:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E ProcessGroupHCCL.cpp:481] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupHCCL.cpp:805] [Rank 2] HCCL watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600570 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 2] HCCL watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600570 milliseconds before timing out.
[E ProcessGroupHCCL.cpp:421] [Rank 5] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600671 milliseconds before timing out.
[E ProcessGroupHCCL.cpp:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E ProcessGroupHCCL.cpp:481] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupHCCL.cpp:805] [Rank 5] HCCL watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600671 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 5] HCCL watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600671 milliseconds before timing out.
[E ProcessGroupHCCL.cpp:421] [Rank 3] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600819 milliseconds before timing out.
[E ProcessGroupHCCL.cpp:475] Some HCCL operations have failed or timed out. Due to the asynchronous nature of ASCEND kernels, subsequent NPU operations might run on corrupted/incomplete data.
[E ProcessGroupHCCL.cpp:481] To avoid data inconsistency, we are taking the entire process down.
[E ProcessGroupHCCL.cpp:805] [Rank 3] HCCL watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600819 milliseconds before timing out.
terminate called after throwing an instance of 'std::runtime_error'
  what():  [Rank 3] HCCL watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkHCCL(SeqNum=11, OpType=ALLREDUCE, Timeout(ms)=600000) ran for 600819 milliseconds before timing out.
[2025-07-24 10:06:18,105] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 140194 closing signal SIGTERM
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/data/miniconda3/envs/env-3.9.16/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/data/miniconda3/envs/env-3.9.16/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/data/miniconda3/envs/env-3.9.16/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
[2025-07-24 10:06:18,570] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 1 (pid: 140195) of binary: /data/miniconda3/envs/env-3.9.16/bin/python
Traceback (most recent call last):
  File "/data/miniconda3/envs/env-3.9.16/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/torch/distributed/run.py", line 806, in main
    run(args)
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/data/miniconda3/envs/env-3.9.16/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
=======================================================
pretrain_vlm.py FAILED
-------------------------------------------------------
Failures:
[1]:
  time      : 2025-07-24_10:06:18
  host      : bore-prod-wadeqin-20250724092955-worker-0
  rank      : 2 (local_rank: 2)
  exitcode  : -6 (pid: 140196)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 140196
[2]:
  time      : 2025-07-24_10:06:18
  host      : bore-prod-wadeqin-20250724092955-worker-0
  rank      : 3 (local_rank: 3)
  exitcode  : -6 (pid: 140197)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 140197
[3]:
  time      : 2025-07-24_10:06:18
  host      : bore-prod-wadeqin-20250724092955-worker-0
  rank      : 4 (local_rank: 4)
  exitcode  : -6 (pid: 140198)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 140198
[4]:
  time      : 2025-07-24_10:06:18
  host      : bore-prod-wadeqin-20250724092955-worker-0
  rank      : 5 (local_rank: 5)
  exitcode  : -6 (pid: 140199)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 140199
[5]:
  time      : 2025-07-24_10:06:18
  host      : bore-prod-wadeqin-20250724092955-worker-0
  rank      : 6 (local_rank: 6)
  exitcode  : -6 (pid: 140200)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 140200
[6]:
  time      : 2025-07-24_10:06:18
  host      : bore-prod-wadeqin-20250724092955-worker-0
  rank      : 7 (local_rank: 7)
  exitcode  : -6 (pid: 140201)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 140201
-------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-07-24_10:06:18
  host      : bore-prod-wadeqin-20250724092955-worker-0
  rank      : 1 (local_rank: 1)
  exitcode  : -6 (pid: 140195)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 140195
=======================================================
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/data/miniconda3/envs/env-3.9.16/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/data/miniconda3/envs/env-3.9.16/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/data/miniconda3/envs/env-3.9.16/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/data/miniconda3/envs/env-3.9.16/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Ascend/MindSpeed-MM
暂停

内容风险标识

评论 (2)

Ascend/MindSpeed-MM暂停 .gitee-modal { width: 500px !important; }

内容风险标识