sd3/3.5 数据读取失败

报错：数据读取失败

Traceback (most recent call last):
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 1854, in _prepare_split_single
    for _, table in generator:
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 82, in _generate_tables
    parquet_file = pq.ParquetFile(f)
  File "/usr/local/python3.10/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 317, in __init__
    self.reader.open(
  File "pyarrow/_parquet.pyx", line 1514, in pyarrow._parquet.ParquetReader.open
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/nfs/appnfs/sxh/code/SD/diffusers/./examples/dreambooth/train_dreambooth_sd3.py", line 1814, in <module>
    main(args)
  File "/home/nfs/appnfs/sxh/code/SD/diffusers/./examples/dreambooth/train_dreambooth_sd3.py", line 1344, in main
    train_dataset = DreamBoothDataset(
  File "/home/nfs/appnfs/sxh/code/SD/diffusers/./examples/dreambooth/train_dreambooth_sd3.py", line 691, in __init__
    dataset = load_dataset(
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/load.py", line 2154, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 1000, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 1741, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 1897, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
12/06/2024 11:45:10 - INFO - datasets - PyTorch version 2.1.0 available.

Generating train split:   0%|          | 0/833 [00:00<?, ? examples/s]
Generating train split:   0%|          | 0/833 [00:00<?, ? examples/s]
Traceback (most recent call last):
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 1854, in _prepare_split_single
    for _, table in generator:
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 82, in _generate_tables
    parquet_file = pq.ParquetFile(f)
  File "/usr/local/python3.10/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 317, in __init__
    self.reader.open(
  File "pyarrow/_parquet.pyx", line 1514, in pyarrow._parquet.ParquetReader.open
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/nfs/appnfs/sxh/code/SD/diffusers/./examples/dreambooth/train_dreambooth_sd3.py", line 1814, in <module>
    main(args)
  File "/home/nfs/appnfs/sxh/code/SD/diffusers/./examples/dreambooth/train_dreambooth_sd3.py", line 1344, in main
    train_dataset = DreamBoothDataset(
  File "/home/nfs/appnfs/sxh/code/SD/diffusers/./examples/dreambooth/train_dreambooth_sd3.py", line 691, in __init__
    dataset = load_dataset(
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/load.py", line 2154, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 1000, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 1741, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 1897, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
12/06/2024 11:45:12 - INFO - datasets - PyTorch version 2.1.0 available.

Generating train split:   0%|          | 0/833 [00:00<?, ? examples/s]
Generating train split:   0%|          | 0/833 [00:00<?, ? examples/s]
Traceback (most recent call last):
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 1854, in _prepare_split_single
    for _, table in generator:
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 82, in _generate_tables
    parquet_file = pq.ParquetFile(f)
  File "/usr/local/python3.10/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 317, in __init__
    self.reader.open(
  File "pyarrow/_parquet.pyx", line 1514, in pyarrow._parquet.ParquetReader.open
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/nfs/appnfs/sxh/code/SD/diffusers/./examples/dreambooth/train_dreambooth_sd3.py", line 1814, in <module>
    main(args)
  File "/home/nfs/appnfs/sxh/code/SD/diffusers/./examples/dreambooth/train_dreambooth_sd3.py", line 1344, in main
    train_dataset = DreamBoothDataset(
  File "/home/nfs/appnfs/sxh/code/SD/diffusers/./examples/dreambooth/train_dreambooth_sd3.py", line 691, in __init__
    dataset = load_dataset(
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/load.py", line 2154, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 924, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 1000, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 1741, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/usr/local/python3.10/lib/python3.10/site-packages/datasets/builder.py", line 1897, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset
[2024-12-06 11:45:13,759] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 13485 closing signal SIGTERM
[2024-12-06 11:45:13,760] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 13486 closing signal SIGTERM
[2024-12-06 11:45:13,760] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 13487 closing signal SIGTERM
[2024-12-06 11:45:13,760] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 13488 closing signal SIGTERM
[2024-12-06 11:45:13,760] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 13489 closing signal SIGTERM
[2024-12-06 11:45:13,760] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 13490 closing signal SIGTERM
[2024-12-06 11:45:13,760] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 13491 closing signal SIGTERM
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[2024-12-06 11:45:16,893] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 7 (pid: 13492) of binary: /usr/local/python3.10/bin/python3.10
Traceback (most recent call last):
  File "/usr/local/python3.10/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/python3.10/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/usr/local/python3.10/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1091, in launch_command
    deepspeed_launcher(args)
  File "/usr/local/python3.10/lib/python3.10/site-packages/accelerate/commands/launch.py", line 787, in deepspeed_launcher
    distrib_run.run(args)
  File "/usr/local/python3.10/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/usr/local/python3.10/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/python3.10/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
./examples/dreambooth/train_dreambooth_sd3.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-12-06_11:45:13
  host      : 1883d8a979b5
  rank      : 7 (local_rank: 7)
  exitcode  : 1 (pid: 13492)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/usr/local/python3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/usr/local/python3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/usr/local/python3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/usr/local/python3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/usr/local/python3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/usr/local/python3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Ascend/MindSpeed-MM
暂停

内容风险标识

评论 (5)

Ascend/MindSpeed-MM暂停 .gitee-modal { width: 500px !important; }

内容风险标识