【西安AICC】AssertionErrorAssertionError: : One or both of the .idx and .bin files cannot be found at the path prefix NoneOne or both of the .idx and .bin files cannot be found ...

一、问题现象（附报错日志上下文）：
模型微调传入数据集所在目录会报错：`AssertionErrorAssertionError: : One or both of the .idx and .bin files cannot be found at the path prefix NoneOne or both of the .idx and .bin files cannot be found at the path prefix None`

二、软件版本:
-- CANN 版本 (e.g., CANN 3.0.x，5.x.x):  8.0.RC2
--Tensorflow/Pytorch/MindSpore 版本:MindSpeed-llm 1.0.0
--Python 版本 (e.g., Python 3.7.5):3.9
-- MindStudio版本 (e.g., MindStudio 2.0.0 (beta3)):-
--操作系统版本 (e.g., Ubuntu 18.04):Euler

三、测试步骤：
1.参考README进行模型转换，数据集转换，配置pretrain脚本，并执行
2.测试模型：Qwen2.5-3B/7B
3.BUG反馈： **在微调过程中必须需要指定数据集idx/bin文件前缀全部内容，否则会报错** ，这种低级的反人类操作能出现在现在版本中，真是令人意外，无疑增加了使用成本，消耗用户耐心，降低产品满意度。。。
![输入图片说明](https://foruda.gitee.com/images/1737163160624763920/f9def148_8013481.png "屏幕截图")

四、日志信息:

```
> building train, validation, and test datasets ...
 > datasets target sizes (minimum size):
    train:      8000
    validation: 0
    test:       0
INFO:megatron.core.datasets.blended_megatron_dataset_config:mock = False
**************************************************
/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/blended_megatron_dataset_builder.py 214
 dataset_path: /home/ma-user/work/minying/demo/MindSpeed-LLM/dataset/alpaca
**************************************************
/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/gpt_dataset.py 228 /home/ma-user/work/minying/demo/MindSpeed-LLM/dataset/alpaca
/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/indexed_dataset.py **************************************************
path_prefix: /home/ma-user/work/minying/demo/MindSpeed-LLM/dataset/alpaca
**************************************************
idx_path: /home/ma-user/work/minying/demo/MindSpeed-LLM/dataset/alpaca.idx
bin_path: /home/ma-user/work/minying/demo/MindSpeed-LLM/dataset/alpaca.bin
INFO:megatron.core.datasets.blended_megatron_dataset_config:Let split_matrix = [(0, 1.0), None, None]
> building train, validation, and test datasets for GPT ...
**************************************************
/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/blended_megatron_dataset_builder.py 214
 dataset_path: /home/ma-user/work/minying/demo/MindSpeed-LLM/dataset/alpaca
**************************************************
/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/gpt_dataset.py 228 /home/ma-user/work/minying/demo/MindSpeed-LLM/dataset/alpaca
/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/indexed_dataset.py **************************************************
path_prefix: /home/ma-user/work/minying/demo/MindSpeed-LLM/dataset/alpaca
**************************************************
idx_path: /home/ma-user/work/minying/demo/MindSpeed-LLM/dataset/alpaca.idx
bin_path: /home/ma-user/work/minying/demo/MindSpeed-LLM/dataset/alpaca.bin
Traceback (most recent call last):
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/pretrain_gpt.py", line 228, in <module>
Traceback (most recent call last):
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/pretrain_gpt.py", line 228, in <module>
    main()
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/pretrain_gpt.py", line 221, in main
    main()
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/pretrain_gpt.py", line 221, in main
    pretrain(train_valid_test_datasets_provider,
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/mindspeed_llm/training/training.py", line 339, in pretrain
    pretrain(train_valid_test_datasets_provider,
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/mindspeed_llm/training/training.py", line 339, in pretrain
    train_args, test_data_iterator_list = build_train_args(args, timers, train_valid_test_dataset_provider,
train_args, test_data_iterator_list = build_train_args(args, timers, train_valid_test_dataset_provider,  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/mindspeed_llm/training/training.py", line 250, in build_train_args

File "/home/ma-user/work/minying/demo/MindSpeed-LLM/mindspeed_llm/training/training.py", line 250, in build_train_args
    = build_train_valid_test_data_iterators(
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/training/training.py", line 1390, in build_train_valid_test_data_iterators
    = build_train_valid_test_data_iterators(
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/training/training.py", line 1390, in build_train_valid_test_data_iterators
    build_train_valid_test_data_loaders(
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/training/training.py", line 1351, in build_train_valid_test_data_loaders
    build_train_valid_test_data_loaders(
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/training/training.py", line 1351, in build_train_valid_test_data_loaders
    train_ds, valid_ds, test_ds = build_train_valid_test_datasets(
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/training/training.py", line 1321, in build_train_valid_test_datasets
    train_ds, valid_ds, test_ds = build_train_valid_test_datasets(
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/training/training.py", line 1321, in build_train_valid_test_datasets
    return build_train_valid_test_datasets_provider(train_valid_test_num_samples)
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/pretrain_gpt.py", line 205, in train_valid_test_datasets_provider
    return build_train_valid_test_datasets_provider(train_valid_test_num_samples)
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/pretrain_gpt.py", line 205, in train_valid_test_datasets_provider
    train_ds, valid_ds, test_ds = BlendedMegatronDatasetBuilder(
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/blended_megatron_dataset_builder.py", line 75, in build
    train_ds, valid_ds, test_ds = BlendedMegatronDatasetBuilder(
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/blended_megatron_dataset_builder.py", line 75, in build
        return self._build_blended_dataset_splits()return self._build_blended_dataset_splits()

File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/blended_megatron_dataset_builder.py", line 98, in _build_blended_dataset_splits
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/blended_megatron_dataset_builder.py", line 98, in _build_blended_dataset_splits
    return self._build_megatron_dataset_splits(blend[0], split, self.sizes)
      File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/blended_megatron_dataset_builder.py", line 216, in _build_megatron_dataset_splits
return self._build_megatron_dataset_splits(blend[0], split, self.sizes)
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/blended_megatron_dataset_builder.py", line 216, in _build_megatron_dataset_splits
    low_level_dataset = self.cls.build_low_level_dataset(dataset_path, self.config)
      File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/gpt_dataset.py", line 230, in build_low_level_dataset
low_level_dataset = self.cls.build_low_level_dataset(dataset_path, self.config)
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/gpt_dataset.py", line 230, in build_low_level_dataset
        return IndexedDataset(dataset_path, multimodal=False, mmap=config.mmap_bin_files)return IndexedDataset(dataset_path, multimodal=False, mmap=config.mmap_bin_files)

File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/indexed_dataset.py", line 353, in __init__
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/indexed_dataset.py", line 353, in __init__
    self.initialize(path_prefix, multimodal, mmap)
      File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/indexed_dataset.py", line 373, in initialize
self.initialize(path_prefix, multimodal, mmap)
  File "/home/ma-user/work/minying/demo/MindSpeed-LLM/megatron/core/datasets/indexed_dataset.py", line 373, in initialize
        assert os.path.exists(idx_path) and os.path.exists(assert os.path.exists(idx_path) and os.path.exists(

AssertionErrorAssertionError: : One or both of the .idx and .bin files cannot be found at the path prefix NoneOne or both of the .idx and .bin files cannot be found at the path prefix None

[ERROR] 2025-01-18-09:10:22 (PID:1498308, Device:1, RankID:1) ERR99999 UNKNOWN application exception
```

Ascend/MindSpeed-LLM
暂停

内容风险标识

评论 (5)

Ascend/MindSpeed-LLM暂停 .gitee-modal { width: 500px !important; }

内容风险标识