ModuleNotFoundError: No module named 'mindspeed.features_manager'

一、问题现象（附报错日志上下文）：
/opt/conda/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:301: ImportWarning: 
    *************************************************************************************************************
    The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
    The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
    The backend in torch.distributed.init_process_group set to hccl now..
    The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
    The device parameters have been replaced with npu in the function below:
    torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.set_default_device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
    *************************************************************************************************************
    
  warnings.warn(msg, ImportWarning)
/opt/conda/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
  warnings.warn(msg, RuntimeWarning)
Traceback (most recent call last):
  File "/workspace/testcode/ModelLink/./preprocess_data.py", line 36, in <module>
    from mindspeed_llm.training.tokenizer import build_tokenizer
  File "/workspace/testcode/ModelLink/mindspeed_llm/__init__.py", line 16, in <module>
    from mindspeed_llm.tasks import megatron_adaptor
  File "/workspace/testcode/ModelLink/mindspeed_llm/tasks/megatron_adaptor.py", line 23, in <module>
    from mindspeed.features_manager import FEATURES_LIST
ModuleNotFoundError: No module named 'mindspeed.features_manager'
[ERROR] 2025-03-07-20:18:44 (PID:17427, Device:-1, RankID:-1) ERR99999 UNKNOWN application exception
/opt/conda/lib/python3.10/tempfile.py:869: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpztwpexka'>
  _warnings.warn(warn_message, ResourceWarning)

二、软件版本:
-- CANN 版本 (CANN 8.0 RC3):  
--Tensorflow/Pytorch/MindSpore 版本: pytorch2.1.0; torch_npu2.1.0.post8
-- Megatron-LM core_r0.6.0
-- MindSpeed 969686ff 
-- Linux version 4.19.90-2107.6.0.0192.8.oe1.bclinux.aarch64 (mockbuild@kojibuilder)
--Python 版本 (Python 3.10.14):

三、测试步骤：
按照USER_GUIDE处理预训练数据集过程中 
python ./preprocess_data.py \
    --input ./dataset/train-00000-of-00042-d964455e17e96d5a.parquet \
    --tokenizer-name-or-path ./model_from_hf/llama-2-hf \
    --tokenizer-type PretrainedFromHF \
    --handler-name GeneralPretrainHandler \
    --output-prefix ./dataset/enwiki \
    --json-keys text \
    --workers 4 \
    --log-interval 1000

Ascend/MindSpeed
暂停

内容风险标识

评论 (4)

Ascend/MindSpeed暂停 .gitee-modal { width: 500px !important; }

内容风险标识