mindformers实现baichuan2-13b全量微调，单机8卡训练运行run_singlenode.sh 时报错

[ERROR] 2024-01-31 16:07:43,930 [mindformers/tools/cloud_adapter/cloud_monitor.py:43] wrapper: Traceback (most recent call last):
  File "/root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindformers/tools/cloud_adapter/cloud_monitor.py", line 34, in wrapper
    result = run_func(*args, **kwargs)
  File "/root/baichuan2/mindspore/mindformers-dev/research/baichuan2/run_baichuan2.py", line 169, in main
    trainer.finetune(finetune_checkpoint=ckpt, auto_trans_ckpt=config.auto_trans_ckpt, resume_training=resume)
  File "/root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindspore/_checkparam.py", line 1313, in wrapper
    return func(*args, **kwargs)
  File "/root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindformers/trainer/trainer.py", line 485, in finetune
    self.trainer.train(
  File "/root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindformers/trainer/causal_language_modeling/causal_language_modeling.py", line 97, in train
    self.training_process(
  File "/root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindformers/trainer/base_trainer.py", line 710, in training_process
    transform_and_load_checkpoint(config, model, network, dataset)
  File "/root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 317, in transform_and_load_checkpoint
    build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict)
  File "/root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindformers/trainer/utils.py", line 435, in build_model
    model.build(train_dataset=dataset, epoch=config.runner_config.epochs,
  File "/root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindspore/train/model.py", line 1274, in build
    self._init(train_dataset, valid_dataset, sink_size, epoch)
  File "/root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindspore/train/model.py", line 529, in _init
    train_network.compile(*inputs)
  File "/root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindspore/nn/cell.py", line 997, in compile
    _cell_graph_executor.compile(self, phase=self.phase,
  File "/root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindspore/common/api.py", line 1547, in compile
    result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode())
RuntimeError: Create forward communication group between all pipeline stages failed, the rank_list is: [const vector]{0, 2, 4, 6}

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/frontend/parallel/pipeline_transformer/pipeline_transformer.cc:378 CreateForwardGroup

----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------

# In file /root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindspore/train/dataset_helper.py:101
    def construct(self):

# In file /root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindspore/train/dataset_helper.py:103
        return self.network(*outputs)
               ^

# In file /root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindformers/wrapper/wrapper.py:245
        if self.opt_shard:

# In file /root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindformers/wrapper/wrapper.py:246
            grads = self.grad_reducer(grads)

# In file /root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindformers/wrapper/wrapper.py:252
        if self.use_clip_grad:

# In file /root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindformers/wrapper/wrapper.py:253
            grads, _ = self.clip_grad_norm(grads)
            ^

# In file /root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindspore/train/dataset_helper.py:103
        return self.network(*outputs)
               ^

# In file /root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindformers/wrapper/wrapper.py:263
        cond = self.get_overflow_status(self.status, grads)
               ^

# In file /root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindspore/nn/wrap/loss_scale.py:532
        if self.gpu_target:

# In file /root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindspore/nn/wrap/loss_scale.py:534
        elif self.ascend_910b_target:

# In file /root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindspore/nn/wrap/loss_scale.py:540
            overflow = self._get_ascend_overflow_status_on_saturation_mode(status, compute_output)
                       ^

# In file /root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindspore/nn/wrap/loss_scale.py:496
        if self.is_distributed:

# In file /root/.local/conda/envs/baichuan/lib/python3.9/site-packages/mindspore/nn/wrap/loss_scale.py:498
            flag_reduce = self.allreduce(get_status)
                          ^

显卡8张910b 32G

Ascend-cann-toolkit_7.0.0
mindspore 2.2.11
MindFormers dev

GVP MindSpore/mindformers
关闭

内容风险标识

评论 (32)

GVPMindSpore/mindformers关闭

内容风险标识