Qwen2VL多机多卡训练后mm_to_hf转模型问题

### 1.上下文
Qwen2-VL-72B，双机16卡LoRA微调（tp2 pp8)已经完成，finetune脚本里的设置：
```
NNODES=2
NPUS_PER_NODE=8
TP=2
PP=8
CP=1
MBS=1
```
一台机器save_dir里得到mp_rank_00_000~mp_rank_00_0003，mp_rank_01_000~mp_rank_00_003，
另一台机器save_dir里得到mp_rank_00_004~mp_rank_00_0007，mp_rank_01_004~mp_rank_00_007，
把两者放到一个文件夹，得到
![输入图片说明](https://foruda.gitee.com/images/1748244115487856075/ecadaa50_8930315.png "屏幕截图")

然后使用mmconvert做mm_to_hf，把mm模型转回hf模型：
`mm-convert  Qwen2VLConverter mm_to_hf --cfg.save_hf_dir "/works/mm_to_hf/Qwen2-VL-72B-Instruct_tp2_pp8" --cfg.mm_dir "/workspaces/hw_test/MindSpeed-MM/examples/qwen2vl/save_dir/Qwen2-VL-72B-Instruct" --cfg.hf_config.hf_dir "/workspaces/hw_test/MindSpeed-MM/ckpt1/hf_path/Qwen2-VL-72B-Instruct" --cfg.parallel_config.llm_pp_layers [5,11,11,11,11,11,11,9] --cfg.parallel_config.vit_pp_layers [32,0,0,0,0,0,0,0] --cfg.parallel_config.tp_size 2`

### 2. 报错

#### 问题1：
mmconvert报错：
```
Traceback (most recent call last):
  File "/root/miniconda3/envs/hw_test_bak/bin/mm-convert", line 8, in <module>
    sys.exit(main())
  File "/workspaces/hw_test/MindSpeed-MM/checkpoint/convert_cli.py", line 126, in main
    jsonargparse.CLI(Converter.subclasses)
  File "/root/miniconda3/envs/hw_test_bak/lib/python3.10/site-packages/jsonargparse/_cli.py", line 23, in CLI
    return auto_cli(*args, _stacklevel=3, **kwargs)
  File "/root/miniconda3/envs/hw_test_bak/lib/python3.10/site-packages/jsonargparse/_cli.py", line 125, in auto_cli
    return _run_component(component, init.get(subcommand))
  File "/root/miniconda3/envs/hw_test_bak/lib/python3.10/site-packages/jsonargparse/_cli.py", line 210, in _run_component
    return component(**cfg)
  File "/workspaces/hw_test/MindSpeed-MM/checkpoint/convert_cli.py", line 59, in mm_to_hf
    qwen2vl_mm_to_hf.main(cfg)
  File "/workspaces/hw_test/MindSpeed-MM/checkpoint/qwen2vl_mm_to_hf.py", line 182, in main
    state_dict = convert_mm_to_hf(state_dict, config)
  File "/workspaces/hw_test/MindSpeed-MM/checkpoint/qwen2vl_mm_to_hf.py", line 124, in convert_mm_to_hf
    q_chunk, k_chunk, v_chunk = torch.split(chunk, [q_size, k_size, v_size], dim=0)
  File "/root/miniconda3/envs/hw_test_bak/lib/python3.10/site-packages/torch/functional.py", line 187, in split
    return tensor.split(split_size_or_sections, dim)
  File "/root/miniconda3/envs/hw_test_bak/lib/python3.10/site-packages/torch/_tensor.py", line 864, in split
    return torch._VF.split_with_sizes(self, split_size, dim)
RuntimeError: split_with_sizes expects split_sizes to sum exactly to 1 (input tensor's size at dimension 0), but got split_sizes=[1024, 128, 128]
[ERROR] 2025-05-26-15:17:15 (PID:1319604, Device:0, RankID:-1) ERR99999 UNKNOWN application exception
```
请问在哪里有配置错误的地方吗？

#### 问题2：
我需要用mm微调完的模型做evaluation，但是似乎evaluation的时候又需要加载hf的模型（qwen2vl_pipeline.py）？所以我得把mm微调完的模型再转回hf的格式，才能做evaluation，是这样吗？

Ascend/MindSpeed-MM
暂停

内容风险标识

评论 (6)

Ascend/MindSpeed-MM暂停 .gitee-modal { width: 500px !important; }

内容风险标识