执行微调时，LLM切分为0时出现learning_rate=None assertion报错

一、问题现象（附报错日志上下文）：
 [2025-09-04 12:02:56] iteration        1/   10000 | consumed samples:           48 | elapsed time per iteration (ms): 16061.3 | learning rate: 1.000000E-09 | global batch size:    48 | loss: 1.409824E+00 | loss scale: 1.0 | grad norm: 8.030 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
[Rank 1] (after 1 iterations) memory (MB) | allocated: 41407.015625 | max allocated: 41407.01611328125 | reserved: 43564.0 | max reserved: 43564.0
[Rank 3] (after 1 iterations) memory (MB) | allocated: 49425.904296875 | max allocated: 49425.927734375 | reserved: 50636.0 | max reserved: 50636.0
[Rank 2] (after 1 iterations) memory (MB) | allocated: 40065.0771484375 | max allocated: 40065.07763671875 | reserved: 42048.0 | max reserved: 42048.0
[rank0]: Traceback (most recent call last):
[rank0]:   File "/mnt/fangcr/ascendc-api-adv/examples/pipeline/vector_chain/MindSpeed_mm_test/MindSpeed-MM/pretrain_vlm.py", line 191, in <module>
[rank0]:     pretrain(
[rank0]:   File "/mnt/fangcr/ascendc-api-adv/examples/pipeline/vector_chain/MindSpeed_mm_test/MindSpeed-MM/mindspeed_mm/training.py", line 240, in pretrain
[rank0]:     iteration, num_floating_point_operations_so_far = train(
[rank0]:   File "/mnt/fangcr/ascendc-api-adv/examples/pipeline/vector_chain/MindSpeed_mm_test/MindSpeed-MM/mindspeed_mm/training.py", line 480, in train
[rank0]:     report_memory_flag = training_log(
[rank0]:   File "/mnt/fangcr/ascendc-api-adv/examples/pipeline/vector_chain/MindSpeed_mm_test/MindSpeed-MM/mindspeed_mm/training.py", line 888, in training_log
[rank0]:     raise AssertionError
[rank0]: AssertionError

二、软件版本:
-- CANN 版本 (e.g., ):  CANN 5
--Tensorflow/Pytorch/MindSpore 版本: Pytorch 2.7.1 
--Python 版本 (e.g., Python 3.7.5): Python 3.10
-- MindStudio版本 (e.g., MindStudio 2.0.0 (beta3)):
--操作系统版本 (e.g., Ubuntu 18.04): Ubuntu 22.04 aarch

三、测试步骤：
在完成对于Qwen2.5vl的权重切分与数据处理后执行微调脚本时，正常加载数据与模型，但是开始训练出现如上文所述assertionError,在training.py中定位到是因为learning_rate=None的缘故，切分参数为vit=[32,0,0,0] llm=[0,8,10,10]，改成llm=[1,7,10,10]后可以执行，可以问下这样的切分是否合理还是lr的加载存在问题？

Ascend/MindSpeed-MM
暂停

内容风险标识

评论 (4)

Ascend/MindSpeed-MM暂停 .gitee-modal { width: 500px !important; }

内容风险标识