Qwen-72B, lora微调开启重计算后报错，不开重计算HBM占用很大

一、问题现象（附报错日志上下文）：
Qwen-72B lora微调开启重计算，backward时出现梯度缺失报错，以下是报错信息：

```
Traceback (most recent call last):
  File "pretrain_gpt.py", line 264, in <module>
Traceback (most recent call last):
  File "pretrain_gpt.py", line 264, in <module>
Traceback (most recent call last):
  File "pretrain_gpt.py", line 264, in <module>
Traceback (most recent call last):
  File "pretrain_gpt.py", line 264, in <module>
Traceback (most recent call last):
  File "pretrain_gpt.py", line 264, in <module>
Traceback (most recent call last):
  File "pretrain_gpt.py", line 264, in <module>
Traceback (most recent call last):
  File "pretrain_gpt.py", line 264, in <module>
Traceback (most recent call last):
  File "pretrain_gpt.py", line 264, in <module>
                    pretrain(train_valid_test_datasets_provider,pretrain(train_valid_test_datasets_provider,pretrain(train_valid_test_datasets_provider,    pretrain(train_valid_test_datasets_provider,    pretrain(train_valid_test_datasets_provider,

pretrain(train_valid_test_datasets_provider,
pretrain(train_valid_test_datasets_provider,
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 180, in pretrain
      File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 180, in pretrain
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 180, in pretrain

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 180, in pretrain

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 180, in pretrain
pretrain(train_valid_test_datasets_provider,  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 180, in pretrain
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 180, in pretrain

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 180, in pretrain
    iteration = train(forward_step_func,
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 784, in train
        iteration = train(forward_step_func,        iteration = train(forward_step_func,        iteration = train(forward_step_func,iteration = train(forward_step_func,

iteration = train(forward_step_func,iteration = train(forward_step_func,
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 784, in train
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 784, in train
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 784, in train

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 784, in train
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 784, in train
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 784, in train
    train_step(forward_step_func,
      File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 447, in train_step
iteration = train(forward_step_func,
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 784, in train
    train_step(forward_step_func,
    train_step(forward_step_func,
train_step(forward_step_func,
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 447, in train_step
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 447, in train_step
        train_step(forward_step_func,train_step(forward_step_func,  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 447, in train_step

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 447, in train_step
train_step(forward_step_func,  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 447, intrain_step

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 447, in train_step
        losses_reduced = forward_backward_func(losses_reduced = forward_backward_func(

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 1242, in forward_backward_pipelining_without_interleaving
        losses_reduced = forward_backward_func(losses_reduced = forward_backward_func(

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 1242, in forward_backward_pipelining_without_interleaving
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 1242, in forward_backward_pipelining_without_interleaving
    losses_reduced = forward_backward_func(  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 1242, in forward_backward_pipelining_without_interleaving

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 1242, inforward_backward_pipelining_without_interleaving
    losses_reduced = forward_backward_func(losses_reduced = forward_backward_func(

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 1242, in forward_backward_pipelining_without_interleaving
      File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 1242, inforward_backward_pipelining_without_interleaving
train_step(forward_step_func,
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/training.py", line 447, in train_step
    losses_reduced = forward_backward_func(
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 1242, in forward_backward_pipelining_without_interleaving
    input_tensor_grad = backward_step(
      File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 251, in backward_step
input_tensor_grad = backward_step(
input_tensor_grad = backward_step(input_tensor_grad = backward_step(    input_tensor_grad = backward_step(input_tensor_grad = backward_step(input_tensor_grad = backward_step(

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 251, in backward_step
input_tensor_grad = backward_step(

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 251, in backward_step
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 251, in backward_step

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 251, in backward_step
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 251, in backward_step
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 251, in backward_step
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 251, in backward_step
custom_backward(output_tensor[0], output_tensor_grad[0])
custom_backward(output_tensor[0], output_tensor_grad[0])  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 138, in custom_backward

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 138, in custom_backward
    custom_backward(output_tensor[0], output_tensor_grad[0])
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 138, in custom_backward
    custom_backward(output_tensor[0], output_tensor_grad[0])
    custom_backward(output_tensor[0], output_tensor_grad[0])custom_backward(output_tensor[0], output_tensor_grad[0])custom_backward(output_tensor[0], output_tensor_grad[0])  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 138, in custom_backward
Variable._execution_engine.run_backward(

File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 138, in custom_backward
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 138, in custom_backward
  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 138, in custom_backward
RuntimeError        : Variable._execution_engine.run_backward(Variable._execution_engine.run_backward(element 0 of tensorsdoes not require grad and does not have a grad_fn

RuntimeErrorRuntimeError: : element 0 of tensors does not require grad and does not have a grad_fnelement 0 of tensors does not require grad and does not have a grad_fn

Variable._execution_engine.run_backward(    Variable._execution_engine.run_backward(custom_backward(output_tensor[0], output_tensor_grad[0])Variable._execution_engine.run_backward(
Variable._execution_engine.run_backward(

RuntimeErrorRuntimeError  File "/home/ma-user/work/aicc/code_store/ModelLink-0304/megatron/core/pipeline_parallel/schedules.py", line 138, in custom_backward
: RuntimeError: RuntimeError: element 0 of tensors does not require grad and does not have a grad_fnelement 0 of tensors does not require grad and does not have a grad_fn: element 0 of tensors does not require grad and does not have a grad_fn

element 0 of tensors does not require grad and does not have a grad_fn

Variable._execution_engine.run_backward(
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
[2024-03-27 05:18:53,929] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 1925858) of binary: /root/miniconda3/envs/py38_torch210/bin/python
```

二、软件版本:
-- CANN 版本:  7.0.0.alpha003
--Pytorch 版本: 2.1.0
--Python 版本 (e.g., Python 3.7.5): 3.8.18
--操作系统版本 (e.g., Ubuntu 18.04): EulerOS 2.0 (SP8)

三、测试步骤：
[qwen训练脚本添加如下lora参数和重计算参数](https://gitee.com/ascend/ModelLink/blob/master/examples/qwen/pretrain_qwen_72b_ptd.sh)

```shell
--recompute-granularity full
--recompute-method block
--recompute-num-layers 16   #这里PP_size=5
--lora-target-modules query_key_value dense gate_proj dense_h_to_4h dense_4h_to_h
--lora-r 16
--lora-alpha 32
--lora-register-forward-hook word_embeddings input_layernorm
```
在N卡上也出现过类似使用lora+重计算报错，通过将torch.utils.checkpoint.checkpoint()函数里use_reentrant=False或者training_args.ddp_find_unused_parameters=False解决，这里怎么解决呢？

此外，单节点8卡lora微调qwen-72b，HBM占用为什么很高以至于OOM，按理来说每卡只有9B的参数，64GB的HBM应该够啊

**bk_origin_23分支里老的适配代码单机8卡可以lora微调Qwen-72B,开重计算也没问题

四、日志信息:

Ascend/MindSpeed-LLM
暂停

内容风险标识

评论 (1)

Ascend/MindSpeed-LLM暂停 .gitee-modal { width: 500px !important; }

内容风险标识