169 Star 1K Fork 241

MindSpore / community

 / 详情

BaseTunerLayer类的get_base_layer()方法循环调用导致Python对象时超过的最大递归深度。

DONE
Question
创建于  
2024-04-25 19:16
name about labels
Bug Report Use this template for reporting a bug kind/bug

Describe the current behavior / 问题描述 (Mandatory / 必填)

在微调模型时候,出现了因为BaseTunerLayer类的get_base_layer()方法的循环引用,导致调用Python对象时超过的最大递归深度。

Environment / 环境信息 (Mandatory / 必填)

  • Hardware Environment(Ascend/GPU/CPU) / 硬件环境: GPU

Please delete the backend not involved / 请删除不涉及的后端:
/device ascend/GPU/CPU/kirin/等其他芯片

  • Software Environment / 软件环境 (Mandatory / 必填):
    -- MindSpore version (e.g., 1.7.0.Bxxx) :2.2.11
    -- Python version (e.g., Python 3.7.5) :3.9

  • Excute Mode / 执行模式 (Mandatory / 必填)(PyNative/Graph):

对MindNLP进行编包装包
然后在mindnlp/llm/peft/里面选择一个模型放到一个新的目录下
使用有MindNLP包的环境执行里面的train.py进行微调训练,例子如下:
aiyuxin@mindspore:/data1/yy/train_gpt_bigcode$ ./run_peft.sh

Related testcase / 关联用例 (Mandatory / 必填)

# 出问题的内部代码
class BaseTunerLayer(ABC):
    def get_base_layer(self) -> nn.Cell:
        base_layer = self
        while hasattr(base_layer, "base_layer"):
            base_layer = base_layer.base_layer
        return base_layer

    @property
    def weight(self) -> Tensor:
        # This is required for some transformers code, e.g. for T5, weight is accessed as:
        #     self.wo.weight
        # where "wo" is the adapter layer.
        # https://github.com/huggingface/transformers/blob/78f6ed6c70b29c1560780e3869a7ad4c6b3d2710/src/transformers
        # /models/t5/modeling_t5.py#L292
        base_layer = self.get_base_layer()
        weight = base_layer.weight
        return weight

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

先对MindNLP进行编包装包

然后在mindnlp/llm/peft/里面选择一个模型放到一个新的目录下

使用有MindNLP包的环境执行里面的train.py进行微调训练

aiyuxin@mindspore:/data1/yy/train_gpt_bigcode$ ./run_peft.sh

Describe the expected behavior / 预期结果 (Mandatory / 必填)

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

(yy_env) daiyuxin@mindspore:/data1/yy/train_gpt_bigcode$ ./run_peft.sh
Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
Loading model cost 0.614 seconds.
Prefix dict has been built successfully.
/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/datasets/load.py:2516: FutureWarning: 'use_auth_token' was deprecated in favor of 'token' in version 2.14.0 and will be removed in 3.0.0.
You can remove this warning by passing 'token=<use_auth_token>' instead.
warnings.warn(
Size of the train set: 5875. Size of the validation set: 30
0%| | 0/400 [00:00<?, ?it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (7483 > 2048). Running this sequence through the model will result in indexing errors
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 400/400 [00:03<00:00, 112.99it/s]
The character to token ratio of the dataset is: 3.31
FIM is not supported by tokenizer, disabling FIM
FIM is not supported by tokenizer, disabling FIM
Traceback (most recent call last):
File "/data1/yy/train_gpt_bigcode/train.py", line 187, in
main(args)
File "/data1/yy/train_gpt_bigcode/train.py", line 171, in main
model = get_peft_model(model, peft_config)
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/mapping.py", line 97, in get_peft_model
return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](model, peft_config, adapter_name=adapter_name)
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/peft_model.py", line 390, in init
super().init(model, peft_config, adapter_name)
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/peft_model.py", line 80, in init
self.base_model = PEFT_TYPE_TO_MODEL_MAPPING[peft_config.peft_type](
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/tuners/lora.py", line 161, in init
super().init(model, config, adapter_name)
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/tuners/tuners_utils.py", line 100, in init
self.inject_adapter(self.model, adapter_name)
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/tuners/tuners_utils.py", line 250, in inject_adapter
self._create_and_replace(peft_config, adapter_name, target, target_name, parent, **optionnal_kwargs)
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/tuners/lora.py", line 262, in _create_and_replace
new_module = self._create_new_module(lora_config, adapter_name, target, **kwargs)
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/tuners/lora.py", line 539, in _create_new_module
new_module = Linear(adapter_name, in_features, out_features, has_bias=bias, **kwargs)
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/tuners/lora.py", line 691, in init
nn.Dense.init(self, in_features, out_features, **kwargs)
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/injection.py", line 587, in init
self.weight = Parameter(initializer(
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindspore/nn/cell.py", line 966, in setattr
self._set_attr_for_parameter(name, value)
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindspore/nn/cell.py", line 859, in _set_attr_for_parameter
self.insert_param_to_cell(name, value)
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindspore/nn/cell.py", line 1224, in insert_param_to_cell
if hasattr(self, param_name) and param_name not in self._params:
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/tuners/tuners_utils.py", line 329, in weight
weight = base_layer.weight
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/tuners/tuners_utils.py", line 329, in weight
weight = base_layer.weight
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/tuners/tuners_utils.py", line 329, in weight
weight = base_layer.weight
[Previous line repeated 974 more times]
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/tuners/tuners_utils.py", line 328, in weight
base_layer = self.get_base_layer()
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindnlp/peft/tuners/tuners_utils.py", line 317, in get_base_layer
while hasattr(base_layer, "base_layer"):
File "/home/daiyuxin/miniconda3/envs/yy_env/lib/python3.9/site-packages/mindspore/nn/cell.py", line 440, in getattr
"The '{}' object has no attribute '{}'.".format(type(self).name, name)
RecursionError: maximum recursion depth exceeded while calling a Python object

Special notes for this issue/备注 (Optional / 选填)

评论 (6)

Yaung 创建了Bug-Report

Please assign maintainer to check this issue.
请为此issue分配处理人。
@fangwenyi @chengxiaoli @Shawny

感谢您的提问,您可以评论//mindspore-assistant更快获取帮助:

  1. 如果您刚刚接触MindSpore,或许您可以在教程找到答案
  2. 如果您是资深Pytorch用户,您或许需要:
  1. 如果您遇到动态图问题,可以设置set_context(pynative_synchronize=True)查看报错栈协助定位
  2. 模型精度调优问题可参考官网调优指南
  3. 如果您反馈的是框架BUG,请确认您在ISSUE中提供了MindSpore版本、使用的后端类型(CPU、GPU、Ascend)、环境、训练的代码官方链接以及可以复现报错的代码的启动方式等必要的定位信息
  4. 如果您已经定位出问题根因,欢迎提交PR参与MindSpore开源社区,我们会尽快review
Yaung 修改了描述
Yaung 修改了描述
Yaung 修改了描述
Yaung 修改了描述
Yaung 修改了描述
Yaung 修改了描述
Yaung 修改了描述

mindnlp建议去对应代码仓进行提问,如您怀疑是框架问题建议简化用例后再提交ISSUE

我是运行 mindnlp/llm/peft/ 里面模型的微调例子出现的问题。

Shawny 负责人设置为Shawny
Shawny 任务类型Bug-Report 修改为Question
Shawny 关联项目设置为MindSpore Issue Assistant
Shawny 计划开始日期设置为2024-04-28
Shawny 计划截止日期设置为2024-05-28
Shawny 添加了
 
mindspore-assistant
标签
Shawny 任务状态TODO 修改为VALIDATION
Yaung 修改了描述

我是运行 mindnlp/llm/peft/ 里面模型的微调例子出现的问题。

@Yaung mindnlp的问题麻烦移步mindnlp代码仓提问,会有开发同事进行回复

您好,由于问题单没有回复暂时关闭,如您仍有疑问,可以反馈下具体信息,并将ISSUE状态修改为WIP,我们这边会进一步跟踪,谢谢

Shawny 任务状态VALIDATION 修改为DONE

登录 后才可以发表评论

状态
负责人
项目
里程碑
Pull Requests
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
开始日期   -   截止日期
-
置顶选项
优先级
预计工期 (小时)
参与者(3)
8108889 shawny233 1628167362
1
https://gitee.com/mindspore/community.git
git@gitee.com:mindspore/community.git
mindspore
community
community

搜索帮助

344bd9b3 5694891 D2dac590 5694891