2.4K Star 8.2K Fork 4.4K

GVPMindSpore / mindspore

 / 详情

MindSpore Droupout静态图出错

WIP
Bug-Report
创建于  
2024-04-13 16:14
name about labels
Bug Report Use this template for reporting a bug kind/bug

Describe the current behavior / 问题描述 (Mandatory / 必填)

mindspore 静态图使用 Dropout报错

Environment / 环境信息 (Mandatory / 必填)

  • Hardware Environment / 硬件环境:GPU

  • Software Environment / 软件环境 (Mandatory / 必填):
    -- MindSpore version : 2.2.13
    -- Python version : 3.9.19
    -- OS platform and distribution : Ubuntu 20.04.6 LTS

  • Excute Mode / 执行模式 (Mandatory / 必填):Graph

Related testcase / 关联用例 (Mandatory / 必填)

MindNLP

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

1.git clone https://github.com/mindspore-lab/mindnlp.git
2.cd mindnlp/llm/parallel/convbert
3.python train.py

Describe the expected behavior / 预期结果 (Mandatory / 必填)

训练脚本启动失败

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
Loading model cost 0.619 seconds.
Prefix dict has been built successfully.
The following parameters in checkpoint files are not loaded:
['embeddings.position_ids']
The following parameters in models are missing parameter:
['qa_outputs.weight', 'qa_outputs.bias']
[ERROR] PIPELINE(3792830,7fd2f04f54c0,python):2024-04-12-21:42:20.160.153 [mindspore/ccsrc/pipeline/jit/ps/fallback.cc:494] GeneratePyExecuteNodeWithScriptSrc] Not found PyExecute input. script: out, _ = dropout_op(input)
Traceback (most recent call last):
  File "/home/daiyuxin/neoming_convbert/dev/mindnlp/llm/parallel/convbert/train.py", line 104, in <module>
    main(args)
  File "/home/daiyuxin/neoming_convbert/dev/mindnlp/llm/parallel/convbert/train.py", line 69, in main
    model.build(ds, epoch=args.num_epochs)
  File "/home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindspore/train/model.py", line 1274, in build
    self._init(train_dataset, valid_dataset, sink_size, epoch)
  File "/home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindspore/train/model.py", line 529, in _init
    train_network.compile(*inputs)
  File "/home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindspore/nn/cell.py", line 997, in compile
    _cell_graph_executor.compile(self, phase=self.phase,
  File "/home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindspore/common/api.py", line 1547, in compile
    result = self._graph_executor.compile(obj, args, kwargs, phase, self._use_vm_mode())
RuntimeError: In the inputs of operation 'getitem', there are unsupported syntax in graph mode. Those codes would be fallen back to python interpreter, which is not supported for operation 'getitem'.

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/frontend/operator/composite/multitype_funcgraph.cc:194 PrintMatchFailLog

----------------------------------------------------
- The Traceback of Net Construct Code:
----------------------------------------------------
# 0 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindspore/train/dataset_helper.py:103
        return self.network(*outputs)
               ^
# 1 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindspore/nn/wrap/cell_wrapper.py:417
        if not self.sense_flag:
# 2 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindspore/nn/wrap/cell_wrapper.py:418
            return self._no_sens_impl(*inputs)
                   ^
# 3 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindspore/nn/wrap/cell_wrapper.py:437
        if self.return_grad:
# 4 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindspore/nn/wrap/cell_wrapper.py:433
        loss = self.network(*inputs)
               ^
# 5 In file /home/daiyuxin/neoming_convbert/dev/mindnlp/llm/parallel/convbert/train.py:42
        output = self._backbone(
# 6 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindnlp/transformers/models/convbert/graph_convbert.py:576
        outputs = self.convbert(
# 7 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindnlp/transformers/models/convbert/graph_convbert.py:543
        hidden_states = self.encoder(
# 8 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindnlp/transformers/models/convbert/graph_convbert.py:543
        hidden_states = self.encoder(
                        ^
# 9 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindnlp/transformers/models/convbert/graph_convbert.py:485
        for i, layer_cell in enumerate(self.layer):
        ^
# 10 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindnlp/transformers/models/convbert/graph_convbert.py:487
            layer_outputs = layer_cell(
# 11 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindnlp/transformers/models/convbert/graph_convbert.py:444
        self_attention_outputs = self.attention(
# 12 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindnlp/transformers/models/convbert/graph_convbert.py:380
        self_outputs = self.self(
# 13 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindnlp/transformers/models/convbert/graph_convbert.py:297
        attention_probs = ops.dropout(attention_probs, p=self.dropout_p)
                          ^
# 14 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindspore/ops/function/nn_func.py:1287
    if training is False:
    ^
# 15 In file /home/daiyuxin/miniconda3/envs/neoming_convbert/lib/python3.9/site-packages/mindspore/ops/function/nn_func.py:1293
    out, _ = dropout_op(input)
    ^
 (See file '/home/daiyuxin/neoming_convbert/dev/mindnlp/llm/parallel/convbert/rank_0/om/analyze_fail.ir' for more details. Get instructions about `analyze_fail.ir` at https://www.mindspore.cn/search?inputValue=analyze_fail.ir)
[WARNING] MD(3792830,7fd2f04f54c0,python):2024-04-12-21:42:23.473.235 [mindspore/ccsrc/minddata/dataset/engine/datasetops/data_queue_op.cc:163] ~DataQueueOp]
preprocess_batch: 3;
batch_queue: 0, 0, 3, 2, 2, 1, 16;
            push_start_time -> push_end_time
2024-04-12-21:42:17.436.851 -> 2024-04-12-21:42:17.835.979
2024-04-12-21:42:17.836.002 -> 2024-04-12-21:42:17.836.285
2024-04-12-21:42:17.836.289 -> 2024-04-12-21:42:23.472.459
For more details, please refer to the FAQ at https://www.mindspore.cn/docs/en/master/faq/data_processing.html.

Special notes for this issue/备注 (Optional / 选填)

评论 (9)

neoming 创建了Bug-Report

Please assign maintainer to check this issue.
请为此issue分配处理人。
@fangwenyi @chengxiaoli @Shawny

感谢您的提问,您可以评论//mindspore-assistant更快获取帮助:

  1. 如果您刚刚接触MindSpore,或许您可以在教程找到答案
  2. 如果您是资深Pytorch用户,您或许需要:
  1. 如果您遇到动态图问题,可以设置set_context(pynative_synchronize=True)查看报错栈协助定位
  2. 模型精度调优问题可参考官网调优指南
  3. 如果您反馈的是框架BUG,请确认您在ISSUE中提供了MindSpore版本、使用的后端类型(CPU、GPU、Ascend)、环境、训练的代码官方链接以及可以复现报错的代码的启动方式等必要的定位信息
  4. 如果您已经定位出问题根因,欢迎提交PR参与MindSpore开源社区,我们会尽快review
neoming 修改了描述
neoming 修改了描述

输入图片说明 您好,请提供在如图位置生成的analyze_fail.ir文件

好的,晚点提供出来

i-robot 添加了
 
foruda
标签
Shawny 任务状态TODO 修改为WIP
Shawny 负责人设置为李良灿
Shawny 关联项目设置为MindSpore Issue Assistant
Shawny 计划开始日期设置为2024-04-15
Shawny 计划截止日期设置为2024-05-15
Shawny 添加了
 
mindspore-assistant
标签
Shawny 添加了
 
sig/ops
标签
Shawny 添加了
 
sig/graph-engine
标签
i-robot 添加了
 
pan
标签

您好,由于问题单没有回复,我们后续会关闭,如您仍有疑问,可以反馈下具体信息,并将ISSUE状态修改为WIP,我们这边会进一步跟踪,谢谢

您好,由于问题单没有回复,我们后续会关闭,如您仍有疑问,可以反馈下具体信息,并将ISSUE状态修改为WIP,我们这边会进一步跟踪,谢谢

该问题新版本已修复,可以尝试使用2.2以后的版本,或将 out, _ = dropout_op(input)改为取下标的方式如out = dropout_op(input)[0]

登录 后才可以发表评论

状态
负责人
项目
里程碑
Pull Requests
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
开始日期   -   截止日期
-
置顶选项
优先级
预计工期 (小时)
参与者(3)
8105229 neoming 1614668116
Python
1
https://gitee.com/mindspore/mindspore.git
git@gitee.com:mindspore/mindspore.git
mindspore
mindspore
mindspore

搜索帮助

344bd9b3 5694891 D2dac590 5694891