2.4K Star 8.2K Fork 4.4K

GVPMindSpore / mindspore

 / 详情

[CT][MS][OP]Asin report RuntimeError with dtype is float64 at ascend pynative mode

DONE
Bug-Report
创建于  
2021-11-18 16:34

Environment

  • Hardware Environment(Ascend/GPU/CPU): Ascend+pynative

Uncomment only one /device <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/device ascend

  • Software Environment:
    -- MindSpore version (source or binary):
    -- Python version (e.g., Python 3.7.5):
    -- OS platform and distribution (e.g., Linux Ubuntu 16.04):
    -- GCC/Compiler version (if compiled from source):

Related testcase

def test_asin_input_1x12x8x4x16x10x24_fp64():
def test_asin_input_20x7x13x9x88_fp64():
    def test_asin_input_20x7x88_fp64():
        fact = AsinFactory(input_shape=(20, 7, 88), dtype=np.float64)
        fact.forward_cmp()
>       fact.grad_cmp()

Steps to reproduce the issue

  1. 执行用例

Describe the current behavior

ascend+pynative , dtype是fp64时, 报错RuntimeError,具体信息如下:
ascend图模式下是pass的, cpu上也是pass的

    def test_asin_input_20x7x88_fp64():
        fact = AsinFactory(input_shape=(20, 7, 88), dtype=np.float64)
        fact.forward_cmp()
>       fact.grad_cmp()
test_asin.py:123:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../share/ops/primitive/asin_ops.py:72: in grad_cmp
    input_grad_mindspore = self.grad_mindspore_impl()
../share/ops/primitive/asin_ops.py:50: in grad_mindspore_impl
    input_grad = grad_net(input_ms, out_grad)
/root/archiconda3/envs/caory_3.7/lib/python3.7/site-packages/mindspore/nn/cell.py:483: in __call__
    raise err
/root/archiconda3/envs/caory_3.7/lib/python3.7/site-packages/mindspore/nn/cell.py:480: in __call__
    output = self.run_construct(cast_inputs, kwargs)
/root/archiconda3/envs/caory_3.7/lib/python3.7/site-packages/mindspore/nn/cell.py:363: in run_construct
    output = self.construct(*cast_inputs, **kwargs)
../share/grad.py:27: in construct
    return self.grad(self.network)(*inputs)
/root/archiconda3/envs/caory_3.7/lib/python3.7/site-packages/mindspore/ops/composite/base.py:395: in after_grad
    return grad_(fn)(*args, **kwargs)
/root/archiconda3/envs/caory_3.7/lib/python3.7/site-packages/mindspore/common/api.py:78: in wrapper
    results = fn(*arg, **kwargs)
/root/archiconda3/envs/caory_3.7/lib/python3.7/site-packages/mindspore/ops/composite/base.py:384: in after_grad
    out = _pynative_executor(fn, *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <mindspore.common.api._PynativeExecutor object at 0xffff9ae19450>, obj = Asin<>
args = (Tensor(shape=[20, 7, 88], dtype=Float64, value=
[[[ 2.64771472e-01,  1.84741329e-01,  3.00367079e-01 ...  1.70422975e...5e-01],
  [-3.12819027e-01,  4.64458469e-01, -6.54240419e-01 ...  1.23265315e+00,  7.13418732e-01,  7.93989728e-01]]]))
kwargs = {}

    def __call__(self, obj, *args, **kwargs):
        args = args + tuple(kwargs.values())
>       return self._executor(obj, args)
E       RuntimeError: mindspore/ccsrc/backend/session/ascend_session.cc:1734 SyncStream] Sync stream error!
[ERROR] DEVICE(69583,fffe30ff91e0,python3.7):2021-11-17-14:49:02.062.226 [mindspore/ccsrc/runtime/device/ascend/ascend_kernel_runtime.cc:612] TaskFailCallback] Execute TaskFailCallback failed. task_fail_info or current_graph_ is nullptr
[ERROR] DEVICE(69583,fffe30ff91e0,python3.7):2021-11-17-14:49:02.063.968 [mindspore/ccsrc/runtime/device/ascend/ascend_kernel_runtime.cc:612] TaskFailCallback] Execute TaskFailCallback failed. task_fail_info or current_graph_ is nullptr
[ERROR] DEVICE(69583,fffe30ff91e0,python3.7):2021-11-17-14:49:02.064.057 [mindspore/ccsrc/runtime/device/ascend/ascend_kernel_runtime.cc:958] SyncStream] Call runtime rtStreamSynchronize error.
[CRITICAL] SESSION(69583,fffe30ff91e0,python3.7):2021-11-17-14:49:02.064.072 [mindspore/ccsrc/backend/session/ascend_session.cc:1734] SyncStream] Sync stream error!
[ERROR] SESSION(69583,fffe30ff91e0,python3.7):2021-11-17-14:49:02.064.265 [mindspore/ccsrc/backend/session/ascend_session.cc:1763] ReportErrorMessage] Ascend error occurred, error message:
E39999: Inner Error!
        Aicpu kernel execute failed, device_id=3, stream_id=3, task_id=5, fault so_name=libaicpu_kernels.so, fault kernel_name=RunCpuKernel, fault op_name=, extend_info=[FUNC:GetError][FILE:stream.cc][LINE:712]
        Stream synchronize failed, stream = 0xfffe240a6640[FUNC:StreamSynchronize][FILE:logger.cc][LINE:285]
        rtStreamSynchronize execute failed, reason=[aicpu exception][FUNC:ReportFuncErrorReason][FILE:error_message_manage.cc][LINE:41]


Describe the expected behavior

执行pass无异常

Related log / screenshot

Special notes for this issue

评论 (4)

caoruyue1 创建了Bug-Report
caoruyue1 添加了
 
sig/ops
标签
caoruyue1 添加了
 
kind/bug
标签
caoruyue1 添加了device/ascend(已删除)标签
展开全部操作日志

Please add labels (comp or sig), also you can visit https://gitee.com/mindspore/community/blob/master/sigs/dx/docs/labels.md to find more.
为了让代码尽快被审核,请您为Pull Request打上 组件(comp)或兴趣组(sig) 标签,打上标签的PR可以直接推送给责任人进行审核。
更多的标签可以查看https://gitee.com/mindspore/community/blob/master/sigs/dx/docs/labels.md
以组件相关代码提交为例,如果你提交的是data组件代码,你可以这样评论:
//comp/data
当然你也可以邀请data SIG组来审核代码,可以这样写:
//sig/data
另外你还可以给这个PR标记类型,例如是bugfix或者是特性需求:
//kind/bug or //kind/feature
恭喜你,你已经学会了使用命令来打标签,接下来就在下面的评论里打上标签吧!

问题回归不通过

caoruyue1 添加协作者caoruyue1
caoruyue1 负责人caoruyue1 修改为liangxhao
caoruyue1 取消协作者caoruyue1
caoruyue1 任务状态VALIDATION 修改为WIP
liangxhao 任务状态WIP 修改为VALIDATION
liangxhao 添加协作者liangxhao
liangxhao 负责人liangxhao 修改为caoruyue1

问题原因:
AsinFactory方式下,so文件名识别错误

解决方案:
写死so文件名

caoruyue1 任务状态VALIDATION 修改为DONE
caoruyue1 移除了device/ascend(已删除)标签

登录 后才可以发表评论

状态
负责人
项目
里程碑
Pull Requests
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
开始日期   -   截止日期
-
置顶选项
优先级
预计工期 (小时)
参与者(5)
Python
1
https://gitee.com/mindspore/mindspore.git
git@gitee.com:mindspore/mindspore.git
mindspore
mindspore
mindspore

搜索帮助