[CT][MS][CI]<test_dvm.py::test_easy_fuse_dvm>failed in gate

### Describe the current behavior / 问题描述 (Mandatory / 必填)
 2.3分支
 test_dvm.py::test_easy_fuse_dvm failed in gate
用例已下至level1,修复后需要重新加回level0观察

### Environment / 环境信息 (Mandatory / 必填)
- **Hardware Environment(`Ascend`/`GPU`/`CPU`)  / 硬件环境**: 
> Please delete the backend not involved / 请删除不涉及的后端:
> /device ascend910B

- **Software Environment / 软件环境 (Mandatory / 必填)**:
-- MindSpore version (e.g., 1.7.0.Bxxx) :
-- Python version (e.g., Python 3.7.5) :
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):
-- GCC/Compiler version (if compiled from source):

- **Excute Mode / 执行模式 (Mandatory / 必填)(`PyNative`/`Graph`)**: 
> Please delete the mode not involved / 请删除不涉及的模式:
> /mode pynative
> /mode graph

### Related testcase / 关联用例 (Mandatory / 必填)
 [gate faild]tests/st/ops/graph_kernel
test_dvm.py::test_easy_fuse_dvm
### Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)
1. 
2.  
3.

### Describe the expected behavior / 预期结果 (Mandatory / 必填)
 
### Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

```
 [2024-04-01T03:36:35.674Z] F
[2024-04-01T03:36:35.674Z] 
[2024-04-01T03:36:35.674Z] =================================== FAILURES ===================================
[2024-04-01T03:36:35.674Z] _________________ test_easy_fuse_dvm[float32-shape11-shape21] __________________
[2024-04-01T03:36:35.674Z] 
[2024-04-01T03:36:35.674Z] shape1 = (44, 1, 47, 1), shape2 = (1, 34, 1, 91)
[2024-04-01T03:36:35.674Z] dtype = <class 'numpy.float32'>
[2024-04-01T03:36:35.674Z] 
[2024-04-01T03:36:35.674Z]     @pytest.mark.level0
[2024-04-01T03:36:35.674Z]     @pytest.mark.platform_arm_ascend910b_training
[2024-04-01T03:36:35.674Z]     @pytest.mark.env_onecard
[2024-04-01T03:36:35.674Z]     @pytest.mark.parametrize("shape1, shape2", [((32, 1024), (32, 1024)), ((44, 1, 47, 1), (1, 34, 1, 91))])
[2024-04-01T03:36:35.674Z]     @pytest.mark.parametrize("dtype", [np.float16, np.float32])
[2024-04-01T03:36:35.674Z]     def test_easy_fuse_dvm(shape1, shape2, dtype):
[2024-04-01T03:36:35.674Z]         """
[2024-04-01T03:36:35.674Z]         Feature: easy test case for graph_kernel in Ascend.
[2024-04-01T03:36:35.674Z]         Description: ascend test case, use graph_kernel execute ops.
[2024-04-01T03:36:35.674Z]         Expectation: the result match with close graph_kernel result
[2024-04-01T03:36:35.674Z]         """
[2024-04-01T03:36:35.674Z]         os.environ["GRAPH_OP_RUN"] = "1"
[2024-04-01T03:36:35.674Z]         context.set_context(mode=context.GRAPH_MODE, device_target="Ascend")
[2024-04-01T03:36:35.674Z] >       fuse(shape1, shape2, dtype)
[2024-04-01T03:36:35.674Z] 
[2024-04-01T03:36:35.674Z] test_dvm.py:98: 
[2024-04-01T03:36:35.674Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[2024-04-01T03:36:35.674Z] 
[2024-04-01T03:36:35.674Z] shape1 = (44, 1, 47, 1), shape2 = (1, 34, 1, 91)
[2024-04-01T03:36:35.674Z] dtype = <class 'numpy.float32'>
[2024-04-01T03:36:35.674Z] 
[2024-04-01T03:36:35.674Z]     def fuse(shape1, shape2, dtype):
[2024-04-01T03:36:35.674Z]         i0 = Tensor(np.random.uniform(1, 2, shape1).astype(dtype))
[2024-04-01T03:36:35.674Z]         i1 = Tensor(np.random.uniform(1, 2, shape2).astype(dtype))
[2024-04-01T03:36:35.674Z]         expect = get_output(ComplexNet, [i0, i1], enable_graph_kernel=False)
[2024-04-01T03:36:35.674Z]         output = get_output(ComplexNet, [i0, i1], enable_graph_kernel=True)
[2024-04-01T03:36:35.674Z]         if dtype == np.float32:
[2024-04-01T03:36:35.674Z]             eps = 1e-5
[2024-04-01T03:36:35.674Z]         elif dtype == np.float16:
[2024-04-01T03:36:35.674Z]             eps = 1e-3
[2024-04-01T03:36:35.674Z]         else:
[2024-04-01T03:36:35.674Z]             eps = 0
[2024-04-01T03:36:35.674Z]         assert np.allclose(expect[0].asnumpy(), output[0].asnumpy(), eps, eps)
[2024-04-01T03:36:35.674Z] >       assert np.allclose(expect[1].asnumpy(), output[1].asnumpy(), eps, eps)
[2024-04-01T03:36:35.674Z] E       assert False
[2024-04-01T03:36:35.674Z] E        +  where False = <function allclose at 0xffff092ee830>(array([[[[60.648937],\n         [62.913143],\n         [61.323658],\n         [58.64773 ],\n         [61.983624],\n        ... [62.22449 ],\n         [63.420677],\n         [63.9927  ],\n         [64.333916],\n         [62.56211 ]]]], dtype=float32), array([[[[ 60.648945],\n         [ 62.91314 ],\n         [ 61.32366 ],\n         [ 58.64773 ],\n         [ 61.983624],\n   ....85232 ],\n         [118.41847 ],\n         [125.704445],\n         [124.2419  ],\n         [118.46937 ]]]], dtype=float32), 1e-05, 1e-05)
[2024-04-01T03:36:35.674Z] E        +    where <function allclose at 0xffff092ee830> = np.allclose
[2024-04-01T03:36:35.674Z] E        +    and   array([[[[60.648937],\n         [62.913143],\n         [61.323658],\n         [58.64773 ],\n         [61.983624],\n        ... [62.22449 ],\n         [63.420677],\n         [63.9927  ],\n         [64.333916],\n         [62.56211 ]]]], dtype=float32) = <bound method Tensor.asnumpy of Tensor(shape=[1, 1, 47, 1], dtype=Float32, value=\n[[[[ 6.06489372e+01],\n   [ 6.29131432e+01],\n   [ 6.13236580e+01],\n   ...\n   [ 6.39926987e+01],\n   [ 6.43339157e+01],\n   [ 6.25621109e+01]]]])>()
[2024-04-01T03:36:35.675Z] E        +      where <bound method Tensor.asnumpy of Tensor(shape=[1, 1, 47, 1], dtype=Float32, value=\n[[[[ 6.06489372e+01],\n   [ 6.29131432e+01],\n   [ 6.13236580e+01],\n   ...\n   [ 6.39926987e+01],\n   [ 6.43339157e+01],\n   [ 6.25621109e+01]]]])> = Tensor(shape=[1, 1, 47, 1], dtype=Float32, value=\n[[[[ 6.06489372e+01],\n   [ 6.29131432e+01],\n   [ 6.13236580e+01],\n   ...\n   [ 6.39926987e+01],\n   [ 6.43339157e+01],\n   [ 6.25621109e+01]]]]).asnumpy
[2024-04-01T03:36:35.675Z] E        +    and   array([[[[ 60.648945],\n         [ 62.91314 ],\n         [ 61.32366 ],\n         [ 58.64773 ],\n         [ 61.983624],\n   ....85232 ],\n         [118.41847 ],\n         [125.704445],\n         [124.2419  ],\n         [118.46937 ]]]], dtype=float32) = <bound method Tensor.asnumpy of Tensor(shape=[1, 1, 47, 1], dtype=Float32, value=\n[[[[ 6.06489449e+01],\n   [ 6.29131393e+01],\n   [ 6.13236618e+01],\n   ...\n   [ 1.25704445e+02],\n   [ 1.24241898e+02],\n   [ 1.18469368e+02]]]])>()
[2024-04-01T03:36:35.675Z] E        +      where <bound method Tensor.asnumpy of Tensor(shape=[1, 1, 47, 1], dtype=Float32, value=\n[[[[ 6.06489449e+01],\n   [ 6.29131393e+01],\n   [ 6.13236618e+01],\n   ...\n   [ 1.25704445e+02],\n   [ 1.24241898e+02],\n   [ 1.18469368e+02]]]])> = Tensor(shape=[1, 1, 47, 1], dtype=Float32, value=\n[[[[ 6.06489449e+01],\n   [ 6.29131393e+01],\n   [ 6.13236618e+01],\n   ...\n   [ 1.25704445e+02],\n   [ 1.24241898e+02],\n   [ 1.18469368e+02]]]]).asnumpy
[2024-04-01T03:36:35.675Z] 
[2024-04-01T03:36:35.675Z] test_dvm.py:81: AssertionError
[2024-04-01T03:36:35.675Z] =============================== warnings summary ===============================
```
test_dvm.py::test_easy_fuse_dvm	7	7	r2.3	#66497	https://build.mindspore.cn/blue/rest/organizations/jenkins/pipelines/Gitee_Gate_Compile_And_Test/runs/101333/nodes/759/log

#67761	https://build.mindspore.cn/blue/rest/organizations/jenkins/pipelines/Gitee_Gate_Compile_And_Test/runs/101414/nodes/170/log

#67653	https://build.mindspore.cn/blue/rest/organizations/jenkins/pipelines/Gitee_Gate_Compile_And_Test/runs/101577/nodes/237/log

#67772	https://build.mindspore.cn/blue/rest/organizations/jenkins/pipelines/Gitee_Gate_Compile_And_Test/runs/101627/nodes/170/log

#67751	https://build.mindspore.cn/blue/rest/organizations/jenkins/pipelines/Gitee_Gate_Compile_And_Test/runs/101664/nodes/759/log

#67485	https://build.mindspore.cn/blue/rest/organizations/jenkins/pipelines/Gitee_Gate_Compile_And_Test/runs/101859/nodes/759/log

#67120	https://build.mindspore.cn/blue/rest/organizations/jenkins/pipelines/Gitee_Gate_Compile_And_Test/runs/102131/nodes/759/log

### Special notes for this issue/备注 (Optional / 选填)

加回level0门禁后还是挂的《又下掉了= =》
test_dvm.py::test_easy_fuse_dvm 2.3分支随机挂#68169	https://build.mindspore.cn/blue/rest/organizations/jenkins/pipelines/Gitee_Gate_Compile_And_Test/runs/106825/nodes/759/log
https://build.mindspore.cn/blue/rest/organizations/jenkins/pipelines/Gitee_Gate_Compile_And_Test/runs/106981/nodes/759/log

#68202	https://build.mindspore.cn/blue/rest/organizations/jenkins/pipelines/Gitee_Gate_Compile_And_Test/runs/107321/nodes/170/log

#67873	https://build.mindspore.cn/blue/rest/organizations/jenkins/pipelines/Gitee_Gate_Compile_And_Test/runs/106131/nodes/237/log

#68100	https://build.mindspore.cn/blue/rest/organizations/jenkins/pipelines/Gitee_Gate_Compile_And_Test/runs/106391/nodes/237/log

```
[2024-04-11T15:22:12.714Z]     @pytest.mark.level0
[2024-04-11T15:22:12.714Z]     @pytest.mark.platform_arm_ascend910b_training
[2024-04-11T15:22:12.714Z]     @pytest.mark.env_onecard
[2024-04-11T15:22:12.714Z]     @pytest.mark.parametrize("shape1, shape2", [((32, 1024), (32, 1024)), ((44, 1, 47, 1), (1, 34, 1, 91))])
[2024-04-11T15:22:12.714Z]     @pytest.mark.parametrize("dtype", [np.float16, np.float32])
[2024-04-11T15:22:12.714Z]     def test_easy_fuse_dvm(shape1, shape2, dtype):
[2024-04-11T15:22:12.714Z]         """
[2024-04-11T15:22:12.714Z]         Feature: easy test case for graph_kernel in Ascend.
[2024-04-11T15:22:12.714Z]         Description: ascend test case, use graph_kernel execute ops.
[2024-04-11T15:22:12.714Z]         Expectation: the result match with close graph_kernel result
[2024-04-11T15:22:12.714Z]         """
[2024-04-11T15:22:12.714Z]         os.environ["GRAPH_OP_RUN"] = "1"
[2024-04-11T15:22:12.714Z]         context.set_context(mode=context.GRAPH_MODE, device_target="Ascend")
[2024-04-11T15:22:12.714Z] >       fuse(shape1, shape2, dtype)
[2024-04-11T15:22:12.714Z] 
[2024-04-11T15:22:12.714Z] test_dvm.py:101: 
[2024-04-11T15:22:12.714Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[2024-04-11T15:22:12.714Z] 
[2024-04-11T15:22:12.714Z] shape1 = (44, 1, 47, 1), shape2 = (1, 34, 1, 91)
[2024-04-11T15:22:12.714Z] dtype = <class 'numpy.float32'>
[2024-04-11T15:22:12.714Z] 
[2024-04-11T15:22:12.714Z]     def fuse(shape1, shape2, dtype):
[2024-04-11T15:22:12.714Z]         np.random.seed(1)
[2024-04-11T15:22:12.714Z]         i0 = Tensor(np.random.uniform(1, 2, shape1).astype(dtype))
[2024-04-11T15:22:12.714Z]         i1 = Tensor(np.random.uniform(1, 2, shape2).astype(dtype))
[2024-04-11T15:22:12.714Z]         expect = get_output(ComplexNet, [i0, i1], enable_graph_kernel=False)
[2024-04-11T15:22:12.714Z]         expects = [e.asnumpy() for e in expect]
[2024-04-11T15:22:12.714Z]         output = get_output(ComplexNet, [i0, i1], enable_graph_kernel=True)
[2024-04-11T15:22:12.714Z]         outputs = [o.asnumpy() for o in output]
[2024-04-11T15:22:12.714Z]         if dtype == np.float32:
[2024-04-11T15:22:12.714Z]             eps = 1e-4
[2024-04-11T15:22:12.714Z]         elif dtype == np.float16:
[2024-04-11T15:22:12.714Z]             eps = 1e-3
[2024-04-11T15:22:12.714Z]         else:
[2024-04-11T15:22:12.714Z]             eps = 0
[2024-04-11T15:22:12.714Z]         np.testing.assert_allclose(expects[0], outputs[0], eps, eps)
[2024-04-11T15:22:12.714Z] >       np.testing.assert_allclose(expects[1], outputs[1], eps, eps)
[2024-04-11T15:22:12.714Z] E       AssertionError: 
[2024-04-11T15:22:12.714Z] E       Not equal to tolerance rtol=0.0001, atol=0.0001
[2024-04-11T15:22:12.714Z] E       
[2024-04-11T15:22:12.714Z] E       Mismatched elements: 39 / 47 (83%)
[2024-04-11T15:22:12.714Z] E       Max absolute difference: 58.18592
[2024-04-11T15:22:12.714Z] E       Max relative difference: 0.49074903
[2024-04-11T15:22:12.714Z] E        x: array([[[[62.988182],
[2024-04-11T15:22:12.714Z] E                [63.732666],
[2024-04-11T15:22:12.714Z] E                [60.08432 ],...
[2024-04-11T15:22:12.714Z] E        y: array([[[[ 62.98819 ],
[2024-04-11T15:22:12.714Z] E                [ 63.73267 ],
[2024-04-11T15:22:12.714Z] E                [ 60.08432 ],...
[2024-04-11T15:22:12.714Z] 
[2024-04-11T15:22:12.714Z] test_dvm.py:84: AssertionError
[2024-04-11T15:22:12.714Z] =============================== warnings summary ===============================
```

GVP MindSpore/mindspore

内容风险标识

评论 (5)

GVPMindSpore/mindspore

内容风险标识