Describe the current behavior

GPU A100 CUDA:11.6 算子测试运行效率低，测试了ops.add()、ops.BatchMatMul()运行15-18min才出结果
同步对比测试用例在V100：CUDA 11.4和NPU 910 在10-20s出结果。
相同机器和镜像环境，安装pytorch进行算子测试，均1s内出结果

Environment

• GPU A100
  nvcc: NVIDIA (R) Cuda compiler driver
  Copyright (c) 2005-2022 NVIDIA Corporation
  Built on Tue_Mar__8_18:18:20_PST_2022
  Cuda compilation tools, release 11.6, V11.6.124
  Build cuda_11.6.r11.6/compiler.31057947_0

• Software Environment:
  -- MindSpore 1.6.1
  -- Python 3.7.5
  -- Ubuntu 20.04.4 LTS
  -- gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

• Excute Mode:

docker 环境下

Related testcase / 关联用例 (Mandatory / 必填)

from mindspore import Tensor
import numpy as np
import mindspore
import mindspore.ops as ops
x = Tensor(np.ones(shape=[2, 4, 1, 3]), mindspore.float32)
y = Tensor(np.ones(shape=[2, 4, 3, 4]), mindspore.float32)
batmatmul = ops.BatchMatMul()
print(batmatmul(x, y))

或者

import numpy as np
from mindspore import Tensor
import mindspore.ops as ops
import mindspore.context as context

context.set_context(device_target="GPU")
x = Tensor(np.ones([1,3,3,4]).astype(np.float32))
y = Tensor(np.ones([1,3,3,4]).astype(np.float32))
print(ops.add(x, y))

两个测试用例运行时长均在15-18分钟左右，可看到显存分配占用过程。
同步对比测试用例在V100：CUDA 11.4和NPU 910 在10-20s出结果。

Steps to reproduce the issue

1. 登陆A100机器
2. pull docker镜像:docker pull hengtao2/mindspore:gpu_ms_1.6.1
3. 启动镜像：docker run -it --runtime=nvidia --cap-add=IPC_LOCK --device=/dev/infiniband/uverbs0 --device=/dev/infinibandev/infiniband/rdma_cm --privileged --net=host -v /home//:/tmp Images_ID:mindspore:gpu_ms_1.6.1 /bin/bash
4. 运行测试用例：如上两例

Describe the expected behavior

算子运行效率在秒级

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

算子运行测试截图

Special notes for this issue

GPU A100运行paugu模型出现如果所示报错，定位到测试相关算子，发现算子运行效率低