[ST][MS][大集群专项]pangu_sigma网络单卡模拟编译kbk流程，第一次编译正常，第二次编译失败

| name       | about	                             | labels   |
| ---------- | ------------------------------------- | -------- |
| Bug Report | Use this template for reporting a bug | kind/bug |

### Describe the current behavior / 问题描述 (Mandatory / 必填)
pangu_sigma网络单卡模拟编译kbk流程，第一次编译正常，第二次编译失败
 
### Environment / 环境信息 (Mandatory / 必填)
- **Hardware Environment(`Ascend`/`GPU`/`CPU`)  / 硬件环境**: 
> Please delete the backend not involved / 请删除不涉及的后端:
> /device ascend

- **Software Environment / 软件环境 (Mandatory / 必填)**:
-- MindSpore version (e.g., 1.7.0.Bxxx) :
-- Python version (e.g., Python 3.7.5) :
-- OS platform and distribution (e.g., Linux Ubuntu 16.04):
-- GCC/Compiler version (if compiled from source): 
master_20240515061516_049592165b92

- **Excute Mode / 执行模式 (Mandatory / 必填)(`PyNative`/`Graph`)**: 
> Please delete the mode not involved / 请删除不涉及的模式:
> /mode pynative
> /mode graph

### Related testcase / 关联用例 (Mandatory / 必填)
pangu_sigma_230b_ascend910b_4096_4096_FA_fp16_065/pangu_sigma_230b_ascend910b_4096_4096_0204_001.py
 
### Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)
1. get code from pangu
2. 设置参数：
    RANK_ID: 204
    RANK_SIZE: 4096
    apply_fine_micro_interleaved: false
    attention_type: FA
    compile_cache: true
    data_parallel: 16
    device_id: 5
    fused_qkv: false
    logits_comp_type: fp16
    micro_interleaved_size: 2
    micro_size: 160
    mode: 230B
    model_parallel: 8
    num_layers: 64
    optimizer_shard: 1
    param_init_type: fp16
    per_batch_size: 2
    pipeline_parallel: 32
    select_recompute: -1
    sequence_parallel_num: 1
    sequence_partition: true
    设置ENABLE_CELL_REUSE=1，开启kbk：
    from mindspore import JitConfig
    jitconfig = JitConfig(jit_level="O1")
3. 开始网络训练
4. 验证网络第二次编译是否正常

### Describe the expected behavior / 预期结果 (Mandatory / 必填)
第二次编译正常
 
### Related log / screenshot / 日志 / 截图 (Mandatory / 必填)
```
[ERROR] DRV(3546076,python):2024-05-23-17:04:05.791.422 [ascend][curpid: 3546076, 3546076][drv][devmm][drvMemGetAttribute 1562]<errno:0, 0> Addr isn't alloced. (va=0x12bb40dc9600)
[ERROR] RUNTIME(3546076,python):2024-05-23-17:04:05.791.477 [npu_driver.cc:2088]3546076 PointerGetAttributes:report error module_type=1, module_name=EL9999
[ERROR] RUNTIME(3546076,python):2024-05-23-17:04:05.791.488 [npu_driver.cc:2088]3546076 PointerGetAttributes:[drv api] drvMemGetAttribute failed: drvRetCode=11!
[ERROR] RUNTIME(3546076,python):2024-05-23-17:04:05.791.706 [logger.cc:681]3546076 PointerGetAttributes:Get pointer attributes failed
[ERROR] RUNTIME(3546076,python):2024-05-23-17:04:05.791.719 [api_error.cc:1048]3546076 MemcpyAsync:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3546076,python):2024-05-23-17:04:05.791.727 [api_error.cc:1048]3546076 MemcpyAsync:Memory async failed, get dst pointer attributes failed, retCode=0x7020010
[ERROR] RUNTIME(3546076,python):2024-05-23-17:04:05.791.791 [api_c.cc:1320]3546076 rtMemcpyAsync:ErrCode=507899, desc=[driver error:internal error], InnerCode=0x7020010
[ERROR] RUNTIME(3546076,python):2024-05-23-17:04:05.791.802 [error_message_manage.cc:53]3546076 FuncErrorReason:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3546076,python):2024-05-23-17:04:05.791.816 [error_message_manage.cc:53]3546076 FuncErrorReason:rtMemcpyAsync execute failed, reason=[driver error:internal error]
[ERROR] ASCENDCL(3546076,python):2024-05-23-17:04:05.791.853 [memory.cpp:336]3546076 aclrtMemcpyAsync: asynchronized memcpy failed, kind = 3, runtime result = 507899
[ERROR] DEVICE(3546076,ffff99944020,python):2024-05-23-17:04:05.791.905 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_kernel_runtime.cc:624] MemcpyAsync] Call runtime rtMemcpyAsync error.
[ERROR] DEVICE(3546076,ffff99944020,python):2024-05-23-17:04:05.792.198 [mindspore/ccsrc/plugin/device/ascend/hal/device/ascend_device_address.cc:763] AsyncDeviceToDevice] MemcpyAsync failed!
[CRITICAL] RUNTIME_FRAMEWORK(3546076,ffff99944020,python):2024-05-23-17:04:05.795.126 [mindspore/ccsrc/runtime/graph_scheduler/actor/output_actor.cc:425] UpdateOutputDeviceAddress] Sync device to device failed, device type: 2, output node: Default/data-4192
EL9999: Inner Error!
EL9999: 2024-05-23-17:04:05.791.592  [drv api] drvMemGetAttribute failed: drvRetCode=11![FUNC:PointerGetAttributes][FILE:npu_driver.cc][LINE:2088]
        TraceBack (most recent call last):
        Memory async failed, get dst pointer attributes failed, retCode=0x7020010[FUNC:MemcpyAsync][FILE:api_error.cc][LINE:1048]
        rtMemcpyAsync execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
        asynchronized memcpy failed, kind = 3, runtime result = 507899[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)
Enable overflow is: True
------ train start -------
================actual_epoch_num is: 2
Exception in training: Sync device to device failed, device type: 2, output node: Default/data-4192

----------------------------------------------------
- Ascend Error Message:
----------------------------------------------------
EL9999: Inner Error!
EL9999: 2024-05-23-17:04:05.791.592  [drv api] drvMemGetAttribute failed: drvRetCode=11![FUNC:PointerGetAttributes][FILE:npu_driver.cc][LINE:2088]
        TraceBack (most recent call last):
        Memory async failed, get dst pointer attributes failed, retCode=0x7020010[FUNC:MemcpyAsync][FILE:api_error.cc][LINE:1048]
        rtMemcpyAsync execute failed, reason=[driver error:internal error][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
        asynchronized memcpy failed, kind = 3, runtime result = 507899[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

(Please search "CANN Common Error Analysis" at https://www.mindspore.cn for error code description)

----------------------------------------------------
- C++ Call Stack: (For framework developers)
----------------------------------------------------
mindspore/ccsrc/runtime/graph_scheduler/actor/output_actor.cc:425 UpdateOutputDeviceAddress
```
 
### Special notes for this issue/备注 (Optional / 选填)
走给施梁

GVP MindSpore/mindspore
关闭

内容风险标识

评论 (2)

GVPMindSpore/mindspore关闭

内容风险标识