ERR00100 PTA call acl api failed

一、问题现象（附报错日志上下文）：

```
[E OpParamMaker.cpp:273] call aclnnMaskedSelect failed, detail:EI9999: Inner Error!
EI9999  Memory async copy failed, device_id=1, stream_id=4, task_id=6, flip_num=0, copy_type=0, memcpy_type=0, copy_data_type=0, length=293[FUNC:GetError][FILE:stream.cc][LINE:1455]
        TraceBack (most recent call last):
        rtStreamSynchronize execute failed, reason=[task exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
        Assert ((rtStreamSynchronize(stream)) == 0) failed
        Assert ((extInfoHandle_->UpdateOutputShapeFromExtInfo(outputs_, stream)) == OK) failed
        launch failed for MaskedSelect, errno:361001.

[ERROR] 2024-06-18-20:00:36 (PID:3037885, Device:1, RankID:-1) ERR01005 OPS internal error
Exception raised from operator() at third_party/op-plugin/op_plugin/ops/base_ops/opapi/MaskedSelectKernelNpuOpApi.cpp:49 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x68 (0xfffdbd9ed898 in /tmp/data/miniconda/envs/test/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x6c (0xfffdbd9a62a8 in /tmp/data/miniconda/envs/test/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0xa7ae70 (0xfffbd799ae70 in /tmp/data/miniconda/envs/test/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #3: <unknown function> + 0xe26ad0 (0xfffbd7d46ad0 in /tmp/data/miniconda/envs/test/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #4: <unknown function> + 0x56a1c0 (0xfffbd748a1c0 in /tmp/data/miniconda/envs/test/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: <unknown function> + 0x56a5e8 (0xfffbd748a5e8 in /tmp/data/miniconda/envs/test/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: <unknown function> + 0x5684c0 (0xfffbd74884c0 in /tmp/data/miniconda/envs/test/lib/python3.8/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: <unknown function> + 0x946ec (0xfffdbda146ec in /tmp/data/miniconda/envs/test/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x87ac (0xfffdd40287ac in /lib64/libpthread.so.0)
frame #9: <unknown function> + 0xd548c (0xfffdd3e4548c in /lib64/libc.so.6)

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/data/miniconda/envs/test/lib/python3.8/site-packages/torch/_tensor.py", line 431, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
  File "/tmp/data/miniconda/envs/test/lib/python3.8/site-packages/torch/_tensor_str.py", line 664, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
  File "/tmp/data/miniconda/envs/test/lib/python3.8/site-packages/torch/_tensor_str.py", line 595, in _str_intern
    tensor_str = _tensor_str(self, indent)
  File "/tmp/data/miniconda/envs/test/lib/python3.8/site-packages/torch/_tensor_str.py", line 347, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
  File "/tmp/data/miniconda/envs/test/lib/python3.8/site-packages/torch/_tensor_str.py", line 137, in __init__
    nonzero_finite_vals = torch.masked_select(
RuntimeError: The Inner error is reported as above.
 Since the operator is called asynchronously, the stacktrace may be inaccurate. If you want to get the accurate stacktrace, pleace set the environment variable ASCEND_LAUNCH_BLOCKING=1.
[ERROR] 2024-06-18-20:00:36 (PID:3037885, Device:1, RankID:-1) ERR00100 PTA call acl api failed
```

二、软件版本:
- CANN 版本: CANN 7.1.0.5.220:7.0.1.1
- Python 版本: 3.8.19
- torch: 2.1.0
- torch-npu: 2.1.0.post3
- 操作系统版本: CTyunOS release 22.06.2 aarch64
- npu-smi: 23.0.3, NPU 910B3 * 8

三、测试步骤：

```python
import torch
import torch_npu
a = torch.randn(2, 3, device="npu:0")
b = torch.randn(2, 3, device="npu:1")
# 第一次 to 一定没问题, 第二次 to 一定出错: ERR00100 PTA call acl api failed
b.to("npu:0") # 没问题
a.to("npu:1") # 出错
# 或者调换顺序
# a.to("npu:1") # 没问题
# b.to("npu:0") # 出错
```

四、日志信息:
cat debug/plog/plog-3045211_20240618201012127.log
```
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:11.962.116 [msprof_callback_impl.cpp:324] >>> (tid:3045211) Started to register profiling ctrl callback.
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:11.962.237 [msprof_callback_impl.cpp:331] >>> (tid:3045211) Started to register profiling hash id callback.
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:11.962.242 [prof_atls_plugin.cpp:83] >>> (tid:3045211) RegisterProfileCallback, callback type is 7
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:11.962.246 [msprof_callback_impl.cpp:338] >>> (tid:3045211) Started to register profiling enable host freq callback.
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:11.962.249 [prof_atls_plugin.cpp:83] >>> (tid:3045211) RegisterProfileCallback, callback type is 8
[EVENT] RUNTIME(3045211,python):2024-06-18-20:10:12.186.735 [runtime.cc:4400] 3045211 GetVisibleDevices: ASCEND_RT_VISIBLE_DEVICES param was not set
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:12.186.992 [prof_atls_plugin.cpp:160] >>> (tid:3045211) Module[7] register callback of ctrl handle.
[EVENT] IDEDD(3045211,python):2024-06-18-20:10:21.066.995 [adx_dump_record.cpp:220][tid:3045853]>>> start dump thread, remote dump record temp path : /root/.
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:21.067.191 [prof_atls_plugin.cpp:160] >>> (tid:3045211) Module[48] register callback of ctrl handle.
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:21.067.206 [prof_atls_plugin.cpp:160] >>> (tid:3045211) Module[45] register callback of ctrl handle.
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:21.192.415 [prof_atls_plugin.cpp:160] >>> (tid:3045211) Module[6] register callback of ctrl handle.
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:21.567.367 [msprof_callback_impl.cpp:79] >>> (tid:3045211) MsprofCtrlCallback called, type: 255
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:21.567.557 [ai_drv_dev_api.cpp:333] >>> (tid:3045211) Succeeded to DrvGetApiVersion version: 0x72313
[EVENT] RUNTIME(3045211,python):2024-06-18-20:10:21.718.952 [device.cc:341] 3045211 Init: isDoubledie:0, topologytype:0
[EVENT] RUNTIME(3045211,python):2024-06-18-20:10:21.753.619 [npu_driver.cc:5370] 3045868 GetDeviceStatus: GetDeviceStatus status=1.
[TRACE] GE(3045211,python):2024-06-18-20:10:21.763.452 [status:INIT] [ge_api.cc:206]3045211 GEInitializeImpl:GEInitialize start
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:21.864.174 [msprof_callback_impl.cpp:79] >>> (tid:3045211) MsprofCtrlCallback called, type: 255
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:21.864.185 [ai_drv_dev_api.cpp:333] >>> (tid:3045211) Succeeded to DrvGetApiVersion version: 0x72313
[TRACE] GE(3045211,python):2024-06-18-20:10:21.932.940 [status:RUNNING] [ge_api.cc:270]3045211 GEInitializeImpl:Initializing environment
[EVENT] TUNE(3045211,python):2024-06-18-20:10:22.682.748 [cann_kb_pyfunc_mgr.cpp:72][CANNKB][Tid:3045211]"CannKbPyfuncMgr: Enter PyObjectInit, reference_ is 0!"
[EVENT] TUNE(3045211,python):2024-06-18-20:10:22.682.768 [handle_manager.cpp:115][CANNKB][Tid:3045211]"Start to run init functions to load dynamic python lib!"
[EVENT] TUNE(3045211,python):2024-06-18-20:10:22.682.815 [handle_manager.cpp:407][CANNKB][Tid:3045211]"Init functions of loading dynamic python lib end!"
[EVENT] TUNE(3045211,python):2024-06-18-20:10:22.682.820 [cann_kb_pyfunc_mgr.cpp:24][CANNKB][Tid:3045211]"CANN_KB_Py has already been initialized."
[EVENT] TUNE(3045211,python):2024-06-18-20:10:24.400.680 [cann_kb_pyfunc_mgr.cpp:117][CANNKB][Tid:3045211]"CannKbPyfuncMgr: Run PyObjectInit successfully!"
[EVENT] TBE(3045211,python):2024-06-18-20:10:24.510.688 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:312][repository_manager] get_compiler_core_num core_num = [8].
[EVENT] TBE(3045211,python):2024-06-18-20:10:27.275.795 [../../../../../../latest/python/site-packages/te_fusion/parallel_compilation.py:552][init] The time cost of random buffer compile is [164817] micro second.
[EVENT] TBE(3045211,python):2024-06-18-20:10:27.307.095 [../../../../../../latest/python/site-packages/te_fusion/parallel_compilation.py:552][init] The time cost of random buffer compile is [18003] micro second.
[TRACE] GE(3045211,python):2024-06-18-20:10:29.715.290 [status:STOP] [ge_api.cc:313]3045211 GEInitializeImpl:GEInitialize finished
[EVENT] PROFILING(3045211,python):2024-06-18-20:10:29.763.854 [prof_atls_plugin.cpp:160] >>> (tid:3045211) Module[61] register callback of ctrl handle.
[EVENT] OP(3045211,python):2024-06-18-20:10:29.836.345 [op_cache.cpp:553][NNOP][GetCacheSizeLimit][3045211] OpName:[aclnnInplaceNormal_1] cachelimit is 10000
[EVENT] RUNTIME(3045211,python):2024-06-18-20:10:29.996.875 [device.cc:341] 3045211 Init: isDoubledie:0, topologytype:0
[ERROR] RUNTIME(3045211,python):2024-06-18-20:10:58.227.126 [engine.cc:3918]3046329 ProcLogicCqReport:Task run failed, device_id=1, stream_id=2, task_id=6, sqe_type=17(pcie dma), errType=0x1(task exception), sqSwStatus=0x170200
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.839.729 [stream.cc:3266]3046329 EnterFailureAbort:stream_id=2 enter failure abort.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.839.765 [stream.cc:1452]3046329 GetError:Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.839.771 [task_info.cc:2748]3046329 DoCompleteSuccessForMemcpyAsyncTask:mem async copy error, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.839.787 [task_info.cc:2706]3046329 PrintErrorInfoForMemcpyAsyncTask:Memory async copy failed, device_id=1, stream_id=2, task_id=6, flip_num=0, copy_type=0, memcpy_type=0, copy_data_type=0, length=293
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.839.792 [task_info.cc:2721]3046329 PrintErrorInfoForMemcpyAsyncTask:Memory async copy failed, device_id=1, stream_id=2, task_id=6, flip_num=0, copy_type=0, memcpy_type=0, copy_data_type=0, length=293, src_addr=0xfffc78018c80, dst_addr=0x1240c0063000.
[ERROR] IDEDD(3045211,python):2024-06-18-20:11:05.839.810 [dump_args.cpp:27][tid:3046329]>>> [Dump][Exception] args exception info is invalid.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.839.844 [engine.cc:3776]3046329 StarsResumeRtsq:stop scheduling in abort failure mode: stream_id=2, sq_id=2, sq_head=6, task_id=6, taskType=5.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.839.887 [stream.cc:3056]3046329 StarsWaitForTask:Task Wait: device_id=1, stream_id=2 is Abort, runningState=0
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.839.892 [stream.cc:1452]3046329 GetError:Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.839.895 [stream.cc:1455]3046329 GetError:report error module_type=2, module_name=EI9999
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.839.899 [stream.cc:1455]3046329 GetError:Memory async copy failed, device_id=1, stream_id=2, task_id=6, flip_num=0, copy_type=0, memcpy_type=0, copy_data_type=0, length=293
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.839.977 [logger.cc:475]3046329 StreamSynchronize:Stream synchronize failed, stream_id=2
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.840.006 [api_c.cc:810]3046329 rtStreamSynchronize:ErrCode=507001, desc=[task exception], InnerCode=0x7150057
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.840.010 [error_message_manage.cc:50]3046329 FuncErrorReason:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.840.018 [error_message_manage.cc:50]3046329 FuncErrorReason:rtStreamSynchronize execute failed, reason=[task exception]
[ERROR] OP(3045211,python):2024-06-18-20:11:05.840.050 [aicpu_ext_info_handle.cpp:351][NNOP][UpdateOutputShapeFromExtInfo][3046329] errno[361001] OpName:[aclnnMaskedSelect_42_MaskedSelect] Assert ((rtStreamSynchronize(stream)) == 0) failed
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.840.096 [stream.cc:1452]3046329 GetError:Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.840.100 [logger.cc:475]3046329 StreamSynchronize:Stream synchronize failed, stream_id=2
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.840.110 [api_c.cc:810]3046329 rtStreamSynchronize:ErrCode=507001, desc=[task exception], InnerCode=0x7150057
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.840.113 [error_message_manage.cc:50]3046329 FuncErrorReason:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.840.120 [error_message_manage.cc:50]3046329 FuncErrorReason:rtStreamSynchronize execute failed, reason=[task exception]
[ERROR] OP(3045211,python):2024-06-18-20:11:05.840.136 [aicpu_ext_info_handle.cpp:351][NNOP][UpdateOutputShapeFromExtInfo][3046329] errno[361001] OpName:[aclnnMaskedSelect_42_MaskedSelect] Assert ((rtStreamSynchronize(stream)) == 0) failed
[ERROR] OP(3045211,python):2024-06-18-20:11:05.840.149 [aicpu_task.cpp:222][NNOP][Run][3046329] errno[361001] OpName:[aclnnMaskedSelect_42_MaskedSelect] Assert ((extInfoHandle_->UpdateOutputShapeFromExtInfo(outputs_, stream)) == OK) failed
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.840.163 [stream.cc:1452]3046329 GetError:Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.840.167 [logger.cc:475]3046329 StreamSynchronize:Stream synchronize failed, stream_id=2
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.840.175 [api_c.cc:810]3046329 rtStreamSynchronize:ErrCode=507001, desc=[task exception], InnerCode=0x7150057
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.840.179 [error_message_manage.cc:50]3046329 FuncErrorReason:report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:05.840.186 [error_message_manage.cc:50]3046329 FuncErrorReason:rtStreamSynchronize execute failed, reason=[task exception]
[ERROR] OP(3045211,python):2024-06-18-20:11:05.840.201 [aicpu_ext_info_handle.cpp:351][NNOP][UpdateOutputShapeFromExtInfo][3046329] errno[361001] OpName:[aclnnMaskedSelect_42_MaskedSelect] Assert ((rtStreamSynchronize(stream)) == 0) failed
[ERROR] OP(3045211,python):2024-06-18-20:11:05.840.218 [op_executor.cpp:247][NNOP][Run][3046329] errno[361001] OpName:[aclnnMaskedSelect_42_MaskedSelect] launch failed for MaskedSelect, errno:361001.
[ERROR] APP(3045211,python):2024-06-18-20:11:05.849.117 [log_inner.cpp:76]aclAppLog call vsnprintf_s failed
[ERROR] APP(3045211,python):2024-06-18-20:11:05.849.129 [log_inner.cpp:76]3046329 OpParamMaker.cpp:ExecFunc:277: "[PTA]:"Custom hand fail! name=aclnnMaskedSelect, ret=0x0x186a0""
[ERROR] APP(3045211,python):2024-06-18-20:11:05.852.130 [log_inner.cpp:76]3046329 NPUQueue.cpp:ReadQueue:271: "[PTA]:"---Thread---281469337137632: device = 1, write_idx = 6, read_idx = 5, status = 1, ret = 100000""
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.601.507 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.601.522 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.601.529 [context.cc:309]3045211 Synchronize:[INIT][DEFAULT]sync stream fail, stream_id=2, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.601.650 [context.cc:315]3045211 Synchronize:[INIT][DEFAULT]Synchronize streams, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.601.657 [logger.cc:877]3045211 DeviceSynchronize:[INIT][DEFAULT]Device synchronize failed.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.601.678 [api_c.cc:1912]3045211 rtDeviceSynchronize:[INIT][DEFAULT]ErrCode=507001, desc=[task exception], InnerCode=0x7150057
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.601.683 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.601.693 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]rtDeviceSynchronize execute failed, reason=[task exception]
[ERROR] ASCENDCL(3045211,python):2024-06-18-20:11:08.601.733 [device.cpp:284]3045211 aclrtSynchronizeDevice: [INIT][DEFAULT]wait for compute device to finish failed, runtime result = 507001.
[ERROR] APP(3045211,python):2024-06-18-20:11:08.609.660 [log_inner.cpp:76]3045211 InitNpuBindings.cpp:THPModule_npu_shutdown:59: "[PTA]:"NPU shutdown synchronize device failed.""
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.609.687 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.609.693 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.609.699 [context.cc:309]3045211 Synchronize:[INIT][DEFAULT]sync stream fail, stream_id=2, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.609.725 [context.cc:315]3045211 Synchronize:[INIT][DEFAULT]Synchronize streams, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.609.730 [logger.cc:877]3045211 DeviceSynchronize:[INIT][DEFAULT]Device synchronize failed.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.609.742 [api_c.cc:1912]3045211 rtDeviceSynchronize:[INIT][DEFAULT]ErrCode=507001, desc=[task exception], InnerCode=0x7150057
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.609.748 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.609.757 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]rtDeviceSynchronize execute failed, reason=[task exception]
[ERROR] ASCENDCL(3045211,python):2024-06-18-20:11:08.609.779 [device.cpp:284]3045211 aclrtSynchronizeDevice: [INIT][DEFAULT]wait for compute device to finish failed, runtime result = 507001.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.614.530 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.614.539 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.614.544 [context.cc:309]3045211 Synchronize:[INIT][DEFAULT]sync stream fail, stream_id=2, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.614.551 [context.cc:315]3045211 Synchronize:[INIT][DEFAULT]Synchronize streams, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.614.556 [logger.cc:877]3045211 DeviceSynchronize:[INIT][DEFAULT]Device synchronize failed.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.614.566 [api_c.cc:1912]3045211 rtDeviceSynchronize:[INIT][DEFAULT]ErrCode=507001, desc=[task exception], InnerCode=0x7150057
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.614.571 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.614.580 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]rtDeviceSynchronize execute failed, reason=[task exception]
[ERROR] ASCENDCL(3045211,python):2024-06-18-20:11:08.614.601 [device.cpp:284]3045211 aclrtSynchronizeDevice: [INIT][DEFAULT]wait for compute device to finish failed, runtime result = 507001.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.615.473 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.615.481 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.615.486 [context.cc:309]3045211 Synchronize:[INIT][DEFAULT]sync stream fail, stream_id=2, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.615.492 [context.cc:315]3045211 Synchronize:[INIT][DEFAULT]Synchronize streams, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.615.497 [logger.cc:877]3045211 DeviceSynchronize:[INIT][DEFAULT]Device synchronize failed.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.615.508 [api_c.cc:1912]3045211 rtDeviceSynchronize:[INIT][DEFAULT]ErrCode=507001, desc=[task exception], InnerCode=0x7150057
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.615.513 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.615.522 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]rtDeviceSynchronize execute failed, reason=[task exception]
[ERROR] ASCENDCL(3045211,python):2024-06-18-20:11:08.615.542 [device.cpp:284]3045211 aclrtSynchronizeDevice: [INIT][DEFAULT]wait for compute device to finish failed, runtime result = 507001.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.616.279 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.616.287 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.616.292 [context.cc:309]3045211 Synchronize:[INIT][DEFAULT]sync stream fail, stream_id=2, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.616.299 [context.cc:315]3045211 Synchronize:[INIT][DEFAULT]Synchronize streams, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.616.303 [logger.cc:877]3045211 DeviceSynchronize:[INIT][DEFAULT]Device synchronize failed.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.616.314 [api_c.cc:1912]3045211 rtDeviceSynchronize:[INIT][DEFAULT]ErrCode=507001, desc=[task exception], InnerCode=0x7150057
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.616.319 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.616.328 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]rtDeviceSynchronize execute failed, reason=[task exception]
[ERROR] ASCENDCL(3045211,python):2024-06-18-20:11:08.616.348 [device.cpp:284]3045211 aclrtSynchronizeDevice: [INIT][DEFAULT]wait for compute device to finish failed, runtime result = 507001.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.082 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.091 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.096 [context.cc:309]3045211 Synchronize:[INIT][DEFAULT]sync stream fail, stream_id=2, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.102 [context.cc:315]3045211 Synchronize:[INIT][DEFAULT]Synchronize streams, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.107 [logger.cc:877]3045211 DeviceSynchronize:[INIT][DEFAULT]Device synchronize failed.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.118 [api_c.cc:1912]3045211 rtDeviceSynchronize:[INIT][DEFAULT]ErrCode=507001, desc=[task exception], InnerCode=0x7150057
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.122 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.131 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]rtDeviceSynchronize execute failed, reason=[task exception]
[ERROR] ASCENDCL(3045211,python):2024-06-18-20:11:08.617.151 [device.cpp:284]3045211 aclrtSynchronizeDevice: [INIT][DEFAULT]wait for compute device to finish failed, runtime result = 507001.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.877 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.885 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.890 [context.cc:309]3045211 Synchronize:[INIT][DEFAULT]sync stream fail, stream_id=2, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.896 [context.cc:315]3045211 Synchronize:[INIT][DEFAULT]Synchronize streams, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.901 [logger.cc:877]3045211 DeviceSynchronize:[INIT][DEFAULT]Device synchronize failed.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.912 [api_c.cc:1912]3045211 rtDeviceSynchronize:[INIT][DEFAULT]ErrCode=507001, desc=[task exception], InnerCode=0x7150057
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.917 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.617.925 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]rtDeviceSynchronize execute failed, reason=[task exception]
[ERROR] ASCENDCL(3045211,python):2024-06-18-20:11:08.617.945 [device.cpp:284]3045211 aclrtSynchronizeDevice: [INIT][DEFAULT]wait for compute device to finish failed, runtime result = 507001.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.618.681 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.618.689 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.618.694 [context.cc:309]3045211 Synchronize:[INIT][DEFAULT]sync stream fail, stream_id=2, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.618.700 [context.cc:315]3045211 Synchronize:[INIT][DEFAULT]Synchronize streams, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.618.705 [logger.cc:877]3045211 DeviceSynchronize:[INIT][DEFAULT]Device synchronize failed.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.618.715 [api_c.cc:1912]3045211 rtDeviceSynchronize:[INIT][DEFAULT]ErrCode=507001, desc=[task exception], InnerCode=0x7150057
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.618.720 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.618.728 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]rtDeviceSynchronize execute failed, reason=[task exception]
[ERROR] ASCENDCL(3045211,python):2024-06-18-20:11:08.618.758 [device.cpp:284]3045211 aclrtSynchronizeDevice: [INIT][DEFAULT]wait for compute device to finish failed, runtime result = 507001.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.619.481 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.619.489 [stream.cc:1452]3045211 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x20b, [memcpy exception].
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.619.494 [context.cc:309]3045211 Synchronize:[INIT][DEFAULT]sync stream fail, stream_id=2, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.619.501 [context.cc:315]3045211 Synchronize:[INIT][DEFAULT]Synchronize streams, retCode=0x7150057.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.619.505 [logger.cc:877]3045211 DeviceSynchronize:[INIT][DEFAULT]Device synchronize failed.
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.619.516 [api_c.cc:1912]3045211 rtDeviceSynchronize:[INIT][DEFAULT]ErrCode=507001, desc=[task exception], InnerCode=0x7150057
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.619.521 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(3045211,python):2024-06-18-20:11:08.619.529 [error_message_manage.cc:50]3045211 FuncErrorReason:[INIT][DEFAULT]rtDeviceSynchronize execute failed, reason=[task exception]
[ERROR] ASCENDCL(3045211,python):2024-06-18-20:11:08.619.549 [device.cpp:284]3045211 aclrtSynchronizeDevice: [INIT][DEFAULT]wait for compute device to finish failed, runtime result = 507001.
[EVENT] PROFILING(3045211,python):2024-06-18-20:11:09.582.614 [msprof_callback_impl.cpp:79] >>> (tid:3045211) MsprofCtrlCallback called, type: 3
[EVENT] PROFILING(3045211,python):2024-06-18-20:11:09.582.728 [ai_drv_dev_api.cpp:333] >>> (tid:3045211) Succeeded to DrvGetApiVersion version: 0x72313
[TRACE] GE(3045211,python):2024-06-18-20:11:09.582.844 [status:INIT] [ge_api.cc:362]3045211 GEFinalize:GEFinalize start
[EVENT] PROFILING(3045211,python):2024-06-18-20:11:09.582.851 [msprof_callback_impl.cpp:79] >>> (tid:3045211) MsprofCtrlCallback called, type: 3
[EVENT] PROFILING(3045211,python):2024-06-18-20:11:09.582.854 [ai_drv_dev_api.cpp:333] >>> (tid:3045211) Succeeded to DrvGetApiVersion version: 0x72313
[TRACE] GE(3045211,python):2024-06-18-20:11:09.582.990 [status:RUNNING] [ge_api.cc:374]3045211 GEFinalize:Finalizing environment
[EVENT] TUNE(3045211,python):2024-06-18-20:11:10.168.152 [cann_kb_pyfunc_mgr.cpp:127][CANNKB][Tid:3045211]"CannKbPyfuncMgr: enter PyObjectDeinit function, reference_[1]"
[EVENT] TUNE(3045211,python):2024-06-18-20:11:10.168.168 [cann_kb_pyfunc_mgr.cpp:138][CANNKB][Tid:3045211]"CannKbPyfuncMgr: PyObjectDeinit function end successfully!"
[TRACE] GE(3045211,python):2024-06-18-20:11:10.170.072 [status:STOP] [ge_api.cc:402]3045211 GEFinalize:GEFinalize finished
[EVENT] RUNTIME(3045211,python):2024-06-18-20:11:11.846.038 [runtime.cc:1706] 3045211 ~Runtime: deconstruct runtime.
[EVENT] DRV(3045211,python):2024-06-18-20:11:11.974.671 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 150]DEV_MEM dev0 mem stats(Bytes): module_name=RUNTIME, module_id=7 current_alloced_size=0, alloced_peak_size=35176448.
[EVENT] DRV(3045211,python):2024-06-18-20:11:11.974.685 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 150]DEV_MEM dev0 mem stats(Bytes): module_name=APP, module_id=33 current_alloced_size=2097152, alloced_peak_size=2097152.
[EVENT] DRV(3045211,python):2024-06-18-20:11:11.974.690 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 150]DEV_MEM dev0 mem stats(Bytes): module_name=AICPU, module_id=36 current_alloced_size=0, alloced_peak_size=4096.
[EVENT] DRV(3045211,python):2024-06-18-20:11:11.974.696 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 156]DEV_MEM dev0 Cached_size:37748736Bytes
[EVENT] DRV(3045211,python):2024-06-18-20:11:11.974.700 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 150]DEV_MEM dev1 mem stats(Bytes): module_name=RUNTIME, module_id=7 current_alloced_size=30138368, alloced_peak_size=34340864.
[EVENT] DRV(3045211,python):2024-06-18-20:11:11.974.704 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 150]DEV_MEM dev1 mem stats(Bytes): module_name=APP, module_id=33 current_alloced_size=2097152, alloced_peak_size=2097152.
[EVENT] DRV(3045211,python):2024-06-18-20:11:11.974.708 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 156]DEV_MEM dev1 Cached_size:7610368Bytes
[EVENT] DRV(3045211,python):2024-06-18-20:11:11.974.761 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 150]HOST_MEM mem stats(Bytes): module_name=RUNTIME, module_id=7 current_alloced_size=11341824, alloced_peak_size=22683648.
[EVENT] DRV(3045211,python):2024-06-18-20:11:11.974.765 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 156]HOST_MEM Cached_size:13824000Bytes
[EVENT] DRV(3045211,python):2024-06-18-20:11:12.179.819 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 150]DEV_MEM dev0 mem stats(Bytes): module_name=RUNTIME, module_id=7 current_alloced_size=0, alloced_peak_size=35176448.
[EVENT] DRV(3045211,python):2024-06-18-20:11:12.179.833 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 150]DEV_MEM dev0 mem stats(Bytes): module_name=APP, module_id=33 current_alloced_size=0, alloced_peak_size=2097152.
[EVENT] DRV(3045211,python):2024-06-18-20:11:12.179.838 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 150]DEV_MEM dev0 mem stats(Bytes): module_name=AICPU, module_id=36 current_alloced_size=0, alloced_peak_size=4096.
[EVENT] DRV(3045211,python):2024-06-18-20:11:12.179.843 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 156]DEV_MEM dev0 Cached_size:0Bytes
[EVENT] DRV(3045211,python):2024-06-18-20:11:12.179.846 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 150]DEV_MEM dev1 mem stats(Bytes): module_name=RUNTIME, module_id=7 current_alloced_size=0, alloced_peak_size=34340864.
[EVENT] DRV(3045211,python):2024-06-18-20:11:12.179.850 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 150]DEV_MEM dev1 mem stats(Bytes): module_name=APP, module_id=33 current_alloced_size=2097152, alloced_peak_size=2097152.
[EVENT] DRV(3045211,python):2024-06-18-20:11:12.179.854 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 156]DEV_MEM dev1 Cached_size:37748736Bytes
[EVENT] DRV(3045211,python):2024-06-18-20:11:12.179.888 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 150]HOST_MEM mem stats(Bytes): module_name=RUNTIME, module_id=7 current_alloced_size=0, alloced_peak_size=22683648.
[EVENT] DRV(3045211,python):2024-06-18-20:11:12.179.892 [ascend][curpid: 3045211, 3045211][drv][devmm][_svm_mem_stats_show 156]HOST_MEM Cached_size:25165824Bytes
```

Ascend/pytorch
暂停

内容风险标识

评论 (4)

Ascend/pytorch暂停 .gitee-modal { width: 500px !important; }

内容风险标识