Baichuan2-7B for PyTorch的训练报错 RuntimeError: ACL stream synchronize failed, error code:507014 fault kernel_name=55/47_-1_55_MatMul_tvmbin

一、问题现象（附报错日志上下文）：
训练报错了

AllReduce error, see details in Ascend logs.AllReduce error, see details in Ascend logs.AllReduce error, see details in Ascend logs.AllReduce error, see details in Ascend logs.AllReduce error, see details in Ascend logs.AllReduce error, see details in Ascend logs.EZ9999: Inner Error!
EZ9999  Kernel task happen error, retCode=0x25, [aicore timeout].[FUNC:PreCheckTaskErr][FILE:task_info.cc][LINE:1574]
        TraceBack (most recent call last):
        The error from device(0), serial number is 1, there is an aicore error, core id is 0, error code = 0, dump info: pc start: 0x10001241401ba000, current: 0x1241401ba2e8, vec error info: 0xdfffd7e, mte error info: 0, ifu error info: 0x1bf6eb496f780, ccu error info: 0, cube error info: 0xe6, biu error info: 0, aic error mask: 0x65000200d000288, para base: 0x1241804b3400, errorStr: time out or trap error.[FUNC:PrintCoreErrorInfo][FILE:device_error_proc.cc][LINE:533]
        The extend info from device(0), serial number is 1, there is aicore error, core id is 0, aicore int: 0x11, aicore error2: 0, axi clamp ctrl: 0, axi clamp state: 0x1717, biu status0: 0x101e44800000000, biu status1: 0x940002092a0000, clk gate mask: 0x1000, dbg addr: 0, ecc en: 0, mte ccu ecc 1bit error: 0, vector cube ecc 1bit error: 0, run stall: 0x1, dbg data0: 0, dbg data1: 0, dbg data2: 0, dbg data3: 0, dfx data: 0[FUNC:PrintCoreErrorInfo][FILE:device_error_proc.cc][LINE:564]
        The device(0), core list[0-0], error code is:[FUNC:PrintCoreInfoErrMsg][FILE:device_error_proc.cc][LINE:587]
        coreId( 0):               0    [FUNC:PrintCoreInfoErrMsg][FILE:device_error_proc.cc][LINE:601]
        Aicore kernel execute failed, device_id=2, stream_id=10, report_stream_id=10, task_id=4813, flip_num=0, fault kernel_name=55/47_-1_55_MatMul_tvmbin, program id=48, hash=10615614166334013548.[FUNC:GetError][FILE:stream.cc][LINE:1473]        [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1473]
        rtStreamSynchronizeWithTimeout execute failed, reason=[aicore timeout][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
        synchronize stream failed, runtime result = 507014[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

DEVICE[2] PID[2224108]:
EXCEPTION STREAM:
  Exception info:TGID=2224108, model id=65535, stream id=10, stream phase=3
  Message info[0]:read SOC_AIC_SC_REG_RUN_STALL_REG=0x0
    Other info[0]:time=2023-09-28-02:17:57.991.269, function=process_aic_task_kill, line=1746, error code=0x1
EXCEPTION TGID:
  Exception info:TGID=2224108
  Message info[0]:SOC_AIC_SC_REG_RUN_STALL_REG=0x1
    Other info[0]:time=2023-09-28-02:17:57.991.306, function=process_aic_task_kill, line=1779, error code=0x1
EXCEPTION STREAM:
  Exception info:TGID=2224108, model id=65535, stream id=10, stream phase=3
  Message info[0]:RTS_HWTS: aicore timeout, slot_id=0, stream_id=10, task_id=4813
    Other info[0]:time=2023-09-28-02:17:58.010.648, function=hwts_timeout_recored_err_msg, line=1894, error code=0x25
EXCEPTION STREAM:
  Exception info:TGID=2224108, model id=65535, stream id=10, stream phase=3
  Message info[0]:RTS_HWTS: slot_id=0, stream_id=10, task_id=4813, result=37
    Other info[0]:time=2023-09-28-02:17:58.010.670, function=send_timeout_cq_msg_with_result, line=1383, error code=0x25Traceback (most recent call last):
  File "train_npu.py", line 38, in <module>
    train()
  File "/media/sda/shengteng_test/ModelZoo-PyTorch/PyTorch/built-in/foundation/Baichuan2/7B/train.py", line 128, in train
    trainer.train()
  File "/root/miniconda3/envs/foundation_torch_1.11.0/lib/python3.7/site-packages/transformers/trainer.py", line 1667, in train
    ignore_keys_for_eval=ignore_keys_for_eval,
  File "/root/miniconda3/envs/foundation_torch_1.11.0/lib/python3.7/site-packages/transformers/trainer.py", line 1931, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/root/miniconda3/envs/foundation_torch_1.11.0/lib/python3.7/site-packages/transformers/trainer.py", line 2720, in training_step
    loss = self.deepspeed.backward(loss)
  File "/root/miniconda3/envs/foundation_torch_1.11.0/lib/python3.7/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/root/miniconda3/envs/foundation_torch_1.11.0/lib/python3.7/site-packages/deepspeed/runtime/engine.py", line 1845, in backward
    self.optimizer.backward(loss, retain_graph=retain_graph)
  File "/root/miniconda3/envs/foundation_torch_1.11.0/lib/python3.7/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1901, in backward
    self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)
  File "/media/sda/datastore/big_model/DeepSpeed/deepspeed_npu/adaptor_runtime_fp16_loss_scaler.py", line 10, in backward
    clear_npu_overflow_flag()
  File "/root/miniconda3/envs/foundation_torch_1.11.0/lib/python3.7/site-packages/torch_npu/npu/utils.py", line 429, in clear_npu_overflow_flag
    float_status = torch.zeros(8).npu()
  File "/root/miniconda3/envs/foundation_torch_1.11.0/lib/python3.7/site-packages/torch_npu/utils/device_guard.py", line 45, in wrapper
    return func(*args, **kwargs)
  File "/root/miniconda3/envs/foundation_torch_1.11.0/lib/python3.7/site-packages/torch_npu/utils/tensor_methods.py", line 74, in _npu
    return torch_npu._C.npu(self, *args, **kwargs)
RuntimeError: ACL stream synchronize failed, error code:507014
[W NPUStream.cpp:392] Warning: NPU warning, error code is 507014[Error]:
[Error]: The aicore execution times out.
        Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore timeout][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
EH9999  wait for compute device to finish failed, runtime result = 507014.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)
AllReduce error, see details in Ascend logs.AllReduce error, see details in Ascend logs.EZ9999: Inner Error!

RuntimeError: ACL stream synchronize failed, error code:507014
在最后的reduce阶段报错了 超时了
[Error]: The aicore execution times out.
        Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore timeout][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
EH9999  wait for compute device to finish failed, runtime result = 507014.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)

环境是按照官方配置的

二、软件版本:
-- driver版本
driver的版本 Version=23.0.rc2
-- CANN 版本 (e.g., CANN 6.3.RC2.alpha005):
CANN 7.0.RC1.alpha003
--Tensorflow/Pytorch/MindSpore 版本:
Pytorch
--Python 版本 (e.g., Python 3.7.5):
3.7.11
-- Pytorch版本 (e.g., 1.8.1 ):
1.11.0
-- Pytorch_torch版本 :
torch_npu-1.11.0.post1-cp37-cp37m-linux_aarch64.whl
--操作系统版本 (e.g., Ubuntu 18.04):
Ubuntu 18.04

三、测试步骤：
https://gitee.com/ascend/ModelZoo-PyTorch/tree/master/PyTorch/built-in/foundation/Baichuan2/7B#%E8%AE%AD%E7%BB%83%E6%A8%A1%E5%9E%8B

按照官方文档操作
但是机器的是8个910B，每个卡32GB 不是64GB版本 8卡训练报显存不足的问题，调整参数使用deepspeed的offload模式
修改的配置如下：

能跑起来，但是运行一段时间之后报错

三、测试步骤：
https://gitee.com/ascend/ModelZoo-PyTorch/tree/master/PyTorch/built-in/foundation/Baichuan2/7B#%E8%AE%AD%E7%BB%83%E6%A8%A1%E5%9E%8B

四、日志信息:
~/ascend/log/debug/plog# 路径下的报错日志

[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.751.915 [task_info.cc:1574]2224109 PreCheckTaskEr
r:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.751.992 [task_info.cc:1574]2224109 PreCheckTaskEr
r:[INIT][DEFAULT]Kernel task happen error, retCode=0x25, [aicore timeout].
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.752.438 [task_info.cc:1536]2224109 PrintErrorInfo
ForDavinciTask:[INIT][DEFAULT]Aicore kernel execute failed, device_id=3, stream_id=10, report_stream_i
d=10, task_id=4813, flip_num=0, fault kernel_name=55/47_-1_55_MatMul_tvmbin, program id=48, hash=10615
614166334013548.
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.752.596 [task_info.cc:1479]2224109 GetArgsInfo:[I
NIT][DEFAULT][AIC_INFO] args(0 to 4) after execute:0x1248f4800200, 0x12485b100000, 0x1248d5d00000, 0x1
24140010000,
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.752.639 [task_info.cc:1540]2224109 PrintErrorInfo
ForDavinciTask:[INIT][DEFAULT][AIC_INFO] after execute:args print end
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.753.918 [engine.cc:1570]2224109 ReportExceptProc:
[INIT][DEFAULT]Task exception! device_id=3, stream_id=10, task_id=4813, type=0(KERNEL_AICORE), failure
mode =1, retCode=0x25, [aicore timeout]
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.754.493 [device_error_proc.cc:533]2224109 PrintCo
reErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.754.515 [device_error_proc.cc:533]2224109 PrintCo
reErrorInfo:[INIT][DEFAULT]The error from device(0), serial number is 1, there is an aicore error, cor
e id is 0, error code = 0, dump info: pc start: 0x10001241401ba000, current: 0x1241401ba318, vec error
 info: 0x1f4fcbff, mte error info: 0, ifu error info: 0x3377eb9d3ef00, ccu error info: 0, cube error i
nfo: 0x77, biu error info: 0, aic error mask: 0x65000200d000288, para base: 0x1241804b3400, errorStr:
time out or trap error.
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.754.596 [device_error_proc.cc:564]2224109 PrintCo
reErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.754.613 [device_error_proc.cc:564]2224109 PrintCo
reErrorInfo:[INIT][DEFAULT]The extend info from device(0), serial number is 1, there is aicore error,
core id is 0, aicore int: 0x11, aicore error2: 0, axi clamp ctrl: 0, axi clamp state: 0x1717, biu stat
us0: 0x101e44800000000, biu status1: 0x940002092a0000, clk gate mask: 0x1000, dbg addr: 0, ecc en: 0,
mte ccu ecc 1bit error: 0, vector cube ecc 1bit error: 0, run stall: 0x1, dbg data0: 0, dbg data1: 0,
dbg data2: 0, dbg data3: 0, dfx data: 0
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.754.746 [stream.cc:3106]2224109 EnterFailureAbort
:[INIT][DEFAULT]stream_id=10 enter failure abort.
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.754.767 [stream.cc:1470]2224109 GetError:[INIT][D
EFAULT]Stream Synchronize failed, stream_id=10, retCode=0x25, [aicore timeout].
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.754.784 [stream.cc:1473]2224109 GetError:[INIT][D
EFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.754.799 [stream.cc:1473]2224109 GetError:[INIT][D
EFAULT]Aicore kernel execute failed, device_id=3, stream_id=10, report_stream_id=10, task_id=4813, fli
p_num=0, fault kernel_name=55/47_-1_55_MatMul_tvmbin, program id=48, hash=10615614166334013548.
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.754.829 [stream.cc:1473]2224109 GetError:[INIT][D
EFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:17:58.754.845 [stream.cc:1473]2224109 GetError:[INIT][D
EFAULT][AIC_INFO] after execute:args print end
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.871.867 [engine.cc:1926]2224109 SyncTask:[INIT][D
EFAULT]Task Wait:stream_id=10 is ABORT.
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.871.923 [stream.cc:1470]2224109 GetError:[INIT][D
EFAULT]Stream Synchronize failed, stream_id=10, retCode=0x25, [aicore timeout].
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.871.952 [logger.cc:472]2224109 StreamSynchronize:
[INIT][DEFAULT]Stream synchronize failed, stream_id=10
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.872.040 [api_c.cc:808]2224109 rtStreamSynchronize
WithTimeout:[INIT][DEFAULT]ErrCode=507014, desc=[aicore timeout], InnerCode=0x7150025
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.872.058 [error_message_manage.cc:50]2224109 FuncE
rrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.872.089 [error_message_manage.cc:50]2224109 FuncE
rrorReason:[INIT][DEFAULT]rtStreamSynchronizeWithTimeout execute failed, reason=[aicore timeout]
[ERROR] ASCENDCL(2224109,python3):2023-09-28-02:18:03.872.224 [stream.cpp:143]2224109 aclrtSynchronize
StreamWithTimeout: [INIT][DEFAULT]synchronize stream failed, runtime result = 507014
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.874.930 [device_msg_handler.cc:156]2224109 Handle
MsgInHostBuf:[INIT][DEFAULT]
DEVICE[3] PID[2224109]:
EXCEPTION STREAM:
  Exception info:TGID=2224109, model id=65535, stream id=10, stream phase=3
  Message info[0]:read SOC_AIC_SC_REG_RUN_STALL_REG=0x0
    Other info[0]:time=2023-09-28-02:17:58.313.259, function=process_aic_task_kill, line=1746, error c
ode=0x1
EXCEPTION TGID:
  Exception info:TGID=2224109
  Message info[0]:SOC_AIC_SC_REG_RUN_STALL_REG=0x1
    Other info[0]:time=2023-09-28-02:17:58.313.294, function=process_aic_task_kill, line=1779, error c
ode=0x1
EXCEPTION STREAM:
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.874.959 [device_msg_handler.cc:156]2224109 Handle
MsgInHostBuf:[INIT][DEFAULT]
  Exception info:TGID=2224109, model id=65535, stream id=10, stream phase=3
  Message info[0]:RTS_HWTS: aicore timeout, slot_id=2, stream_id=10, task_id=4813
    Other info[0]:time=2023-09-28-02:17:58.332.637, function=hwts_timeout_recored_err_msg, line=1894,
error code=0x25
EXCEPTION STREAM:
  Exception info:TGID=2224109, model id=65535, stream id=10, stream phase=3
  Message info[0]:RTS_HWTS: slot_id=2, stream_id=10, task_id=4813, result=37
    Other info[0]:time=2023-09-28-02:17:58.332.659, function=se
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.874.975 [device_msg_handler.cc:156]2224109 Handle
MsgInHostBuf:[INIT][DEFAULT]nd_timeout_cq_msg_with_result, line=1383, error code=0x25
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.888.115 [stream.cc:2366]2224109 WaitForTask:[INIT
][DEFAULT]Task Wait: device_id=3, stream_id=10 is Abort, RunningState=0
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.888.130 [stream.cc:1470]2224109 GetError:[INIT][D
EFAULT]Stream Synchronize failed, stream_id=10, retCode=0x25, [aicore timeout].
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.888.138 [stream.cc:1470]2224109 GetError:[INIT][D
EFAULT]Stream Synchronize failed, stream_id=10, retCode=0x25, [aicore timeout].
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.888.144 [context.cc:306]2224109 Synchronize:[INIT
][DEFAULT]sync stream fail, stream_id=10, retCode=0x7150025.
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.888.189 [context.cc:311]2224109 Synchronize:[INIT
][DEFAULT]Synchronize streams, retCode=0x7150025.
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.888.195 [logger.cc:862]2224109 DeviceSynchronize:
[INIT][DEFAULT]Device synchronize failed.
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.888.217 [api_c.cc:1762]2224109 rtDeviceSynchroniz
e:[INIT][DEFAULT]ErrCode=507014, desc=[aicore timeout], InnerCode=0x7150025
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.888.224 [error_message_manage.cc:50]2224109 FuncE
rrorReason:[INIT][DEFAULT]report error module_type=3, module_name=EE8888
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:03.888.234 [error_message_manage.cc:50]2224109 FuncE
rrorReason:[INIT][DEFAULT]rtDeviceSynchronize execute failed, reason=[aicore timeout]
[ERROR] ASCENDCL(2224109,python3):2023-09-28-02:18:03.888.272 [device.cpp:285]2224109 aclrtSynchronize
Device: [INIT][DEFAULT]wait for compute device to finish failed, runtime result = 507014.
[ERROR] APP(2224109,python3):2023-09-28-02:18:03.889.124 [log_inner.cpp:76]2224109 $(notdir $(abspath
$<)):THPModule_npu_shutdown:78: "[PTA]:"NPU shutdown synchronize device failed.""
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:14.974.300 [stream.cc:2366]2224109 WaitForTask:[INIT
][DEFAULT]Task Wait: device_id=3, stream_id=10 is Abort, RunningState=0
[ERROR] RUNTIME(2224109,python3):2023-09-28-02:18:14.974.342 [stream.cc:1470]2224109 GetError:[INIT][D
EFAULT]Stream Synchronize failed, stream_id=10, retCode=0x25, [aicore timeout].
请根据自己的运行环境参考以下方式搜集日志信息，如果涉及到算子开发相关的问题，建议也提供UT/ST测试和单算子集成测试相关的日志。

日志提供方式:
将日志打包后作为附件上传。若日志大小超出附件限制，则可上传至外部网盘后提供链接。

获取方法请参考wiki：
https://gitee.com/ascend/modelzoo/wikis/%E5%A6%82%E4%BD%95%E8%8E%B7%E5%8F%96%E6%97%A5%E5%BF%97%E5%92%8C%E8%AE%A1%E7%AE%97%E5%9B%BE?sort_id=4097825

Ascend/modelzoo
暂停

内容风险标识

评论 (5)

Ascend/modelzoo暂停 .gitee-modal { width: 500px !important; }

内容风险标识