RuntimeError: ACL stream synchronize failed, error code:507035

一、问题现象（附报错日志上下文）：
 **使用torch_npu + ray跑多卡的模型，报以下错误：** 
Traceback (most recent call last):
  File "main.py", line 23, in <module>
    outputs = llm.generate(prompts, sampling_params)
  File "/home/shanyizhou/shuwen/xds/src/inference/vllm/vllm/entrypoints/llm.py", line 173, in generate
    return self._run_engine(use_tqdm)
  File "/home/shanyizhou/shuwen/xds/src/inference/vllm/vllm/entrypoints/llm.py", line 196, in _run_engine
    step_outputs = self.llm_engine.step()
  File "/home/shanyizhou/shuwen/xds/src/inference/vllm/vllm/engine/llm_engine.py", line 683, in step
    blocks_to_copy=scheduler_outputs.blocks_to_copy,
  File "/home/shanyizhou/shuwen/xds/src/inference/vllm/vllm/engine/llm_engine.py", line 907, in _run_workers
    all_outputs = ray.get(all_outputs)
  File "/usr/local/lib64/python3.7/site-packages/ray/_private/auto_init_hook.py", line 24, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib64/python3.7/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib64/python3.7/site-packages/ray/_private/worker.py", line 2547, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RuntimeError): ray::RayWorker.execute_method() (pid=214078, ip=172.17.0.34, actor_id=a35394f273ee3154c78ae8fa08000000, repr=<vllm.engine.ray_utils.RayWorker object at 0xfffc006da690>)
  File "/usr/local/lib64/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 88, in dumps
    cp.dump(obj)
  File "/usr/local/lib64/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 733, in dump
    return Pickler.dump(self, obj)
  File "/usr/local/lib64/python3.7/site-packages/torch_npu/utils/storage.py", line 67, in _reduce_ex
    tmp_tensor = self.cpu()
RuntimeError: ACL stream synchronize failed, error code:507035

**查看/root/ascend/log后，有以下报错（太长了仅展示部分）：** 
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.375.953 [engine.cc:3670]214075 ProcLogicCqReport:[INIT][DEFAULT]CQE error, device_id=4, stream_id=2, task_id=3008, sqe_type=0, errType=0x1(task exception), sqSwStatus=0
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.567 [device_error_proc.cc:1090]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.614 [device_error_proc.cc:1090]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]The error from device(chipId:4, dieId:0), serial number is 868, there is an aivec error exception, core id is 47, error code = 0x800000, dump info: pc start: 0x1240c03a918c, current: 0x1240c03aacac, vec error info: 0x4000000085, mte error info: 0xab06000013, ifu error info: 0x1ff7636a84200, ccu error info: 0x126919220000085, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x1241006f0000.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.679 [device_error_proc.cc:1102]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.685 [device_error_proc.cc:1102]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x6000013, fixp_error1 info: 0xab fsmId:0, tslot:7, thread:0, ctxid:0, blk:1, sublk:0, subErrType:4.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.701 [device_error_proc.cc:1090]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.706 [device_error_proc.cc:1090]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]The error from device(chipId:4, dieId:0), serial number is 868, there is an aivec error exception, core id is 0, error code = 0x800000, dump info: pc start: 0x1240c03a918c, current: 0x1240c03aacac, vec error info: 0x4000000085, mte error info: 0xab06000013, ifu error info: 0x36263508067c0, ccu error info: 0x111f481820000085, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x1241006f0000.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.719 [device_error_proc.cc:1102]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.723 [device_error_proc.cc:1102]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x6000013, fixp_error1 info: 0xab fsmId:1, tslot:7, thread:0, ctxid:0, blk:4, sublk:0, subErrType:4.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.733 [device_error_proc.cc:1090]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.737 [device_error_proc.cc:1090]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]The error from device(chipId:4, dieId:0), serial number is 868, there is an aivec error exception, core id is 1, error code = 0x800000, dump info: pc start: 0x1240c03a918c, current: 0x1240c03aacac, vec error info: 0x4000000085, mte error info: 0xab06000013, ifu error info: 0x77633fa730080, ccu error info: 0x7fbb80b020000085, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x1241006f0000.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.750 [device_error_proc.cc:1102]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.754 [device_error_proc.cc:1102]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x6000013, fixp_error1 info: 0xab fsmId:1, tslot:7, thread:0, ctxid:0, blk:5, sublk:0, subErrType:4.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.763 [device_error_proc.cc:1090]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.767 [device_error_proc.cc:1090]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]The error from device(chipId:4, dieId:0), serial number is 868, there is an aivec error exception, core id is 46, error code = 0x800000, dump info: pc start: 0x1240c03a918c, current: 0x1240c03aacac, vec error info: 0x4000000085, mte error info: 0xab06000013, ifu error info: 0x47fe7b28058c0, ccu error info: 0xb890d4a920000085, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x1241006f0000.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.778 [device_error_proc.cc:1102]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.782 [device_error_proc.cc:1102]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x6000013, fixp_error1 info: 0xab fsmId:1, tslot:7, thread:0, ctxid:0, blk:0, sublk:0, subErrType:4.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.791 [device_error_proc.cc:1090]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.795 [device_error_proc.cc:1090]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]The error from device(chipId:4, dieId:0), serial number is 868, there is an aivec error exception, core id is 48, error code = 0x800000, dump info: pc start: 0x1240c03a918c, current: 0x1240c03aacac, vec error info: 0x4000000085, mte error info: 0xab06000013, ifu error info: 0x678ec8a31c0, ccu error info: 0x855b25ef20000085, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x1241006f0000.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.807 [device_error_proc.cc:1102]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.811 [device_error_proc.cc:1102]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x6000013, fixp_error1 info: 0xab fsmId:1, tslot:7, thread:0, ctxid:0, blk:2, sublk:0, subErrType:4.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.821 [device_error_proc.cc:1090]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.825 [device_error_proc.cc:1090]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]The error from device(chipId:4, dieId:0), serial number is 868, there is an aivec error exception, core id is 49, error code = 0x800000, dump info: pc start: 0x1240c03a918c, current: 0x1240c03aacac, vec error info: 0x4000000085, mte error info: 0xab06000013, ifu error info: 0x6ef3c38f47c40, ccu error info: 0x8d94901120000085, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x1241006f0000.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.836 [device_error_proc.cc:1102]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.840 [device_error_proc.cc:1102]214075 ProcessStarsCoreErrorInfo:[INIT][DEFAULT]The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x6000013, fixp_error1 info: 0xab fsmId:1, tslot:7, thread:0, ctxid:0, blk:3, sublk:0, subErrType:4.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.850 [stream.cc:2981]214075 EnterFailureAbort:[INIT][DEFAULT]stream_id=2 enter failure abort.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.859 [task_info.cc:1615]214075 SetStarsResultForDavinciTask:[INIT][DEFAULT]AIV Kernel happen error, retCode=0x31.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.865 [task_info.cc:1517]214075 PreCheckTaskErr:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.869 [task_info.cc:1517]214075 PreCheckTaskErr:[INIT][DEFAULT]Kernel task happen error, retCode=0x31, [vector core exception].
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.880 [stream.cc:1416]214075 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x31, [vector core exception].
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.885 [stream.cc:1419]214075 GetError:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.461.889 [stream.cc:1419]214075 GetError:[INIT][DEFAULT]AIV Kernel happen error, retCode=0x31.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.462.199 [task_info.cc:1479]214075 PrintErrorInfoForDavinciTask:[INIT][DEFAULT]Aicore kernel execute failed, device_id=4, stream_id=2, report_stream_id=2, task_id=3008, flip_num=0, fault kernel_name=pos_encoding_float16, program id=1, hash=5861005151946346259.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.462.243 [task_info.cc:1422]214075 GetArgsInfo:[INIT][DEFAULT][AIC_INFO] args(0 to 13) after execute:0x124047e7fc00, 0x12480b62fa00, 0x12480b62fa00, 0x12480b62fa00, 0x124041302a00, 0x124880000000, 0x1248c0000000, 0x124047e7fe00, 0x80000000006, 0x60000000080, 0x60000000600, 0x200000008, 0x8000000080,  
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.462.254 [task_info.cc:1483]214075 PrintErrorInfoForDavinciTask:[INIT][DEFAULT][AIC_INFO] after execute:args print end
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.463.405 [engine.cc:3549]214075 StarsResumeRtsq:[INIT][DEFAULT]stop scheduling in abort failure mode: stream_id=2, sq_id=2, sq_head=3008, task_id=3008, taskType=66.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.463.417 [stream.cc:1416]214075 GetError:[INIT][DEFAULT]Stream Synchronize failed, stream_id=2, retCode=0x31, [vector core exception].
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.463.422 [stream.cc:1419]214075 GetError:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.463.426 [stream.cc:1419]214075 GetError:[INIT][DEFAULT]Aicore kernel execute failed, device_id=4, stream_id=2, report_stream_id=2, task_id=3008, flip_num=0, fault kernel_name=pos_encoding_float16, program id=1, hash=5861005151946346259.
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.463.440 [stream.cc:1419]214075 GetError:[INIT][DEFAULT]report error module_type=5, module_name=EZ9999
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.463.444 [stream.cc:1419]214075 GetError:[INIT][DEFAULT][AIC_INFO] after execute:args print end
[ERROR] RUNTIME(214075,orker.execute_method):2024-01-10-11:04:55.463.452 [logger.cc:466]214075 StreamSynchronize:[INIT][DEFAULT]Stream synchronize failed

二、软件版本:
-- CANN 版本 7.0.0
--Python 版本 1.11.0

Ascend/pytorch
暂停

内容风险标识

评论 (2)

Ascend/pytorch暂停 .gitee-modal { width: 500px !important; }

内容风险标识