模型推理出现507035

使用场景：
单卡多进程，用embedding模型向量化文档，文档内容比较大，所以i好像当遇到单个任务量的时候就会报错

报错内容：
2024-06-14 07:08:57,509 - embeddings_api.py[line:39] - ERROR: Sync:/usr1/02/workspace/j_yxiCvvHE/pytorch/torch_npu/csrc/framework/OpCommand.cpp:162 NPU error, error code is 507035
[Error]: The vector core execution is abnormal. 
        Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue
EZ9999: Inner Error!
EZ9999  The error from device(chipId:6, dieId:0), serial number is 4, there is an aivec error exception, core id is 12, error code = 0x800000, dump info: pc start: 0x1240c022e984, current: 0x1240c022f470, vec error info: 0xe1077f3838, mte error info: 0xbc0302e63c, ifu error info: 0x60054b00a4100, ccu error info: 0x634900005000035, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x1241007d3c00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1138]
        TraceBack (most recent call last):
        The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x302e63c, fixp_error1 info: 0xbc fsmId:0, tslot:0, thread:0, ctxid:0, blk:1, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1150]
        The error from device(chipId:6, dieId:0), serial number is 4, there is an aivec error exception, core id is 13, error code = 0x800000, dump info: pc start: 0x1240c022e984, current: 0x1240c022f470, vec error info: 0x611465ef8f, mte error info: 0xbc0304765a, ifu error info: 0x4bbc3cd00180, ccu error info: 0x19d305dc05000035, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x1241007d3c00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1138]
        The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x304765a, fixp_error1 info: 0xbc fsmId:0, tslot:0, thread:0, ctxid:0, blk:2, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1150]
        The error from device(chipId:6, dieId:0), serial number is 4, there is an aivec error exception, core id is 14, error code = 0x800000, dump info: pc start: 0x1240c022e984, current: 0x1240c022f470, vec error info: 0xf1093482e5, mte error info: 0xbc0305666e, ifu error info: 0x57a89e0b24200, ccu error info: 0xb818104805000035, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x1241007d3c00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1138]
        The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x305666e, fixp_error1 info: 0xbc fsmId:0, tslot:0, thread:0, ctxid:0, blk:3, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1150]
        The error from device(chipId:6, dieId:0), serial number is 4, there is an aivec error exception, core id is 15, error code = 0x800000, dump info: pc start: 0x1240c022e984, current: 0x1240c022f470, vec error info: 0x5119f1dea1, mte error info: 0xbc0306e63c, ifu error info: 0x6d7104df41780, ccu error info: 0x2001813505000035, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd000288, para base: 0x1241007d3c00.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1138]
        The extend info: errcode:(0x800000, 0, 0) errorStr: The DDR address of the MTE instruction is out of range. fixp_error0 info: 0x306e63c, fixp_error1 info: 0xbc fsmId:0, tslot:0, thread:0, ctxid:0, blk:4, sublk:0, subErrType:4.[FUNC:ProcessStarsCoreErrorInfo][FILE:device_error_proc.cc][LINE:1150]
...

Ascend/pytorch
暂停

内容风险标识

评论 (5)

Ascend/pytorch暂停 .gitee-modal { width: 500px !important; }

内容风险标识