glip多线程报错

NonZero
EZ9999: Inner Error!
EZ9999  The error from device(1), serial number is 45, there is an aicore error, core id is 0, error code = 0x800000, dump info: pc start: 0x1000124080f6a000, current: 0x124080f6b738, vec error info: 0x1fdccefc, mte error info: 0x6002c17, ifu error info: 0x27f75f7dffe80, ccu error info: 0, cube error info: 0x3f, biu error info: 0, aic error mask: 0x65000200d000288, para base: 0x1240003fbbc0, errorStr: The DDR address of the MTE instruction is out of range.[FUNC:PrintCoreErrorInfo][FILE:device_error_proc.cc][LINE:523]
        TraceBack (most recent call last):
        The extend info from device(1), serial number is 45, there is aicore error, core id is 0, aicore int: 0x1, aicore error2: 0, axi clamp ctrl: 0, axi clamp state: 0x1717, biu status0: 0x101e44800000000, biu status1: 0x940002092a0000, clk gate mask: 0x1000, dbg addr: 0, ecc en: 0, mte ccu ecc 1bit error: 0, vector cube ecc 1bit error: 0, run stall: 0x1, dbg data0: 0, dbg data1: 0, dbg data2: 0, dbg data3: 0, dfx data: 0[FUNC:PrintCoreErrorInfo][FILE:device_error_proc.cc][LINE:554]
        The dha(mata) info from device(1), dha id is 0, dha status 1 info:0xf[FUNC:ProcessCoreErrorInfo][FILE:device_error_proc.cc][LINE:644]
        The dha(mata) info from device(1), dha id is 1, dha status 1 info:0xf[FUNC:ProcessCoreErrorInfo][FILE:device_error_proc.cc][LINE:644]
        The dha(mata) info from device(1), dha id is 2, dha status 1 info:0xf[FUNC:ProcessCoreErrorInfo][FILE:device_error_proc.cc][LINE:644]
        The dha(mata) info from device(1), dha id is 3, dha status 1 info:0xf[FUNC:ProcessCoreErrorInfo][FILE:device_error_proc.cc][LINE:644]
        The dha(mata) info from device(1), dha id is 4, dha status 1 info:0xf[FUNC:ProcessCoreErrorInfo][FILE:device_error_proc.cc][LINE:644]
        The dha(mata) info from device(1), dha id is 5, dha status 1 info:0xf[FUNC:ProcessCoreErrorInfo][FILE:device_error_proc.cc][LINE:644]
        The dha(mata) info from device(1), dha id is 6, dha status 1 info:0xf[FUNC:ProcessCoreErrorInfo][FILE:device_error_proc.cc][LINE:644]
        The dha(mata) info from device(1), dha id is 7, dha status 1 info:0xf[FUNC:ProcessCoreErrorInfo][FILE:device_error_proc.cc][LINE:644]
        The device(1), core list[0-0], error code is:[FUNC:PrintCoreInfoErrMsg][FILE:device_error_proc.cc][LINE:577]
        coreId( 0):        0x800000    [FUNC:PrintCoreInfoErrMsg][FILE:device_error_proc.cc][LINE:591]
        Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:task_info.cc][LINE:1518]
        Aicore kernel execute failed, device_id=0, stream_id=19, report_stream_id=19, task_id=15477, flip_num=0, fault kernel_name=58_-1_124_NonZero, program id=112, hash=2806487305886966728.[FUNC:GetError][FILE:stream.cc][LINE:1418]
        [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1418]
        rtStreamSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
        synchronize stream failed, runtime result = 507015[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

DEVICE[0] PID[713478]: 
EXCEPTION STREAM:
  Exception info:TGID=3354634, model id=65535, stream id=19, stream phase=3
  Message info[0]:RTS_HWTS: aicore exception, slot_id=30, stream_id=19
    Other info[0]:time=2024-07-23-15:02:47.085.380, function=int_process_hwts_task_exception, line=1706, error code=0x26
Error getting item from queue: ACL stream synchronize failed.
Traceback (most recent call last):
  File "test_demo2_pool.py", line 498, in <module>
    frame, flag = pool.get()
TypeError: cannot unpack non-iterable NoneType object
[W NPUStream.cpp:393] Warning: NPU warning, error code is 507015[Error]: 
[Error]: The aicore execution is abnormal. 
        Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
EH9999  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)
[W NPUStream.cpp:393] Warning: NPU warning, error code is 507015[Error]: 
[Error]: The aicore execution is abnormal. 
        Rectify the fault based on the error information in the log, or you can ask us at follwing gitee link by issues: https://gitee.com/ascend/pytorch/issue
EH9999: Inner Error!
        rtDeviceSynchronize execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:50]
EH9999  wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]
        TraceBack (most recent call last):
 (function npuSynchronizeDevice)

我创建了两个线程去运行glip模型，报错。多线程不加模型推理能正常运行，加上模型推理就报错。

Ascend/pytorch
暂停

内容风险标识

评论 (6)

Ascend/pytorch暂停 .gitee-modal { width: 500px !important; }

内容风险标识