登录
注册
开源
企业版
高校版
搜索
帮助中心
使用条款
关于我们
开源
企业版
高校版
私有云
模力方舟
登录
注册
代码拉取完成，页面将自动刷新
捐赠
捐赠前请先登录
取消
前往登录
扫描微信二维码支付
取消
支付完成
支付提示
将跳转至支付宝完成支付
确定
取消
Watch
不关注
关注所有动态
仅关注版本发行动态
关注但不提醒动态
56
Star
558
Fork
1.4K
Ascend
/
ModelZoo-PyTorch
代码
Issues
360
Pull Requests
200
Wiki
统计
流水线
服务
质量分析
Jenkins for Gitee
腾讯云托管
腾讯云 Serverless
悬镜安全
阿里云 SAE
Codeblitz
SBOM
我知道了，不再自动展开
更新失败，请稍后重试！
移除标识
内容风险标识
本任务被
标识为内容中包含有代码安全 Bug 、隐私泄露等敏感信息，仓库外成员不可访问
Atlas 300I Duo卡服务化 DeepSeek-R1-Distill-Qwen-14B报错
TODO
#IBSEDK
需求
Dr
创建于
2025-03-11 11:30
驱动：24.1.rc2 固件：7.3.0.1.231 使用镜像：swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-300I-Duo-py311-openeuler24.03-lts 已按照https://gitee.com/ascend/ModelZoo-PyTorch/tree/master/MindIE/LLM/DeepSeek/DeepSeek-R1-Distill-Qwen-14B 中的操作步骤，完成了： 1、稀疏量化、量化权重切分及压缩 python3 quant_qwen.py --model_path /root/llm/DeepSeek-R1-Distill-Qwen-14B/ --save_directory /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8S/ --calib_file ../common/cn_en.jsonl --w_bit 4 --a_bit 8 --fraction 0.011 --co_sparse True --device_type npu --use_sigma True --is_lowbit True --sigma_factor 4.0 --anti_method m4 torchrun --nproc_per_node 2 -m examples.convert.model_slim.sparse_compressor --multiprocess_num 4 --model_path /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8S --save_directory /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8SC 2、对话测试 torchrun --nproc_per_node 2 --master_port 20037 -m examples.run_pa --model_path /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8SC --max_output_length 20 上述操作都正常，但是进行服务化推理的时候报错： [2025-03-04 16:16:34,884] [340] [281464127078752] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16]) [2025-03-04 16:16:34,942] [338] [281464018104672] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 32, 128, 16]) [2025-03-04 16:16:34,951] [338] [281464018104672] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16]) [2025-03-04 16:16:34,952] [338] [281464018104672] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281472827437616 id of vcache is 281472827437520 2025-03-04 16:16:42,601 [ERROR] model.py:39 - [Model] >>> Exception:npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015 [ERROR] 2025-03-04-16:16:42 (PID:338, Device:0, RankID:-1) ERR00100 PTA call acl api failed [Error]: The aicore execution is abnormal. Rectify the fault based on the error information in the ascend log. EZ9999: Inner Error! EZ9999: [PID: 338] 2025-03-04-16:16:42.403.969 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356] TraceBack (most recent call last): Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124] [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124] rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53] wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] DEVICE[0] PID[338]: EXCEPTION STREAM: Exception info:TGID=23517, model id=65535, stream id=2, stream phase=3 Message info[0]:RTS_HWTS: Aicore exception, slot_id=25, stream_id=2 Other info[0]:time=2025-03-04-16:16:42.420.163, function=process_hwts_error_exception, line=1320, error code=0x26 Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize return self.python_model.initialize(config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize self.generator = Generator( ^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__ self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode) File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 305, in warm_up npu.synchronize() File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 35, in synchronize return torch_npu._C._npu_synchronize() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015 [ERROR] 2025-03-04-16:16:42 (PID:338, Device:0, RankID:-1) ERR00100 PTA call acl api failed [Error]: The aicore execution is abnormal. Rectify the fault based on the error information in the ascend log. EZ9999: Inner Error! EZ9999: [PID: 338] 2025-03-04-16:16:42.403.969 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356] TraceBack (most recent call last): Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124] [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124] rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53] wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] DEVICE[0] PID[338]: EXCEPTION STREAM: Exception info:TGID=23517, model id=65535, stream id=2, stream phase=3 Message info[0]:RTS_HWTS: Aicore exception, slot_id=25, stream_id=2 Other info[0]:time=2025-03-04-16:16:42.420.163, function=process_hwts_error_exception, line=1320, error code=0x26 2025-03-04 16:16:42,603 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'} 2025-03-04 16:16:42,601 [ERROR] model.py:39 - [Model] >>> Exception:npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015 [ERROR] 2025-03-04-16:16:42 (PID:340, Device:1, RankID:-1) ERR00100 PTA call acl api failed [Error]: The aicore execution is abnormal. Rectify the fault based on the error information in the ascend log. EZ9999: Inner Error! EZ9999: [PID: 340] 2025-03-04-16:16:42.404.559 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356] TraceBack (most recent call last): Aicore kernel execute failed, device_id=1, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124] [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124] rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53] wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] DEVICE[1] PID[340]: EXCEPTION STREAM: Exception info:TGID=23519, model id=65535, stream id=2, stream phase=3 Message info[0]:RTS_HWTS: Aicore exception, slot_id=29, stream_id=2 Other info[0]:time=2025-03-04-16:16:42.421.225, function=process_hwts_error_exception, line=1320, error code=0x26 Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize return self.python_model.initialize(config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize self.generator = Generator( ^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__ self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode) File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 305, in warm_up npu.synchronize() File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 35, in synchronize return torch_npu._C._npu_synchronize() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015 [ERROR] 2025-03-04-16:16:42 (PID:340, Device:1, RankID:-1) ERR00100 PTA call acl api failed [Error]: The aicore execution is abnormal. Rectify the fault based on the error information in the ascend log. EZ9999: Inner Error! EZ9999: [PID: 340] 2025-03-04-16:16:42.404.559 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356] TraceBack (most recent call last): Aicore kernel execute failed, device_id=1, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124] [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124] rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53] wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] DEVICE[1] PID[340]: EXCEPTION STREAM: Exception info:TGID=23519, model id=65535, stream id=2, stream phase=3 Message info[0]:RTS_HWTS: Aicore exception, slot_id=29, stream_id=2 Other info[0]:time=2025-03-04-16:16:42.421.225, function=process_hwts_error_exception, line=1320, error code=0x26 2025-03-04 16:16:42,604 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'} [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! /usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' Daemon is killing...
驱动：24.1.rc2 固件：7.3.0.1.231 使用镜像：swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:1.0.0-300I-Duo-py311-openeuler24.03-lts 已按照https://gitee.com/ascend/ModelZoo-PyTorch/tree/master/MindIE/LLM/DeepSeek/DeepSeek-R1-Distill-Qwen-14B 中的操作步骤，完成了： 1、稀疏量化、量化权重切分及压缩 python3 quant_qwen.py --model_path /root/llm/DeepSeek-R1-Distill-Qwen-14B/ --save_directory /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8S/ --calib_file ../common/cn_en.jsonl --w_bit 4 --a_bit 8 --fraction 0.011 --co_sparse True --device_type npu --use_sigma True --is_lowbit True --sigma_factor 4.0 --anti_method m4 torchrun --nproc_per_node 2 -m examples.convert.model_slim.sparse_compressor --multiprocess_num 4 --model_path /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8S --save_directory /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8SC 2、对话测试 torchrun --nproc_per_node 2 --master_port 20037 -m examples.run_pa --model_path /root/llm/DeepSeek-R1-Distill-Qwen-14B-W8A8SC --max_output_length 20 上述操作都正常，但是进行服务化推理的时候报错： [2025-03-04 16:16:34,884] [340] [281464127078752] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16]) [2025-03-04 16:16:34,942] [338] [281464018104672] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 32, 128, 16]) [2025-03-04 16:16:34,951] [338] [281464018104672] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16]) [2025-03-04 16:16:34,952] [338] [281464018104672] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281472827437616 id of vcache is 281472827437520 2025-03-04 16:16:42,601 [ERROR] model.py:39 - [Model] >>> Exception:npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015 [ERROR] 2025-03-04-16:16:42 (PID:338, Device:0, RankID:-1) ERR00100 PTA call acl api failed [Error]: The aicore execution is abnormal. Rectify the fault based on the error information in the ascend log. EZ9999: Inner Error! EZ9999: [PID: 338] 2025-03-04-16:16:42.403.969 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356] TraceBack (most recent call last): Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124] [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124] rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53] wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] DEVICE[0] PID[338]: EXCEPTION STREAM: Exception info:TGID=23517, model id=65535, stream id=2, stream phase=3 Message info[0]:RTS_HWTS: Aicore exception, slot_id=25, stream_id=2 Other info[0]:time=2025-03-04-16:16:42.420.163, function=process_hwts_error_exception, line=1320, error code=0x26 Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize return self.python_model.initialize(config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize self.generator = Generator( ^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__ self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode) File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 305, in warm_up npu.synchronize() File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 35, in synchronize return torch_npu._C._npu_synchronize() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015 [ERROR] 2025-03-04-16:16:42 (PID:338, Device:0, RankID:-1) ERR00100 PTA call acl api failed [Error]: The aicore execution is abnormal. Rectify the fault based on the error information in the ascend log. EZ9999: Inner Error! EZ9999: [PID: 338] 2025-03-04-16:16:42.403.969 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356] TraceBack (most recent call last): Aicore kernel execute failed, device_id=0, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124] [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124] rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53] wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] DEVICE[0] PID[338]: EXCEPTION STREAM: Exception info:TGID=23517, model id=65535, stream id=2, stream phase=3 Message info[0]:RTS_HWTS: Aicore exception, slot_id=25, stream_id=2 Other info[0]:time=2025-03-04-16:16:42.420.163, function=process_hwts_error_exception, line=1320, error code=0x26 2025-03-04 16:16:42,603 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'} 2025-03-04 16:16:42,601 [ERROR] model.py:39 - [Model] >>> Exception:npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015 [ERROR] 2025-03-04-16:16:42 (PID:340, Device:1, RankID:-1) ERR00100 PTA call acl api failed [Error]: The aicore execution is abnormal. Rectify the fault based on the error information in the ascend log. EZ9999: Inner Error! EZ9999: [PID: 340] 2025-03-04-16:16:42.404.559 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356] TraceBack (most recent call last): Aicore kernel execute failed, device_id=1, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124] [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124] rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53] wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] DEVICE[1] PID[340]: EXCEPTION STREAM: Exception info:TGID=23519, model id=65535, stream id=2, stream phase=3 Message info[0]:RTS_HWTS: Aicore exception, slot_id=29, stream_id=2 Other info[0]:time=2025-03-04-16:16:42.421.225, function=process_hwts_error_exception, line=1320, error code=0x26 Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize return self.python_model.initialize(config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize self.generator = Generator( ^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__ self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode) File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 305, in warm_up npu.synchronize() File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 35, in synchronize return torch_npu._C._npu_synchronize() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015 [ERROR] 2025-03-04-16:16:42 (PID:340, Device:1, RankID:-1) ERR00100 PTA call acl api failed [Error]: The aicore execution is abnormal. Rectify the fault based on the error information in the ascend log. EZ9999: Inner Error! EZ9999: [PID: 340] 2025-03-04-16:16:42.404.559 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356] TraceBack (most recent call last): Aicore kernel execute failed, device_id=1, stream_id=2, report_stream_id=2, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124] [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124] rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53] wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] DEVICE[1] PID[340]: EXCEPTION STREAM: Exception info:TGID=23519, model id=65535, stream id=2, stream phase=3 Message info[0]:RTS_HWTS: Aicore exception, slot_id=29, stream_id=2 Other info[0]:time=2025-03-04-16:16:42.421.225, function=process_hwts_error_exception, line=1320, error code=0x26 2025-03-04 16:16:42,604 [ERROR] model.py:42 - [Model] >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0'} [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! [ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared! /usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /usr/lib64/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' Daemon is killing...
评论 (
3
)
登录
后才可以发表评论
状态
TODO
TODO
WIP
DONE
CLOSED
REJECTED
负责人
未设置
标签
未设置
项目
未立项任务
未立项任务
里程碑
未关联里程碑
未关联里程碑
Pull Requests
未关联
未关联
关联的 Pull Requests 被合并后可能会关闭此 issue
分支
未关联
未关联
master
ci-pipeline
开始日期 - 截止日期
-
置顶选项
不置顶
置顶等级：高
置顶等级：中
置顶等级：低
优先级
不指定
严重
主要
次要
不重要
预计工期
（小时）
参与者（1）
Python
1
https://gitee.com/ascend/ModelZoo-PyTorch.git
git@gitee.com:ascend/ModelZoo-PyTorch.git
ascend
ModelZoo-PyTorch
ModelZoo-PyTorch
点此查找更多帮助
搜索帮助
Git 命令在线学习
如何在 Gitee 导入 GitHub 仓库
Git 仓库基础操作
企业版和社区版功能对比
SSH 公钥设置
如何处理代码冲突
仓库体积过大，如何减小？
如何找回被删除的仓库数据
Gitee 产品配额说明
GitHub仓库快速导入Gitee及同步更新
什么是 Release（发行版）
将 PHP 项目自动发布到 packagist.org
仓库举报
回到顶部
登录提示
该操作需登录 Gitee 帐号，请先登录后再操作。
立即登录
没有帐号，去注册
