采用自动回退量化Qwen3-235B-A22B-Thinking-2507的W8A8版本装载模型报错

一、问题现象（附报错日志上下文）：
因为quant_qwen_moe_w8a8.py默认配置量化出来的Qwen3-235B-A22B-Thinking-2507有比较明显的精度问题，会时不时出现“游戏副本”的字样，因此尝试使用回退linear层的方式重新量化，设置自动回退层级为L5。量化可以成功，但是装载模型报错。

二、软件版本:
运行镜像为 mindie:2.1.RC1.B152-800I-A2-py3.11-openeuler24.03-lts-aarch64

三、测试步骤：
采用镜像：mindie:2.0.T18.B010-800I-A2-py3.11-openeuler24.03-lts-aarch64
修改：example/Qwen3-MOE/quant_qwen_moe_w8a8.py
自动回退设置为L5
    calibrator = Calibrator(model,
                            quant_config,
                            calib_data=dataset_calib,
                            disable_level="L5",
                            mix_cfg={"*.mlp.*": "w8a8_dynamic", "*": "w8a8"})

执行
python3 quant_qwen_moe_w8a8.py --model_path /Model/Qwen3-235B-A22B-Thinking-2507 --save_path /Model/Qwen3-235B-A22B-Thinking-2507-L5 --trust_remote_code True

四、日志信息:

生成模型后运行报错，运行镜像为 mindie:2.1.RC1.B152-800I-A2-py3.11-openeuler24.03-lts-aarch64
机器为1台A800 I2,
加载模型报错，日志为：

[2025-08-30 11:59:33.518] [47420] [281470512787808] [llmmodels] [ERROR] [acl_nn_operation.cpp:142] gmmNode call SetAclNNWorkspaceExecutor fail, error:161002
[2025-08-30 11:59:33.518] [47420] [281470512787808] [llmmodels] [ERROR] [acl_nn_operation.cpp:115] gmmNode call CreateAclNNOpCache fail, error:12
[2025-08-30 11:59:33.518] [47420] [281470512787808] [llmmodels] [ERROR] [acl_nn_operation.cpp:59] gmmNode call UpdateAclNNOpCache, error:12
[2025-08-30 11:59:33.520] [47445] [281471276740960] [llmmodels] [ERROR] [acl_nn_operation.cpp:142] gmmNode call SetAclNNWorkspaceExecutor fail, error:161002
[2025-08-30 11:59:33.520] [47445] [281471276740960] [llmmodels] [ERROR] [acl_nn_operation.cpp:115] gmmNode call CreateAclNNOpCache fail, error:12
[2025-08-30 11:59:33.520] [47445] [281471276740960] [llmmodels] [ERROR] [acl_nn_operation.cpp:59] gmmNode call UpdateAclNNOpCache, error:12

[2025-08-30 11:59:33.030+0800] [47445] [281473340731744] [batchscheduler] [ERROR] [model.py:61] : [Model]       >>> Exception:Setup fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to find the first error. For more details, see the MindIE official document.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 59, in initialize
    return self.python_model.initialize(config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 133, in initialize
    self.generator = Generator(
                     ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 299, in __init__
    self.cache_manager = self.warm_up(
                         ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 456, in warm_up
    npu_mem = self.__warmup_standard(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 626, in __warmup_standard
    npu_mem = self.__warmup_prefill(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 614, in __warmup_prefill
    npu_mem = self.__auto_warmup(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, is_prefill=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 708, in __auto_warmup
    self.__execute_warm_up(cache_manager, input_metadata, dummy=True)
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 508, in __execute_warm_up
    raise e
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 499, in __execute_warm_up
    self.generator_backend._warm_up(model_inputs, inference_mode=self.inference_mode,
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 570, in _warm_up
    super()._warm_up(model_inputs, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_backend.py", line 247, in _warm_up
    logits = self.forward(model_inputs, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 69, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 230, in forward
    logits = self._forward(model_inputs, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 621, in _forward
    logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 125, in forward
    result = self.forward_from_model_inputs(model_inputs, npu_cache, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 197, in forward_from_model_inputs
    result = self.forward_tensor(
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 237, in forward_tensor
    result = self.model_runner.forward(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 310, in forward
    res = self.model.forward(**kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 536, in forward
    logits = self.execute_ascend_operator(acl_inputs, acl_param, is_prefill)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 467, in execute_ascend_operator
    acl_model_out = self.acl_encoder_operation.execute(acl_inputs, acl_param)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Setup fail, enable log: export ASDOPS_LOG_LEVEL=ERROR, export ASDOPS_LOG_TO_STDOUT=1 to find the first error. For more details, see the MindIE official document.

[2025-08-30 11:59:33.030+0800] [47420] [281472577433952] [batchscheduler] [ERROR] [model.py:64] : [MIE04E13030A] [Model]        >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0', 'memPoolId': '-1'}
[2025-08-30 11:59:33.030+0800] [47445] [281473340731744] [batchscheduler] [ERROR] [model.py:64] : [MIE04E13030A] [Model]        >>> return initialize error result: {'status': 'error', 'npuBlockNum': '0', 'cpuBlockNum': '0', 'memPoolId': '-1'}

Ascend/msit
暂停

内容风险标识

评论 (3)

Ascend/msit暂停 .gitee-modal { width: 500px !important; }

内容风险标识