LLMDet_tiny模型转成om格式成功，但推理失败

背景说明：想在atlas 200DK A2开发者套件上运行LLMDet-tiny这个模型。
模型在：https://huggingface.co/iSEE-Laboratory/llmdet_tiny
编译onnx的系统：windows10 
编译onnx的系统使用的python版本：3.10.8
python依赖详情如下

```
asttokens==3.0.0
certifi==2025.8.3
charset-normalizer==3.4.3
clip @ git+https://github.com/ultralytics/CLIP.git@a6238961f9fab312d99c7709f8c7485b3f9a9b50
colorama==0.4.6
coloredlogs==15.0.1
comm==0.2.3
contourpy==1.3.2
cycler==0.12.1
Cython==3.1.3
debugpy==1.8.16
decorator==5.2.1
exceptiongroup==1.3.0
executing==2.2.0
filelock==3.19.1
flatbuffers==25.2.10
fonttools==4.59.1
fsspec==2025.7.0
ftfy==6.3.1
huggingface-hub==0.34.4
humanfriendly==10.0
idna==3.10
iniconfig==2.1.0
intel-openmp==2021.4.0
ipykernel==6.30.1
ipython==8.37.0
jedi==0.19.2
Jinja2==3.1.6
jupyter_client==8.6.3
jupyter_core==5.8.1
kiwisolver==1.4.9
MarkupSafe==3.0.2
matplotlib==3.10.5
matplotlib-inline==0.1.7
mkl==2021.4.0
ml_dtypes==0.5.3
modelscope==1.29.2
mpmath==1.3.0
nest-asyncio==1.6.0
networkx==3.4.2
numpy==1.26.4
onnx==1.17.0
onnx-ir==0.1.7
onnxruntime==1.22.1
onnxscript==0.4.0
onnxslim==0.1.65
opencv-python==4.12.0.88
optimum==1.27.0
packaging==25.0
pandas==2.3.2
parso==0.8.5
pillow==11.3.0
platformdirs==4.3.8
pluggy==1.6.0
prompt_toolkit==3.0.51
protobuf==6.32.0
psutil==7.0.0
pure_eval==0.2.3
py-cpuinfo==9.0.0
Pygments==2.19.2
pyparsing==3.2.3
pyreadline3==3.5.4
pytest==8.4.1
python-dateutil==2.9.0.post0
pytz==2025.2
pywin32==311
PyYAML==6.0.2
pyzmq==27.0.2
regex==2025.7.33
requests==2.32.5
safetensors==0.6.2
scipy==1.15.3
six==1.17.0
stack-data==0.6.3
sympy==1.14.0
tbb==2021.13.1
tokenizers==0.21.4
tomli==2.2.1
torch==2.3.0
torchvision==0.18.0
tornado==6.5.2
tqdm==4.67.1
traitlets==5.14.3
# Editable install with no version control (transformers==4.55.0)
-e d:\projects\video\try_yolo_world\transformers_src\transformers-4.55.0
typing_extensions==4.14.1
tzdata==2025.2
ultralytics==8.3.183
ultralytics-thop==2.0.16
urllib3==2.5.0
wcwidth==0.2.13
```
其中transformers使用了4.55.0版本，不然无法读取到MMGroundingDinoForObjectDetection这个类

将onnx转成om的系统：Ubuntu 22.04.5 LTS  x86_64架构
将onnx转成om的系统使用的cann工具包：Ascend-cann-toolkit_8.0.RC1_linux- **x86_64** .run
将onnx转成om的系统使用的python版本：3.10.12

python依赖详情如下

```
aclruntime @ file:///home/wangheyu/aclruntime-0.0.2-cp39-cp39-linux_aarch64.whl#sha256=af3be0a0fb0c74dabb0c3e7307dd2054673f635d22958011fa860f71a42d5dd4
action-msgs==1.2.1
actionlib-msgs==4.2.3
ais-bench @ file:///home/wangheyu/ais_bench-0.0.2-py3-none-any.whl#sha256=ff55373a11d9975eaad497a230c9fb0d93856dc184790b8f168143c9c5f1cccd
ament-cmake-test==1.3.4
ament-copyright==0.12.6
ament-cppcheck==0.12.6
ament-cpplint==0.12.6
ament-flake8==0.12.6
ament-index-python==1.4.0
ament-lint==0.12.6
ament-lint-cmake==0.12.6
ament-package==0.14.0
ament-pep257==0.12.6
ament-uncrustify==0.12.6
ament-xmllint==0.12.6
anyio==4.10.0
argon2-cffi==25.1.0
argon2-cffi-bindings==25.1.0
arrow==1.3.0
ascendctools @ file:///usr/local/Ascend/ascend-toolkit/8.0.RC1.alpha002/toolkit/tools/ascendctools-0.1.0-py3-none-any.whl
asttokens==3.0.0
async-lru==2.0.5
attrs==25.3.0
auto-tune @ file:///root/selfgz273739916/compiler/lib64/auto_tune-0.1.0-py3-none-any.whl
babel==2.17.0
beautifulsoup4==4.13.5
bleach==6.2.0
builtin-interfaces==1.2.1
cartographer-ros-msgs==2.0.9000
certifi==2025.8.3
cffi==1.17.1
charset-normalizer==3.4.3
comm==0.2.3
composition-interfaces==1.2.1
contourpy==1.3.0
cv-bridge==3.2.1
cycler==0.12.1
dataflow @ file:///root/selfgz273739916/compiler/lib64/dataflow-0.0.1-py3-none-any.whl
debugpy==1.8.16
decorator==5.2.1
defusedxml==0.7.1
diagnostic-msgs==4.2.3
diagnostic-updater==3.1.2
domain-coordinator==0.10.0
exceptiongroup==1.3.0
executing==2.2.1
fastjsonschema==2.21.2
filelock==3.19.1
fonttools==4.59.2
fqdn==1.5.1
fsspec==2025.9.0
geometry-msgs==4.2.3
h11==0.16.0
hccl @ file:///root/selfgz273739916/compiler/lib64/hccl-0.1.0-py3-none-any.whl
hccl-parser @ file:///usr/local/Ascend/ascend-toolkit/8.0.RC1.alpha002/toolkit/tools/hccl_parser-0.1-py3-none-any.whl
httpcore==1.0.9
httpx==0.28.1
idna==3.10
importlib_metadata==8.7.0
importlib_resources==6.5.2
interactive-markers==2.3.2
ipykernel==6.30.1
ipython==8.18.1
isoduration==20.11.0
jedi==0.19.2
Jinja2==3.1.6
json5==0.12.1
jsonpointer==3.0.0
jsonschema==4.25.1
jsonschema-specifications==2025.4.1
jupyter-events==0.12.0
jupyter-lsp==2.3.0
jupyter_client==8.6.3
jupyter_core==5.8.1
jupyter_server==2.17.0
jupyter_server_terminals==0.5.3
jupyterlab==4.4.7
jupyterlab_pygments==0.3.0
jupyterlab_server==2.27.3
kiwisolver==1.4.7
lark==1.2.2
laser-geometry==2.4.0
launch==1.0.4
launch-ros==0.19.4
launch-testing==1.0.4
launch-testing-ros==0.19.4
launch-xml==1.0.4
launch-yaml==1.0.4
lifecycle-msgs==1.2.1
map-msgs==2.1.0
MarkupSafe==3.0.2
matplotlib==3.9.4
matplotlib-inline==0.1.7
message-filters==4.3.3
mistune==3.1.4
mpmath==1.3.0
msadvisor @ file:///usr/local/Ascend/ascend-toolkit/8.0.RC1.alpha002/tools/msadvisor/python/msadvisor-1.0.0-cp37-abi3-linux_aarch64.whl
nav-msgs==4.2.3
nav2-msgs==1.1.6
nav2-simple-commander==1.0.0
nbclient==0.10.2
nbconvert==7.16.6
nbformat==5.10.4
nest-asyncio==1.6.0
networkx==3.2.1
notebook==7.4.5
notebook_shim==0.2.4
numpy==2.0.2
op-compile-tool @ file:///root/selfgz273739916/compiler/lib64/op_compile_tool-0.1.0-py3-none-any.whl
op-gen @ file:///usr/local/Ascend/ascend-toolkit/8.0.RC1.alpha002/toolkit/tools/op_gen-0.1-py3-none-any.whl
op-test-frame @ file:///usr/local/Ascend/ascend-toolkit/8.0.RC1.alpha002/toolkit/tools/op_test_frame-0.1-py3-none-any.whl
opc-tool @ file:///root/selfgz273739916/compiler/lib64/opc_tool-0.1.0-py3-none-any.whl
opencv-python==4.12.0.88
osrf-pycommon==2.0.2
overrides==7.7.0
packaging==25.0
pandocfilters==1.5.1
parso==0.8.5
pcl-msgs==1.0.0
pexpect==4.9.0
pillow==11.3.0
platformdirs==4.4.0
polars==1.33.0
prometheus_client==0.22.1
prompt_toolkit==3.0.52
psutil==7.0.0
ptyprocess==0.7.0
pure_eval==0.2.3
py-cpuinfo==9.0.0
pycparser==2.22
Pygments==2.19.2
pyparsing==3.2.3
python-dateutil==2.9.0.post0
python-json-logger==3.3.0
PyYAML==6.0.2
pyzmq==27.0.2
rcl-interfaces==1.2.1
rclpy==3.3.8
rcutils==5.1.3
referencing==0.36.2
requests==2.32.5
resource-retriever==3.1.1
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rfc3987-syntax==1.1.0
rmw-dds-common==1.6.0
ros2action==0.18.6
ros2bag==0.15.5
ros2cli==0.18.6
ros2component==0.18.6
ros2doctor==0.18.6
ros2interface==0.18.6
ros2launch==0.19.4
ros2lifecycle==0.18.6
ros2multicast==0.18.6
ros2node==0.18.6
ros2param==0.18.6
ros2pkg==0.18.6
ros2run==0.18.6
ros2service==0.18.6
ros2topic==0.18.6
rosbag2-interfaces==0.15.5
rosbag2-py==0.15.5
rosbridge-library==1.3.1
rosbridge-msgs==1.3.1
rosbridge-test-msgs==1.3.1
rosgraph-msgs==1.2.1
rosidl-adapter==3.1.4
rosidl-cli==3.1.4
rosidl-cmake==3.1.4
rosidl-generator-c==3.1.4
rosidl-generator-cpp==3.1.4
rosidl-generator-dds-idl==0.8.1
rosidl-generator-py==0.14.4
rosidl-parser==3.1.4
rosidl-runtime-py==0.9.3
rosidl-typesupport-c==2.0.0
rosidl-typesupport-cpp==2.0.0
rosidl-typesupport-fastrtps-c==2.2.0
rosidl-typesupport-fastrtps-cpp==2.2.0
rosidl-typesupport-introspection-c==3.1.4
rosidl-typesupport-introspection-cpp==3.1.4
rpds-py==0.27.1
rpyutils==0.2.1
schedule-search @ file:///root/selfgz273739916/compiler/lib64/schedule_search-0.1.0-py3-none-any.whl
scipy==1.13.1
Send2Trash==1.8.3
sensor-msgs==4.2.3
sensor-msgs-py==4.2.3
shape-msgs==4.2.3
six==1.17.0
sniffio==1.3.1
soupsieve==2.8
sros2==0.10.4
stack-data==0.6.3
statistics-msgs==1.2.1
std-msgs==4.2.3
std-srvs==4.2.3
stereo-msgs==4.2.3
sympy==1.14.0
te @ file:///root/selfgz273739916/compiler/lib64/te-0.4.0-py3-none-any.whl
terminado==0.18.1
tf-transformations==1.0.0
tf2-geometry-msgs==0.25.2
tf2-kdl==0.25.2
tf2-msgs==0.25.2
tf2-py==0.25.2
tf2-ros-py==0.25.2
tf2-tools==0.25.2
tinycss2==1.4.0
tomli==2.2.1
torch==2.8.0
torchvision==0.23.0
tornado==6.5.2
tqdm==4.67.1
traitlets==5.14.3
trajectory-msgs==4.2.3
turtlesim==1.4.2
types-python-dateutil==2.9.0.20250822
typing_extensions==4.15.0
ultralytics==8.3.192
ultralytics-thop==2.0.17
unique-identifier-msgs==2.2.1
uri-template==1.3.0
urllib3==2.5.0
vision-msgs==4.1.0
visualization-msgs==4.2.3
wcwidth==0.2.13
webcolors==24.11.1
webencodings==0.5.1
websocket-client==1.8.0
x30-rtk-calibra
```

atlas 200DK A2 开发者套件上使用的cann工具包：Ascend-cann-toolkit_8.0.RC1_linux- **aarch64** .run
开发者套件上的npu信息为
(run_om_venv) [root@davinci-mini run_yoloe_om]# npu-smi info
+--------------------------------------------------------------------------------------------------------+
| npu-smi 24.1.rc2                                 Version: 24.1.rc2                                     |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU     Name                  | Health          | Power(W)     Temp(C)           Hugepages-Usage(page) |
| Chip    Device                | Bus-Id          | AICore(%)    Memory-Usage(MB)                        |
+===============================+=================+======================================================+
| 0       310B1                 | OK              | 8.3          57                15    / 15            |
| 0       0                     | NA              | 0            1557 / 7545                             |
+===============================+=================+======================================================+

实验一：转成静态输入的onnx，并尝试推理
转onnx使用的代码

```
import traceback

sorted_keys_dict = {
    "pixel_values": 1,
    "input_ids": 2,
    "token_type_ids": 3,
    "attention_mask": 4,
    "pixel_mask": 5
}

if __name__ == '__main__':
    import torch
    from transformers import AutoModelForZeroShotObjectDetection, AutoProcessor
    from transformers.image_utils import load_image

# Prepare processor and model
    model_id = r"D:\projects\video\try_llmdet\llmdet_tiny_back"
    device = "cuda" if torch.cuda.is_available() else "cpu"
    processor = AutoProcessor.from_pretrained(model_id)
    model = AutoModelForZeroShotObjectDetection.from_pretrained(model_id).to(device)

# Prepare inputs
    image_url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    image = load_image(image_url)
    image = image.resize((640,640))
    text_labels = [["person","face","trash","smoke","fire"]]

# Run inference
    inputs = processor(images=image, text=text_labels, return_tensors="pt").to(device)
    with torch.no_grad():
        outputs = model(**inputs)

# Postprocess outputs
    results = processor.post_process_grounded_object_detection(
        outputs,
        threshold=0.2,
        target_sizes=[(image.height, image.width)]
    )

# Retrieve the first image result
    result = results[0]
    for box, score, labels in zip(result["boxes"], result["scores"], result["labels"]):
        box = [round(x, 2) for x in box.tolist()]
        print(f"Detected {labels} with confidence {round(score.item(), 3)} at location {box}")

keys = []
    values = []
    for key in sorted_keys_dict:
        keys.append(key)
        value = inputs[key]
        values.append(value)
        print(f"{key}: {value.shape}")

try:
        torch.onnx.export(
            model,
            tuple(values),
            rf"D:\projects\video\try_llmdet\llmdet_tiny_oonx\model_{i}_static_input.onnx",
            input_names=keys,
            output_names=["logits", "pred_boxes"],
            opset_version=16,
            do_constant_folding=True,
        )
    except Exception as e:

print(f"按照version 16 转 onnx 发生异常，重新保存")
        traceback.print_exception(e)
```
转换后的netron可视化截图为
![netron可视化结果](https://foruda.gitee.com/images/1757039227363549009/891a9a44_10624706.png "屏幕截图")
转om使用的shell脚本如下，

```
atc --mode=0 \
--framework=5 \
--model="/mnt/d/projects/video/try_llmdet/llmdet_tiny_oonx/model_16.onnx" \
--input_shape="pixel_values:1,3,800,800;input_ids:1,12;token_type_ids:1,12;attention_mask:1,12;pixel_mask:1,800,800" \
--input_format=ND \
--display_model_info=1 \
--output="/home/wangheyu/llmdet_om/llmdet_tiny_om_static_input" \
--host_env_os="linux" \
--host_env_cpu="aarch64" \
--soc_version "Ascend310B1" \
--log=info
```

运行代码如下

```
import os
os.environ['ASCEND_GLOBAL_LOG_LEVEL'] = '1'  # 0: DEBUG, 1: INFO, 2: WARNING, 3: ERROR
os.environ['GLOG_v'] = '2'  # 设置CANN日志级别

model_path = "/home/wangheyu/run_om/model_om/llmdet_tiny/llmdet_tiny_om_static_input_linux_aarch64.om"
from ais_bench.infer.interface import InferSession
model = InferSession(0, model_path)
```
加载时，报错如下

```
[INFO] acl init success
[INFO] open device 0 success
[INFO] create new context
[ACL ERROR] E19999: Inner Error!
E19999  Param:registry_holder is nullptr, check invalid[FUNC:GetOrCreateRegistry][FILE:op_impl_space_registry.cc][LINE:69]
        TraceBack (most recent call last):
        Assert ((space_registry->GetOrCreateRegistry(so_list, root_model->GetSoInOmInfo())) == ge::SUCCESS) failed[FUNC:CreateModelDesc][FILE:model_converter.cc][LINE:509]
        Assert ((CreateModelDesc(root_model)) == ge::SUCCESS) failed[FUNC:ConvertGeModelToExecuteGraph][FILE:model_converter.cc][LINE:417]
        Failed to lowering to execute graph[FUNC:LoadToModelV2ExecutorBuilder][FILE:api.cc][LINE:56]
        Assert ((error_code) == ge::SUCCESS) failed[FUNC:LoadExecutorFromModelData][FILE:api.cc][LINE:103]
        [Model][FromData]call gert::LoadExecutorFromModelDataWithMem load model from data failed, ge result[1343225857][FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

[ERROR] load model from file failed, model file is /home/wangheyu/run_om/model_om/llmdet_tiny/llmdet_tiny_om_static_input_linux_aarch64.om
[WARN] Check failed:processModel->LoadModelFromFile(modelPath), ret:1

[WARN] no model had been loaded, unload failed
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/wangheyu/run_om_venv/lib/python3.9/site-packages/ais_bench/infer/interface.py", line 82, in __init__
    self.session = aclruntime.InferenceSession(self.model_path, self.device_id, self.options)
RuntimeError: [1][ACL: invalid parameter] 
```

Ascend/modelzoo
暂停

内容风险标识

评论 (1)

Ascend/modelzoo暂停 .gitee-modal { width: 500px !important; }

内容风险标识