If you encounter an issue when using MindSpore Lite, you can view logs first. In most scenarios, you can locate the issue based on the error information reported in logs. You can set the environment variable GLOG_v to adjust the log level to print more debug logs. The following describes how to locate and rectify common faults.
- The log line number may vary in different versions. In the following example, the line number in the error log information is represented by "**".
- Only common information is listed in the example logs. Other information related to specific scenarios is displayed as "****".
The model path is incorrect or the file is damaged. The error log information is as follows:
WARNING: Logging before InitGoogleLogging() is written to STDERR
[WARNING] LITE(11979,7fbdc90a8ec0,converter_lite):2021-12-13-16:20:49.506.071 [mindspore/lite/tools/common/protobuf_utils.cc:94] ReadProtoFromBinaryFile] Parse ***.onnx failed.
[ERROR] LITE(11979,7fbdc90a8ec0,converter_lite):2021-12-13-16:20:49.506.122 [mindspore/lite/build/tools/converter/parser/onnx/onnx_op_parser.cc:3079] InitOriginModel] Read onnx model file failed, model path: ./ml_audio_kit_vocals_resunet.onnx
[ERROR] LITE(11979,7fbdc90a8ec0,converter_lite):2021-12-13-16:20:49.506.131 [mindspore/lite/build/tools/converter/parser/onnx/onnx_op_parser.cc:3026] Parse] init origin model failed.
[ERROR] LITE(11979,7fbdc90a8ec0,converter_lite):2021-12-13-16:20:49.506.137 [mindspore/lite/tools/converter/converter.cc:64] BuildFuncGraph] Get funcGraph failed for fmk: ONNX
[ERROR] LITE(11979,7fbdc90a8ec0,converter_lite):2021-12-13-16:20:49.506.143 [mindspore/lite/tools/converter/converter.cc:133] Convert] Parser/Import model return nullptr
[ERROR] LITE(11979,7fbdc90a8ec0,converter_lite):2021-12-13-16:20:49.506.162 [mindspore/lite/tools/converter/converter.cc:209] RunConverter] CONVERT RESULT FAILED:-1 Common error code.
CONVERT RESULT FAILED:-1 Common error code.
Analysis: According to the error information, an error is reported during model import and the model import exits before the conversion process starts.
Solution: Check whether the model path in the command entered during model conversion is correct. If the path is correct, check whether the model is damaged. If the damaged file cannot be parsed, the system exits.
Unsupported operators exist. The error log information is as follows:
[mindspore/lite/tools/converter/converter.cc:**] BuildFuncGraph] Get funcGraph failed for fmk: ****
[mindspore/lite/tools/converter/converter.cc:**] Converter] Parser/Import model return nullptr
[mindspore/lite/tools/converter/converter_context.h:**] PrintOps] ===========================================
[mindspore/lite/tools/converter/converter_context.h:**] PrintOps] UNSUPPORTED OP LIST:
[mindspore/lite/tools/converter/converter_context.h:**] PrintOps] FMKTYPE: ****, OP TYPE: ****
[mindspore/lite/tools/converter/converter_context.h:**] PrintOps] ===========================================
[mindspore/lite/tools/converter/converter.cc:**] RunConverter] CONVERT RESULT FAILED:-300 Failed to find operator.
Unsupported operators exist. The error log information is as follows:
[mindspore/lite/tools/converter/parser/caffe/caffe_model_parser.cc:**] ConvertLayers] parse node **** failed.
For dynamic shape models, you need to set --inputShape=<INPUTSHAPE>
listed in the Parameter Description. For example:
./converter_lite --fmk=ModelType --modelFile=ModelFilePath --outputFile=ConvertedModelPath --configFile=/mindspore/lite/tools/converter/quantizer/config/full_quant.cfg --inputShape=intput_1:1,224,224,3;intput_2:1,48;
For a multi-batch model, the data preprocessing function cannot be directly used. You need to preprocess the calibration dataset in advance and then set the calibration dataset in the BIN
format.
The model file is incorrect. The error log information is as follows:
[mindspore/lite/src/lite_model.cc:**] ConstructModel] The model buffer is invalid and fail to create graph.
[mindspore/lite/src/lite_model.cc:**] ImportFromBuffer] construct model failed.
The model file is incompatible with the inference package version. The following error information is displayed in logs:
[mindspore/lite/src/lite_model.cc:**] ConstructModel] Maybe this is a model transferred out using the conversion tool before 1.1.0.
[mindspore/lite/src/lite_model.cc:**] ImportFromBuffer] construct model failed.
The model file is incompatible with the inference package version. The following error information is displayed in logs:
WARNING [mindspore/lite/src/lite_model.cc:**] ConstructModel] model version is MindSpore Lite 1.2.0, inference version is MindSpore Lite 1.5.0 not equal
[mindspore/lite/src/litert/infer_manager.cc:**] KernelInferShape] Get infershape func failed! type: ****
[mindspore/lite/src/scheduler.cc:**] ScheduleNodeToKernel] FindBackendKernel return nullptr, name: ****, type: ****
[mindspore/lite/src/scheduler.cc:**] ScheduleSubGraphToKernels] schedule node return nullptr, name: ****, type: ****
[mindspore/lite/src/scheduler.cc:**] ScheduleMainSubGraphToKernels] Schedule subgraph failed, index: 0
[mindspore/lite/src/scheduler.cc:**] ScheduleGraphToKernels] ScheduleSubGraphToSubGraphKernel failed
[mindspore/lite/src/scheduler.cc:**] Schedule] Schedule graph to kernels failed.
[mindspore/lite/src/lite_session.cc:**] CompileGraph] Schedule kernels failed: -1.
The model input is a dynamic shape. The following error information is displayed in logs:
[mindspore/lite/src/common/tensor_util.cc:**] CheckTensorsInvalid] The shape of tensor contains negative dimension, check the model and assign the input shape with method Resize().
[mindspore/lite/src/lite_session.cc:**] RunGraph] CheckInputs failed.
The model file is incompatible with the inference package version. The following error information is displayed in logs:
ERROR [mindspore/lite/src/lite_session.cc:1539] LoadModelByBuff] Please enable runtime convert.
ERROR [mindspore/lite/src/lite_session.cc:1598] LoadModelAndCompileByPath] Read model file failed
ERROR [mindspore/lite/src/cxx_api/model/model_impl.cc:93] Build] Init session failed
ERROR [mindspore/lite/tools/benchmark/benchmark_unified_api.cc:845] RunBenchmark] ms_model_.Build failed while running
ERROR [mindspore/lite/tools/benchmark/run_benchmark.cc:80] RunBenchmark] Run Benchmark Q888_CV_new_detect.pb.ms Failed : -1
ms_model_.Build failed while running Run Benchmark Q888_CV_new_detect.pb.ms Failed : -1
The model input is a dynamic shape. The following error information is displayed in logs:
WARNING [mindspore/lite/src/litert/kernel/opencl/kernel/arithmetic_self.cc:40] CheckSpecs] only support dim = 4 or 2 but your dim = 3
ERROR [mindspore/lite/src/litert/kernel/opencl/opencl_kernel.cc:222] ReSize] ReSize failed for check kernel specs!
ERROR [mindspore/lite/src/inner_kernel.cc:81] Execute] run kernel PreProcess failed, name: Exp_1234
ERROR [mindspore/lite/src/litert/gpu/opencl/opencl_executor.cc:70] RunOrTune] run kernel failed, name: Exp_1234
ERROR [mindspore/lite/src/litert/kernel/opencl/opencl_subgraph.cc:574] Execute] Run opencl executor failed: -1
ERROR [mindspore/lite/src/lite_mindrt.h:58] RunKernel] run kernel failed, name: GpuSubGraph4_8
WARNING [mindspore/lite/src/litert/gpu/opencl/opencl_allocator.cc:475] MapBuffer] Host ptr no need map
WARNING [mindspore/lite/src/litert/gpu/opencl/opencl_allocator.cc:525] UnmapBuffer] Host ptr do not mapped
Map buffer errors
WARNING [mindspore/lite/src/litert/gpu/opencl/opencl_allocator.cc:494] MapBuffer] Map buffer failed, can not found buffer or already mapped, dev_ptr=0x7244929ff0, host_ptr=0x722fbacd80
ERROR [mindspore/lite/src/litert/kernel/arm/base/strided_slice.cc:179] FastRun] input_ptr_ must not be null!
ERROR [mindspore/lite/src/inner_kernel.cc:88] Execute] run kernel failed, name: Slice_1147
ERROR [mindspore/lite/src/sub_graph_kernel.cc:223] Execute] run kernel failed, name: Slice_1147
ERROR [mindspore/lite/src/lite_mindrt.h:56] RunKernel] run kernel failed, name: CpuFP32SubGraph0_1
ERROR [mindspore/lite/src/mindrt_executor.cc:193] Run] MindrtRun failed
ERROR [mindspore/lite/src/lite_session.cc:709] RunGraph] RunGraph failed : -1
ERROR [mindspore/lite/src/cxx_api/model/model_impl.cc:294] Predict] Run graph failed.
ERROR [mindspore/lite/tools/benchmark/benchmark_unified_api.cc:721] MarkAccuracy] Inference error
Inference error Run MarkAccuracy error: -1
If the model input is a dynamic shape or the model has a shape operator, the following error information about dimensions is displayed in the log:
ERROR [mindspore/lite/src/delegate/tensorrt/tensorrt_runtime.h:31] log] Parameter check failed at: optimizationProfile.cpp::setDimensions::119, condition: std::all_of(dims.d, dims.d + dims.nbDims, [](int x) { return x >= 0; })
ERROR [mindspore/lite/src/delegate/tensorrt/tensorrt_subgraph.cc:219] ParseInputDimsProfile] setDimensions of kMIN failed for input
ERROR [mindspore/lite/src/delegate/tensorrt/tensorrt_runtime.h:31] log] xxx: xxx size cannot have negative dimension, size = [-1]
--inputShape=<INPUTSHAPE>
in the conversion command to specify the shape information of the input tensor. If you need to change the input shape during inference, you can set the inputShapes parameter when using the benchmark tool or call the Resize method when using MindSpore Lite for integration and development. Note: The shape dimension of the Resize input must be less than or equal to the dimension of the Build model.Offline broadcast operators do not support resizing. An error message is displayed, indicating that the input dimension of an operator does not match. For example:
ERROR [mindspore/lite/src/delegate/tensorrt/tensorrt_runtime.h:31] log] xxx: dimensions not compatible for xxx
ERROR [mindspore/lite/src/delegate/tensorrt/tensorrt_runtime.h:31] log] shapeMachine.cpp (252) - Shape Error in operator(): broadcast with incompatible Dimensions
ERROR [mindspore/lite/src/delegate/tensorrt/tensorrt_runtime.h:31] log] Instruction: CHECK_BROADCAST xx xx
ERROR [mindspore/lite/src/delegate/tensorrt/tensorrt_subgraph.cc:500] Execute] TensorRT execute failed.
--inputShape=<INPUTSHAPE>
. Take the ones like operator as an example. 1 is broadcast to the corresponding constant tensor based on the input shape information. In this case, when the input is resized to different dimensions, an error is reported for operators (such as concat and matmul) that are sensitive to the input tensor dimensions on the network.Failed to build an NPU graph. A tool is used to capture background logs and search for MS_LITE in the logs. The following error information is displayed:
MS_LITE : [mindspore/lite/src/delegate/npu/npu_subgraph.cc:**] BuildIRModel] Build IR model failed.
MS_LITE : [mindspore/lite/src/delegate/npu/npu_subgraph.cc:**] Init] Build IR model failed.
MS_LITE : [mindspore/lite/src/delegate/npu/npu_graph.cc:**] CreateNPUSubgraphKernel] NPU Subgraph Init failed.
MS_LITE : [mindspore/lite/src/delegate/npu/npu_delegate.cc:**] Build] Create NPU Graph failed.
(1) Search for the keyword E AI_FMK in the log file. If the following error log is found before the "MS_LITE" error is reported:
AI_FMK : /conv_base_op_builder.cpp CheckShapeMatch(**)::"Input Channel ** does not match convolution weight channel ** * group **."
AI_FMK : /conv_base_op_builder.cpp Init(**)::"Shape of op **** does not match"
AI_FMK : /op_builder.cpp BuildOutputDesc(**)::""Init" failed. Node: ****."
This indicates that the Transpose operator for tensor format conversion is inserted or omitted during graph construction. As a result, the input shape of the convolution does not match the weight format.
(2) Search for the keyword E AI_FMK in the log file. If the following error log is found before the "MS_LITE" error is reported:
AI_FMK : /model_compatibility_check.cpp GetIRGraphCompatibilityCheckResult(**)::"Node **** type **** don't support!"
AI_FMK : /model_compatibility_check.cpp CheckIRGraphCompatibility(**)::"CompleteExecuteDeviceConfig CheckIRGraphCompatibility failed"
The error message indicates that the operator is not supported due to operator compatibility issues between HiAI ROM and HiAI DDK used by MindSpore Lite. You can update your phone system to update HiAI ROM, replace unsupported operators, or provide feedback in the open-source community.
NPU inference fails. A tool is used to capture background logs and search for MS_LITE in the logs. The following error information is displayed:
MS_LITE : [mindspore/lite/src/delegate/npu/npu_executor.cc:**] Run] NPU Process failed. code is 1
MS_LITE : [mindspore/lite/src/delegate/npu/npu_graph.cc:**] Execute] NPU Subgraph **** execute failed.
MS_LITE : [mindspore/lite/src/lite_mindrt.h:**] RunKernel] run kernel failed, name: ****
Search for the keyword E AI_FMK in the log file. If the following error log is found before the MS_LITE error is reported:
AI_FMK : /common_memory_allocator.cpp Allocate(**)::"Call rt api failed, ret: ****"
Search for DEVMM. If the following error information is displayed in the first 10 lines of the previous logs:
/vendor/bin/hiaiserver: [DEVMM][E] DevmmJudgeAllocSize:** the alloc memory size exceeds the specification limit, already alloc total size = 0x3ff95000
This indicates that the memory application of the NPU exceeds the limit. Check whether the model file is large or whether a tensor with a large shape exists in the model. According to official HiAI requirements, the size of a single NPU's subgraph should not exceed 200 MB, and the memory requested for the array should not exceed the NPU's video memory. For example, in the log in this example, the NPU can apply for a maximum of 1 GB video memory. If you still need to run the NPU, adjust the model structure, split the model, or adjust the shape of the tensor.
When MindSpore Lite is used for integration, the post-processing effect of the inference result is not ideal. How do I locate the issue of inference accuracy?
Check whether the input data is correct. In MindSpore Lite 1.3.0 and earlier versions, the input data format of the MS model is NHWC. In MindSpore Lite 1.5.0 and later versions, the inputDataFormat parameter can be set to NHWC or NCHW. Ensure that the input data format is the same as that required by the MS model.
Use the benchmark tool provided by MindSpore Lite to test the accuracy. If the following logs are displayed, the accuracy may be incorrect. Otherwise, the MindSpore Lite inference accuracy is correct. In this case, you need to check whether the data pre- and post-processing processes are correct.
Mean bias of all nodes/tensors is too big: **
Compare output error -1
Run MarkAccuracy error: -1
If the accuracy of the entire network inference performed by MindSpore Lite is incorrect, you can use the Dump function of the benchmark tool to save the output of the operator layer and compare the output with the inference result of the original framework to further locate the operator with incorrect accuracy.
For operators with accuracy issues, you can download the MindSpore source code to check the operator implementation and construct the corresponding single-operator network for debugging and fault locating. You can also commit an issue in the MindSpore community to MindSpore Lite developers for troubleshooting.
What do I do if the FP32 inference result is correct but the FP16 inference result contains the NaN or Inf value?
What do I do if the NaN or Inf value exists in the FP32 and FP16 inference results of MindSpore Lite?
Analysis: Operators that perform division operations exist on the entire network. During inference, if the division operation is performed and the divisor is 0, the NaN value may display. Take the following network structure as an example. If the network is used in the scenario where the input data is not normalized and the input data ranges from 0 to 255, the NaN value is displayed. The reason is that the input data is large and the output value of matmul is large when the input data is not normalized. As a result, the output value of the Tanh activation function is 1. As a result, the Div operator is divided by 0. However, if the network input data is normalized and the Tanh activation function is not 1, the network inference data does not have the NaN value.
Solution: If the input data is too large, you are advised to normalize the network input data during training. If the NaN value still exists after the input data is normalized, you need to use the Dump function of the benchmark tool to save the output of the operator layer to confirm the operator where the value overflow occurs and analyze the issue.
Why is the actual inference performance the same as that of the CPU after the device is specified as the NPU?
If the device does not support the NPU but the NPU is specified in the context, the model automatically runs on a CPU instead of the NPU. In this case, the inference performance is the same as that of the CPU. You can use a tool (such as adb logcat) to capture background logs and search for the keyword MS_LITE in the logs to check whether the device supports NPUs. The common prompts and descriptions are as follows:
MS_LITE : [mindspore/lite/src/delegate/npu/npu_manager.cc:**] IsSupportNPU] The current devices NOT SUPPORT NPU.
If the log contains only the preceding information, check whether your device is a Huawei device with a HiSilicon Kirin processor. If not, NPUs are not supported.
MS_LITE : [mindspore/lite/src/delegate/npu/npu_manager.cc:**] IsKirinChip] Unsupported KirinChip ***.
MS_LITE : [mindspore/lite/src/delegate/npu/npu_manager.cc:**] IsSupportNPU] The current devices NOT SUPPORT NPU.
If the log contains the preceding information, your device uses the Kirin chip, but the chip model does not support NPUs. Currently, the following Kirin chips support NPUs: Kirin 810, Kirin 820, Kirin 985, and later versions.
MS_LITE : [mindspore/lite/src/delegate/npu/npu_manager.cc:**] CheckDDKVerGreatEqual] DDK Version 100.***.***.*** less than 100.320.011.019.
MS_LITE : [mindspore/lite/src/delegate/npu/npu_manager.cc:**] IsSupportNPU] The current devices NOT SUPPORT NPU.
If the log contains the preceding information, your device meets the hardware requirements, but the HiAI ROM version does not meet the requirements. As a result, the NPU operator cannot run. MindSpore Lite requires that the HiAI ROM version be later than 100.320.011.018.
MS_LITE : [mindspore/lite/src/delegate/npu/op/convolution_npu.cc:**] GetNPUConvOp] NPU does not support runtime inference shape.
MS_LITE : [mindspore/lite/src/delegate/npu/op/npu_op.h:** GetNPUOp] NPU does not support runtime inference shape.
If either of the preceding two messages appears in the log for multiple times, check whether the model input is a dynamic shape and whether the input shape is specified before inference. If yes, the model cannot run on the NPU, and the program automatically switches to the CPU for execution.
Why is the actual inference performance poorer than that of the CPU after the device is specified as the NPU?
(1) Check whether there are a large number of Pad or StridedSlice operators in the model. The array format of the NPU is different from that of the CPU. The operation of these operators in the NPU involves array rearrangement. Therefore, the NPU has no advantage over the CPU and even is inferior to the CPU. If you need to run such an operator on the NPU, you are advised to remove or replace the operator. (2) Use a tool (such as adb logcat) to capture background logs and search for the keyword BuildIRModel build successfully. It is found that related logs appear multiple times, indicating that the model is partitioned into multiple NPU-related subgraphs during online graph construction. Generally, subgraph partitioning is caused by the existence of Transpose and/or unsupported NPU operators in the graph. Currently, a maximum of 20 subgraphs can be partitioned. The more the subgraphs, the more time the NPU takes. You are advised to refer to the NPU operators supported by MindSpore Lite and avoid unsupported operators during model building. Alternatively, commit an issue to MindSpore Lite developers.
When the static library is used, an error message is displayed, indicating that the Creator function of the parameter cannot be found. The following error information is displayed in logs:
ERROR [mindspore\lite\src\ops\populate\populate_register.h:**] GetParameterCreator] Unsupported parameter type in Create : **
ERROR [mindspore\lite\src\scheduler.cc:**] InferNodeShape] parameter generator is nullptr.
ERROR [mindspore\lite\src\scheduler.cc:**] InferSubGraphShape] InferShape failed, name: **, type: **
ERROR [mindspore\lite\src\scheduler.cc:**] SchedulePreProcess] op infer shape failed.
ERROR [mindspore\lite\src\lite_session.cc:**] CompileGraph] Schedule kernels failed: -500
Analysis: When the static library is linked, not all symbols in the static library are imported by default. The Creator function of parameter is registered with the singleton object through the global static object.
Solution: When linking the static library built by Visual Studio, choose "Properties > Linker > Command Line > Additional Options" and add /WHOLEARCHIVE:libmindspore-lite.lib.
The model verification fails. The error log information is as follows:
ERROR [mindspore\lite\src\lite_model.cc:**] ModelVerify] Model does not have inputs.
ERROR [mindspore\lite\src\lite_model.cc:**] ConstructModel] ModelVerify failed.
ERROR [mindspore\lite\src\lite_model.cc:**] ImportFromBuffer] construct model failed.
Analysis: The read model file is incomplete.
Solution: When using the Visual Studio compiler, add std::ios::binary to the model stream to be read.
When Xcode uses a framework package to build an app, an error is reported indicating that a parameter is not supported when the GetParameterCreator function is executed. The log is as follows:
ERROR [mindspore/lite/src/ops/populate/populate_register.h:46] GetParameterCreator] Unsupported parameter type in Create : **
ERROR [mindspore/lite/src/scheduler.cc:208] InferNodeShape] parameter generator is nullptr.
ERROR [mindspore/lite/src/scheduler.cc:266] InferSubGraphShape] InferShape failed, name: **, type: **
ERROR [mindspore/lite/src/scheduler.cc:78] SchedulePreProcess] op infer shape failed.
ERROR [mindspore/lite/src/lite_session.cc:508] CompileGraph] Schedule kernels failed: -500
Analysis: When the static library in the framework is linked, not all symbols in the static library are imported. The Creator function of parameter is registered with the singleton object through the global static object.
Solution: When using the framework package to build an app in Xcode, choose "Build Settings > Linking > Other Linker Flags" and add the "mindspore_lite.framework/mindspore_lite" path.
When Xcode is used to build an app using the framework package, the following error message is displayed:
Undefined symbol:
mindspore::session::LiteSession::CreateSession(**);
Analysis: The corresponding symbol is not found, and the framework package is not correctly imported to the Xcode project.
Solution: When using the framework package to build an app in Xcode, choose "Build Settings > Search Paths > Framework Search Paths" and add the "mindspore_lite.framework" path. In addition, choose "Build Settings > Search Paths > User Header Search Paths" and add the "mindspore_lite.framework/Headers" path.
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。