doc/samples.md · anolis/intel-openvino

# 1. Setup Anolis OS environment

- Download [AnolisOS-23.1-x86_64-dvd.iso](https://mirrors.openanolis.cn/anolis/23/isos/GA/x86_64/AnolisOS-23.1-x86_64-dvd.iso) and install it onto the system.

- Setup an Anolis OS 23 docker and pass the GPU device to the container

```shell
# docker pull openanolis/anolisos:23
#
# docker run -it -v /home/aubrey/work:/root/work -v /srv/models/:/root/models \
--device=/dev/dri --ipc=host \
-e http_proxy=$http_proxy -e https_proxy=$https_proxy -e no_proxy=$no_proxy \
--name ov docker.io/openanolis/anolisos:23 bash
```

# 2. Install Intel GPU drivers and graphics compute runtimes

- Setup EPAO repository

```shell
# yum install -y anolis-epao-release
```

- Install Intel GPU drivers and graphics compute runtimes

```shell
# yum install intel-gmmlib intel-gsc intel-igc-cm intel-igc-core intel-igc-opencl \
intel-level-zero-gpu intel-ocloc intel-opencl level-zero libmetee ocl-icd
```

# 3. Verify Intel GPU drivers and graphics compute runtimes

```shell
# clinfo --list
Platform #0: Intel(R) OpenCL Graphics
 `-- Device #0: Intel(R) Arc(TM) A770 Graphics
Platform #1: Intel(R) OpenCL Graphics
 `-- Device #0: Intel(R) UHD Graphics 770
```

# 4. Install OpenVINO packages from Anolis OS repository

- List all OpenVINO packages

```shell
# yum list all | grep openvino
libopenvino.x86_64                                       2024.3.0-1.an23                     @EPAO   
libopenvino-auto-batch-plugin.x86_64                     2024.3.0-1.an23                     @EPAO   
libopenvino-auto-plugin.x86_64                           2024.3.0-1.an23                     @EPAO   
libopenvino-devel.x86_64                                 2024.3.0-1.an23                     @EPAO   
libopenvino-intel-cpu-plugin.x86_64                      2024.3.0-1.an23                     @EPAO   
libopenvino-intel-gpu-plugin.x86_64                      2024.3.0-1.an23                     @EPAO   
libopenvino_ir_frontend.x86_64                           2024.3.0-1.an23                     @EPAO   
libopenvino_onnx_frontend.x86_64                         2024.3.0-1.an23                     @EPAO   
libopenvino_paddle_frontend.x86_64                       2024.3.0-1.an23                     @EPAO   
libopenvino_pytorch_frontend.x86_64                      2024.3.0-1.an23                     @EPAO   
libopenvino_tensorflow_frontend.x86_64                   2024.3.0-1.an23                     @EPAO   
libopenvino_tensorflow_lite_frontend.x86_64              2024.3.0-1.an23                     @EPAO   
openvino-samples.noarch                                  2024.3.0-1.an23                     @EPAO   
libopenvino-hetero-plugin.x86_64                         2024.3.0-1.an23                     EPAO    
```

- Install basic OpenVINO packages

```shell
# yum install libopenvino libopenvino-intel-cpu-plugin libopenvino-intel-gpu-plugin openvino-samples libopenvino-devel
```

# 5. Build OpenVINO samples

- Install the build required tools
```shell
# yum install -y cmake gcc g++ wget
```

- Build the samples

```shell
# cd /usr/share/openvino/samples/cpp/
# ./build_samples.sh

Setting environment variables for building samples...
-- The C compiler identification is GNU 12.3.0
-- The CXX compiler identification is GNU 12.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for C++ include unistd.h
-- Looking for C++ include unistd.h - found
-- Looking for C++ include stdint.h
-- Looking for C++ include stdint.h - found
-- Looking for C++ include sys/types.h
-- Looking for C++ include sys/types.h - found
-- Looking for C++ include fnmatch.h
-- Looking for C++ include fnmatch.h - found
-- Looking for strtoll
-- Looking for strtoll - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Configuring done (1.2s)
-- Generating done (0.0s)
-- Build files have been written to: /root/openvino_cpp_samples_build
[ 18%] Built target format_reader
[ 29%] Built target gflags_nothreads_static
[ 40%] Built target ie_samples_utils
[ 45%] Built target model_creation_sample
[ 59%] Built target sync_benchmark
[ 62%] Built target classification_sample_async
[ 56%] Built target throughput_benchmark
[ 67%] Built target hello_classification
[ 72%] Built target hello_query_device
[ 78%] Built target hello_nv12_input_classification
[ 94%] Built target benchmark_app
[100%] Built target hello_reshape_ssd
[100%] Built target ov_samples

Build completed, you can find binaries for all samples in the /root/openvino_cpp_samples_build/intel64/Release subfolder.
```

# 6. Run hello_query_device sample

```shell
# cd /root/openvino_cpp_samples_build/intel64/Release
#
# ./hello_query_device 
[ INFO ] Build ................................. 2024.3.0-000--
[ INFO ] 
[ INFO ] Available devices: 
[ INFO ] CPU
[ INFO ] 	SUPPORTED_PROPERTIES: 
[ INFO ] 		Immutable: AVAILABLE_DEVICES : ""
[ INFO ] 		Immutable: RANGE_FOR_ASYNC_INFER_REQUESTS : 1 1 1
[ INFO ] 		Immutable: RANGE_FOR_STREAMS : 1 32
[ INFO ] 		Immutable: EXECUTION_DEVICES : CPU
[ INFO ] 		Immutable: FULL_DEVICE_NAME : 13th Gen Intel(R) Core(TM) i9-13900K
[ INFO ] 		Immutable: OPTIMIZATION_CAPABILITIES : FP32 INT8 BIN EXPORT_IMPORT
[ INFO ] 		Immutable: DEVICE_TYPE : integrated
[ INFO ] 		Immutable: DEVICE_ARCHITECTURE : intel64
[ INFO ] 		Mutable: NUM_STREAMS : 1
[ INFO ] 		Mutable: INFERENCE_NUM_THREADS : 0
[ INFO ] 		Mutable: PERF_COUNT : NO
[ INFO ] 		Mutable: INFERENCE_PRECISION_HINT : f32
[ INFO ] 		Mutable: PERFORMANCE_HINT : LATENCY
[ INFO ] 		Mutable: EXECUTION_MODE_HINT : PERFORMANCE
[ INFO ] 		Mutable: PERFORMANCE_HINT_NUM_REQUESTS : 0
[ INFO ] 		Mutable: ENABLE_CPU_PINNING : YES
[ INFO ] 		Mutable: SCHEDULING_CORE_TYPE : ANY_CORE
[ INFO ] 		Mutable: MODEL_DISTRIBUTION_POLICY : ""
[ INFO ] 		Mutable: ENABLE_HYPER_THREADING : YES
[ INFO ] 		Mutable: DEVICE_ID : ""
[ INFO ] 		Mutable: CPU_DENORMALS_OPTIMIZATION : NO
[ INFO ] 		Mutable: LOG_LEVEL : LOG_NONE
[ INFO ] 		Mutable: CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE : 1
[ INFO ] 		Mutable: DYNAMIC_QUANTIZATION_GROUP_SIZE : 32
[ INFO ] 		Mutable: KV_CACHE_PRECISION : f16
[ INFO ] 		Mutable: AFFINITY : CORE
[ INFO ] 
[ INFO ] GPU.0
[ INFO ] 	SUPPORTED_PROPERTIES: 
[ INFO ] 		Immutable: AVAILABLE_DEVICES : 0 1
[ INFO ] 		Immutable: RANGE_FOR_ASYNC_INFER_REQUESTS : 1 2 1
[ INFO ] 		Immutable: RANGE_FOR_STREAMS : 1 2
[ INFO ] 		Immutable: OPTIMAL_BATCH_SIZE : 1
[ INFO ] 		Immutable: MAX_BATCH_SIZE : 1
[ INFO ] 		Immutable: DEVICE_ARCHITECTURE : GPU: vendor=0x8086 arch=v12.2.0
[ INFO ] 		Immutable: FULL_DEVICE_NAME : Intel(R) UHD Graphics 770 (iGPU)
[ INFO ] 		Immutable: DEVICE_UUID : 868080a7040000000002000000000000
[ INFO ] 		Immutable: DEVICE_LUID : 409a0000499a0000
[ INFO ] 		Immutable: DEVICE_TYPE : integrated
[ INFO ] 		Immutable: DEVICE_GOPS : {f16:1689.6,f32:844.8,i8:3379.2,u8:3379.2}
[ INFO ] 		Immutable: OPTIMIZATION_CAPABILITIES : FP32 BIN FP16 INT8 EXPORT_IMPORT
[ INFO ] 		Immutable: GPU_DEVICE_TOTAL_MEM_SIZE : 53769777152
[ INFO ] 		Immutable: GPU_UARCH_VERSION : 12.2.0
[ INFO ] 		Immutable: GPU_EXECUTION_UNITS_COUNT : 32
[ INFO ] 		Immutable: GPU_MEMORY_STATISTICS : ""
[ INFO ] 		Mutable: PERF_COUNT : NO
[ INFO ] 		Mutable: MODEL_PRIORITY : MEDIUM
[ INFO ] 		Mutable: GPU_HOST_TASK_PRIORITY : MEDIUM
[ INFO ] 		Mutable: GPU_QUEUE_PRIORITY : MEDIUM
[ INFO ] 		Mutable: GPU_QUEUE_THROTTLE : MEDIUM
[ INFO ] 		Mutable: GPU_ENABLE_SDPA_OPTIMIZATION : YES
[ INFO ] 		Mutable: GPU_ENABLE_LOOP_UNROLLING : YES
[ INFO ] 		Mutable: GPU_DISABLE_WINOGRAD_CONVOLUTION : NO
[ INFO ] 		Mutable: CACHE_DIR : ""
[ INFO ] 		Mutable: CACHE_MODE : optimize_speed
[ INFO ] 		Mutable: PERFORMANCE_HINT : LATENCY
[ INFO ] 		Mutable: EXECUTION_MODE_HINT : PERFORMANCE
[ INFO ] 		Mutable: COMPILATION_NUM_THREADS : 32
[ INFO ] 		Mutable: NUM_STREAMS : 1
[ INFO ] 		Mutable: PERFORMANCE_HINT_NUM_REQUESTS : 0
[ INFO ] 		Mutable: INFERENCE_PRECISION_HINT : f16
[ INFO ] 		Mutable: ENABLE_CPU_PINNING : NO
[ INFO ] 		Mutable: DEVICE_ID : 0
[ INFO ] 		Mutable: DYNAMIC_QUANTIZATION_GROUP_SIZE : 0
[ INFO ] 
[ INFO ] GPU.1
[ INFO ] 	SUPPORTED_PROPERTIES: 
[ INFO ] 		Immutable: AVAILABLE_DEVICES : 0 1
[ INFO ] 		Immutable: RANGE_FOR_ASYNC_INFER_REQUESTS : 1 2 1
[ INFO ] 		Immutable: RANGE_FOR_STREAMS : 1 4
[ INFO ] 		Immutable: OPTIMAL_BATCH_SIZE : 1
[ INFO ] 		Immutable: MAX_BATCH_SIZE : 1
[ INFO ] 		Immutable: DEVICE_ARCHITECTURE : GPU: vendor=0x8086 arch=v12.55.8
[ INFO ] 		Immutable: FULL_DEVICE_NAME : Intel(R) Arc(TM) A770 Graphics (dGPU)
[ INFO ] 		Immutable: DEVICE_UUID : 8680a056080000000300000000000000
[ INFO ] 		Immutable: DEVICE_LUID : 30bb5b7cfd7f0000
[ INFO ] 		Immutable: DEVICE_TYPE : discrete
[ INFO ] 		Immutable: DEVICE_GOPS : {f16:0,f32:19660.8,i8:0,u8:0}
[ INFO ] 		Immutable: OPTIMIZATION_CAPABILITIES : FP32 BIN FP16 INT8 GPU_HW_MATMUL EXPORT_IMPORT
[ INFO ] 		Immutable: GPU_DEVICE_TOTAL_MEM_SIZE : 16225243136
[ INFO ] 		Immutable: GPU_UARCH_VERSION : 12.55.8
[ INFO ] 		Immutable: GPU_EXECUTION_UNITS_COUNT : 512
[ INFO ] 		Immutable: GPU_MEMORY_STATISTICS : ""
[ INFO ] 		Mutable: PERF_COUNT : NO
[ INFO ] 		Mutable: MODEL_PRIORITY : MEDIUM
[ INFO ] 		Mutable: GPU_HOST_TASK_PRIORITY : MEDIUM
[ INFO ] 		Mutable: GPU_QUEUE_PRIORITY : MEDIUM
[ INFO ] 		Mutable: GPU_QUEUE_THROTTLE : MEDIUM
[ INFO ] 		Mutable: GPU_ENABLE_SDPA_OPTIMIZATION : YES
[ INFO ] 		Mutable: GPU_ENABLE_LOOP_UNROLLING : YES
[ INFO ] 		Mutable: GPU_DISABLE_WINOGRAD_CONVOLUTION : NO
[ INFO ] 		Mutable: CACHE_DIR : ""
[ INFO ] 		Mutable: CACHE_MODE : optimize_speed
[ INFO ] 		Mutable: PERFORMANCE_HINT : LATENCY
[ INFO ] 		Mutable: EXECUTION_MODE_HINT : PERFORMANCE
[ INFO ] 		Mutable: COMPILATION_NUM_THREADS : 32
[ INFO ] 		Mutable: NUM_STREAMS : 1
[ INFO ] 		Mutable: PERFORMANCE_HINT_NUM_REQUESTS : 0
[ INFO ] 		Mutable: INFERENCE_PRECISION_HINT : f16
[ INFO ] 		Mutable: ENABLE_CPU_PINNING : NO
[ INFO ] 		Mutable: DEVICE_ID : 1
[ INFO ] 		Mutable: DYNAMIC_QUANTIZATION_GROUP_SIZE : 0
[ INFO ] 
```

# 7. Run Benchmark Tool to estimate deep learning inference performance on CPU and GPU

- Download Machine learning models from the Intel(R) Deep Learning Streamer Pipeline Zoo.

```shell
# pwd
/root/openvino_cpp_samples_build/intel64/Release
#
# wget https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/resnet50-binary-0001/FP16-INT1/resnet50-binary-0001.bin
#
# wget https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/resnet50-binary-0001/FP16-INT1/resnet50-binary-0001.xml
```

- Run benchmark_app on CPU

```shell
# ./benchmark_app -m resnet50-binary-0001.xml  -niter 1 -d CPU  -hint latency
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.3.0-000--
[ INFO ] 
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2024.3.0-000--
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 28.50 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[Step 6/11] Configuring input of the model
[ INFO ] Model batch size: 1
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 93.51 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   NETWORK_NAME: torch-jit-export
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
[ INFO ]   NUM_STREAMS: 1
[ INFO ]   INFERENCE_NUM_THREADS: 8
[ INFO ]   PERF_COUNT: NO
[ INFO ]   INFERENCE_PRECISION_HINT: f32
[ INFO ]   PERFORMANCE_HINT: LATENCY
[ INFO ]   EXECUTION_MODE_HINT: PERFORMANCE
[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ]   ENABLE_CPU_PINNING: YES
[ INFO ]   SCHEDULING_CORE_TYPE: ANY_CORE
[ INFO ]   MODEL_DISTRIBUTION_POLICY: 
[ INFO ]   ENABLE_HYPER_THREADING: NO
[ INFO ]   EXECUTION_DEVICES: CPU
[ INFO ]   CPU_DENORMALS_OPTIMIZATION: NO
[ INFO ]   LOG_LEVEL: LOG_NONE
[ INFO ]   CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1
[ INFO ]   DYNAMIC_QUANTIZATION_GROUP_SIZE: 32
[ INFO ]   KV_CACHE_PRECISION: f16
[ INFO ]   AFFINITY: CORE
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Test Config 0
[ INFO ] 0  ([N,C,H,W], u8, [1,3,224,224], static):	random (image/numpy array is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 1 inference requests, limits: 1 iterations)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 4.41 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ CPU ]
[ INFO ] Count:               1 iterations
[ INFO ] Duration:            2.85 ms
[ INFO ] Latency:
[ INFO ]    Median:           2.85 ms
[ INFO ]    Average:          2.85 ms
[ INFO ]    Min:              2.85 ms
[ INFO ]    Max:              2.85 ms
[ INFO ] Throughput:          350.90 FPS
```

- Run benchmark_app on GPU

```shell
[root@154fe6015c5b Release]# ./benchmark_app -m resnet50-binary-0001.xml  -niter 1 -d GPU.1  -hint latency
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.3.0-000--
[ INFO ] 
[ INFO ] Device info:
[ INFO ] GPU
[ INFO ] Build ................................. 2024.3.0-000--
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 9.75 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : f32 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 5/11] Resizing model to match image sizes and given batch
[Step 6/11] Configuring input of the model
[ INFO ] Model batch size: 1
[ INFO ] Network inputs:
[ INFO ]     0 (node: 0) : u8 / [N,C,H,W] / [1,3,224,224]
[ INFO ] Network outputs:
[ INFO ]     1463 (node: 1463) : f32 / [...] / [1,1000]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 901.34 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   NETWORK_NAME: torch-jit-export
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 1
[ INFO ]   PERF_COUNT: NO
[ INFO ]   ENABLE_CPU_PINNING: NO
[ INFO ]   MODEL_PRIORITY: MEDIUM
[ INFO ]   GPU_HOST_TASK_PRIORITY: MEDIUM
[ INFO ]   GPU_QUEUE_PRIORITY: MEDIUM
[ INFO ]   GPU_QUEUE_THROTTLE: MEDIUM
[ INFO ]   GPU_ENABLE_LOOP_UNROLLING: YES
[ INFO ]   GPU_DISABLE_WINOGRAD_CONVOLUTION: NO
[ INFO ]   CACHE_DIR: 
[ INFO ]   CACHE_MODE: optimize_speed
[ INFO ]   PERFORMANCE_HINT: LATENCY
[ INFO ]   EXECUTION_MODE_HINT: PERFORMANCE
[ INFO ]   COMPILATION_NUM_THREADS: 32
[ INFO ]   NUM_STREAMS: 1
[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ]   INFERENCE_PRECISION_HINT: f16
[ INFO ]   DEVICE_ID: 1
[ INFO ]   EXECUTION_DEVICES: GPU.1
[ INFO ]   DYNAMIC_QUANTIZATION_GROUP_SIZE: 0
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Test Config 0
[ INFO ] 0  ([N,C,H,W], u8, [1,3,224,224], static):	random (image/numpy array is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 1 inference requests, limits: 1 iterations)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 8.15 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices: [ GPU.1 ]
[ INFO ] Count:               1 iterations
[ INFO ] Duration:            4.43 ms
[ INFO ] Latency:
[ INFO ]    Median:           4.43 ms
[ INFO ]    Average:          4.43 ms
[ INFO ]    Min:              4.43 ms
[ INFO ]    Max:              4.43 ms
[ INFO ] Throughput:          225.89 FPS
```

anolis/intel-openvino

简介

发行版

贡献者

近期动态

anolis/intel-openvino .gitee-modal { width: 500px !important; }

简介

发行版

贡献者

近期动态

搜索帮助

anolis/intel-openvino