# AtkOps **Repository Path**: ruanlt/AtkOps ## Basic Information - **Project Name**: AtkOps - **Description**: No description available - **Primary Language**: Unknown - **License**: BSD-3-Clause - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 9 - **Created**: 2020-03-10 - **Last Updated**: 2024-06-02 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README EN|[CN](README.zh.md) # High Efficient Development Tool of Customer Operators This document describes how to efficiently develop the operator development tool of the TBE custom operator. The current operator development tool is applicable to the TensorFlow and Caffe operators and runs on the Atlas 300. When the TensorFlow and Caffe models in the industry are converted into models applicable to the Ascend 310 chip, unsupported operators may be encountered. This is the network analysis process. Develop custom operators for operators that are not supported and generate plug-ins for operators. This is the automatic development process of operators and plug-ins. Verify the developed operators by comparing the data generated by the operators on the Ascend 310 with the data generated by the operators on the TensorFlow or Caffe. The custom operator is integrated into the OMG conversion by the developed plug-in, This is the process of plug-in verification. This tool can automatically develop and verify the preceding processes. [TOC] ## Supported Product Atlas 300 (Model 3010) - all modules support Atlas 300 (Model 3000) - operator project and plug-in generation module support ## Supported Version 1.3.2.B893 1.31.T15.B150 1.32.T7.B070 Supported versions are obtained by the following command: ```bash npu-smi info ``` ## Tool Dependency jq, TensorFlow / Caffe, Make, CMake, Python2 ## Directory Structure The directory structure of the operator high efficient development tool is as follows. ```bash . ├── common │ ├── customOp │ ├── davinci_infer │ ├── __init__.py │ ├── op_develop_files │ └── op_verify_files ├── config.json ├── convert2davinci.py ├── env.conf ├── get_caffe_model.py ├── get_tf_model_and_data.py ├── op_verify.py ├── net_verify.py ├── README.md ├── README.zh.md └── run.sh ``` The following table describes the functions of each file or directory. | File/Directory | Function | | :----------------------- | ------------------------------------------------------------ | | customOp | Used to store custom operator templates | | davinci_infer | The resulting C++ code based on the Ascend 310, invoked by the net_verify.py file. You can ignore this file. | | op_develop_files | Invoked by the run.sh file. You can ignore this file. | | op_verify_files | Used to store the single-operator models generated by the get_caffe_model.py and get_tf_model_and_data.py files | | config.json | The configuration file of the tool. You need to modify this file. | | convert2davinci.py | Used for model analysis for generating a list of unsupported operations and their weight parameters, and model conversion | | env.conf | The configuration file of environment variables. C31 DDK users may need to modify this file. (For details, refer to FAQs.) | | get_caffe_model.py | Used to extract the custom operator layer from the Caffe network and generate a single-operator model | | get_tf_model_and_data.py | Used to store the TensorFlow single-operator model generated by users. You need to modify this file. | | op_verify.py | Used to verify the correctness of the operator (single-operator verification) | | net_verify.py | Used to verify the correctness of the operator and plug-in (single-operator network verification) | | run.sh | Used to generate the custom operator project and plug-in | Therefore, this tool consists of six modules: (1) Network analysis module: Perform the model analysis for generating a list of unsupported operations and their parameters and weights, and model conversion. (2) Plug-in automatic generation module: Generate operator projects and plug-ins. (3) Operator verification module: Verify the correctness of operators. (4) Plug-in verification module: Verify plug-ins by using the model conversion. (5) Network operator module (depending on plug-ins): Perform the model conversion and Ascend 310-based inference on a single-operator model based on operators and plug-ins, and verify the correctness of operators and plug-ins at the same time. (6) Custom operator reference template: Prepare a series of implemented custom operator projects for reference. The following describes the process of using modules (1) to (5). ## Usage Process Modules (1) to (5) are independent of each other. You can use some or all of them as required. If you only want to develop operators, go to the second module: operator project and plug-in generation module. You need to set the environment variables for the five modules. That is, change the value of DDK_PATH in the config.json file to the installed DDK path, and then run the following command: ```bash source env.conf ``` For the DDK of the C30 version, run the preceding command directly. However, for the C31 version, users may need to modify the content of the env.conf file (for details, see FAQs). ### Network Model Analysis Module #### Caffe Operator The following describes the automatic process of Caffe network model analysis. **1 Configure the config.json file.** Configure the following fields in the config.json file: - framework Set to caffe. - pycaffe_path Set the Pycaffe path. - net_prototxt Enter the network prototxt path. - net_caffemodel Enter the network caffemodel path. **2 Perform a network OMG conversion.** Run the following command: ```bash python convert2davinci.py 0 ``` 1) After the command is executed, if all operators on the network are supported, the following information is displayed: ``` convert model...... Please wait Model convert success. D Model : /home/xxx ``` D Model indicates the path of the net_prototxt file. 2) After the command is executed, if the OMG operation fails because of the unsupported operator, the following information is displayed: ```bash convert model...... Please wait Model convert failed. Unsupported ops list: xxx xxx You could check all ops in /home/xxx/xxx_ops.txt ``` xxx_ops.txt is a list of unsupported operators. It is stored in the ./common/op_verify_files/caffe_files path. The path also contains the detailed parameters and weights information about the unsupported operators, which are displayed as .json files. Unsupported operators are also displayed on the screen. You can select the custom operator to be implemented, configure it in the config.json file, and create and verify the custom operator project. #### TensorFlow Operator The following describes the automatic process of TensorFlow network model analysis. **1 Configure the config.json file.** Configure the following fields in the config.json file: - framework Set to tensorflow. - net_pb Enter the network pb path. **2 Perform a network OMG conversion.** Run the following command: ```bash python convert2davinci.py 0 ``` 1) After the command is executed, if all operators on the network are supported, the following information is displayed: ``` convert model...... Please wait Model convert success. D Model : /home/xxx ``` D Model indicates the path of the net_pb file. 2) After the command is executed, if the OMG operation fails because of the unsupported operator, the following information is displayed: ```bash convert model...... Please wait Model convert failed. Unsupported ops list: xxx xxx You could check all ops in /home/xxx/xxx_ops.txt ``` xxx_ops.txt is a list of unsupported operators. It is stored in the ./common/op_verify_files/tensorflow_files path. The path also contains the detailed parameters and weights information about the unsupported operators, which are displayed as .json files. Unsupported operators are also displayed on the screen. You can select the custom operator to be implemented, configure it in the config.json file, and create and verify the custom operator project. ### Operator Project and Plug-In Generation Module #### Caffe Operator Project Without Weights The procedure for automatically generating a Caffe operator project is as follows. Step 1 Modify the config.json configuration file. ```bash "operator_name": "custom_Upsample", "framework": "caffe", "DDK_PATH": "/home/mindstudio/tools/che/ddk/ddk", "custom_caffe_proto_file": "/home/mindstudio/caffe.proto", "same_input_output_shape": "False", "input_num": "1", ``` Step 1 sets the custom operator name and operator framework to the Caffe custom operator, configures the DDK path, and customized caffe.proto file path, and sets input_num to 1. Set same_input_output_shape to False, because the input and output shapes are different. The caffe.proto parameter contains the parameter information about the custom operator to be developed. After the preceding information is configured, the operator project is automatically generated. Other parameters do not need to be configured. Step 2 Run bash run.sh or . run.sh to generate an operator project. ```bash . ├── operator │ ├── custom_UpsampleOutputWeightShape.py │ └── custom_Upsample.py └── plugin ├── custom_Upsample_parser_C30.cpp ├── libcaffe_custom_Upsample_layer.so ├── lib_caffe_parser.so ├── Makefile ├── proto │ └── caffe │ ├── caffe.pb.cc │ ├── caffe.pb.h │ ├── caffe.pb.o │ └── caffe.proto └── tmp └── obj └── custom_Upsample_parser_C30.o ``` After the command is executed, you only need to develop the operator file custom_Upsample.py and output weight shape file custom_UpsampleOutputWeightShape.py in the operator directory. The plugin directory does not need to be modified. After all files in the operator directory are developed, you can run . run.sh to compile the plug-in. Step 3 After the operator file is developed, run . run.sh again to compile the plug-in. In addition, if the input and output shapes of the custom operator are the same, set same_input_output_shape in the config.json file to True. In this way, complete compilation of the plug-in can be completed in step 2, and only operator files need to be developed in step 3. After the operator development and plug-in compilation are complete, you can integrate the custom operator into the network for model conversion or operator and plug-in verification (see the following sections). #### Caffe Operator Project with Weights The preceding three steps are also applicable to the operator project with weights. However, the weight information needs to be added to the options configured in step 1. The configured items are as follows. ```bash "operator_name": "custom_Upsample", "framework": "caffe", "DDK_PATH": "/home/mindstudio/tools/che/ddk/ddk", "custom_caffe_proto_file": "/home/mindstudio/caffe.proto", "same_input_output_shape": "False", "input_num": "2", "weight_num": "2", "weight_num_param": [ { "weight_1": "5", "weight_2": "3" } ], "filter_caffe_param": ["min_size", "img_w", "img_size", "flip", "max_size"], ``` In the preceding information, weight_num is set to 2, weight_1 is set to 5 for the shape dimension of the first weight, and weight_2 is set to 3 for the shape dimension of the second weight. This tool is generated based on the custom operator parameter in the caffe.proto file. If there are too many parameters, you can filter the parameters. The parameters to be filtered are configured in the filter_caffe_param field. If the parameter filtering function is not required, you can delete the filter_caffe_param field or do not configure it.　 filter_caffe_param specifies the operator parameters to be filtered. For details, see the .json file generated after the model analysis of the network in the above section. The model analysis can generate a list of operators that are not supported for the Caffe network and provide detailed parameter and weight information about the unsupported operators in .json files. #### TensorFlow Operator Project Step 1 Modify the config.json configuration file. ```bash "operator_name": "batch_matmul", "framework": "tensorflow", "DDK_PATH": "/home/mindstudio/tools/che/ddk/ddk", "same_input_output_shape": "False", "input_num": "2", "tf_op_origin_name": "BatchMatMul", "tensorflow_param": [ { "param": "adj_x", "type": "bool" }, { "param": "adj_y", "type": "bool" } ``` Set tf_op_origin_name to the name of the native operator in TensorFlow, and set tensorflow_param to the parameters contained in the custom operator and the data type of the parameters. Step 2 Run bash run.sh or . run.sh to generate an operator project. ```bash . ├── operator │ ├── batch_matmulOutputWeightShape.py │ └── batch_matmul.py └── plugin ├── batch_matmul_tf_parser.cpp ├── libtf_batch_matmul.so ├── makefile ├── proto │ └── tensorflow │ ├── attr_value.pb.cc │ ├── attr_value.pb.h │ ├── function.pb.cc │ ├── function.pb.h │ ├── graph.pb.cc │ ├── graph.pb.h │ ├── node_def.pb.cc │ ├── node_def.pb.h │ ├── op_def.pb.cc │ ├── op_def.pb.h │ ├── resource_handle.pb.cc │ ├── resource_handle.pb.h │ ├── tensor.pb.cc │ ├── tensor.pb.h │ ├── tensor_shape.pb.cc │ ├── tensor_shape.pb.h │ ├── types.pb.cc │ ├── types.pb.h │ ├── versions.pb.cc │ └── versions.pb.h └── tmp └── obj └── batch_matmul_tf_parser.o ``` You need to develop only the operator file batch_matmul.py and output weight shape file batch_matmulOutputWeightShape.py. If the input and output shapes are the same, set same_input_output_shape to True. In this case, you can perform step 2 to complete the compilation of the plug-in. Otherwise, go to step 3. Step 3 After the operator file is developed, run bash run.sh or . run.sh again to compile the plug-in. In addition, input_num herein may be configured as a positive integer, or may be configured to auto, indicating that the number of inputs is variable, but a specific input shape dimension is fixed to four dimensions. input_num can also be set to autoAll, indicating that the number of input parameters and the dimension of the input shape are not fixed. The operator parameters specified by tensorflow_param can be obtained from the json file generated by the model analysis of the network. The model analysis can generate the list of unsupported operators and provide detailed parameter information about the unsupported operators in .json files. After the operator development and plug-in compilation are complete, you can integrate the custom operator into the network for model conversion or operator and plug-in verification (see the following sections). ### Operator Verification Module This module is used to verify the correctness of the operator file after the operator is developed. #### Caffe Operator The following describes the automatic verification process of the Caffe operator. **1 Configure the config.json file.** Configure the following fields in the config.json file: - framework Set to caffe. - DDK_PATH The installation directory of the DDK - caffe_operator_type The Caffe custom operator type name - pycaffe_path The Pyhon path of the Caffe - net_prototxt The network prototxt path - operator_path The single-operator path - single_operator_run_cfg The single-operator verification configuration precision_deviation indicates the precision deviation, which also indicates the allowed range of the relative error of a single data. The value range is (0, 1). A smaller precision deviation indicates higher precision. statistical_discrepancy indicates the statistics deviation, which also indicates the percentage of the data volume whose precision deviation that does not meet the threshold in the entire data set. The value range is (0, 1). A smaller value indicates higher precision. The tool determines whether the verification is successful based on these two error counters. **2 Generate a single-operator model.** Run the following command: ```bash python get_caffe_model.py ``` The get_caffe_model.py file generates the prototxt file of the single-operator model and the caffemodel file of the corresponding random weight in the common/op_verify_files/caffe_files directory based on the net_prototxt and caffe_operator_type fields of the network. Therefore, to generate the Caffe single-operator model, you need to configure pycaffe_path, net_prototxt, and caffe_operator_type files first. **3 Add a test case to the operator file.** You need to add a test case to the TBE operator file. The parameters of the test case must be consistent with those of the custom operator layer of the single-operator model generated in step 2. The dtype in the parameters must be the same as the dtype of the single_operator_run_cfg. Note that the operator interface parameters and input shape in the test case are obtained from the single-operator model generated in step 3. If the test case does not match the single-operator model, the single-operator verification fails，use the npu-smi tool to restart the chip when needed. **4 Verify the single operator.** Run the following command: ```bash python op_verify.py ``` After the command is executed successfully, the following information is displayed: the output data on the Ascend 310, expected output data on the Caffe, average relative error rate, maximum relative error rate, and whether the precision requirements (precision_deviation and statistical_discrepancy) configured in the config.json file are met. In addition, the single_op_run directory is created in the operator_path directory of the operator, and the result text files on Ascend 310 and Caffe are written to the single_op_run directory. #### TensorFlow Operator The following describes the automatic verification process of the TensorFlow operator. **1 Configure the config.json file.** To perform the single-operator verification on the TensorFlow operator, configure the following fields in the config.json file: - framework Set to tensorflow. - DDK_PATH The installation directory of the DDK - operator_path The operator path - single_operator_run_cfg The single-operator verification configuration **2 Modify the get_tf_model_and_data.py file.** You need to modify the TFGenModelData() interface in the get_tf_model_and_data.py file. This file will be invoked in the op_verify.py file to generate the input and expected output of the TensorFlow operator and the single-operator model. The generation path of a single-operator model is common/op_verify_files/tensorflow_files/. You need to configure the get_tf_model_and_data.py file including: 1) Enter the number of tensors and shapes. ```python # Enter the number of tensors and shapes. shape_x = (7, 2, 3, 4) shape_y = (7, 2, 3, 4) x = tf.placeholder(tf.float32, shape=shape_x, name='x') y = tf.placeholder(tf.float32, shape=shape_y, name='y') ``` 2) Modify the function of the TensorFlow operator. ```python # Modify the function of the TensorFlow operator. name = "pow" op = tf.pow(x, y, name=name) ``` 3) Configure the input data (usually random data). ```python # Configure the input data (usually random data). input_x = np.random.randint(1, 5, size=shape_x).astype(np.float32, copy=False) - 8 input_y = np.random.randint(1, 5, size=shape_y).astype(np.float32, copy=False) ``` 4) Modify the return statement based on the input and output quantity. ```python # In this example, there are two inputs and one output. return [input_x, input_y], [expect,] ``` Note that this tool supports only single-output operators. **3 Add a test case to the operator file.** You need to add a test case to the TBE operator file. The parameters of the test case must be consistent with those of the TFGenModelData() interface in step 2. The dtype in the parameters must be the same as the dtype of the single_operator_run_cfg. Note that the operator interface parameters and input shape in the test case are obtained from the single-operator model generated in step 2. If the test case does not match the single-operator model, the single-operator verification fails. **4 Verify the single operator.** Run the following command: ```bash python op_verify.py ``` After the command is executed successfully, the following information is displayed: the output data on the Ascend 310 and the TensorFlow, average relative error rate, maximum relative error rate, and whether the precision requirements (precision_deviation and statistical_discrepancy) configured in the config.json file are met. In addition, the single_op_run directory is created in the operator_path directory of the operator, and the result text files on Ascend 310 and TensorFlow are written to the single_op_run directory. ### Plug-In Verification Module In the plug-in verification, the operators and plug-ins are developed by default. If the operator verification is complete, configure plugin_path in the config.json file, and then run the following command: ``` python convert2davinci.py 1 ``` If the operator verification is not complete, perform the following steps for the plug-in verification: #### Caffe Operator The following describes the automatic verification process of the Caffe operator and plug-in. **1 Configure the environment variable.** Run the following command to configure the environmental variable: ```bash source env.conf ``` This environment variable can be used for the operator and plug-in verification. **2 Configure the config.json file.** To perform the single-operator verification on the Caffe operator, the operator and plug-in files have been developed by default and can be configured in the config.json file. The fields to be configured are as follows. - framework Set to caffe. - DDK_PATH Installation directory of the DDK, which needs to be configured to $home/tools/che/ddk/ddk. - caffe_operator_type The Caffe custom operator type name - pycaffe_path The Python path of the Caffe - net_prototxt The network prototxt path - operator_path The single-operator path - plugin_path The single-operator plug-in path - single_operator_run_cfg The single-operator verification configuration **3 Generate a single-operator model.** Run the following command: ```bash python get_caffe_model.py ``` The get_caffe_model.py file generates the prototxt file of the single-operator model and the caffemodel file of the corresponding random weight in the common/op_verify_files/caffe_files directory based on the net_prototxt and caffe_operator_type fields of the network. **4 Verify the single operator and plug-in.** Run the following command: ```bash python convert2davinci.py 1 ``` The input parameter is set to 1, which indicates that the convert2davinci.py file will be used for the OMG conversion of the single-operator model that is integrated by the custom operator. The converted model is stored in common/op_verify_files/caffe_files/ and the path of the OM file is displayed. #### TensorFlow Operator The following describes the automatic verification process of the TensorFlow operator and plug-in. **1 Configure the environment variable.** Run the following command to configure the environmental variable: ```bash source env.conf ``` **2 Configure the config.json file.** To perform the single-operator verification on the TensorFlow operator, configure the following fields in the config.json file: - framework Set to tensorflow. - DDK_PATH The installation directory of the DDK - operator_path The operator path - plugin_path The single-operator plug-in path - single_operator_run_cfg The single-operator verification configuration **3 Modify the get_tf_model_and_data.py file.** Run the following command to generate a single-operator model: ```shell python get_tf_model_and_data.py ``` You need to modify the TFGenModelData() interface in the get_tf_model_and_data.py file. This file will be invoked in the op_verify.py file to generate the input and expected output of the TensorFlow operator and the single-operator model. The generation path of a single-operator model is common/op_verify_files/tensorflow_files/. **5 Verify the single operator and plug-in.** Run the following command: ```bash python convert2davinci.py 1 ``` After the command is successfully executed, the converted model is stored in common/op_verify_files/caffe_files/ and the path of the OM file is displayed. ### Operator Network-Wide (Depending on Plug-Ins) Verification Module This tool is used to verify the correctness of the custom operator and plug-in after the operator and plug-in are developed. This module performs the following three steps: (1) Generate a single-operator model for the Caffe or TensorFlow. For the Caffe operator, the get_caffe_model.py file needs to be invoked to generate the prototxt of the single-operator model and the caffemodel of the corresponding random weight in the common/op_verify_files/caffe_files directory. For the TensorFlow operator, the get_tf_model_and_data.py file needs to be invoked (you need to modify this file) to generate the .pb file of the single-operator model in the common/op_verify_files/tensorflow_files/ directory. (2) Perform the OMG model conversion for the single-operator model of (1) based on operators and plug-ins to generate the OM model. (3) Construct random data, send the data to the OM model and Caffe/TensorFlow model, print the results of the two models, and obtain the relative error. You need to compile davinci_infer for both the Caffe and TensorFlow operators. **1 Compile davinci_infer**. In the common/davinci_infer directory, run the following command: ```bash sh build.sh ``` #### Caffe Operator After the preceding steps are performed, perform the following steps to verify the network (depending on plug-ins) of the Caffe operator: **1 Configure the config.json file.** To perform the single-operator verification on the Caffe operator, configure the following fields in the config.json file: - framework Set to caffe. - DDK_PATH The installation directory of the DDK - caffe_operator_type The Caffe custom operator type name - pycaffe_path The Pyhon path of the Caffe - net_prototxt The network prototxt path - single_operator_run_cfg The single-operator verification configuration You only need to set precision_deviation and statistical_discrepancy. - plugin_path The operator plug-in path. Note that the plug-in must have been compiled. **2 Generate a single-operator model.** Run the following command: ```bash python get_caffe_model.py ``` The get_caffe_model.py file generates the prototxt file of the single-operator model and the caffemodel file of the corresponding random weight in the common/op_verify_files/caffe_files directory based on the net_prototxt and caffe_operator_type fields of the config.json file. **3 Single-operator model OMG, Da Vinci inference and verification** Run the following command: ``` python net_verify.py ``` After the command is successfully executed, the single-operator model (prototxt and caffemodel generated in step 2) is model-converted in common/op_verify_files/caffe_files, and an OM model with the same name as caffe_operator_type is generated in this directory. Then, random input is constructed and sent to the OM and Caffe models for inference. The output of the OM and Caffe models and the average and maximum error rates of the two models are printed on the console. The net_verify directory is created in the operator project, and the outputs of the OM and Caffe models are written to this directory. #### TensorFlow Operator After configuring the environment variable and compiling davinci_infer, perform the following steps to verify the TensorFlow operator on the network (depending on plug-ins): **1 Configure the config.json file.** Configure the following fields in the config.json file: - framework Set to tensorflow. - DDK_PATH The installation directory of the DDK - single_operator_run_cfg The single-operator verification configuration You only need to set precision_deviation and statistical_discrepancy. - plugin_path The operator plug-in path. Note that the plug-in must have been compiled. **2 Modify the get_tf_model_and_data.py file.** This file is used to generate the input and expected output of the TensorFlow operator and the pb model of a single operator. You need to modify the following information: 1) Enter the number of tensors and shapes. ```python # Enter the number of tensors and shapes. shape_x = (7, 2, 3, 4) shape_y = (7, 2, 3, 4) x = tf.placeholder(tf.float32, shape=shape_x, name='x') y = tf.placeholder(tf.float32, shape=shape_y, name='y') ``` 2) Modify the function of the TensorFlow operator. ```python # Modify the function of the TensorFlow operator. name = "pow" op = tf.pow(x, y, name=name) ``` 3) Configure the input data (usually random data). ```python # Configure the input data (usually random data). input_x = np.random.randint(1, 5, size=shape_x).astype(np.float32, copy=False) - 8 input_y = np.random.randint(1, 5, size=shape_y).astype(np.float32, copy=False) ``` 4) Modify the return statement based on the input and output quantity. ```python # In this example, there are two inputs and one output. return [input_x, input_y], [expect,] ``` Note that this tool supports only single-output operators. **4 Single-operator model OMG, Da Vinci inference and verification** Run the following command: ``` python net_verify.py ``` After the command is successfully executed, the single-operator pb model is model-converted in common/op_verify_files/tensorflow_files, and an OM model with the same name as caffe_operator_type is generated in this directory. Note that this will first clear the original pb and OM models in common/op_verify_files/tensorflow_files. Then, random input is constructed and sent to the OM model and pb model for inference. The output of the OM model and pb model, and the average and maximum error rates between the two models are displayed on the console. In addition, the net_verify directory is created in the operator project, write the output of the OM and pb models to this directory. ### Demo 1. Provide a network or use the get_tf_model_and_data.py file to generate a single-operator model to replace the network. Modify the file as follows. ```bash import os import numpy as np import tensorflow as tf def TFGenModelData(gen_pb_model=True): os.environ["CUDA_VISIBLE_DEVICES"] = '' with tf.Session(graph=tf.Graph()) as sess: shape_x = (7, 2, 6, 9) shape_y = (7, 2, 9, 6) x = tf.placeholder(tf.float32, shape=shape_x, name='x') y = tf.placeholder(tf.float32, shape=shape_y, name='y') name = "batch_matmul" op = tf.matmul(x, y, name=name) input_x = np.random.randint(1, 5, size=shape_x).astype(np.float32, copy=False) - 8 input_y = np.random.randint(1, 5, size=shape_y).astype(np.float32, copy=False) feed_dict = {x: input_x, y: input_y} sess.run(tf.global_variables_initializer()) expect = sess.run(op, feed_dict) if gen_pb_model: pwd = os.getcwd() os.chdir("./common/op_verify_files/tensorflow_files") for filename in os.listdir("./"): if filename.endswith(".om") or filename.endswith(".pb"): os.remove(filename) graph = tf.compat.v1.graph_util.convert_variables_to_constants( sess, sess.graph_def, [name]) with tf.gfile.FastGFile('tf_' + name + '.pb', mode='wb') as f: f.write(graph.SerializeToString()) os.chdir(pwd) return [input_x, input_y], [expect, ] if __name__ == "__main__": TFGenModelData() ``` 2. **Execute the python get_tf_model_and_data.py command.** Obtain the following files in ./common/op_verify_files/tensorflow_files: ```bash . ├── README.md ├── README.zh.md └── tf_batch_matmul.pb ``` This section uses the single-operator model tf_batch_matmul.pb as an example to describe the model analysis procedure. 3. [Model Analysis] Configure two fields of the config.json file. ```bash "framework": "tensorflow", "net_pb": "./common/op_verify_files/tensorflow_files/tf_batch_matmul.pb", ``` **Execute the python convert2davinci.py 0 command.** Analyze the model configured in net_pb. The following information is displayed: ```bash Unsupported ops list: BatchMatMulV2 ``` Obtain the following files in ./common/op_verify_files/tensorflow_files: ```bash . ├── README.md ├── README.zh.md ├── support_tf_batch_matmul.json ├── tf_batch_matmul_ops.txt ├── tf_batch_matmul.pb └── unsupport_tf_batch_matmul.json ``` tf_batch_matmul.pb is a model for analyzing the network. unsupport_tf_custom_batch_matmul.json lists the operators that are not supported by the network and their parameters. ```bash [ { "tensorflow_param": [ { "default": "false", "type": "bool", "param": "adj_y" }, { "default": "false", "type": "bool", "param": "adj_x" } ], "tf_op_origin_name": "BatchMatMulV2" } ``` In addition, the unsupported operator BatchMatMulV2 and its two parameters adj_x and adj_y in the tf_batch_matmul.pb model are obtained. **At this point, the model analysis module of the network is complete. ** The BatchMatMulV2 operator is the unsupported TensorFlow operator after OMG conversion. The BatchMatMulV2 operator is planned to be developed based on the TBE and named as custom_batch_matmul. 4. [Development operators] Add or modify the following fields in the config.json file: ```bash "operator_name": "custom_batch_matmul", "DDK_PATH": "/home/xx/C31/ddk/ddk/", "same_input_output_shape": "False", "input_num": "2", "tf_op_origin_name": "BatchMatMul", "tensorflow_param": [ { "param": "adj_x", "type": "bool" }, { "param": "adj_y", "type": "bool" } ], "project_path": "/home/xxx/mindstudio", ``` **Execute the bash run.sh command.** Obtain the custom_batch_matmul operator project in /home/xxx/mindstudio. ```bash ├── operator │ ├── custom_batch_matmulOutputWeightShape.py │ └── custom_batch_matmul.py └── plugin ├── custom_batch_matmul_tf_parser.cpp ├── libtf_custom_batch_matmul.so ├── makefile ├── proto └── tmp ``` Develop the operator file custom_batch_matmul.py. Refer to: ./common/customOp/tensorflowOp/custom_batch_matmul/operator/custom_batch_matmul.py If same_input_output_shape is set to False, the input and output shapes are different. Therefore, the output shape logic needs to be developed, which is obtained from the code in the custom_batch_matmul.py file. The custom_batch_matmulOutputWeightShape.py file limits the output of shape in four dimensions. The output shape logic is described as follows. ```python def OutputShapecustom_batch_matmul(shape_1, shape_2, adj_x, adj_y): """ TODO: Please add code here to obtain the output shape. """ if not adj_x and not adj_y: output_shape = (shape_1[0], shape_1[1], shape_1[2], shape_2[3]) elif not adj_x and adj_y: output_shape = (shape_1[0], shape_1[1], shape_1[2], shape_2[2]) elif adj_x and not adj_y: output_shape = (shape_1[0], shape_1[1], shape_1[3], shape_2[3]) else: output_shape = (shape_1[0], shape_1[1], shape_1[3], shape_2[2]) return output_shape ``` After the operator file and operator output shape logic file are developed, go back to the root directory of the tool and perform the operator verification and plug-in verification. To perform the plug-in verification, you need to generate the single-operator model of the operator. Therefore, you need to modify the get_tf_model_and_data.py file, which has been modified in step 1. 5. [Plug-in verification] Add the following fields to the config.json file: ```bash "plugin_path": "/home/xxx/mindstudio/custom_batch_matmul/plugin" ``` **Execute the source env.conf command.** **Execute the python convert2davinci.py 1 command.** If the following information is displayed, the model conversion is successful: ```bash Model parsing is complete. (1/4) Graph optimization is complete. (2/4) Model building is complete. (3/4) Offline model saving is complete. (4/4) OMG generate offline model success. Model convert success. ``` At this point, the correctness of the plug-in is verified by using the OMG model transformation. 6. [Operator verification] Add the following fields to the config.json file: ```bash "operator_path": "/home/xxx/mindstudio/custom_batch_matmul/operator/custom_batch_matmul.py", "single_operator_run_cfg": { "dtype": "float32", "precision_deviation": "0.2", "statistical_discrepancy": "0.2" } ``` Add a test case to the end of the operator file. ```python if __name__ == "__main__": shape_1 = (7, 2, 6, 9) shape_2 = (7, 2, 9, 6) adj_x = False adj_y = False dtype = "float32" custom_batch_matmul(shape_1, shape_2, dtype, adj_x, adj_y, need_build = True) ``` The value of dtype in the test case must be the same as the value of dtype in the config.json file. The values of shape_1 and shape_2 in the test case must be the same as the data in the single-operator model, that is, the data in the get_tf_model_and_data.py file. **Execute the python op_verify.py command.** The running data and comparison result of the operator (independent of plug-ins) in Ascend 310 and TensorFlow are displayed. 7. [Verification of the operator that depends on the plug-in] Compile the operator in the ./common/davinci_infer directory. This module supports only four-dimensional input and output. **Execute the bash build.sh command.** ``` - do [do_pre_build] [CC] src/out/visual_infer_main.o [CC] src/out/host/data_recv.o [CC] src/out/host/visual_infer_host.o [CC] src/out/host/util.o [CC] src/out/host/raw_data_host.o [CC] src/out/host/raw_data_mutil_input_host.o [CC] src/out/device/sample_data.o [CC] src/out/config_parser/config_parser.o [LD] src/out/DavinciInfer - do [do_build] make success copy success ``` The preceding correct compilation result is displayed. After the inference module is compiled, go back to the root directory of the tool. **Execute the python net_verify.py command.** The comparison results of the operator (depending on the plug-in) in Ascend 310 and TensorFlow are displayed. In addition, if the operator output shape is not 4-dimensional, you can modify the plug-in code. If you want to develop the output shape logic in the plug-in, set same_input_output_shape to True and modify the plug-in. At this point, the operator and plug-in verification module are completed. ## FAQ (1) When I use the operator verification module, the system displays RuntimeError: ('compile cce error :', OSError(2, 'No such file or directory')). Or when I use operator network module (depending on plug-ins) to perform an inference on the Da Vinci model, the system displays a message indicating that the permission is insufficient. You can run the tool as the root or HwHiAiUser user. Another method is to add the current user to the HwHiAiUser group. Then switch the default user group of the current user to HwHiAiUser. After the preceding operations, the current user can perform an inference on the Da Vinci model. (2) The DDK of the C31 version compilation tool fails. This is because the DDK of the C31 version divides a separate lib library. The lib library can be decompressed to any path, and this tool needs to use the .so file in the lib library. Therefore, for the C31 version, you need to modify the NPU_HOST_LIB and NPU_DEVICE_LIB environment variables in the env.conf file, point them to the correct path, and then execute the source env.conf file. (3) What fields are contained in the config.json configuration file? What should I pay attention to? The following describes all the configuration fields in the config.json file and the precautions. ```None { "operator_name": Set to the name of the Caffe/TensorFlow operator. "framework": Set only to tensorflow or caffe. "DDK_PATH": Configure the DDK path. For details, see /home/muser/tools/che/ddk/ddk. "custom_caffe_proto_file": Set to the path of the caffe.proto file that contains custom operators. "same_input_output_shape": Set only to False or True. "input_num": Set only to positive integers such as 2, auto and autoAll. "weight_num": Set only to non-negative integers, including 0. This field can be deleted and only the Caffe operator is supported. "weight_num_param" [ No weight, it can be deleted. Only the Caffe operator is supported. "weight_1": Set only to non-negative integers. This field can be deleted. "weight_2": Set only to non-negative integers. This field can be deleted. } ] "filter_caffe_param": Filter the parameters of the Caffe operator in caffe.proto. "tf_op_origin_name": Indicates the operator name in TensorFlow. "tensorflow_param": [ No parameter, it can be deleted. { "param": The first parameter name of the operator "type": The data type of the first parameter of the operator }， { "param": The second parameter name of the operator "type": The data type of the second parameter of the operator } ] "project_path": Specify the path for generating the operator project. This field can be deleted. (By default, the operator project is generated in the root directory of the tool.) "caffe_operator_type": Indicates the operator name in Caffe, which is used for operator verification and data verification. - pycaffe_path Set the Pycaffe path. "net_prototxt": Indicates the path of the prototxt file on the Caffe network, which is used for model analysis on the network. "net_caffemodel": Indicates the path of the caffemodel file on the Caffe network, which is used for model analysis on the network. "net_pb": Indicates the path of the .pb file on the TensorFlow network, which is used for model analysis on the network. "operator_path": Indicates the Caffe/TensorFlow operator file path, which is used for operator verification that does not depend on plug-ins. "plugin_path": Indicates the Caffe/TensorFlow plug-in file path, which is used for operator verification and operator verification on the network. "single_operator_run_cfg": { Indicates the configuration parameters of the operator verification and operator network verification modules. "dtype": "precision_deviation": "statistical_discrepancy": } } ```