Distributed inference means use multiple devices for prediction. If data parallel or integrated save is used in training, the method of distributed inference is same with the Single-card inference. It is noted that each device should load one same checkpoint file.
This tutorial would focus on the process that the model slices are saved on each device in the distributed training process, and the model is reloaded according to the predication strategy in the inference stage. In view of the problem that there are too many parameters in the super large scale neural network model, the model can not be fully loaded into a single device for inference, so multiple devices can be used for distributed inference.
Download Distributed inference sample code.
Execute training, and generate the checkpoint file and the model strategy file.
- The distributed training tutorial and sample code can be referred to Distributed Parallel Training Example (Ascend).
- In the distributed Inference scenario, during the training phase, the
integrated_save
ofCheckpointConfig
interface should be set toFalse
, which means that each device only saves the slice of model instead of the full model.parallel_mode
ofset_auto_parallel_context
interface should be set toauto_parallel
orsemi_auto_parallel
. The parallel mode is either auto parallel or semi-automatic parallelism.- In addition, you need to specify
strategy_ckpt_save_file
to indicate the path of the strategy file.- If pipeline distributed inference is used, then the pipeline parallel training also must be used. And the
device_num
andpipeline_stages
used for pipeline training and inference must be the same. While applying pipeline inference,micro_size
is 1 and there is no need to callPipelineCell
. Please refer to Pipeline Parallel.
Set context and infer inference strategy according to the inference data.
set_auto_parallel_context(full_batch=True, parallel_mode='semi_auto_parallel', strategy_ckpt_load_file='./train_strategy.ckpt')
network = Net()
model = Model(network)
predict_data = create_predict_data()
predict_strategy = model.infer_predict_layout(predict_data)
In the inference data:
full_batch
: whether to load the dataset in full or not. When True
, it indicates full load, and data of each device is the same. It must be set to True
in this scenario.parallel_mode
: parallel mode, it must be auto_parallel
or semi_auto_parallel
.strategy_ckpt_load_file
: file path of the strategy generated in the training phase, which must be set in the distributed inference scenario.create_predict_data
: user-defined interface that returns predication data whose type is Tensor
.infer_predict_layout
: generates predication strategy based on predication data.Load checkpoint files, and load the corresponding model slice into each device based on the inference strategy.
ckpt_file_list = create_ckpt_file_list()
load_distributed_checkpoint(network, ckpt_file_list, predict_strategy)
In the preceding information:
create_ckpt_file_list
:user-defined interface that returns a list of checkpoint file path in order of rank id.load_distributed_checkpoint
:merges model slices, then splits it according to the predication strategy, and loads it into the network.For pipeline inference, each
stage
only needs to load the checkpoint file of self_stage.The
load_distributed_checkpoint
interface supports that predict_strategy isNone
, which is single device inference, and the process is different from distributed inference. The detailed usage can be referred to the reference link.
Execute inference and get the inference result.
model.predict(predict_data)
When the super-large-scale neural network model has too many parameters, the MindIR format model cannot be completely loaded into a single card for inference. Multi-card should be used for distributed inference. In this case, multiple MindIR files need to be exported before the inference task. The specific methods are as follows:
First, you need to prepare checkpoint files and training strategy files.
The checkpoint file is generated during the training process. For the detailed use of checkpoint, refer to Use CheckPoint
The training strategy file needs to be generated by setting the context during training. The context configuration items are as follows:
set_auto_parallel_context(strategy_ckpt_save_file='train_strategy.ckpt')
In this way, after training, a training strategy file named train_strategy.ckpt
will be generated in the set directory.
Before exporting the MindIR file, it is necessary to load the checkpoint file, the distributed training checkpoint file, the training strategy and the reasoning strategy need to be combined, so the reasoning strategy file can be generated. The code to generate the inference strategy is as follows:
predict_strategy = model.infer_predict_layout(predict_data)
Then, use the method of loading distributed checkpoints to load the previously trained parameters to the network. Code show as below:
load_distributed_checkpoint(model, ckpt_file_list, predict_strategy)
Finally, you can export the MindIR file in the distributed inference scenario.
The core code is as follows:
# Configure the strategy file generated during the training process in the context
set_auto_parallel_context(strategy_ckpt_load_file='train_strategy.ckpt')
# Define network structure
network = Net()
model = Model(network)
# Get the reasoning strategy file
predict_strategy = model.infer_predict_layout(predict_data)
# Create checkpoint list
ckpt_file_list = create_ckpt_file_list()
# Load distributed parameters
load_distributed_checkpoint(model, ckpt_file_list, predict_strategy)
# Export distributed MindIR file
export(net, Tensor(input), file_name='net', file_format='MINDIR')
In the case of multi-card training and single-card inference, the usage of exporting MindIR is the same as that of single host.
Distributed scenario exports MindIR file sample code: distributed_export
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。