TorchServe is a flexible and easy to use tool for serving PyTorch models.
For full documentation, see Model Server for PyTorch Documentation.
Conda instructions are provided in more detail, but you may also use pip
and virtualenv
if that is your preference.
Note: Java 11 is required. Instructions for installing Java 11 for Ubuntu or macOS are provided in the Install with Conda section.
To use pip
to install TorchServe and the model archiver:
pip install torch torchtext torchvision sentencepiece
pip install torchserve torch-model-archiver
Ubuntu
sudo apt-get install openjdk-11-jdk
conda create --name torchserve torchserve torch-model-archiver pytorch torchtext torchvision -c pytorch -c powerai
conda create --name torchserve torchserve torch-model-archiver pytorch torchtext torchvision cudatoolkit=10.1 -c pytorch -c powerai
source activate torchserve
macOS
brew tap AdoptOpenJDK/openjdk
brew cask install adoptopenjdk11
conda create --name torchserve torchserve torch-model-archiver pytorch torchtext torchvision -c pytorch -c powerai
source activate torchserve
Now you are ready to package and serve models with TorchServe.
If you plan to develop with TorchServe and change some of the source code, you must install it from source code. First, clone the repo with:
git clone https://github.com/pytorch/serve
cd serve
Then make your changes executable with this command:
pip install -e .
cd serve/model-archiver
pip install -e .
pip install -U -e .
For information about the model archiver, see detailed documentation.
This section shows a simple example of serving a model with TorchServe. To complete this example, you must have already installed TorchServe and the model archiver.
To run this example, clone the TorchServe repository and navigate to the root of the repository:
git clone https://github.com/pytorch/serve.git
cd serve
Then run the following steps from the root of the repository.
To serve a model with TorchServe, first archive the model as a MAR file. You can use the model archiver to package a model. You can also create model stores to store your archived models.
Create a directory to store your models.
mkdir ~/model_store
cd ~/model_store
Download a trained model.
wget https://download.pytorch.org/models/densenet161-8d451a50.pth
Archive the model by using the model archiver. The extra-files
param uses fa file from the TorchServe
repo, so update the path if necessary.
torch-model-archiver --model-name densenet161 --version 1.0 --model-file ~/serve/examples/image_classifier/densenet_161/model.py --serialized-file ~/model_store/densenet161-8d451a50.pth --extra-files ~/serve/examples/image_classifier/index_to_name.json --handler image_classifier
For more information about the model archiver, see Torch Model archiver for TorchServe
After you archive and store the model, use the torchserve
command to serve the model.
torchserve --start --model-store ~/model_store --models ~/model_store/densenet161=densenet161.mar
After you execute the torchserve
command above, TorchServe runs on your host, listening for inference requests.
Note: If you specify model(s) when you run TorchServe, it automatically scales backend workers to the number equal to available vCPUs (if you run on a CPU instance) or to the number of available GPUs (if you run on a GPU instance). In case of powerful hosts with a lot of compute resoures (vCPUs or GPUs). This start up and autoscaling process might take considerable time. If you want to minimize TorchServe start up time you avoid registering and scaling the model during start up time and move that to a later point by using corresponding Management API, which allows finer grain control of the resources that are allocated for any particular model).
To test the model server, send a request to the server's predictions
API.
Complete the following steps:
curl
to download one of these cute pictures of a kitten
and use the -o
flag to name it kitten.jpg
for you.curl
to send POST
to the TorchServe predict
endpoint with the kitten's image.The following code completes all three steps:
curl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg
curl -X POST http://127.0.0.1:8080/predictions/densenet161 -T kitten.jpg
The predict endpoint returns a prediction response in JSON. It will look something like the following result:
[
{
"tiger_cat": 0.46933549642562866
},
{
"tabby": 0.4633878469467163
},
{
"Egyptian_cat": 0.06456148624420166
},
{
"lynx": 0.0012828214094042778
},
{
"plastic_bag": 0.00023323034110944718
}
]
You will see this result in the response to your curl
call to the predict endpoint, and in the server logs in the terminal window running TorchServe. It's also being logged locally with metrics.
Now you've seen how easy it can be to serve a deep learning model with TorchServe! Would you like to know more?
To stop the currently running TorchServe instance, run the following command:
torchserve --stop
You see output specifying that TorchServe has stopped.
git clone https://github.com/pytorch/serve.git
cd serve
The following are examples on how to use the build_image.sh
script to build Docker images to support CPU or GPU inference.
To build the TorchServe image for a CPU device using the master
branch, use the following command:
./build_image.sh
To create a Docker image for a specific branch, use the following command:
./build_image.sh -b <branch_name>
To create a Docker image for a GPU device, use the following command:
./build_image.sh --gpu
To create a Docker image for a GPU device with a specific branch, use following command:
./build_image.sh -b <branch_name> --gpu
To run your TorchServe Docker image and start TorchServe inside the container with a pre-registered resnet-18
image classification model, use the following command:
./start.sh
We welcome all contributions!
To learn more about how to contribute, see the contributor guide here.
To file a bug or request a feature, please file a GitHub issue. For filing pull requests, please use the template here. Cheers!
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。