How to Serve PyTorch Models with TorchServe

TL;DR

Torserve is a performant and scalable model serving solution for PyTorch, allowing developers to integrate and deploy their trained models for inference through an HTTP or HTTPS API.

Transcript

welcome to pytorch 2021 hackathon my name is hamid i'm on partner engineering team from pytorch at facebook ai my job is to help ml developers like you to improve their solutions with pytorch using its best and greatest features i'm also one of the co-maintainers of twitch serve today i'm excited to tell you about torsor and give you some example o... Read More

Key Insights

• 🚂 Model serving, integrating trained models into larger systems, and making them available for inferences is an important step in machine learning development.
• 🛟 Torserve is the native model serving solution for PyTorch, offering performance and scalability for wrapping PyTorch models in HTTP or HTTPS APIs.
• 😒 Torserve provides default handlers and the flexibility to write custom handlers to accommodate various applications and use cases.
• 😑 It supports pre-trained models from third-party libraries and offers various performance optimizations for advanced users.
• 😶‍🌫️ Torserve has integrations with Kubeflow, MLflow, and major cloud providers like AWS, Google Cloud, and Azure.

Questions & Answers

Q: What is Torserve?

Torserve is a model serving solution for PyTorch that allows developers to integrate trained models into larger systems and expose them through an HTTP or HTTPS API for running inferences.

Q: What is a handler in Torserve?

A handler in Torserve is a Python script that includes all the model initialization, pre-processing, inference, and post-processing code. It can be a default handler provided by Torserve or a custom handler created by the developer.

Q: Can I use pre-trained models from third-party libraries with Torserve?

Yes, Torserve supports the use of pre-trained models from third-party libraries like Hugging Face. You can write a custom handler that fits your needs and leverages the pre-trained model for inference.

Q: Does Torserve support model scaling and management?

Yes, Torserve provides APIs for registering, unregistering, and scaling up or down the number of workers for your model. It also offers an inference API management API for configuring and monitoring the model serving process.

Key Insights:

• Model serving, integrating trained models into larger systems, and making them available for inferences is an important step in machine learning development.
• Torserve is the native model serving solution for PyTorch, offering performance and scalability for wrapping PyTorch models in HTTP or HTTPS APIs.
• Torserve provides default handlers and the flexibility to write custom handlers to accommodate various applications and use cases.
• It supports pre-trained models from third-party libraries and offers various performance optimizations for advanced users.
• Torserve has integrations with Kubeflow, MLflow, and major cloud providers like AWS, Google Cloud, and Azure.
• Torserve is used in production by companies like Metroid, Toyota Research Institute, and Dynabench platform for research and benchmarking purposes.

Summary & Key Takeaways

• PyTorch partner engineering team member explains Torserve, a model serving solution for integrating trained models into larger systems and making them available for running inferences.

• Torserve offers default handlers for various applications such as image classification, segmentation, object detection, and text classification, along with the possibility to create custom handlers.

• The video demonstrates how to install and use Torserve, showcasing examples of image classification and NLP using pre-trained models and custom handlers.