Scaling Machine Learning Inference for Multi-Tenant SaaS Use Cases with Amazon SageMaker

Scaling Machine Learning Inference for Multi-Tenant SaaS Use Cases with Amazon SageMaker

Introduction:
Machine learning inference is a crucial aspect of many multi-tenant SaaS applications. However, as the number of models and customer requests increase, optimizing memory utilization and reducing response time become significant challenges. In this article, we will explore how Amazon SageMaker's multi-model features and innovative strategies employed by Zendesk can help scale machine learning inference for multi-tenant SaaS use cases.

Lazy Loading of Models:
One of the key features of SageMaker is the lazy loading of models, where models are loaded into memory when invoked for the first time. This approach optimizes memory utilization but can lead to response time spikes known as the cold start problem. For Zendesk, this posed a challenge, but they overcame it by implementing a preloading functionality on top of the SageMaker endpoint provisioning. This ensures that models are loaded into memory before serving production traffic, effectively reducing response time spikes.

Unloading Infrequently Used Models:
To achieve consistent low latency on all models and prevent "noisy neighbors" from impacting less active models, Zendesk collaborates with AWS to add new features. These features enable more explicit per-model management and address the issue of unloading infrequently used models from memory. Zendesk has also right-sized their multi-model endpoint fleet to minimize the unloading of too many models. As a result, Zendesk can serve predictions to all customers with low latency, around 100 milliseconds, while achieving 90% cost savings compared to dedicated endpoints.

Benefits of Multi-Model Endpoints:
Amazon SageMaker's multi-model endpoints offer a scalable and cost-effective solution for hosting a large number of models on a shared serving container. This approach is ideal for applications with a mix of frequently and infrequently accessed models. By using a shared serving container and the same fleet of resources, multi-model endpoints improve endpoint utilization and reduce hosting costs compared to single-model endpoints. SageMaker manages the loading of models in memory and scales them based on traffic patterns, reducing deployment overhead.

Important Features and Recommendations:
When using multi-model endpoints, several features and recommendations should be considered. These include:

1. AWS PrivateLink and VPCs: Utilize AWS PrivateLink and Virtual Private Clouds (VPCs) for enhanced security and isolation.

2. Auto Scaling: Leverage auto scaling capabilities to dynamically adjust resources based on traffic demands.

3. Serial Inference Pipelines and A/B Testing: Take advantage of serial inference pipelines and A/B testing to improve the efficiency and accuracy of model inference.

Instance Recommendations for Multi-Model Endpoints:
When selecting a SageMaker ML instance type for a multi-model endpoint, certain factors should be considered:

1. Provision Sufficient Amazon EBS Capacity: Ensure that you allocate enough Amazon Elastic Block Store (EBS) capacity to accommodate all the models that need to be served.

2. Balance Performance and Cost: Find the right balance between performance and cost by provisioning an appropriate instance capacity. Avoid over-provisioning, which can lead to unnecessary expenses.

3. Storage Volume Considerations: For containers running in MultiModel mode, the storage volume provisioned for instances is larger than in SingleModel mode. This allows for caching more models on the instance storage volume, enhancing performance.

Conclusion:
Scaling machine learning inference for multi-tenant SaaS use cases requires careful consideration of memory utilization, response time spikes, and cost optimization. By leveraging Amazon SageMaker's multi-model endpoints and implementing innovative strategies like preloading and explicit per-model management, companies like Zendesk can achieve low latency, cost savings, and efficient resource utilization. Incorporating features like AWS PrivateLink, auto scaling, and serial inference pipelines further enhance the scalability and effectiveness of multi-model endpoints. By following these actionable recommendations, organizations can successfully scale their machine learning inference for multi-tenant SaaS applications.