Scaling Machine Learning Inference and Tenant Management in Multi-Tenant SaaS Environments

Scaling Machine Learning Inference and Tenant Management in Multi-Tenant SaaS Environments

Introduction:
In the fast-paced world of software as a service (SaaS), companies are constantly innovating to provide efficient and scalable solutions to their customers. Two key areas that require careful consideration in SaaS offerings are scaling machine learning inference and tenant management. In this article, we will explore how companies like Zendesk and CyberArk have tackled these challenges using Amazon Web Services (AWS) and discuss actionable advice for implementing similar solutions in your own SaaS platform.

Scaling Machine Learning Inference for Multi-Tenant SaaS:
One of the common challenges faced by SaaS companies is optimizing memory utilization while ensuring low response times for machine learning inference. Amazon SageMaker, a popular ML platform offered by AWS, provides a multi-model feature called lazy loading. This feature allows models to be loaded into memory only when invoked for the first time, optimizing memory usage. However, this can result in response time spikes, commonly known as the cold start problem.

Zendesk, a customer service software provider, encountered this issue and overcame it by implementing a preloading functionality on top of the SageMaker endpoint provisioning. This approach loads the models into memory before serving production traffic, effectively reducing response time spikes. Additionally, Zendesk collaborates with AWS to add new features for more explicit per-model management, ensuring consistent low latency for all models and preventing "noisy neighbors" from impacting less active models. Right-sizing the multi-model endpoint fleet further minimizes model unloading, enabling Zendesk to serve predictions to its customers with low latency, around 100 milliseconds, while achieving significant cost savings compared to dedicated endpoints.

Tenant Management in SaaS Environments:
Tenant management is another critical aspect of SaaS offerings, as it involves creating and managing isolated environments for each customer or tenant. CyberArk, a leader in privileged access security, built a robust tenant management service for its SaaS offering using AWS services.

The tenant creation process at CyberArk involves several steps. A delivery team member authenticates to CyberArk's identity provider (IdP) and receives a JSON Web Token (JWT) with claims that describe their permissions. The team member then sends a request to the Amazon API Gateway REST API, providing the relevant tenant configuration details. The API Gateway triggers a Lambda function, which validates the authentication and authorization based on the JWT claims. This Lambda function generates a unique tenant ID, stores the configuration details in an Amazon DynamoDB table, and sets the tenant status to "create in progress." The tenant creation event triggers an AWS Lambda function called the "dispatcher," which utilizes AWS Step Functions for orchestration.

AWS Step Functions provides a visual workflow service that simplifies the orchestration of tasks and supports timeout and wait functionality. In the context of tenant management, the Step Functions workflow waits for all business services to finish the tenant creation process and responds with a success or failure status. This approach ensures a streamlined and reliable tenant creation process for CyberArk's SaaS platform.

Actionable Advice for Scaling Machine Learning Inference and Tenant Management:

1. Optimize Memory Utilization: Implement lazy loading of models in your machine learning infrastructure to optimize memory usage. However, be aware of the potential cold start problem and consider preloading models to reduce response time spikes.

2. Collaborate with Cloud Providers: Work closely with cloud providers, like AWS, to leverage their expertise and explore new features or enhancements that can improve per-model management, ensure consistent low latency, and mitigate the impact of "noisy neighbors."

3. Utilize Orchestration Services: Leverage orchestration services like AWS Step Functions to streamline and automate complex workflows, such as tenant management. These services provide visual representations of workflows, support error handling, and enable easy integration with other AWS services.

Conclusion:
Scaling machine learning inference and managing tenants in a multi-tenant SaaS environment are critical challenges that require careful consideration and implementation. By following best practices like optimizing memory utilization, collaborating with cloud providers, and utilizing orchestration services, companies can achieve efficient and scalable solutions for their SaaS offerings.