Cloud deployment - Poolside

Supported environments

Amazon EKS
Deploy model inference with Helm on Amazon EKS, using IRSA for object storage and an Application Load Balancer for ingress.

Red Hat OpenShift
Deploy model inference with Helm on your OpenShift cluster.

Upstream Kubernetes
Deploy model inference with Helm on your self-managed Kubernetes cluster, such as RKE2 or Charmed Kubernetes.

Architecture

Cloud deployment includes:

• One Deployment and Service per model. Each model server downloads its checkpoint from object storage on startup and serves an OpenAI-compatible API.
• Each model is exposed at its own hostname through an ingress or OpenShift Route that routes directly to its vLLM service.
• Optionally, the Poolside documentation site, deployed in-cluster from the bundle. See Set up offline documentation.

You are responsible for sending requests to the inference endpoints and for any authentication or routing in front of them. To expose a shared endpoint with centralized access controls, routing, virtual keys, budgets, and spend tracking, see Use LiteLLM with Poolside model inference.

Operational considerations

• Service availability: All external services your deployment depends on, including object storage and the container registry, must be reachable from within the cluster. The cluster must have access to compatible GPU hardware.
• Backup and recovery: You are responsible for backup and recovery for the infrastructure and external services in your environment, such as object storage, container registry contents, and Kubernetes configuration.