## Supported environments

**Amazon EKS**  
Deploy model inference with Helm on Amazon EKS, using IRSA for object storage and an Application Load Balancer for ingress.

**Red Hat OpenShift**  
Deploy model inference with Helm on your OpenShift cluster.

**Upstream Kubernetes**  
Deploy model inference with Helm on your self-managed Kubernetes cluster, such as RKE2 or Charmed Kubernetes.

## Architecture

Cloud deployment includes:

- One `Deployment` and `Service` per model. Each model server downloads its checkpoint from object storage on startup and serves an OpenAI-compatible API.
- Each model is exposed at its own hostname through an ingress or OpenShift Route that routes directly to its vLLM service.
- Optionally, the Poolside documentation site, deployed in-cluster from the bundle. See [Set up offline documentation](https://docs.poolside.ai/deployment/cloud/set-up-offline-documentation).

You are responsible for sending requests to the inference endpoints and for any authentication or routing in front of them. To expose a shared endpoint with centralized access controls, routing, virtual keys, budgets, and spend tracking, see [Use LiteLLM with Poolside model inference](https://docs.poolside.ai/deployment/inference-gateways/litellm).

## Operational considerations

- **Service availability**: All external services your deployment depends on, including object storage and the container registry, must be reachable from within the cluster. The cluster must have access to compatible GPU hardware.
- **Backup and recovery**: You are responsible for backup and recovery for the infrastructure and external services in your environment, such as object storage, container registry contents, and Kubernetes configuration.