Follow these steps to deploy Poolside model inference on your Amazon EKS cluster and serve models through an OpenAI-compatible API. For an overview of this deployment approach and architecture, see [Amazon EKS deployment](https://docs.poolside.ai/deployment/cloud/aws-eks/overview). This guide deploys the Poolside inference chart. Each model becomes its own `Deployment`, `Service`, and `Ingress`, reachable at its own hostname through a shared Application Load Balancer.

## Prerequisites

Poolside distributes the Helm deployment bundle as a `.tar.gz` archive. Extract it before you start:

```
tar -xzf <bundle-name>.tar.gz
cd <bundle-name>
```

Confirm that you are working from the root of the extracted bundle. The bundle root contains the following directories:

```
./scripts/
./containers/
./charts/
```

### Required AWS infrastructure

You provision the following AWS foundation before you deploy the chart. For a turnkey foundation that provisions all of it, apply the Terraform reference architecture in the [`poolsideai/reference_architectures`](https://github.com/poolsideai/reference_architectures/tree/main/aws) repository, or reproduce the same architecture in your own infrastructure-as-code. For the architecture diagram and design decisions, see [Reference architecture](https://docs.poolside.ai/deployment/cloud/aws-eks/reference-architecture).

- **EKS cluster**, Kubernetes 1.29 or later, with an IAM OIDC provider enabled.
- **GPU node group** with enough GPU memory for the models you deploy.
- **NVIDIA GPU Operator** to expose GPUs to the cluster. Run it in one of two modes, depending on your AMI:
  - If the AMI already includes the NVIDIA driver and container toolkit, run the GPU Operator in device-plugin-only mode.
  - If the AMI ships without drivers, run the full GPU Operator.
- **AWS Load Balancer Controller**, installed and running in the cluster.
- **Amazon S3 bucket** for the model checkpoints. Server-side encryption with a KMS key is recommended.
- **Amazon ECR** to host the bundled container images.
- **AWS Certificate Manager certificate** covering the hostnames you assign to the models.

### Workstation tools

Install the following tools on the host you use to run the deployment:
- `helm` 3.12 or later
- `kubectl`
- `skopeo`
- `aws` CLI
- `jq`
- `tar`
- `curl`
- `eksctl` (optional)

**Disk space**: Stage the deployment from a host with tens of GB free. The extracted bundle is roughly 20 GB.

## Step 1: Create the namespace

The inference stack runs in a single namespace:

```
kubectl create namespace poolside-models
```

## Step 2: Upload the container images to Amazon ECR

Copy the images into Amazon ECR. For each image name from the bundle:
```
for image_name in $(find ./containers -name "*.tar" -type f -exec basename {} .tar \; | sed 's/__.*//' | sort -u); do
  aws ecr describe-repositories --repository-names "$image_name" --region <aws-region> >/dev/null 2>&1 \
    || aws ecr create-repository --repository-name "$image_name" --region <aws-region>
done
```

Authenticate `skopeo` to your ECR registry:
```
aws ecr get-login-password --region <aws-region> \
  | skopeo login --username AWS --password-stdin <account-id>.dkr.ecr.<aws-region>.amazonaws.com
```
Upload the images with the provided script:
```
chmod +x ./scripts/upload_images.sh
./scripts/upload_images.sh <account-id>.dkr.ecr.<aws-region>.amazonaws.com
```

## Step 3: Upload model checkpoints to S3

The model servers download their checkpoints from S3 on pod startup. Upload the checkpoints to your S3 bucket:
```
aws s3 cp ./checkpoints s3://<bucket-name>/checkpoints --recursive --region <aws-region>
```

## Step 4: Create the IRSA role

Save the permissions and trust policy to respective JSON files. Create the policy and role:
```
aws iam create-policy \
  --policy-name inference-pod-policy \
  --policy-document file://inference-pod-policy.json

aws iam create-role \
  --role-name inference-pod-role \
  --assume-role-policy-document file://inference-pod-trust.json

aws iam attach-role-policy \
  --role-name inference-pod-role \
  --policy-arn arn:aws:iam::<account-id>:policy/inference-pod-policy
```

## Step 5: Create the API key secret (recommended for internet-facing)

Create a secret containing the API key:
```
kubectl create secret generic vllm-auth \
  --from-literal=VLLM_API_KEY=<vllm-api-key> \
  -n poolside-models
```

## Step 6: Configure the values file

Create an `inference_values.yaml` file in the bundle root:
```yaml
fullnameOverride: inference

image:
  registry: <account-id>.dkr.ecr.<aws-region>.amazonaws.com

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/inference-pod-role

authentication:
  secretName: vllm-auth

ingress:
  enabled: true
  className: alb
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing

models:
  laguna-m:
    model: s3://<bucket-name>/checkpoints/laguna-m
    modelName: Lagunam
    modelType: agent
    gpus: 4
  laguna-xs:
    model: s3://<bucket-name>/checkpoints/laguna-xs
    modelName: Lagunaxs
    modelType: agent
    gpus: 1
```

## Step 7: Install the chart

Install the `inference` chart:
```
helm install inference ./charts/inference \
  --namespace poolside-models \
  -f ./inference_values.yaml
```

## Step 8: Verify the deployment

Check model pods:
```
kubectl get pods -n poolside-models
```

## Step 9: Call the inference API

Each model serves an API at its own hostname.

Send a request to a model:
```
curl https://<model-hostname>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<served-model-name>",
    "messages": [{"role": "user", "content": "Write a function that reverses a string."}]
  }'
```

## TLS

The load balancer terminates TLS with the ACM certificate and serves models over HTTPS.