## Overview Use this guide to change the set of models served by a running `inference` release: adding a new model, replacing a model’s checkpoint, or removing a model. You edit your `inference_values.yaml` file and run `helm upgrade`; the chart reconciles the model Deployments, Services, and Ingress objects to match. You can make these changes on their own against the current chart version, or apply them as part of a chart upgrade to a new Poolside bundle. To upgrade the chart, see [Upgrade on Amazon EKS](https://docs.poolside.ai/deployment/cloud/aws-eks/upgrade); make the model edits described here in the same `inference_values.yaml` file before you run `helm upgrade`. ## Prerequisites - A working deployment completed with the [Install on Amazon EKS](https://docs.poolside.ai/deployment/cloud/aws-eks/install) guide. - The customized `inference_values.yaml` file you used to install. - The new model checkpoint, provided by Poolside. - Workstation tools: - `helm` `3.12` or later - `kubectl`, configured for your EKS cluster - `aws` CLI, to upload checkpoints to S3 - `jq`, to parse JSON responses from the inference API ## Downtime Adding a model does not affect models that are already serving. Updating a checkpoint rolls that model’s Deployment, and the model server re-downloads the checkpoint from S3 on restart, so expect a delay before it becomes ready again. Plan a maintenance window for single-replica models. ## Add a model Extract the new checkpoint archive as described in [Upload model checkpoints to S3](https://docs.poolside.ai/deployment/cloud/aws-eks/install#step-3-upload-model-checkpoints-to-s3) so its files sit at the prefix root, then upload it to your S3 bucket. Use a distinct prefix per model: ``` aws s3 cp ./checkpoints/ s3:///checkpoints/ \ --recursive \ --region ``` For checkpoint upload details such as concurrency throttling, see [Upload model checkpoints to S3](https://docs.poolside.ai/deployment/cloud/aws-eks/install#step-3-upload-model-checkpoints-to-s3). Add a new key under `models` in your `inference_values.yaml` file. Give the model its own `ingressHost`, covered by the ACM certificate referenced in `ingress.annotations`: Example: inference_values.yaml ``` models: # ...existing models... : model: s3:///checkpoints/ modelName: modelType: agent gpus: ingressHost: ``` Apply the change with `helm upgrade`: ``` helm upgrade inference ./charts/inference \ --namespace poolside-models \ -f ./inference_values.yaml ``` The chart creates a new `Deployment`, `Service`, and `Ingress` named `inference-` for the model. Confirm the new pod starts and the ingress is created: ``` kubectl get pods -n poolside-models kubectl get ingress inference- -n poolside-models ``` Create a DNS record for the new `ingressHost`, pointing it at the load balancer address. ## Update a model checkpoint Upload the new checkpoint to a new, versioned prefix rather than overwriting the existing one. A new path lets `helm upgrade` detect the change and roll the Deployment automatically, and it lets you roll back by pointing at the previous path. Extract the archive first, as in [Step 3](https://docs.poolside.ai/deployment/cloud/aws-eks/install#step-3-upload-model-checkpoints-to-s3), so the files sit at the prefix root: ``` aws s3 cp ./checkpoints/- s3:///checkpoints/- \ --recursive \ --region ``` Point the model’s `model` field at the new path in your `inference_values.yaml` file. Update `modelName` only if the served model name changes: Example: inference_values.yaml ``` models: laguna-m: model: s3:///checkpoints/laguna-m- modelName: Laguna modelType: agent gpus: 4 ingressHost: ``` Apply the change: ``` helm upgrade inference ./charts/inference \ --namespace poolside-models \ -f ./inference_values.yaml ``` The model’s Deployment rolls, and the init container downloads the new checkpoint on startup. Watch the rollout: ``` kubectl rollout status deploy/inference- -n poolside-models ``` If you reuse the same S3 path instead of a versioned one, `helm upgrade` detects no change to the values and does not restart the model. Force a restart so the init container re-downloads the checkpoint: ``` kubectl rollout restart deploy/inference- -n poolside-models ``` ## Remove a model Delete the model’s key from `models` in your `inference_values.yaml` file, then apply the change: ``` helm upgrade inference ./charts/inference \ --namespace poolside-models \ -f ./inference_values.yaml ``` The chart removes that model’s `Deployment`, `Service`, and `Ingress`. Confirm the resources are gone: ``` kubectl get deploy,svc,ingress -n poolside-models -l app.kubernetes.io/component=inference ``` If you no longer need the model’s checkpoint, delete it from the bucket: ``` aws s3 rm s3:///checkpoints/ --recursive --region ``` You can also remove the model’s DNS record once the ingress is gone. ## Verification Confirm a model serves traffic, where `` is the `ingressHost` of that model: ``` curl -s https:///v1/models \ -H "Authorization: Bearer " \ | jq -r '.data[].id' ``` If API key authentication is off, omit the `Authorization` header.