Use LiteLLM with Poolside model inference - Poolside

Use LiteLLM as an OpenAI-Compatible Gateway

Use LiteLLM as an OpenAI-compatible gateway in front of Poolside model inference when you need centralized routing, virtual API keys, budgets, spend tracking, cross-provider fallbacks, or a shared endpoint for internal teams. LiteLLM receives OpenAI-compatible requests from clients, reads the model field in each request, and routes the request to the matching upstream model endpoint. In this setup, each upstream endpoint is a Poolside model server running in your Kubernetes cluster.

LiteLLM is third-party software. You are responsible for LiteLLM configuration, access controls, persistence, upgrades, and security hardening. For production deployment guidance, see the LiteLLM production deployment documentation.

How it works

This setup includes:

Clients send requests to LiteLLM instead of calling each Poolside model endpoint directly. LiteLLM routes each request to the Poolside model service that matches the public model name configured in LiteLLM.

Prerequisites

Step 1: Confirm Poolside model inference is running

Check the model pods and services:

kubectl get pods,svc -n <inference-namespace>

Confirm each model pod is ready before you deploy LiteLLM. In the standard Poolside inference chart, each model deployment and service is named inference-<model-key>, where <model-key> is the suffix shown in the kubectl get pods,svc output, such as lagunas in inference-lagunas. Retrieve the served model name from the model deployment:

kubectl describe deploy/inference-<model-key> \
  -n <inference-namespace> | grep served-model-name -A1

If the model has an ingress, you can query the model endpoint instead:

curl -s http://<model-hostname>/v1/models | jq -r '.data[].id'

To find the model hostname, read it from the model ingress:

kubectl get ingress inference-<model-key> \
  -n <inference-namespace> \
  -o jsonpath='{.spec.rules[0].host}'

Step 2: Create the LiteLLM namespace

Create a namespace for LiteLLM:

kubectl create namespace litellm

Step 3: Create LiteLLM secrets

Create a master key for authenticating to LiteLLM:

kubectl create secret generic litellm-masterkey \
  --from-literal=masterkey="sk-$(openssl rand -hex 24)" \
  -n litellm

Create the LiteLLM environment secret:

kubectl create secret generic litellm-env \
  --from-literal=LITELLM_SALT_KEY="sk-$(openssl rand -hex 24)" \
  -n litellm

Store LITELLM_SALT_KEY securely and do not change it after you add models. LiteLLM uses this key to encrypt provider credentials stored in the database.

Step 4: Configure LiteLLM Helm values

Create a values.yaml file for the LiteLLM Helm chart. This complete example uses the secrets you created in Step 3:

Example: values.yaml

# Existing Secret that holds the LiteLLM master key.
masterkeySecretName: litellm-masterkey
masterkeySecretKey: masterkey

# Existing Secret that holds LITELLM_SALT_KEY.
environmentSecrets:
  - litellm-env

proxy_config:
  model_list: []
  general_settings:
    master_key: os.environ/PROXY_MASTER_KEY
    store_model_in_db: true

postgresql:
  architecture: standalone
  image:
    registry: docker.io
    repository: bitnamilegacy/postgresql
    tag: 16.2.0-debian-12-r6
  auth:
    username: litellm
    database: litellm
    password: "<postgres-password>"
    postgres-password: "<postgres-password>"

redis:
  enabled: false

# Expose the LiteLLM API and admin UI through an Ingress.
ingress:
  enabled: true
  className: "<ingress-class-name>"
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  hosts:
    - host: "<litellm-hostname>"
      paths:
        - path: /
          pathType: ImplementationSpecific

Replace:

This configuration leaves model_list empty and sets store_model_in_db: true so you can add Poolside models from the LiteLLM admin UI.

Step 5: Install LiteLLM

Install the LiteLLM Helm chart:

helm upgrade --install litellm oci://ghcr.io/berriai/litellm-helm \
  --namespace litellm \
  -f values.yaml

The install creates the LiteLLM proxy, a PostgreSQL pod, and a database migration job. Confirm the pods are ready:

kubectl get pods -n litellm

Retrieve the LiteLLM master key:

kubectl get secret litellm-masterkey \
  -n litellm \
  -o go-template='{{index .data "masterkey"}}' | base64 -d

You use this key to sign in to the LiteLLM admin UI and to authenticate test requests to the LiteLLM proxy.

Step 6: Add a Poolside model in LiteLLM

You can add Poolside models through the LiteLLM admin UI or define them in values.yaml. Each model uses three names:

Add a model from the admin UI

  1. Gather two values from your Poolside inference deployment:
    • The served model name from Step 1.
    • The inference service name, inference-<model-key>, used to build the in-cluster API base:
kubectl get svc -n <inference-namespace>
  1. Open the LiteLLM admin UI:
http://<litellm-hostname>/ui/

  1. Sign in with:

    • Username: admin
    • Password: The LiteLLM master key from Step 5

  2. In the navigation menu, select Models + Endpoints.

  3. Select the Add Model tab and then select or enter the following values:

LiteLLM field Value
Provider OpenAI
LiteLLM Model Name(s) Select Custom Model Name (Enter below),
then enter openai/<served-model-name>.
Model Mappings > Public Model Name A name of your choice, such as LagunaS-demo.
API Base http://inference-<model-key>.<inference-namespace>.svc.cluster.local/v1
OpenAI API Key The VLLM_API_KEY configured for Poolside inference.
If the model server does not require authentication,
enter any non-empty value.
  1. Click Test Connect. A successful test confirms that LiteLLM can reach the Poolside model service from inside the cluster.
  2. Click Add Model. The public model name appears on the All Models tab.

Add a model in Helm values

  1. Define the model in values.yaml under proxy_config.model_list:
proxy_config:
     model_list:
    - model_name: "<public-model-name>"
      litellm_params:
        model: "openai/<served-model-name>"
        api_base: "http://inference-<model-key>.<inference-namespace>.svc.cluster.local/v1"
        api_key: "<vllm-api-key>" # The VLLM_API_KEY value, or any non-empty string
  1. Upgrade the chart:
helm upgrade litellm oci://ghcr.io/berriai/litellm-helm \
     --namespace litellm \
     -f values.yaml

Step 7: Test the LiteLLM endpoint

List the models that LiteLLM exposes:

curl -s "http://<litellm-hostname>/v1/models" \
  -H "Authorization: Bearer <litellm-master-key>"

The response includes the public model name you configured in LiteLLM:

{
"data": [
    {
      "id": "<public-model-name>",
      "object": "model"
    }
],
"object": "list"
}

Send a chat completion request through LiteLLM:

curl -s "http://<litellm-hostname>/v1/chat/completions" \
  -H "Authorization: Bearer <litellm-master-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<public-model-name>",
    "messages": [
      {
        "role": "user",
        "content": "What is the capital of France? Answer in one short sentence."
      }
    ],
    "max_tokens": 512,
    "temperature": 0
}'

The setup is working when LiteLLM returns a chat completion response from the Poolside model.

Next steps

Use LiteLLM with Poolside Agent CLI

Follow Install Poolside Agent CLI and choose Connect an OpenAI-compatible provider. Enter:

LiteLLM lists the public model names you configured at /v1/models. Start pool, then press Ctrl+M or use /model to choose a model. See Change the agent.

Use the LiteLLM endpoint from other clients

Point any OpenAI-compatible client, such as the OpenAI SDK, at LiteLLM. Enter:

Set the request’s model value to a public model name configured in LiteLLM.

Operational considerations