## Overview Use this guide to deploy Poolside model inference on a dedicated GPU workstation or server host. The on-premises installation bundle can be used in internet-connected or air-gapped environments. When the bundle is already cached on the host, installation typically takes about one hour. The installation process has the following phases: 1. Prepare the host and install operating-system-specific prerequisites. 2. Install RKE2 infrastructure. 3. Install supporting infrastructure services. 4. Upload model checkpoints. 5. Deploy model inference and ingress. The installation includes: - RKE2 Kubernetes - S3-compatible object storage for model checkpoints - A local container registry - `cert-manager` for self-signed certificates - NVIDIA GPU Operator for GPU access in RKE2 workloads - Model inference workloads for the model checkpoints you provide Model checkpoint files are provided separately based on your deployment. Upload the model checkpoint files during the model upload step. ## Prerequisites Before you begin, ensure that the host meets the following prerequisites: - A supported operating system: - Ubuntu 22.04 LTS - Ubuntu 24.04 LTS - Red Hat Enterprise Linux (RHEL) 9.6 - SUSE Linux Enterprise Server (SLES) 16 - A supported host CPU architecture: - `amd64` (`x86_64`) - `arm64` (`aarch64`) - `sudo` access on the host - The Poolside installation bundle for your host CPU architecture - Poolside model checkpoint files available on the host - The ingress hostnames you plan to expose for model inference - If you use custom TLS certificates, CA and server certificate files with SANs that cover the hostnames described in Step 2 ## Prepare the host Complete the preparation steps for the host operating system before you run the installation steps. Some prerequisite tools provide architecture-specific Linux binaries. Use `amd64` downloads for `x86_64` hosts and `arm64` downloads for `aarch64` hosts. To check the host architecture, run: ``` uname -m ``` ### Prepare Ubuntu These steps apply to both Ubuntu 22.04 LTS and Ubuntu 24.04 LTS. 1. **Install required tools** - Install `kubectl` using `sudo snap install kubectl --classic` - Install `jq` using `sudo apt install -y jq` - Install `yq` (version `v4.49.2` or later) from the [yq releases page](https://github.com/mikefarah/yq/releases/tag/v4.49.2) - Download [`yq_linux_amd64.tar.gz`](https://github.com/mikefarah/yq/releases/download/v4.49.2/yq_linux_amd64.tar.gz) or [`yq_linux_arm64.tar.gz`](https://github.com/mikefarah/yq/releases/download/v4.49.2/yq_linux_arm64.tar.gz) and install it to `/usr/local/bin/yq` - Install `terraform` (version `1.8.5`) from the [Terraform 1.8.5 releases page](https://releases.hashicorp.com/terraform/1.8.5) - Download [`terraform_1.8.5_linux_amd64.zip`](https://releases.hashicorp.com/terraform/1.8.5/terraform_1.8.5_linux_amd64.zip) or [`terraform_1.8.5_linux_arm64.zip`](https://releases.hashicorp.com/terraform/1.8.5/terraform_1.8.5_linux_arm64.zip) and install the binary to `/usr/local/bin/terraform` - `unzip` is required to extract the Terraform binary - Install `skopeo` (version `v1.18` or later) from the [skopeo-binary releases page](https://github.com/lework/skopeo-binary/releases/tag/v1.20.0) - Download [`skopeo-linux-amd64`](https://github.com/lework/skopeo-binary/releases/download/v1.20.0/skopeo-linux-amd64) or [`skopeo-linux-arm64`](https://github.com/lework/skopeo-binary/releases/download/v1.20.0/skopeo-linux-arm64) and install it to `/usr/local/bin/skopeo` 2. **Configure the containers trust policy** Ensure the containers trust policy at `/etc/containers/policy.json` allows `skopeo` to access the RKE2 registry with the minimum required permissions to load container images into the registry during installation. ```json { "default": [ { "type": "insecureAcceptAnything" } ] } ``` 3. **Disable the `nouveau` driver if loaded** Confirm that the `nouveau` graphics driver is not loaded. For instructions, see [Disable the nouveau driver in the NVIDIA documentation](https://docs.nvidia.com/ai-enterprise/deployment/vmware/latest/nouveau.html). Run the following command to check whether the `nouveau` driver is loaded. If the command returns output, follow the next steps to turn off the driver and reboot. ``` lsmod | grep nouveau ``` If the `nouveau` driver is loaded: ``` cat <` with `amd64` or `arm64`: ``` sudo zypper install -y wget wget https://dl.k8s.io/release/v1.35.3/bin/linux//kubectl chmod +x kubectl sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl ``` ``` lsmod | grep nouveau ``` If the `nouveau` driver is loaded: ``` cat < ``` - Refresh the available packages: ``` sudo zypper refresh ``` - Install the driver: ``` sudo zypper install nvidia-open-driver-G07-signed-cuda-kmp-default ``` - Store the installed driver version: ``` version=$(rpm -qa --queryformat '%{VERSION}\n' nvidia-open-driver-G07-signed-cuda-kmp-default | cut -d "_" -f1 | sort -u | tail -n 1) ``` - Install the matching compute utilities and persistence daemon: ``` sudo zypper install -y "nvidia-compute-utils-G07 = ${version}" "nvidia-persistenced = ${version}" ``` ## Install Use the Poolside installation bundle for your host architecture, either `linux/amd64` or `linux/arm64`. You can use the bundle in internet-connected and air-gapped environments. ### Step 0 (optional): Set up an air-gapped installation This configuration is required for air-gapped installations. In internet-connected environments, you can skip this step. To use the local Terraform provider cache included in the bundle, configure Terraform to load providers from the bundled `terraform.d` directory. 1. Locate `poolside-terraform.tfrc` in the root of the unpacked installation bundle. 2. Replace the `$POOLSIDE_INSTALL_DIR` placeholder with the fully qualified path to the bundle’s root directory. 3. For Terraform commands in the installation steps, prefix the command with the Terraform CLI configuration path: ``` TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc terraform ``` Setting this variable ensures that both root and non-root users reference the same cached Terraform providers. ### Step 1: Install RKE2 on the host The `01-infra-rke2` directory contains the Terraform module that installs RKE2 on the host. Using `sudo`, run the following commands from the `01-infra-rke2` directory. You must run the RKE2 installation using `sudo` from the same user account that runs Poolside model inference after deployment. Terraform uses the original user and group IDs from the `sudo` environment to set ownership and permissions required by later installation stages. **Air-gapped environment:** ``` sudo TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc /usr/local/bin/terraform init sudo TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc /usr/local/bin/terraform apply ``` **Internet-connected environment:** ``` sudo /usr/local/bin/terraform init sudo /usr/local/bin/terraform apply ``` If RKE2 certificates or credentials change, re-run this step to refresh the configuration files that restore access for the installation user. ### Step 2: Install supporting infrastructure services The `02-infra-services` directory contains the Terraform module that accesses the RKE2 cluster and deploys the supporting infrastructure required by Poolside model inference. This step installs: - A local container registry - S3-compatible object storage - Ingress and certificate resources for inference endpoints - NVIDIA GPU Operator, deployed as `gpu-operator` Before you run Terraform, complete the following configuration steps. #### 1. Configure ingress hostnames In `02-infra-services/terraform.tfvars`, set `poolside_ingress_hosts` to the model hostnames that you plan to use later in Step 4. The installer uses this value when it creates self-signed certificate SANs. If you use installer-generated self-signed certificates, each model `ingress_host_name` that you configure in Step 4 must match one of the hostnames in `poolside_ingress_hosts`. This lets the installer generate certificates with the required SANs before model inference is deployed. If you use custom TLS certificates, ensure that your certificate SANs include each model `ingress_host_name` that you configure in Step 4. The installer includes `poolside-docs` in certificate SANs by default. Add a documentation hostname to `poolside_ingress_hosts` only if you want to use a different documentation hostname. #### 2. Configure custom TLS certificates Skip this step if you use installer-generated self-signed certificates. If you use custom TLS certificates, you must provide your own CA and server certificate before you run `terraform apply`. The `custom_certificates` and `custom_ca_trust_chain` parameters configure certificates for the TLS-terminating inference and storage services. The `custom_certificates` schema accepts certificate and key entries for `poolside`, `services.storage`, and `services.storage_s3`. You can use one certificate that covers all exposed hostnames, or separate certificates if your Public Key Infrastructure (PKI) requires it. Across all certificates you provide, the Subject Alternative Names (SANs) must cover every hostname that you expose, including: - Every model `ingress_host_name` that you configure in `04-poolside-inference/terraform.tfvars` - Any custom documentation ingress hostname that you configure instead of the default `poolside-docs` hostname - `seaweedfs.poolside.local` - `seaweedfs-s3.poolside.local` 1. Place your CA certificate, server certificate, and private key in a directory accessible to Terraform. The example below uses `/poolside-install/byo-certs/`. The `poolside-install/` subdirectory holds the installation’s persistent state and is preserved across cluster resets, so it is the recommended location for BYO certificate files. ```plaintext /poolside-install/byo-certs/ ├── ca.crt # CA certificate (root, or root and intermediate chain) ├── server.crt # Server certificate signed by the CA └── server.key # Server private key ``` You must reference these files using fully qualified (absolute) paths in the next step. Relative paths are not supported. 2. In `02-infra-services/terraform.tfvars`, set the BYO variables: ```plaintext custom_ca_trust_chain = { root_ca_path = "/poolside-install/byo-certs/ca.crt" } custom_certificates = { poolside = { cert_path = "/poolside-install/byo-certs/server.crt" key_path = "/poolside-install/byo-certs/server.key" } services = { storage = { cert_path = "/poolside-install/byo-certs/server.crt" key_path = "/poolside-install/byo-certs/server.key" } storage_s3 = { cert_path = "/poolside-install/byo-certs/server.crt" key_path = "/poolside-install/byo-certs/server.key" } } } ``` `custom_ca_trust_chain.root_ca_path` must point to the CA that signed `server.crt`. When you run `terraform apply`, the module creates Kubernetes secrets with a `-byo` suffix from these files. #### 3. Run Terraform Using `sudo`, run the following commands from the `02-infra-services` directory. You must run this step using `sudo` from the same user account that runs Poolside model inference after deployment. Terraform uses the original user and group IDs from the `sudo` environment to set the permissions required for RKE2 cluster access in later stages. **Air-gapped environment:** **Internet-connected environment:** ``` sudo /usr/local/bin/terraform init sudo /usr/local/bin/terraform apply ``` This step can take some time to complete. The process loads container images into the local RKE2 registry to support disconnected operation and improve Poolside startup performance. ### Step 3: Upload Poolside models The `03-poolside-model-upload` directory contains the Terraform module that uploads model checkpoints into the deployment’s S3-compatible storage. The module creates a Kubernetes job that syncs model files from a local host directory into the `poolside-models` bucket. 1. Copy the Poolside model checkpoint files for your deployment into the local host directory: ``` /opt/poolside/poolside-model-uploads ``` This is the default location. If you customized the Poolside host volume location in `01-infra-rke2`, use the corresponding directory instead. 2. Run the following commands from the `03-poolside-model-upload` directory. **Air-gapped environment:** ``` TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc terraform init TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc terraform apply ``` **Internet-connected environment:** ``` terraform init terraform apply ``` 3. To upload additional or updated models later, repeat these steps. Uploads are additive and do not remove existing models from the deployment. ### Step 4: Deploy Poolside model inference The `04-poolside-inference` directory contains the Terraform module that deploys the inference containers used to serve Poolside models. 1. In the `04-poolside-inference` directory, update `terraform.tfvars` with the model details you want to deploy. If you use installer-generated self-signed certificates, each model `ingress_host_name` must match one of the hostnames you configured in `poolside_ingress_hosts` during Step 2. If you use custom TLS certificates, the certificate SANs must include each model `ingress_host_name`. Example: Model configuration ```plaintext deployment_name = "poolside-server" models = { agent = { s3_uri = "s3://poolside-models/" ingress_host_name = "poolside-models-agent.poolside.local" gpus = 1 replicas = 1 model_type = "agent" } } ``` 2. Run the following commands from the `04-poolside-inference` directory. **Air-gapped environment:** ``` TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc terraform init TF_CLI_CONFIG_FILE=/poolside-terraform.tfrc terraform apply ``` **Internet-connected environment:** ``` terraform init terraform apply ``` ## Next steps: Post-installation configuration ### Configure local DNS Add hostname resolution on the deployment host. Replace the model ingress hostnames with the `ingress_host_name` values you configured in Step 4. If you expose multiple model ingress hostnames, include each hostname on the same line. ``` cat < seaweedfs.poolside.local seaweedfs-s3.poolside.local EOF ``` For example: ``` cat < ``` - Confirm that model workloads are running: ``` kubectl get pods -n poolside-models ``` - Confirm that the model upload job completed successfully: ``` kubectl get jobs -n poolside-models ``` ## Troubleshooting ### Model pods stuck in `ContainerCreating` - Confirm that the host detects NVIDIA GPU devices: ``` lspci | grep -i nvidia ``` - Confirm that Kubernetes reports GPUs as allocatable: ``` kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{