System checks - Poolside

How it works

dive check (pre- and post-flight)

dive check reads a single values.yaml, which is the same file you pass to the inference Helm chart. It builds the external clients each check needs (Kubernetes API, S3, HTTP) lazily on first use, then runs the selected checks concurrently, up to 8 at a time by default. Results are streamed to stdout as each check finishes; the process exits non-zero if any check returns fail or error.

dive node-checks

dive node-checks is an orchestrator that runs from any machine with cluster access (your laptop, a CI runner, a bastion). It discovers GPU nodes and for each one deploys a short-lived privileged pod that mounts the host’s /sys and /proc read-only. The pod runs the hidden node-check-local subcommand of the same dive binary, performs all 10 host-level checks, and prints a JSON report to its logs. The orchestrator collects those logs, parses the reports, and aggregates them into a single per-node summary.

Prerequisites

Locate the dive binary

When using a Poolside bundle, the dive binary is included as a tarball alongside the other bundle binaries in ./binaries.

Distribute the dive image

For dive node-checks, the cluster nodes must be able to pull the dive image. When using a Poolside bundle, the dive image is included as a tarball in ./containers alongside the other bundle images:

# Upload all bundle images (including dive) to your internal registry
./scripts/upload_images.sh <target-registry>

# Then reference it when running node checks
dive node-checks \
  --namespace dive-node-checks \
  --image <target-registry>/poolside/dive:v1

Air-gapped clusters: In an air-gapped environment, the cluster nodes reach the dive image through the mechanism your environment already relies on, such as an internal registry or an image cached on each node. Make the image available through that mechanism, then pass its reference to --image. Because the check pods set imagePullPolicy: IfNotPresent, a cached image is used without a registry pull even when the tag is :latest.

Run pre-flight checks

Run these checks before installing the Helm chart to catch configuration and environment issues early.

dive check \
  --values values.yaml \
  --namespace poolside-models \
  --pre

Pre-flight check reference

Run post-flight checks

Run these checks after the Helm chart is installed to verify that the inference stack is operating correctly.

dive check \
  --values values.yaml \
  --namespace poolside-models \
  --post

Post-flight check reference

Run GPU node checks

Run these checks to validate that every GPU node’s host configuration is optimal for GPU workloads. This is independent of any Helm values file and can be run before or after installing the inference stack.

dive node-checks \
  --namespace dive-node-checks \
  --image <target-registry>/poolside/dive:v1 \
  --create-namespace

Node check reference

Checks are classified as hard (returns fail) or soft (returns warn):

acs_disabled and numa_gpu_* checks are skipped automatically on nodes with no detected GPUs. gpu_pcie_relaxed_ordering is skipped when no EFA/RDMA devices are present.

Output formats

Both dive check and dive node-checks support --output text (default) and --output json.

JSON output

Machine-readable, suitable for CI pipelines. Each result is an object with name and a nested result object. The result object contains status, message, optional detail, and optional fix:

[
  {
    "name": "inference_gpu_operator_healthy",
    "result": {
      "status": "pass",
      "message": "NVIDIA GPU Operator healthy in \"gpu-operator\" with 4 node(s) having nvidia.com/gpu resources available"
    }
  },
  {
    "name": "inference_nvidia_driver_version",
    "result": {
      "status": "fail",
      "message": "NVIDIA driver version < 580.65.06 on GPU node(s): node-1 (driver 535.104.12)",
      "fix": "Upgrade the NVIDIA driver to >= 580.65.06"
    }
  }
]

Exit codes

CLI reference

dive check

dive check --values <values-file-path> --namespace <namespace> [--pre] [--post] [flags]

dive node-checks

dive node-checks --namespace <namespace> --image <dive-image> [flags]

dive version

dive version

Prints the build version string.

Required RBAC for dive node-checks

The credentials of the user running dive node-checks must allow the following operations: