Inference metrics - Poolside

Model server metrics

The vLLM model server exposes native vLLM metrics on :8080/metrics, under the vllm:*, http_*, and python_* prefixes. There is one Kubernetes Deployment per model, with pods named inference-<uuid>. The examples below resolve the target pod by Helm label at runtime, so they keep working as pods are renamed on redeploy. Core vLLM metrics are present on an idle pod, with counters and histograms starting at 0.

What to watch

To find out Watch Healthy
Is the server overloaded? vllm:num_requests_waiting, vllm:kv_cache_usage_perc Queue near 0, usage below ~0.9
Are responses slow to start? vllm:time_to_first_token_seconds Stays low and steady; a rising trend means slower first tokens
Is streaming smooth after streaming starts? vllm:inter_token_latency_seconds Stays low and steady; a rising trend means choppier streaming
Are requests rejected or preempted? vllm:num_dropped, vllm:num_preemptions 0
Is the prefix cache helping? vllm:prefix_cache_hits_total over vllm:prefix_cache_queries_total Higher is better; no fixed target, but watch for drops
Is throughput holding up? vllm:generation_tokens_total (rate), vllm:num_requests_running Token rate steady under load; a drop while requests are running signals a stall

Latency and throughput

The user-facing performance signals. They are histograms, so alert on high percentiles rather than averages.

Metric Type Description
vllm:time_to_first_token_seconds histogram Time to first token (TTFT), the clearest measure of perceived responsiveness
vllm:inter_token_latency_seconds histogram Gap between successive output tokens (ITL); sets streaming speed after a response starts
vllm:e2e_request_latency_seconds histogram End-to-end request latency, from queue to final token
vllm:request_time_per_output_token_seconds histogram Decode cost normalized per output token
vllm:request_queue_time_seconds histogram Time spent waiting in the queue
vllm:request_prefill_time_seconds histogram Time spent in the prefill phase
vllm:request_decode_time_seconds histogram Time spent in the decode phase
vllm:request_inference_time_seconds histogram Time spent in the running (inference) phase

Load and queue

How saturated the engine is. The first place to look when latency climbs or when deciding whether to scale.

Metric Type Description
vllm:num_requests_waiting gauge Requests queued for admission; the strongest signal you need more capacity
vllm:num_requests_running gauge Requests in the active execution batch; near batch capacity means fully loaded
vllm:num_preemptions gauge Requests preempted by the engine, usually under KV-cache pressure
vllm:num_dropped gauge Requests dropped because the queue exceeded its maximum size
vllm:iteration_tokens_total histogram Tokens processed per engine step, a measure of batching efficiency

KV cache and prefix caching

KV-cache pressure and prefix-cache hit rate largely determine throughput and latency under load.

Metric Type Description
vllm:kv_cache_usage_perc gauge Fraction of the KV cache in use, where 1.0 is 100%; a key saturation signal
vllm:prefix_cache_queries_total counter Prefix-cache lookups, in tokens
vllm:prefix_cache_hits_total counter Prefix-cache hits, in tokens; hits over queries is the local hit rate
vllm:external_prefix_cache_queries_total counter Cross-instance prefix-cache lookups, in tokens
vllm:external_prefix_cache_hits_total counter Cross-instance prefix-cache hits, in tokens
vllm:prompt_tokens_cached_total counter Prompt tokens served from cache, local plus external
vllm:prompt_tokens_recomputed_total counter Cached tokens that had to be recomputed; rising values indicate cache thrashing
vllm:request_prefill_kv_computed_tokens histogram New KV tokens computed during prefill, excluding cached tokens

Token counts

Use these for cost tracking, capacity planning, and understanding workload shape.

Metric Type Description
vllm:prompt_tokens_total counter Cumulative prompt (prefill) tokens processed
vllm:generation_tokens_total counter Cumulative generated tokens; basis for tokens-per-second throughput
vllm:prompt_tokens_by_source_total counter Prompt tokens broken down by source
vllm:request_prompt_tokens histogram Per-request prompt token count
vllm:request_generation_tokens histogram Per-request generated token count
vllm:request_max_num_generation_tokens histogram Per-request maximum requested generation tokens
vllm:request_params_max_tokens histogram Distribution of the max_tokens request parameter
vllm:request_params_n histogram Distribution of the n request parameter

Speculative decoding

Present only when speculative decoding is enabled. Accepted over draft tokens is the acceptance rate; a low rate means speculation is wasting compute.

Metric Type Description
vllm:spec_decode_num_drafts_total counter Speculative-decoding drafts proposed
vllm:spec_decode_num_draft_tokens_total counter Draft tokens proposed
vllm:spec_decode_num_accepted_tokens_total counter Accepted draft tokens
vllm:spec_decode_num_accepted_tokens_per_pos_total counter Accepted draft tokens by draft position

Model FLOPs utilization

Use these to estimate model FLOPs utilization (MFU) and tell whether a workload is compute-bound or memory-bound.

Metric Type Description
vllm:estimated_flops_per_gpu_total counter Estimated floating-point operations per GPU
vllm:estimated_read_bytes_per_gpu_total counter Estimated bytes read from memory per GPU
vllm:estimated_write_bytes_per_gpu_total counter Estimated bytes written to memory per GPU

Runtime and miscellaneous

Metric Type Description
vllm:request_success_total counter Successfully completed requests
vllm:cache_config_info gauge Static cache configuration
vllm:engine_sleep_state gauge Whether the engine is awake or sleeping
vllm:mm_cache_queries_total, vllm:mm_cache_hits_total counter Multi-modal input cache, for multi-modal models only
http_requests_total, http_request_duration_seconds counter, histogram FastAPI HTTP layer in front of the engine
python_*, process_* various Python runtime and process metrics

This list reflects the metrics exposed by the deployed model server. The metric surface can change between releases, so confirm the exact set against a live scrape of your deployment using the commands in the scraping sections.