What Metrics Matter for vLLM Monitoring and Tuning

中文版

Starting a vLLM service is only the first step.

Before production, I care about more concrete questions: how slow is the first token, how stable is decoding, does throughput increase with concurrency, does P99 latency suddenly spike, and is KV Cache approaching the GPU memory limit?

Without monitoring, tuning becomes guesswork.

Start the Service with Clear Goals

Scripts separate startup modes into basic validation, high throughput, low latency, and memory saving. That split is practical because each goal has different parameter choices.

For high throughput, the configuration usually increases concurrent sequences, raises memory utilization, and enables prefix caching:

# 中文注释：高吞吐模式通过增大 batch 和并发序列数，提高 tokens/s
python -m vllm.entrypoints.openai.api_server \
  --model /root/autodl-tmp/models/qwen/Qwen3-0.6B \
  --port 8000 \
  --dtype float16 \
  --max-model-len 4096 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 4096 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --trust-remote-code

For low latency, I pay more attention to Chunked Prefill and batch limits:

# 中文注释：低延迟模式限制批处理规模，并开启分块预填充降低长 prompt 阻塞
python -m vllm.entrypoints.openai.api_server \
  --model /root/autodl-tmp/models/qwen/Qwen3-0.6B \
  --port 8000 \
  --dtype float16 \
  --max-num-seqs 16 \
  --max-num-batched-tokens 512 \
  --enable-chunked-prefill \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code

Chunked Prefill: splitting long prompt prefill into smaller chunks so decode requests are not blocked for too long.

Prefix Caching: reusing KV Cache for shared prompt prefixes. It is useful for multi-turn chat, fixed system prompts, and RAG templates.

Measure TTFT and TPOT Separately

I do not like relying only on end-to-end latency because it mixes different stages.

TTFT, Time To First Token: the delay from request submission to the first generated token. It reflects prefill, queueing, and cold start.

TPOT, Time Per Output Token: the interval between output tokens during decoding. It reflects decode-stage stability.

The script measures these metrics through streaming output:

# 中文注释：发送流式请求，逐个接收 token，并记录每个 token 到达时间
stream = client.chat.completions.create(
    model=MODEL_NAME,
    messages=messages,
    stream=True,
    max_tokens=max_tokens,
    temperature=0.7,
)

# 中文注释：第一个 token 的时间点用于计算 TTFT，后续相邻时间差用于计算 TPOT
for chunk in stream:
    if chunk.choices[0].delta.content:
        now = time.perf_counter()
        token_timestamps.append(now)

One experiment detail is worth noticing: a short prompt had the highest TTFT, about 355ms, while medium and long prompts were only 40ms and 63ms.

This is not a contradiction. It is cold start. The first request may pay for CUDA kernel warmup, memory allocation, and runtime initialization. In production, warmup requests are commonly used to avoid exposing this cost to users.

Concurrency Benchmarking Should Find the Turning Point

The goal of benchmarking is not to prove that more concurrency is always better. The goal is to find the turning point between throughput and tail latency.

The benchmark script uses a thread pool, sends concurrent requests, and calculates QPS, tokens/s, and latency percentiles:

# 中文注释：用线程池模拟指定并发度，同时向 vLLM OpenAI 接口发送请求
with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as executor:
    futures = [
        executor.submit(send_chat_request, prompt, max_tokens)
        for prompt in prompts
    ]

    # 中文注释：每个请求完成后收集结果，用于后续计算延迟分位数和吞吐量
    for future in concurrent.futures.as_completed(futures):
        all_results.append(future.result())

The course benchmark gives a clear baseline:

From 1 to 8 concurrency, throughput increases strongly while P50 latency stays close. At 16, throughput still grows, but P50 and P99 jump. That is the saturation signal.

For production, I would choose a concurrency limit based on acceptable P99, not the highest possible tokens/s.

Prometheus Metrics Close the Loop

vLLM can expose Prometheus-format metrics through /metrics. This is better than a one-off benchmark for long-term observation.

Prometheus: a monitoring system where services expose text metrics, Prometheus collects them periodically, and Grafana can visualize them.

The script extracts core metrics like this:

# 中文注释：定义需要从 Prometheus 文本中提取的 vLLM 核心指标
patterns = {
    "prompt_tokens_total": r"vllm:prompt_tokens_total[^\n]*?\s+([\d.]+)",
    "generation_tokens_total": r"vllm:generation_tokens_total[^\n]*?\s+([\d.]+)",
    "num_requests_running": r"vllm:num_requests_running\s+([\d.]+)",
    "num_requests_waiting": r"vllm:num_requests_waiting\s+([\d.]+)",
    "gpu_cache_usage_perc": r"vllm:gpu_cache_usage_perc\s+([\d.]+)",
}

I focus on these metrics:

num_requests_running: how many requests are actively running.

num_requests_waiting: how many requests are queued. If this keeps growing, traffic exceeds serving capacity.

gpu_cache_usage_perc: GPU KV Cache usage. If it approaches the limit, watch for swapping, OOM, or rising tail latency.

prompt_tokens_total and generation_tokens_total: cumulative input and output tokens. They can be converted into throughput over a time window.

time_to_first_token_seconds and time_per_output_token_seconds: TTFT and TPOT distributions.

In the example, 20 concurrent requests increased generated tokens from 21634 to 24634, adding 3000 tokens. During monitoring, the service produced about 900 tokens every 2 seconds, roughly 450 tokens/s.

Time-series monitoring is more valuable than a single benchmark because it shows whether the system remains stable under load.

Diagnose Bottlenecks by Symptoms

If TTFT is too high, I first check prompt length, queue length, and whether prefill is blocking decode. Common fixes include Chunked Prefill, Prefix Caching, or PD disaggregation.

PD Disaggregation: separating Prefill and Decode into different resources or services, so long prompts do not block short decode workloads.

If TPOT is unstable, I check batch-size variation, speculative decoding acceptance rate, preemption, and swapping.

If GPU utilization is low while memory usage is high, the service may be memory-bound. Possible actions include increasing --max-num-seqs, using FP8 KV Cache, shortening context length, or enabling speculative decoding for small-batch workloads.

FP8 KV Cache: storing KV Cache in lower precision to reduce memory usage and improve throughput. It should still be validated against business quality requirements.

Closing

vLLM tuning is not a fixed parameter recipe. It is a monitoring loop.

My order is:

1. Start with a basic configuration and verify the service works.
2. Measure TTFT and TPOT with streaming requests.
3. Benchmark concurrency to find the throughput and P99 turning point.
4. Use Prometheus to observe queue length, KV Cache usage, token throughput, and latency distributions.
5. Tune for high throughput, low latency, memory saving, or speculative decoding based on the observed bottleneck.

At that point, every parameter change has a reason, and every improvement can be measured.