KV Cache Is the Real Memory Killer in High-Concurrency AI Services
When I look at an LLM service under real traffic, I do not start by asking whether the model can be loaded. I ask a more practical question: what happens when many requests arrive at the same time?
The first instinct is often to focus on model weights. A 7B model in FP16 may take a dozen GB of memory, so an 80GB GPU sounds comfortable. But under high concurrency, the pressure usually comes from another place: KV Cache.
KV Cache: the cached Key and Value tensors produced during autoregressive generation. It avoids recalculating the whole context for every new token, but each request and each generated token consumes GPU memory.
A Small Model Does Not Mean High Concurrency Is Easy
For a typical 7B-level model with hidden size 4096, 32 layers, and FP16 precision, one token of KV Cache is roughly:
2(K+V) x 32(layers) x 4096(hidden size) x 2(FP16 bytes) = 524288 bytes
That is about 0.5MB per token.
If one request has a 4096-token context, its KV Cache is close to 2GB. With 100 concurrent requests, the cache alone can reach 200GB. This already exceeds a single A100 80GB GPU by a large margin.
That is why I treat KV Cache as a first-class capacity metric. Model weights matter, but request growth can turn the cache into the actual memory bottleneck.
Traditional KV Cache Management Wastes Memory
The cache itself is useful. The problem is how it is managed.
Traditional allocation has three common forms of waste.
The first is internal fragmentation. A system may reserve space for 2048 tokens, while the user only generates 500 tokens. The remaining capacity is allocated but unused.
The second is external fragmentation. The GPU may have enough free memory in total, but not enough continuous memory for a new request.
The third is copying overhead in Beam Search or Parallel Sampling.
Beam Search: a decoding method that keeps multiple candidate paths at each step and chooses the best final sequence. It improves stability for some tasks, but it can duplicate shared prefixes.
Parallel Sampling: generating multiple responses from the same prompt at the same time. The prompt prefix is identical, so the KV Cache for that prefix should not be copied repeatedly.
If three branches share the same prompt prefix, storing the same prefix cache three times is pure waste.
PagedAttention Treats GPU Memory Like Pages
PagedAttention: a KV Cache management technique that splits cache storage into fixed-size blocks and uses a mapping table to connect logical token positions to physical GPU blocks.
I like to explain it with a library analogy. A book series does not need to sit on consecutive shelves. As long as a catalog records where each volume is, readers can still read the series in order.
In vLLM, that catalog is the Block Table.
Block Table: the mapping from logical sequence blocks to physical GPU memory blocks. During attention computation, the kernel uses this mapping to find the actual K/V tensors.
Here is a simplified version of the idea:
class BlockTable:
# 中文注释：初始化块表，block_size 表示每个物理块能容纳多少个 token
def __init__(self, block_size=16):
self.block_size = block_size
self.blocks = []
self.num_tokens = 0
# 中文注释：每追加一个 token，就判断当前块是否已满，满了再申请新显存块
def append_token(self):
if self.num_tokens % self.block_size == 0:
new_block = gpu_memory.allocate_block()
self.blocks.append(new_block)
self.num_tokens += 1
# 中文注释：把逻辑位置转换成物理显存位置，attention kernel 依赖这个映射取数据
def get_physical_indices(self, position: int):
block_idx = position // self.block_size
offset = position % self.block_size
physical_block = self.blocks[block_idx]
return physical_block * self.block_size + offset
PagedAttention solves two practical problems.
First, it allocates on demand. If a request generates only 100 tokens and the block size is 16, it needs 7 blocks, not a full preallocated 2048-token region.
Second, it supports Copy-on-Write.
Copy-on-Write: shared cache blocks are reused by multiple branches until a branch actually diverges. Only then does the system allocate a private block for that branch.
This is exactly what Beam Search and Parallel Sampling need.
Continuous Batching Keeps the GPU Busy
Memory management is only half of the story. The other half is scheduling.
Batching: grouping multiple requests into one GPU forward pass. GPUs are good at parallel computation, so batching can significantly improve throughput.
Static batching has a weakness: every request in the batch waits for the longest one. Short requests finish early, but their slots remain idle.
Continuous Batching: after each token generation step, the scheduler checks which requests are done and immediately fills freed slots with new requests.
Iteration-level Scheduling: scheduling at every token generation iteration instead of waiting for a whole request batch to finish.
PagedAttention makes Continuous Batching easier because new requests do not require large continuous memory regions. Any available free blocks can be reused.
How I Judge Whether the System Is Tuned Well
I do not judge a vLLM service by whether it returns an answer once. I look at three metrics.
TTFT, Time To First Token: the time from request submission to the first generated token. It is heavily affected by prefill, queueing, and cold start.
TPOT, Time Per Output Token: the interval between output tokens during decoding. It shows whether the decode stage is stable.
Throughput: generated tokens per second. It reflects the service capacity.
In the course experiment, concurrency from 1 to 8 increased throughput from about 101 tokens/s to 635 tokens/s, while P50 latency only moved from about 900ms to 1037ms. That is Continuous Batching doing its job.
At concurrency 16, P99 latency jumped to about 2085ms. That is the saturation point I care about. Production tuning is not about pushing concurrency forever. It is about finding the point where throughput and tail latency are both acceptable.
Closing
The core challenge in high-concurrency LLM serving is not just loading the model. It is managing KV Cache growth and keeping the GPU consistently utilized.
My usual order is simple: calculate the KV Cache cost, use PagedAttention to reduce fragmentation, and validate the throughput-latency turning point with Continuous Batching.
Once this mental model is clear, vLLM tuning stops being random parameter guessing.