Why Speculative Decoding Speeds Up LLM Inference

中文版

LLMs have a natural bottleneck: they usually generate one token at a time.

This is autoregressive generation. The model needs the previous token before it can generate the next one. Even with a strong GPU, the decode stage can become a long serial process.

Speculative decoding tries to reduce that serial cost.

Speculative Decoding: a method where a smaller and cheaper draft model proposes multiple candidate tokens, and the larger target model verifies them in one pass. Accepted tokens are kept. Rejected tokens are corrected by the target model.

Draft First, Verify Later

I usually explain speculative decoding as a draft-and-review workflow.

Suppose the target model would normally generate a sequence like this:

Step 1: input "I", output "love"
Step 2: input "I love", output "China"
Step 3: input "I love China", output EOS

Traditional decoding needs multiple target-model forward passes.

With speculative decoding, the draft model first proposes several tokens. The target model then verifies those tokens together. If the first two tokens are accepted and the third is rejected, the step is still useful: one target-model pass has confirmed multiple tokens and corrected the wrong one.

Draft Model: the smaller model or lightweight module that proposes candidate tokens quickly.

Target Model: the main model that determines which tokens are accepted and how rejected positions should be corrected.

The important point is that final output quality is still controlled by the target model.

Acceptance Rate Decides the Benefit

Speculative decoding is useful only when the draft is good enough.

Acceptance Rate: the percentage of draft tokens accepted by the target model. If the draft proposes 5 tokens and the target accepts 3, the acceptance rate is 0.6.

Higher acceptance rate means each target-model pass produces more useful tokens. Lower acceptance rate means the draft stage may become wasted work.

There is also a cost balance. A draft model that is too small may be inaccurate. A draft model that is too large may consume the compute savings. The useful point is not “the biggest draft model,” but the cheapest draft model that is accurate enough.

Medusa Avoids a Separate Draft Model

Medusa takes an engineering-friendly route: instead of deploying another draft model, it adds multiple decoding heads to the existing model.

Medusa: an inference acceleration framework that predicts not only the next token, but also several future tokens through additional decoding heads.

This can simplify deployment because there may be no separate draft model to maintain. But the complexity does not disappear. The extra heads need training or adaptation, and verification may require Tree Attention.

Tree Attention: an attention structure used to verify multiple candidate branches at once. It deals with a candidate tree, not just a single linear sequence.

EAGLE Drafts at the Feature Level

EAGLE goes in another direction. It uses intermediate features from the target model to improve draft quality.

EAGLE: a feature-level speculative decoding method. A lightweight draft module predicts future features and tokens, then the target model verifies the candidates.

EAGLE-1 and EAGLE-2 can provide clear speedups, but high-batch production serving introduces another question: is the GPU already compute-bound?

Compute-Bound: the bottleneck is compute capacity. The GPU is already close to fully utilized, so extra draft computation may reduce the benefit.

Memory-Bound: the bottleneck is memory access or bandwidth. There is unused compute capacity, so speculative decoding can make better use of the GPU.

This is the correction I often make: speculative decoding does not always speed things up. Its value depends on batch size, draft cost, acceptance rate, and the current bottleneck.

Why EAGLE-3 Matters

EAGLE-1 and EAGLE-2 have a hidden issue: training and inference are not fully aligned.

During training, the draft module sees clean target-model features. During inference, it may see features influenced by its own previous predictions. Errors can accumulate, and acceptance rate drops over multiple speculative steps.

EAGLE-3 improves this with Training-time Test.

Training-time Test: a training method that simulates multi-step inference during training, so the draft module learns to handle states created by its own predictions.

I think of it as moving from “practicing with the answer sheet” to “taking a closed-book mock exam.” The model becomes more robust when its own intermediate predictions are imperfect.

The course result is meaningful: EAGLE-1/2 may lose most of their benefit at Batch=64, while EAGLE-3 can still keep a throughput gain around 1.38x. That matters for real high-concurrency serving.

Integration with vLLM Is a Systems Problem

Adding speculative decoding to vLLM is not just adding a paper name to a command line.

There are at least three engineering conflicts.

First, the draft stage may prefer small micro-batches, while Continuous Batching tries to keep the main batch dynamically full.

Second, Medusa or EAGLE verification may require Tree Attention, special masks, or a custom attention backend.

Third, both draft and target stages consume memory and may produce KV Cache. Without unified cache management, speculative decoding can amplify memory pressure.

Here is the simplified flow:

# 中文注释：从当前 batch 里挑出适合做推测解码的请求
draft_requests = select_for_speculation(current_batch)

# 中文注释：轻量 EAGLE 模型先生成 3 到 5 步候选 token 树
for step in range(max_draft_tokens):
    draft_features = eagle_model.forward(target_features, previous_tokens)
    draft_tokens = sample(draft_features)

# 中文注释：目标模型并行验证候选 token，并返回被接受的长度
verified_tokens, accepted_length = tree_verify(
    draft_tokens,
    target_model_logits,
)

# 中文注释：只保留被接受 token 对应的 KV blocks，释放被拒绝路径占用的 blocks
update_block_table(request, accepted_length)

The algorithm and the scheduler are tightly coupled. Candidate selection, verification, and cache cleanup all affect the final speedup.

Should You Use It?

Speculative decoding is not magic guessing. It is a trade: use cheaper compute to reduce expensive target-model work.

Before using it in production, I ask four questions:

1. Is the decode stage memory-bound or compute-bound?
2. How cheap is the draft model or draft head?
3. Is the acceptance rate stable?
4. Does throughput improve under large batch, not just single-request latency?

If these questions are unclear, speculative decoding can become extra overhead. If the answers are solid, it is one of the most useful acceleration techniques in LLM serving.