How Weights Become Words: LLM Inference Explained

中文

When I first downloaded an open model, I instinctively treated its multi-gigabyte folder as an offline chatbot. It is not. There is no chat window inside a .safetensors file, and no answer appears until an inference runtime loads the files onto hardware and repeatedly calculates the next token.

One precision point matters. “Open models only release weights, not algorithms” is too absolute. A release may include weights, architecture configuration, tokenizer files, inference code, or even training code; another may include only a subset. Weights are usually the largest thing users download. The Transformer computation and a runtime such as vLLM or Ollama are the other half that turns those static numbers into text.

What is actually inside the folder?

I separate it into three kinds of assets:

• Weights (.safetensors, often sharded): enormous floating-point matrices learned during training. They are long-term statistical memory, not executable behavior by themselves.
• Tokenizer files (tokenizer.json, vocab.json, merges): the dictionary and rules that map text to integer token IDs and back. A model sees IDs, not Chinese characters or English words directly.
• Configuration and conventions (config.json, generation configuration, chat template, special tokens): the contract that says how many layers exist, the hidden width, vocabulary size, and where a conversation starts or ends.

The architecture must match the weights. Using the wrong architecture or tokenizer is like fitting the right gears into the wrong engine: the numbers remain present, but their meaning no longer connects.

From a sentence to vectors

For a prompt such as “Is it raining in Beijing today?”, the runtime first applies the model’s chat template and then calls the tokenizer. The result is a sequence of token IDs. A token can be a character, part of a word, punctuation, or a frequent string—it is not guaranteed to be one word or one character.

Embedding then uses each ID as a row index into a learned table:

[batch, seq_len]  token IDs
        ↓ embedding lookup
[batch, seq_len, hidden_size]  dense vectors

With a hidden size of 4096, each token becomes a vector of 4096 numbers. Higher dimension is not a single “smarter” dial; it is an architecture trade-off among capacity, parameter count, computation, and memory.

Transformer layers: read context, then transform

That vector sequence passes through dozens of Transformer layers. Each layer broadly does two things: attention mixes relevant context across positions; a feed-forward network (FFN) applies a nonlinear transformation at each position. Residual connections preserve a direct information path, while RMSNorm or LayerNorm keeps numerical scale stable.

Q, K, and V are learned context matching

The layer projects each vector into Q (Query), K (Key), and V (Value).

• Q asks what the current position needs.
• K describes what each available position can match.
• V carries the information that a successful match contributes.

Q is compared with K, scaled by 1/√d, normalized with softmax, and used to weight V. The scale keeps dot products from becoming unnecessarily large as dimensionality grows, so softmax does not become extreme too early. It improves training stability; at inference it remains part of the model’s defined computation.

Autoregressive models also use a causal mask: token 10 can read tokens 1–10, never token 11. Multi-head attention runs several Q/K/V projections in parallel, letting different heads model different relationships before their results are combined.

Why the FFN and activation are essential

Attention moves information between positions, but it is not enough on its own. The FFN then expands, activates, and projects each token representation. Modern models often use GeLU or SwiGLU. Without nonlinear activation, any stack of linear layers collapses mathematically into one linear transformation and loses the ability to learn complex patterns.

From hidden vector to next token

After N layers, the last hidden vector goes through the LM Head, a linear projection from hidden size to vocabulary size. Its output is a vector of logits—scores, not probabilities. Softmax turns them into a probability distribution, and a decoding policy chooses one ID:

• Greedy decoding chooses the highest-probability token every time.
• Temperature adjusts how sharp or exploratory the distribution is.
• Top-p sampling draws only from the smallest high-probability set whose cumulative probability reaches p.

The chosen ID is decoded into a text fragment and sent back over a streaming response. It looks like a model emits one character at a time, but the accurate unit is one token at a time; a token can contain several English characters or pieces of code.

Why it stops: EOS, not EOF

This is where file-system terminology causes confusion. EOF is a signal for finishing a file read. It is not the normal signal that stops language-model generation. A model vocabulary defines a special EOS (End Of Sequence) token. During training, the model learns that EOS is often appropriate when an answer has naturally finished. When the runtime samples it, generation normally stops.

Production systems add other stop conditions: max_tokens, configured stop tokens or strings, a disconnected client, a timeout, or policy limits. So stopping is a joint outcome of the model’s probability distribution, special tokens, and runtime policy—not a mysterious end-of-file marker.

What vLLM and Ollama do

They do not rewrite weights into answers. They load model assets, build the matching computation, place tensors on CPU/GPU, invoke optimized kernels, schedule requests, decode tokens, and stream those tokens to a client.

The work has two phases:

1. Prefill processes the full prompt and builds each layer’s K/V state.
2. Decode feeds only the newly generated token, reuses the historical K/V state, predicts another token, and repeats until a stop condition.

Without a KV cache, past K and V would be recomputed for the entire history on every new token. KV caching preserves them. vLLM’s PagedAttention manages that cache in blocks, reducing memory fragmentation and enabling flexible sharing across many concurrent requests. That is a core reason it performs so well as a high-throughput serving engine.

Ollama emphasizes convenient local model management, quantized formats, and an approachable API. vLLM emphasizes server-side scheduling, batching, and throughput. Their internals and format support differ, but both make static model assets run as next-token prediction.

The whole chain in one line

text → chat template + tokenizer → token IDs → embeddings
→ [attention + FFN + residual/norm] × N → LM Head logits
→ decode/sample → streamed tokens → EOS or another stop rule

The useful habit is to ask three separate questions: which assets did I download, does the runtime understand this architecture and tokenizer, and which cache/quantization/decoding policies are in use? Once those are distinct, the black box between “I downloaded a model” and “it started talking” becomes much easier to reason about.

Next, I will go deeper into KV cache memory use, continuous batching, and why PagedAttention can avoid the awkward situation where a GPU appears to have free memory but cannot admit another request.