Send the same question to a language model several times and you may receive different answers. Continue a chat for long enough and an instruction from the beginning may disappear. Calling this a change in the model's mood does not help us diagnose either behavior.
A more useful model is mechanical: an LLM receives tokens, processes the tokens available in a finite context window, and samples what comes next. These three ideas are enough to make later work with APIs, retrieval, tools, and prompts much easier to reason about.
A token is not necessarily a word
Before a model can process text, a tokenizer converts it into token IDs. A token may represent a word, part of a word, punctuation, a Chinese character, or a frequent character sequence.
Two models can split the same sentence differently because tokenizers use different vocabularies and algorithms. BPE, Unigram, and WordPiece are common examples. Rules such as “one Chinese character equals one token” are rough estimates, not accounting methods.
Use the tokenizer associated with the target model, or usage data returned by the service, when you need an exact count. A count produced for model A does not establish the cost or request size for model B.
Generation repeats one decision
For an introductory mental model, a Transformer calculates a probability distribution for the next token given the current context. If the input is “The order arrived late, so support should,” plausible candidates may include “apologize,” “check,” and “explain.”
After selecting one token, the model appends it to the sequence and calculates again. The loop continues until it emits a stopping token or reaches a configured limit.
This is not the same operation as retrieving a document. Generated prose can sound complete even when a statement has no dependable source. Versions, prices, regulations, and current events still require retrieval and source verification.
The context window is a workbench
The context window limits how many tokens a request can hold. System instructions, conversation history, the new question, tool results, and the response budget all consume space.
A chat application appears to remember because it usually sends previous messages again with each request. The model does not automatically preserve every earlier conversation for the application. As a conversation grows, the application may truncate old messages, summarize them, or retrieve selected records from external storage.
An instruction can disappear after many turns because it is no longer included, or because a large amount of unrelated material now competes for attention. Durable information belongs in a database, a knowledge store, or a dedicated memory component. Context is working space, not an archive.
Temperature and Top P change selection
Temperature reshapes the candidate distribution. Lower values tend to favor high-probability candidates and produce more stable output. Higher values allow more alternatives into the result, increasing variation.
Top P keeps a set of candidates whose cumulative probability reaches a threshold and samples from that set. It is not a correctness score. Alibaba Cloud's text-generation documentation recommends adjusting either Temperature or Top P in most cases, rather than changing both and losing a clear cause for the observed behavior.
Classification, extraction, and fixed-format output usually benefit from a stable starting point. Brainstorming may need more variety. Exact values remain model- and task-dependent.
Run a controlled experiment
Use this fixed prompt with one model and one system instruction:
Rewrite “The order arrived late” as a customer-support reply under 20 words.
Return only the reply.
Run it five times with a low Temperature and save every result. Raise only Temperature, leave the other settings unchanged, and run it five more times.
Check more than writing quality:
- How many distinct phrasings appear?
- Does every response stay under 20 words?
- Does the model promise a refund without authorization?
- Do format violations increase with variation?
Changing the model, prompt, Temperature, and Top P together makes the result impossible to attribute. Change one variable at a time and the experiment becomes repeatable.
Check your mental model
You are ready for API work if you can answer these questions:
- Why can token counts differ between models?
- Why does a chat application resend conversation history?
- Why does higher Temperature not mean higher accuracy?
- Why is a context window insufficient for durable memory?
The next article turns these ideas into an API request with environment-based credentials, error handling, and logs that make failures diagnosable.