How LLMs Generate Text

A language model writes its answer one small piece at a time, each time choosing what is likely to come next.

The Generation Loop

1. The model reads your prompt (and the conversation so far).
2. It calculates a probability for every possible next token.
3. It picks one token.
4. It adds that token to the text and repeats from step 1.
5. It stops when it produces a special "end" token or reaches a length limit.

This is why answers appear word by word on screen: the model really is producing them in order.

Probabilities

Suppose the text so far is "Karachi is a big". The model might give:

Here is a toy loop that always picks the most likely next word:

next_word = {
    "Karachi": {"is": 0.9, "has": 0.1},
    "is": {"a": 0.8, "very": 0.2},
    "a": {"big": 0.7, "busy": 0.3},
    "big": {"city": 0.72, "port": 0.15, "place": 0.08, "market": 0.05},
}
text = ["Karachi"]
while text[-1] in next_word:
    options = next_word[text[-1]]
    text.append(max(options, key=options.get))
print(" ".join(text))

Karachi is a big city

Temperature: Safe or Creative

Always picking the top word gives boring, repetitive text. So models sample: they sometimes pick less likely words. A setting called temperature controls how adventurous this is:

• Low temperature (near 0): picks the likeliest words. Good for facts, code and data extraction.
• High temperature (around 1 or more): more varied and creative. Good for stories and brainstorming, but more mistakes.

import math, random

scores = {"city": 2.0, "port": 0.5, "place": 0.0}

def probabilities(temperature):
    exp = {w: math.exp(s / temperature) for w, s in scores.items()}
    total = sum(exp.values())
    return {w: round(v / total, 2) for w, v in exp.items()}

print("Low temperature: ", probabilities(0.5))
print("High temperature:", probabilities(2.0))

random.seed(3)
p = probabilities(2.0)
print("Sampled word:", random.choices(list(p), weights=p.values())[0])

Low temperature:  {'city': 0.94, 'port': 0.05, 'place': 0.02}
High temperature: {'city': 0.54, 'port': 0.26, 'place': 0.2}
Sampled word: city

At low temperature "city" is almost certain; at high temperature the other words get a real chance.

System Prompts and Roles

A conversation sent to a model usually has messages with roles:

• system: hidden instructions from the app maker ("You are a polite support agent for a Karachi bakery. Answer in short sentences.")
• user: what you type
• assistant: the model's earlier replies

The model generates its next reply based on *all* of these together.

Reasoning Models

Some modern models can "think" before answering: they generate hidden working steps first, then the final answer. This helps with maths, logic and planning, but it takes more time and costs more.

Why This Matters for You

• The model predicts likely text; it does not look up a guaranteed truth. That is why it can be wrong.
• Clear prompts make the right answer more likely.
• For exact work (numbers, code, facts), ask for low creativity and check the result.

Exercise