Tokens and Context Windows

Models do not read letters or words. They read tokens, and everything about cost, speed and memory is measured in tokens.

Tokens in One Minute

A token is a small piece of text: a whole short word, part of a long word, a space with a word, or a punctuation mark. The model has a fixed list of tokens (its vocabulary) and turns every text into a list of token numbers.

The Tokens and Context Windows chapter of the Gen AI Tutorial explains the basics. Here we focus on what tokens mean when you build with AI.

A Rough Token Estimate

Each provider has its own tokenizer, so exact counts differ. For English, a common rule of thumb is about 4 characters per token. That is enough for planning:

def estimate_tokens(text):
    # Rough rule for English: about 4 characters per token
    return max(1, round(len(text) / 4))

samples = [
    "Hello",
    "What time does the shop in Saddar open?",
    "Assalam o Alaikum, kya aap ka course online bhi hai?",
]
print("Tokens | Text")
for s in samples:
    print(f"{estimate_tokens(s):>6} | {s}")

Tokens | Text
     1 | Hello
    10 | What time does the shop in Saddar open?
    13 | Assalam o Alaikum, kya aap ka course online bhi hai?

Why Tokens Matter for Builders

Output tokens usually cost more than input tokens. Asking for "a short answer in 3 sentences" saves real money in a busy app.

The Context Window

The context window is the total number of tokens the model can see in one request. It includes:

• The system prompt (your instructions)
• The whole chat history you send
• Any documents you paste in
• The answer the model writes

The model has no memory between requests. When a chatbot "remembers" earlier messages, it is because your app sends the old messages again every time.

Modern models have large context windows, often hundreds of thousands of tokens. But bigger is not always better:

• Every token you send costs money, every time
• Models can miss details buried in the middle of a very long input
• Long requests are slower

So good apps send only what is needed.

Fitting a Chat into a Budget

A common job: keep the system prompt, then keep as many recent messages as fit into a token budget, and drop the oldest ones.

def estimate_tokens(text):
    return max(1, round(len(text) / 4))

def trim_history(messages, budget):
    """Keep the system message, then as many recent messages as fit."""
    system, rest = messages[0], messages[1:]
    kept, used = [], estimate_tokens(system["content"])
    for msg in reversed(rest):               # newest first
        cost = estimate_tokens(msg["content"])
        if used + cost > budget:
            break
        kept.insert(0, msg)
        used += cost
    return [system] + kept, used

chat = [
    {"role": "system", "content": "You are a helpful tutor for Ayesha."},
    {"role": "user", "content": "Explain what a variable is in Python, please."},
    {"role": "assistant", "content": "A variable is a name that stores a value, like age = 18."},
    {"role": "user", "content": "And what is a list?"},
    {"role": "assistant", "content": "A list holds many values in order, like [1, 2, 3]."},
    {"role": "user", "content": "Give me one exercise on lists."},
]
trimmed, used = trim_history(chat, budget=40)
for m in trimmed:
    print(m["role"], ":", m["content"])
print("Estimated tokens:", used)

system : You are a helpful tutor for Ayesha.
user : And what is a list?
assistant : A list holds many values in order, like [1, 2, 3].
user : Give me one exercise on lists.
Estimated tokens: 34

The two oldest messages about variables were dropped, but the system prompt and the latest question stayed.

Other Ways to Save Tokens

• Summarise old messages: replace ten old messages with a three-line summary.
• Retrieve, do not paste: send only the relevant parts of a document (this is RAG, in Part 2).
• Set a maximum output length with a setting like max_tokens.
• Use prompt caching if your provider offers it: a long system prompt that stays the same can be charged at a lower price on repeat requests.

Exercise