Tokens and Context Windows

Language models read and write in tokens, and they can only look at a limited number of tokens at once.

What is a Token?

A token is a small chunk of text: a whole word, part of a word, a space plus a word, or a punctuation mark. Each model has its own way of splitting text (its tokenizer).

A rough rule for English: 1 token is about 4 characters, or about three quarters of a word. So 100 words is roughly 130 tokens. Urdu, Hindi and Arabic scripts usually need more tokens for the same meaning.

"Ayesha studies at Workownix."
might be split as:
["A", "yesha", " studies", " at", " Work", "own", "ix", "."]
-> 8 tokens

Here is a quick estimator using the 4-characters rule:

def estimate_tokens(text):
    return max(1, round(len(text) / 4))

notes = "Photosynthesis is how plants make food using sunlight, water and air."
print("Characters:", len(notes))
print("Estimated tokens:", estimate_tokens(notes))

Characters: 69
Estimated tokens: 17

This is only an estimate. AI companies give exact token counters for their own models.

The Context Window

The context window is the maximum number of tokens the model can consider at one time. It includes:

• the system prompt (hidden instructions),
• the whole conversation so far,
• any files or documents you attached,
• and the reply being written.

Modern models have large windows, often enough for a whole book, but every model still has a limit.

Why Long Chats "Forget"

When a conversation gets longer than the window, the app has to drop or summarise older parts. Even inside the window, models pay less attention to details buried in the middle of very long text.

Here is a simple program that keeps only the newest messages that fit a budget:

messages = ["msg1 " * 30, "msg2 " * 30, "msg3 " * 30, "msg4 " * 30]
budget = 100   # tokens allowed

kept, used = [], 0
for m in reversed(messages):
    cost = round(len(m) / 4)
    if used + cost > budget:
        break
    kept.insert(0, m.split()[0])
    used += cost
print("Kept:", kept, "Tokens used:", used)

Kept: ['msg3', 'msg4'] Tokens used: 76

The oldest messages were dropped. Chat apps do something like this behind the scenes.

Tokens and Cost

AI APIs usually charge per token, often with different prices for input tokens (what you send) and output tokens (what the model writes). This matters when you build apps:

• Send only the text the model needs.
• Ask for short answers when short answers are enough.
• Long documents and long chat histories cost more on every request.

Exercise