Large Language Models
A large language model (LLM) is a very big neural network trained on huge amounts of text to predict what comes next.
What Makes It "Large"?
- It is trained on a massive amount of text: books, websites, code and more.
- It has billions of parameters (the weights you met in the last lesson).
- Training it needs thousands of powerful chips running for weeks.
Models like ChatGPT, Gemini and Claude are built on LLMs.
The Core Trick: Predict the Next Word
An LLM reads some text and predicts the most likely next piece of text. It does this again and again, one piece at a time, to write a full answer.
Here is a toy version that "learns" from three sentences which word usually comes next:
Example
from collections import Counter, defaultdict
text = "chai is hot . chai is sweet . chai is hot"
words = text.split()
next_words = defaultdict(Counter)
for a, b in zip(words, words[1:]):
next_words[a][b] += 1
print(next_words["is"].most_common())
print("Prediction after 'is':", next_words["is"].most_common(1)[0][0])[('hot', 2), ('sweet', 1)]
Prediction after 'is': hotA real LLM is far smarter: it looks at the whole conversation, not just the last word, and it has learned grammar, facts, reasoning patterns and coding styles from its training data.
Tokens
LLMs do not read whole words. They read tokens: small pieces of text. A common English word is often one token; a long or rare word may be split into several. Urdu text usually takes more tokens than the same message in English.
Example
"Workownix teaches AI" might become tokens like:
["Work", "own", "ix", " teaches", " AI"]The amount of text a model can look at once is called its context window, measured in tokens.
How an LLM is Made
| Stage | What happens |
|---|---|
| Pre-training | The model reads enormous amounts of text and learns to predict the next token |
| Fine-tuning | It is trained on good example conversations so it follows instructions |
| Human feedback | People rate answers, and the model learns to prefer helpful, honest, safe replies |
| Adding tools | It is connected to search, code running, files and other apps |
What LLMs Are Good At
- Explaining topics in simple words, in many languages
- Summarising long documents
- Writing and fixing emails, essays and code
- Brainstorming ideas
- Turning messy notes into tables or lists
Limits You Must Know
- Hallucinations: an LLM can state false things confidently, such as a fake reference or a wrong date.
- Knowledge cutoff: it only knows what was in its training data, unless it can search the web.
- No real experience: it predicts text; it has not lived in the world.
- Maths and counting: it can slip on exact calculations unless it uses a calculator or code tool.
Use an LLM to get a first draft or an explanation, then check facts, numbers and names before you rely on them.
Exercise
Exercise
At its core, what does a large language model do?