How Do LLMs Actually Work?

From text to tokens to predictions — the pipeline behind every AI chat, explained in plain English.

↓

WATCH IT

Watch an LLM "Think"

AI OUTPUT

INTERACTIVE

Be the AI: Predict the Next Word

You are the language model. Given the context, what word comes next? 4 rounds — some have obvious answers, some are traps.

🔑

1. Tokenisation

Tap to flip

Text is split into tokens — sub-word chunks, not whole words. “Unhelpful” becomes [“un”, “help”, “ful”]. Common words stay whole; rare words get split. A typical LLM vocabulary is 50,000–100,000 tokens. ^[1]

🗺️

2. Embeddings

Tap to flip

Each token becomes a vector — a list of hundreds or thousands of numbers representing meaning. Similar words (“king”, “queen”) get similar vectors. The model learns these during training. ^[2]

🧠

3. Attention

Tap to flip

The self-attention mechanism lets each token look at every other token and decide what is relevant. In “The bank deposit was large,” attention links “bank” to “deposit” to resolve the meaning. This is the core of the Transformer architecture. ^[3]

🔄

4. Transformer Layers

Tap to flip

The token vectors pass through dozens of Transformer layers (GPT-4 has 96+). Each layer refines the representation by blending attention with feed-forward networks. Deep layers capture more abstract patterns — grammar, reasoning, knowledge.

📊

5. Next-Token Prediction

Tap to flip

The final layer outputs a probability distribution over the entire vocabulary — a score for every possible next token. The model does not “choose” in the human sense; it samples from this distribution. The most probable token is usually selected (depending on temperature). ^[4]

📝

6. Training (Pre-training + Fine-tuning)

Tap to flip

Pre-training: the model reads trillions of tokens of internet text, learning to predict the next token. This is massively expensive (millions of dollars). Fine-tuning (RLHF): humans rank responses, and the model learns to be helpful and safe. RLHF costs <2% of pre-training compute. ^[5]

🔍

Emergent Abilities

Tap to flip

Skills like arithmetic, translation, and reasoning appear suddenly at sufficient scale — they do not improve gradually. A 1B parameter model cannot do math; a 100B model can. This suggests the model is not just memorising, but learning transferable patterns. ^[6]

🎯

Attention Is All You Need

Tap to flip

The 2017 paper “Attention Is All You Need” by Vaswani et al. introduced the Transformer. Before it, models processed text sequentially (RNNs). Attention lets the model look at all tokens simultaneously, enabling parallel training and much deeper understanding of context.

🧱

Scale = Capability

Tap to flip

More parameters + more data + more compute = better performance. This is the scaling laws discovery (Kaplan et al., 2020). It is why GPT-4 (1.7T params) outperforms GPT-2 (1.5B) — not a new algorithm, just more of the same.

🎮

Pattern Matching, Not Understanding

Tap to flip

The model does not understand concepts. It has learned statistical patterns: “after X, Y is likely.” This is why it can write fluent essays but also hallucinate confidently — the patterns it generates are plausible, not verified.

🌟

Context Window

Tap to flip

The model can only “see” a limited number of tokens at once — the context window. Early GPT: 2K tokens. GPT-4: 128K+. Claude: 200K+. Beyond the window, earlier text is forgotten. This is why long conversations sometimes “forget” earlier points.

🔑

Fine-Tuning Shapes Behaviour

Tap to flip

Pre-training produces a model that completes text, not one that answers questions. Fine-tuning with RLHF teaches it to follow instructions, refuse harmful requests, and respond helpfully. This is what makes a chatbot a chatbot.

QUIZ

After reading trillions of words during training

An LLM understands the meaning of the words it generates.

When you type a prompt into ChatGPT

The model searches the internet to find an answer for you.

In the Transformer architecture

Self-attention allows each token to consider every other token in the sequence.

When choosing the next word to output

The model always picks the single most probable next token.

About the training process

RLHF (reinforcement learning from human feedback) costs more compute than pre-training.

About tokenisation

Each token always represents exactly one word.

INTERACTIVE

Temperature: Creativity vs Precision

Slide to see how temperature changes LLM output — from deterministic to creative.

050100

PROTECT YOURSELF

Understanding LLMs: Key Takeaways

Tap to check off each concept you have understood.

LLMs predict the next token

Everything — reasoning, coding, translation — emerges from next-token prediction trained on massive text.

Tokens, not words

Text is split into sub-word tokens. “Unhelpful” = [un + help + ful]. Vocabulary is typically 50K–100K tokens.

Embeddings encode meaning

Each token becomes a vector of numbers. Similar words have similar vectors, learned during training.

Attention connects tokens

Self-attention lets each token weigh the relevance of every other token. This is the core of the Transformer.

More scale = more capability

More parameters, data, and compute produce better models. Emergent abilities appear at sufficient scale.

Temperature controls randomness

Low = deterministic and precise. High = creative and unpredictable. Chatbots default to 0.5–0.7.

LLMs do not understand

They learn statistical patterns, not meaning. Fluency does not equal comprehension. Always verify important outputs.