The Transformer

We now have a matrix of embedding vectors, one row per token. The question is: what does something with those vectors to produce language?

Before Transformers

Before 2017, the answer was recurrent neural networks (RNNs, LSTMs). They processed tokens one at a time, left to right, passing a “hidden state” from each step to the next. You can picture it like a game of telephone: each word whispers some context forward to the next word.

This worked, but had two big problems:

1. Slow training. You can’t parallelize: token 5 can’t be processed until tokens 1-4 are done. On a GPU with thousands of cores sitting idle, this is painful.
2. Forgetting. By the time you reach token 500, the hidden state has been overwritten so many times that it’s mostly forgotten what token 10 said. Long-range dependencies (like a pronoun referring to a noun 200 words earlier) are unreliable.

The Key Insight: Attention

The transformer, introduced in the 2017 paper “Attention Is All You Need,” solved both problems with a single idea: let every token look at every other token directly.

Instead of passing information sequentially through a chain of hidden states, attention lets token 500 directly query token 10 and ask “are you relevant to me?” If yes, it pulls in that information. If no, it ignores it. Every token does this simultaneously, in parallel.

This means:

1. Training is parallelizable. All tokens are processed at once. GPUs are happy.
2. Long-range dependencies are first-class. Token 500 can directly attend to token 10 without information passing through 490 intermediate steps.
3. The architecture scales. Adding more layers and parameters consistently improves performance. This is why we went from millions to billions to trillions of parameters.

Every modern LLM (GPT, LLaMA, Mistral, Claude) is a transformer. What varies is the size, training data, and specific optimizations, but the core architecture is the same.

High-Level Structure

A decoder-only transformer (the kind used by GPT, LLaMA, Mistral) looks like this:

Input tokens → Embeddings + Position → [Transformer Block × N] → Output logits

The “Transformer Block × N” is the heart of the model. A 7B-parameter model typically has 32 of these blocks stacked on top of each other. A 70B model has 80.

Each block contains two sub-layers:

1. Multi-Head Self-Attention: lets every token attend to every other token
2. Feed-Forward Network (FFN): transforms each token independently

Let’s look at each one.

Self-Attention: The Core Mechanism

Attention is the mechanism that lets the model decide which other tokens matter when processing the current token.

The Intuition

Imagine you’re reading the sentence: “The cat sat on the mat because it was tired.”

What does “it” refer to? You (a human) instantly know it refers to “the cat,” not “the mat.” You make this connection by attending to the earlier parts of the sentence.

Attention does the same thing, but mechanically. For each token, the model computes a relevance score against every other token, then takes a weighted average of their information. Tokens that are relevant contribute more; irrelevant tokens contribute nothing.

The Mechanics: Q, K, V

For each token, the model computes three vectors from the embedding:

• Query (Q): “What am I looking for?” (what kind of information does this token need?)
• Key (K): “What do I contain?” (what kind of information does this token offer?)
• Value (V): “What information do I carry?” (the actual content to pass along if selected)

These are computed by multiplying the input by learned weight matrices:

where $X$ is the input matrix (one row per token, each row is the embedding vector) and each $W$ is a learned weight matrix that the model adjusts during training.

Computing Attention Scores

The attention score between two tokens is the dot product of one token’s Query and another’s Key. If you’ve used cosine similarity or dot products before, this is the same idea: vectors pointing in similar directions get high scores.

The full attention formula:

Let’s break this into pieces:

A Concrete Example

Say we’re processing “The cat sat on the mat because it was tired” and we’re computing the output for the token “it”:

1. “it”’s Query dot-producted with each token’s Key gives scores like:
   • “The”: 0.1, “cat”: 0.8, “sat”: 0.2, “on”: 0.05, “the”: 0.1, “mat”: 0.3, “because”: 0.1, “it”: 0.15, “was”: 0.1, “tired”: 0.05
2. After softmax, these become attention weights:
   • “cat” gets ~40% of the attention, “mat” gets ~15%, everything else shares the remaining ~45%
3. The output for “it” is ~40% of “cat”’s Value vector + ~15% of “mat”’s Value vector + small contributions from everything else

The model has learned (through training on billions of sentences) that “it” usually refers to the subject, so the Query for “it” naturally aligns with Keys of subject nouns.

Causal Masking

Multi-Head Attention

A single attention computation can only capture one type of relationship at a time. But language has many simultaneous relationships: syntax, coreference, semantic similarity, proximity.

Multi-head attention solves this by running multiple attention computations in parallel, each with its own set of Q/K/V weight matrices. Each “head” can learn to focus on a different type of relationship:

• One head might track which noun a pronoun refers to
• Another might focus on syntactic structure (subject-verb agreement)
• Another might attend to nearby tokens for local context

Typical models use 32 to 128 heads. Each head works with a smaller dimension ($d_k = d_\text{model} / n_\text{heads}$), so the total computation is roughly the same as one big attention layer.

The outputs of all heads are concatenated and projected back to the model dimension:

Grouped Query Attention (GQA)

Modern models (LLaMA 2/3, Mistral) use an optimization called Grouped Query Attention: instead of giving every head its own Key and Value matrices, multiple query heads share the same K/V matrices. This cuts memory usage during inference (fewer K/V tensors to cache) with minimal quality loss.

You don’t need to understand GQA to fine-tune, but you’ll see it mentioned in model configs and it’s good to know what it means.

Feed-Forward Network

After attention decides which information to gather, the feed-forward network (FFN) decides what to do with it.

Each token passes (independently) through two linear layers with an activation function in between:

If this formula looks intimidating, the intuition is simpler: the FFN is a small neural network applied to each token separately. It takes in a vector, expands it to a larger dimension (typically ~2.7x the model dimension), applies a non-linear transformation, and projects it back down.

Modern models use a variant called SwiGLU (the formula above) instead of the simpler ReLU used in the original transformer. The “gated” part ($\cdot (x \cdot W_3)$) lets the network learn to selectively amplify or suppress different features.

Where Knowledge Lives

Research suggests that factual knowledge primarily lives in the FFN weights. The attention layers are more about routing: deciding which information flows where. The FFN layers are about transforming: applying learned knowledge to the routed information.

This has implications for fine-tuning: when you fine-tune a model on domain-specific data, you’re partly updating the FFN weights to encode new knowledge patterns, and partly updating the attention weights to route information differently for your task.

Putting It All Together

A single transformer block:

Input
  ↓
LayerNorm → Multi-Head Attention → Add residual connection
  ↓
LayerNorm → Feed-Forward Network → Add residual connection
  ↓
Output

Residual connections (the “add” steps) are important: they let the input pass through unchanged and add the attention/FFN output on top. This means each layer only needs to learn what to add, not the entire representation from scratch. Without residual connections, deep networks (32+ layers) would be nearly impossible to train.

Layer normalization scales the values at each step to prevent them from growing too large or too small as they pass through many layers.

Stack 32 of these blocks (for a 7B model) or 80 (for a 70B model), and you have the backbone of a modern LLM. Each layer refines the representation further, building from surface-level patterns (word associations, syntax) in the early layers to abstract reasoning (logic, facts, intent) in the later layers.

Further Reading