Transformers and Large Language Models

Chapter 4 established the foundational building blocks of machine learning: neural networks, optimization, embeddings, and evolutionary algorithms. This chapter focuses on the single most consequential architecture of the modern deep learning era: the transformer. We trace its development from the attention mechanism through the scaling revolution to the emergent capabilities of large language models (LLMs), and close with a careful analysis of the transformer paradigm’s structural limitations—limitations that motivate the DOGMA architecture introduced in Chapter 7.

The transformer is not merely one architecture among many. It is, at the time of writing, the computational substrate upon which virtually all frontier AI systems are built. Understanding it deeply—its mathematical structure, its scaling behavior, its surprising emergent properties, and its fundamental constraints—is prerequisite to understanding why DOGMA proposes a different organizational principle for intelligence.

This chapter is deliberately more detailed than a typical survey. We will work through every matrix multiplication, trace every gradient, and build intuition from concrete numerical examples. If you can reconstruct scaled dot-product attention on a napkin by the end of this chapter, you are ready for DOGMA.

5.1 The Attention Mechanism

5.1.1 From Alignment to Attention

The attention mechanism was originally introduced for neural machine translation, where it allowed the decoder to dynamically focus on relevant parts of the source sentence rather than compressing the entire input into a fixed-length vector [Bahdanau et al., 2015]. The key insight: not all input positions are equally relevant to producing each output position, and the model should learn to selectively attend to the most informative sources.

Consider translating the French sentence “Le chat noir dort sur le tapis” into English. When generating the word “black,” the model should focus on “noir”; when generating “sleeps,” it should focus on “dort.” Before attention, sequence-to-sequence models compressed the entire source sentence into a single fixed-length vector, forcing all information through a bottleneck. Attention removes this bottleneck by allowing the decoder to look back at the full source sequence at every generation step.

5.1.2 Intuition: Queries, Keys, and Values

Before diving into the mathematics, let us build intuition for the three central objects in attention: queries, keys, and values. The analogy to a database lookup is instructive:

• Query (\(\mathbf {Q}\)): “What am I looking for?” Each token produces a query vector that encodes what information it needs from other tokens. Think of it as a search query in a database.
• Key (\(\mathbf {K}\)): “What do I contain?” Each token also produces a key vector that advertises what information it has available. Think of it as the index entry in a database.
• Value (\(\mathbf {V}\)): “What do I actually return?” Each token produces a value vector that is the content to be retrieved. Think of it as the data record that the index points to.

The attention mechanism computes a soft lookup: each query is compared against all keys to produce a relevance score, and the output is a weighted combination of values, where the weights are proportional to the relevance scores. Unlike a hard database lookup that returns a single record, attention returns a blended combination of all records, weighted by relevance.

5.1.3 Scaled Dot-Product Attention

The transformer’s attention mechanism operates on three learned projections of the input: queries \(\mathbf {Q}\), keys \(\mathbf {K}\), and values \(\mathbf {V}\). Given an input matrix \(\mathbf {X} \in \mathbb {R}^{T \times d}\) representing \(T\) tokens of dimension \(d\), these projections are:

\begin {equation} \mathbf {Q} = \mathbf {X}\mathbf {W}_Q, \quad \mathbf {K} = \mathbf {X}\mathbf {W}_K, \quad \mathbf {V} = \mathbf {X}\mathbf {W}_V, \label {eq:qkv} \end {equation}

where \(\mathbf {W}_Q, \mathbf {W}_K \in \mathbb {R}^{d \times d_k}\) and \(\mathbf {W}_V \in \mathbb {R}^{d \times d_v}\) are learned parameter matrices. Scaled dot-product attention then computes:

\begin {equation} \text {Attention}(\mathbf {Q}, \mathbf {K}, \mathbf {V}) = \operatorname {softmax}\!\left (\frac {\mathbf {Q}\mathbf {K}^\top }{\sqrt {d_k}}\right )\mathbf {V}. \label {eq:scaled-attention} \end {equation}

Let us break this computation into four explicit steps:

1.

Compute raw attention scores: \(\mathbf {S} = \mathbf {Q}\mathbf {K}^\top \in \mathbb {R}^{T \times T}\). Entry \(S_{ij}\) measures how much token \(i\)’s query aligns with token \(j\)’s key—i.e., how relevant token \(j\)’s content is to token \(i\)’s current need.

2.

Scale: \(\mathbf {S}' = \mathbf {S} / \sqrt {d_k}\). Without scaling, the dot products grow in magnitude proportionally to \(d_k\) (since each is a sum of \(d_k\) terms). Large magnitudes push the softmax into saturated regions where gradients vanish. Dividing by \(\sqrt {d_k}\) keeps the variance of the scores approximately 1, regardless of the dimension.

3.

Normalize with softmax: \(\mathbf {A} = \operatorname {softmax}(\mathbf {S}')\), applied row-wise. Each row of \(\mathbf {A}\) is a probability distribution over all tokens, with \(A_{ij} \geq 0\) and \(\sum _j A_{ij} = 1\). This converts raw scores into normalized attention weights.

4.

Aggregate values: \(\mathbf {O} = \mathbf {A}\mathbf {V} \in \mathbb {R}^{T \times d_v}\). The output for token \(i\) is a weighted average of all value vectors: \(\mathbf {o}_i = \sum _j A_{ij} \mathbf {v}_j\).

5.1.4 Why the Scaling Factor Matters

To understand the scaling factor \(1/\sqrt {d_k}\) more concretely, suppose the entries of \(\mathbf {q}\) and \(\mathbf {k}\) are independent random variables with mean 0 and variance 1. Then the dot product \(\mathbf {q} \cdot \mathbf {k} = \sum _{j=1}^{d_k} q_j k_j\) has mean 0 and variance \(d_k\) (since each product \(q_j k_j\) contributes variance 1, and the terms are independent). For \(d_k = 64\), the standard deviation of \(\mathbf {q} \cdot \mathbf {k}\) is \(\sqrt {64} = 8\). Dot products of magnitude 8 or larger push softmax outputs toward 0 or 1, creating nearly one-hot attention patterns that produce vanishingly small gradients. Dividing by \(\sqrt {d_k} = 8\) rescales the standard deviation back to 1, keeping softmax in a well-behaved regime.

5.1.5 Worked Example: Self-Attention with 3 Tokens

Let us trace through a complete self-attention computation with \(T = 3\) tokens and \(d_k = d_v = 4\). Suppose after the linear projections, we have:

\begin {equation} \mathbf {Q} = \begin {pmatrix} 1 & 0 & 1 & 0 \\ 0 & 1 & 0 & 1 \\ 1 & 1 & 0 & 0 \end {pmatrix}, \quad \mathbf {K} = \begin {pmatrix} 0 & 1 & 1 & 0 \\ 1 & 0 & 0 & 1 \\ 1 & 1 & 0 & 0 \end {pmatrix}, \quad \mathbf {V} = \begin {pmatrix} 1 & 2 & 3 & 4 \\ 5 & 6 & 7 & 8 \\ 9 & 10 & 11 & 12 \end {pmatrix}. \end {equation}

Step 1: Raw attention scores. Compute \(\mathbf {S} = \mathbf {Q}\mathbf {K}^\top \):

\begin {equation} \mathbf {S} = \begin {pmatrix} \mathbf {q}_1 \cdot \mathbf {k}_1 & \mathbf {q}_1 \cdot \mathbf {k}_2 & \mathbf {q}_1 \cdot \mathbf {k}_3 \\ \mathbf {q}_2 \cdot \mathbf {k}_1 & \mathbf {q}_2 \cdot \mathbf {k}_2 & \mathbf {q}_2 \cdot \mathbf {k}_3 \\ \mathbf {q}_3 \cdot \mathbf {k}_1 & \mathbf {q}_3 \cdot \mathbf {k}_2 & \mathbf {q}_3 \cdot \mathbf {k}_3 \end {pmatrix} = \begin {pmatrix} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 2 \end {pmatrix}. \end {equation}

For example, \(S_{11} = (1)(0) + (0)(1) + (1)(1) + (0)(0) = 1\), and \(S_{33} = (1)(1) + (1)(1) + (0)(0) + (0)(0) = 2\).

Step 2: Scale. With \(d_k = 4\), we divide by \(\sqrt {4} = 2\):

\begin {equation} \mathbf {S}' = \frac {\mathbf {S}}{\sqrt {d_k}} = \begin {pmatrix} 0.5 & 0.5 & 0.5 \\ 0.5 & 0.5 & 0.5 \\ 0.5 & 0.5 & 1.0 \end {pmatrix}. \end {equation}

Step 3: Softmax (row-wise). For rows 1 and 2, all entries are equal (\(0.5\)), so softmax gives uniform weights: \(A_{1j} = A_{2j} = 1/3\) for all \(j\). For row 3:

\begin {equation} A_{31} = A_{32} = \frac {e^{0.5}}{2e^{0.5} + e^{1.0}} \approx \frac {1.649}{3.298 + 2.718} \approx 0.274, \quad A_{33} = \frac {e^{1.0}}{2e^{0.5} + e^{1.0}} \approx \frac {2.718}{6.016} \approx 0.452. \end {equation}

The full attention matrix is: \begin {equation} \mathbf {A} \approx \begin {pmatrix} 0.333 & 0.333 & 0.333 \\ 0.333 & 0.333 & 0.333 \\ 0.274 & 0.274 & 0.452 \end {pmatrix}. \end {equation}

Step 4: Weighted value aggregation. The output \(\mathbf {O} = \mathbf {A}\mathbf {V}\):

\begin {equation} \mathbf {o}_1 = 0.333 \cdot \begin {pmatrix}1\\2\\3\\4\end {pmatrix} + 0.333 \cdot \begin {pmatrix}5\\6\\7\\8\end {pmatrix} + 0.333 \cdot \begin {pmatrix}9\\10\\11\\12\end {pmatrix} = \begin {pmatrix}5.0\\6.0\\7.0\\8.0\end {pmatrix}. \end {equation}

Token 1 attends uniformly, so its output is the average of all value vectors. Token 3, however, attends more heavily to itself (weight 0.452 vs. 0.274), so its output is shifted toward \(\mathbf {v}_3\):

\begin {equation} \mathbf {o}_3 \approx 0.274 \cdot \begin {pmatrix}1\\2\\3\\4\end {pmatrix} + 0.274 \cdot \begin {pmatrix}5\\6\\7\\8\end {pmatrix} + 0.452 \cdot \begin {pmatrix}9\\10\\11\\12\end {pmatrix} \approx \begin {pmatrix}5.70\\6.70\\7.70\\8.70\end {pmatrix}. \end {equation}

5.1.6 Visualizing Self-Attention

Figure 5.1 illustrates the self-attention mechanism with colored arrows showing how tokens selectively attend to each other.

ATCGATCG

5.2 Multi-Head Attention

5.2.1 Why Multiple Heads?

A single attention head computes a single set of attention weights for each pair of tokens. But different aspects of the relationship between two tokens may be relevant simultaneously. For instance, when processing the sentence “The animal didn’t cross the street because it was too tired,” one attention head might need to resolve the pronoun “it” (a syntactic task), while another might need to capture the semantic relationship between “tired” and “animal” (a semantic task). A single head must compress all of these relationships into one set of weights—an information bottleneck.

Multi-head attention solves this by running \(h\) independent attention computations in parallel, each with its own learned projections:

\begin {equation} \text {MultiHead}(\mathbf {Q}, \mathbf {K}, \mathbf {V}) = \operatorname {Concat}(\text {head}_1, \ldots , \text {head}_h)\,\mathbf {W}_O, \label {eq:multihead} \end {equation}

where each head is: \begin {equation} \text {head}_i = \text {Attention}(\mathbf {X}\mathbf {W}_Q^{(i)},\; \mathbf {X}\mathbf {W}_K^{(i)},\; \mathbf {X}\mathbf {W}_V^{(i)}), \end {equation}

and \(\mathbf {W}_O \in \mathbb {R}^{hd_v \times d}\) is a learned output projection.

5.2.2 Head Splitting and Concatenation

In practice, multi-head attention does not require \(h\) separate sets of full-sized projection matrices. Instead, the model dimension \(d\) is split evenly across the \(h\) heads: each head operates on \(d_k = d_v = d / h\) dimensions. This means:

• \(\mathbf {W}_Q^{(i)}, \mathbf {W}_K^{(i)} \in \mathbb {R}^{d \times (d/h)}\) — each head projects from the full \(d\)-dimensional space into a \((d/h)\)-dimensional subspace.
• Each head independently computes attention in its \((d/h)\)-dimensional subspace, producing output of shape \(T \times (d/h)\).
• The \(h\) head outputs are concatenated along the feature dimension, giving shape \(T \times d\).
• The concatenated output is projected through \(\mathbf {W}_O \in \mathbb {R}^{d \times d}\) to produce the final output.

The total number of parameters is the same as a single head with \(d_k = d\): the computation is repartitioned, not increased. With \(h = 12\) heads and \(d = 768\) (as in BERT-base), each head operates on \(d_k = 64\) dimensions.

5.2.3 What Different Heads Learn

Empirical analysis reveals that attention heads specialize in different linguistic functions [Voita et al., 2019]:

• Positional heads: Attend to the immediately preceding or following token—capturing local context.
• Syntactic heads: Attend to syntactically related tokens (e.g., a verb head attends to its subject).
• Rare-word heads: Attend to the least frequent token in the sequence—potentially copying specialized information.
• Delimiter heads: Attend to separator tokens (periods, commas)—capturing sentence boundaries.

This emergent specialization occurs without any explicit supervision: the heads discover useful attention patterns through gradient descent alone.

5.2.4 Multi-Head Attention Diagram

ATCGATCG

5.3 The Transformer Architecture

Vaswani et al. [2017a] introduced the transformer as a sequence-to-sequence architecture composed entirely of attention and feedforward layers, eliminating the recurrence that had characterized prior sequence models. The architecture consists of an encoder stack and a decoder stack, each built from repeated identical layers.

5.3.1 Encoder and Decoder Stacks

Each encoder layer applies two sub-layers:

1.

Multi-head self-attention: Each position attends to all positions in the input.

2.

Position-wise feedforward network (FFN): A two-layer MLP applied independently to each position: \begin {equation} \text {FFN}(\mathbf {x}) = \text {ReLU}(\mathbf {x}\mathbf {W}_1 + \mathbf {b}_1)\mathbf {W}_2 + \mathbf {b}_2. \label {eq:ffn} \end {equation}

The FFN is a critical but often underappreciated component. While attention handles inter-token communication (allowing tokens to share information), the FFN handles intra-token processing (transforming each token’s representation independently). In GPT-3, the FFN inner dimension is \(4d = 49{,}152\) for \(d = 12{,}288\)—meaning the FFN has \(4 \times \) the width of the model dimension. Research suggests that FFN layers function as key-value memories, storing factual associations learned during training.

Each decoder layer adds a third sub-layer: cross-attention over the encoder output, allowing the decoder to condition on the full input sequence while generating the output autoregressively.

5.3.2 Residual Connections and Layer Normalization

Both encoder and decoder employ residual connections [He et al., 2016] around each sub-layer, followed by layer normalization [Ba et al., 2016]:

\begin {equation} \mathbf {y} = \text {LayerNorm}(\mathbf {x} + \text {SubLayer}(\mathbf {x})). \label {eq:residual-ln} \end {equation}

Residual connections add the sub-layer’s input directly to its output: \(\mathbf {x} + \text {SubLayer}(\mathbf {x})\). This creates a “skip connection” that allows gradients to flow directly through the network without passing through every sub-layer. Without residual connections, training transformers with \(L > 6\) layers would be extremely difficult due to vanishing or exploding gradients.

Layer normalization normalizes activations across the feature dimension for each token independently. Given a vector \(\mathbf {x} \in \mathbb {R}^d\):

\begin {equation} \text {LayerNorm}(\mathbf {x}) = \gamma \odot \frac {\mathbf {x} - \mu }{\sigma + \epsilon } + \beta , \label {eq:layernorm} \end {equation}

where \(\mu = \frac {1}{d}\sum _{i=1}^{d} x_i\), \(\sigma = \sqrt {\frac {1}{d}\sum _{i=1}^{d}(x_i - \mu )^2}\), and \(\gamma , \beta \in \mathbb {R}^d\) are learned scale and shift parameters. The small constant \(\epsilon \) (typically \(10^{-5}\)) prevents division by zero.

Modern implementations often use pre-norm placement, applying normalization before the sub-layer rather than after:

\begin {equation} \mathbf {y} = \mathbf {x} + \text {SubLayer}(\text {LayerNorm}(\mathbf {x})), \label {eq:pre-norm} \end {equation}

which improves training stability for very deep models. Pre-norm is now standard in models like GPT-3, LLaMA, and their descendants.

5.3.3 The Complete Transformer Block

Figure 5.3 illustrates the structure of a single transformer decoder block with all components labeled.

ATCGATCG

5.4 Positional Encoding

5.4.1 The Permutation Equivariance Problem

Self-attention is permutation equivariant: if we shuffle the order of input tokens, the output is shuffled in exactly the same way. Mathematically, for any permutation matrix \(\mathbf {P}\):

\begin {equation} \text {Attention}(\mathbf {P}\mathbf {Q}, \mathbf {P}\mathbf {K}, \mathbf {P}\mathbf {V}) = \mathbf {P} \cdot \text {Attention}(\mathbf {Q}, \mathbf {K}, \mathbf {V}). \end {equation}

This means that without additional information, the transformer treats “The cat sat” and “sat cat The” identically. Since word order is essential for meaning, positional information must be injected explicitly.

5.4.2 Sinusoidal Positional Encodings

Vaswani et al. [2017a] proposed sinusoidal positional encodings, where each position \(t\) and dimension \(i\) receives a unique value:

\begin {align} \text {PE}(t, 2i) &= \sin \!\left (\frac {t}{10000^{2i/d}}\right ), \label {eq:pe-sin} \\ \text {PE}(t, 2i+1) &= \cos \!\left (\frac {t}{10000^{2i/d}}\right ). \label {eq:pe-cos} \end {align}

These encodings are added to the input embeddings: \(\mathbf {x}_t' = \mathbf {x}_t + \text {PE}(t)\).

Why sinusoids? The key property is that relative positions can be computed as linear transformations. For any fixed offset \(k\), there exists a matrix \(\mathbf {M}_k\) (independent of position \(t\)) such that \(\text {PE}(t+k) = \mathbf {M}_k \cdot \text {PE}(t)\). This follows from the trigonometric identity \(\sin (\alpha + \beta ) = \sin \alpha \cos \beta + \cos \alpha \sin \beta \). The model can therefore learn to attend to relative positions (e.g., “the token 3 positions back”) using fixed linear operations, without needing to memorize absolute positions.

Multi-frequency structure. Different dimensions encode position at different frequencies. Low dimensions (\(i \approx 0\)) use high frequencies (fast oscillation), capturing fine-grained positional differences. High dimensions (\(i \approx d/2\)) use low frequencies (slow oscillation), capturing long-range positional structure. The model has access to a rich, multi-scale representation of position.

5.4.3 Learned and Relative Position Encodings

Learned positional embeddings. An alternative is to learn a separate embedding vector for each position, stored as a matrix \(\mathbf {E}_{\text {pos}} \in \mathbb {R}^{T_{\max } \times d}\). GPT-2 and BERT use this approach. The downside: the model cannot generalize to positions beyond \(T_{\max }\).

Rotary Position Embeddings (RoPE). Modern models commonly use RoPE [Su et al., 2024], which encode relative distances through rotation matrices applied to query and key vectors. RoPE rotates pairs of dimensions by an angle proportional to position:

\begin {equation} \text {RoPE}(\mathbf {x}_t, t) = \begin {pmatrix} x_1 \cos (t\theta _1) - x_2 \sin (t\theta _1) \\ x_1 \sin (t\theta _1) + x_2 \cos (t\theta _1) \\ \vdots \\ x_{d-1} \cos (t\theta _{d/2}) - x_d \sin (t\theta _{d/2}) \\ x_{d-1} \sin (t\theta _{d/2}) + x_d \cos (t\theta _{d/2}) \end {pmatrix}, \label {eq:rope} \end {equation}

where \(\theta _i = 10000^{-2i/d}\). The key property is that the dot product \(\text {RoPE}(\mathbf {q}_m, m)^\top \text {RoPE}(\mathbf {k}_n, n)\) depends only on \(\mathbf {q}_m\), \(\mathbf {k}_n\), and the relative position \(m - n\), not on the absolute positions \(m\) and \(n\). This enables length generalization: models trained with RoPE can often handle sequences longer than those seen during training.

5.5 GPT and Autoregressive Language Models

5.5.1 Decoder-Only Architecture

While the original transformer used both encoder and decoder stacks, the GPT family [Radford et al., 2019] demonstrated that a decoder-only architecture—using only the masked self-attention stack—is sufficient for powerful language modeling. The key simplification: by training the model to predict the next token given all preceding tokens, the encoder becomes unnecessary.

A decoder-only model with parameters \(\theta \) defines a probability distribution over sequences:

\begin {equation} p_\theta (x_1, x_2, \ldots , x_T) = \prod _{t=1}^{T} p_\theta (x_t \mid x_1, \ldots , x_{t-1}). \label {eq:autoregressive} \end {equation}

This factorization is exact by the chain rule of probability. The model is trained by maximizing the log-likelihood of the training corpus, which is equivalent to minimizing the cross-entropy loss introduced in Equation 4.4 of Chapter 4.

5.5.2 The Next-Token Prediction Objective

The training objective for autoregressive language models is deceptively simple: predict the next token. Given a sequence of tokens \((x_1, x_2, \ldots , x_T)\) from the training corpus, the cross-entropy loss is:

\begin {equation} \mathcal {L}(\theta ) = -\frac {1}{T}\sum _{t=1}^{T} \log p_\theta (x_t \mid x_1, \ldots , x_{t-1}). \label {eq:lm-cross-entropy} \end {equation}

At each position \(t\), the model produces a probability distribution over the entire vocabulary \(\mathcal {V}\) (typically 32,000 to 128,000 tokens). The loss penalizes the model for assigning low probability to the actual next token. Minimizing this loss across trillions of tokens forces the model to develop rich internal representations of language, world knowledge, and reasoning patterns.

5.5.3 Causal Masking

To enforce the autoregressive property during training, a causal mask \(\mathbf {M}\) is applied to the attention scores before the softmax:

\begin {equation} \mathbf {A} = \operatorname {softmax}\!\left (\frac {\mathbf {Q}\mathbf {K}^\top }{\sqrt {d_k}} + \mathbf {M}\right ), \quad M_{ij} = \begin {cases} 0 & \text {if } i \geq j, \\ -\infty & \text {if } i < j. \end {cases} \label {eq:causal-mask} \end {equation}

This ensures that position \(i\) can only attend to positions \(j \leq i\), preventing information leakage from future tokens. Causal masking enables efficient training: the entire sequence can be processed in a single forward pass, with losses computed at every position simultaneously (teacher forcing).

5.6 Training Large Language Models

5.6.1 Training Data Scale

Modern LLMs are trained on colossal datasets. The training corpus for a frontier model typically includes:

• Web text: Crawled and filtered web pages (Common Crawl, typically 1–5 trillion tokens after filtering).
• Books: Digitized books providing long-form, high-quality prose.
• Code: Source code from public repositories (GitHub), which improves reasoning abilities.
• Scientific papers: ArXiv, PubMed, and other academic sources.
• Conversational data: Forums, Q&A sites, and curated dialogue.

The total training corpus for frontier models has grown dramatically: GPT-3 used \(\sim \)300B tokens (2020), LLaMA used 1.4T tokens (2023), and LLaMA-3 used 15T tokens (2024). Data quality, deduplication, and filtering are now recognized as equally important as scale.

5.6.2 Compute Requirements

Training a large language model requires enormous computational resources. The compute budget is measured in FLOPs (floating-point operations). A useful approximation for the total training FLOPs of a transformer is:

\begin {equation} C \approx 6 N D, \label {eq:training-flops} \end {equation}

where \(N\) is the number of parameters and \(D\) is the number of training tokens. The factor of 6 accounts for the forward pass (\(\sim 2ND\)) and backward pass (\(\sim 4ND\)). For LLaMA-3 70B trained on 15T tokens: \(C \approx 6 \times 70 \times 10^9 \times 15 \times 10^{12} = 6.3 \times 10^{24}\) FLOPs.

5.7 Scaling Laws for Language Models

Kaplan et al. [2020a] established that transformer language model performance follows remarkably smooth power-law relationships with three key variables: the number of model parameters \(N\), the dataset size \(D\) (in tokens), and the compute budget \(C\) (in FLOPs).

5.7.1 Power-Law Relationships

The cross-entropy loss \(L\) on held-out data decreases as a power law in each variable, when the other two are not bottlenecked:

\begin {align} L(N) &\approx \left (\frac {N_c}{N}\right )^{\alpha _N}, \quad \alpha _N \approx 0.076, \label {eq:scaling-N} \\ L(D) &\approx \left (\frac {D_c}{D}\right )^{\alpha _D}, \quad \alpha _D \approx 0.095, \label {eq:scaling-D} \\ L(C) &\approx \left (\frac {C_c}{C}\right )^{\alpha _C}, \quad \alpha _C \approx 0.050, \label {eq:scaling-C} \end {align}

where \(N_c, D_c, C_c\) are scaling constants. The loss decreases as a power law over many orders of magnitude, with no sign of saturation within the explored range.

5.7.2 Compute-Optimal Training (Chinchilla)

Hoffmann et al. [2022a] refined these results, showing that for a fixed compute budget, the optimal allocation splits compute roughly equally between model size and training data. Their analysis suggested that many large models (including the original GPT-3) were significantly undertrained: given the compute used, more tokens and fewer parameters would have yielded lower loss.

The Chinchilla scaling law can be expressed as: \begin {equation} N_{\text {opt}} \propto C^{0.50}, \quad D_{\text {opt}} \propto C^{0.50}, \label {eq:ch5-chinchilla} \end {equation}

meaning that when the compute budget doubles, both the model size and the training data should increase by \(\sqrt {2}\). The practical implication: a 70B-parameter model should be trained on approximately 1.4 trillion tokens for compute-optimal performance.

ATCGATCG

5.8 Emergent Abilities of Large Language Models

5.8.1 In-Context Learning

One of the most striking properties of large language models is in-context learning (ICL): the ability to learn new tasks from a few examples provided in the prompt, without any gradient updates [Brown et al., 2020]. Given \(k\) input-output examples \((x_1, y_1), \ldots , (x_k, y_k)\) followed by a new input \(x_{k+1}\), the model generates \(y_{k+1}\) by conditioning on the entire prompt:

\begin {equation} y_{k+1} \sim p_\theta (\cdot \mid x_1, y_1, \ldots , x_k, y_k, x_{k+1}). \label {eq:icl} \end {equation}

ICL is remarkable because it represents a form of meta-learning: the model has learned, during pretraining, a general-purpose algorithm for adapting to new tasks specified by examples. The mechanism by which ICL works remains an active area of research; hypotheses include implicit Bayesian inference, mesa-optimization, and gradient descent in the forward pass [Akyürek et al., 2023].

5.8.2 Few-Shot, One-Shot, and Zero-Shot Learning

Brown et al. [2020] systematically evaluated GPT-3 across dozens of NLP benchmarks in three regimes:

• Zero-shot: Only a task description is provided; no examples.
• One-shot: A single input-output example is provided.
• Few-shot: A small number (typically 2–64) of examples are provided.

Performance improved consistently from zero-shot to few-shot, and larger models showed greater improvement from additional examples. This suggests that scale enables more effective utilization of in-context information.

5.8.3 Chain-of-Thought Reasoning

Wei et al. [2022a] demonstrated that prompting large language models to produce intermediate reasoning steps—a technique called chain-of-thought (CoT) prompting—dramatically improves performance on tasks requiring multi-step reasoning:

\begin {equation} p_\theta (y \mid x) \approx \sum _{r} p_\theta (y \mid r, x) \, p_\theta (r \mid x), \label {eq:cot} \end {equation}

where \(r\) denotes the chain-of-thought reasoning trace. The reasoning trace acts as a “scratchpad” that decomposes complex problems into manageable steps. This capability appears to emerge sharply above certain model sizes, suggesting that it is a threshold phenomenon rather than a gradual improvement.

5.8.4 Instruction Following

Large models trained with instruction tuning develop the ability to follow novel instructions they have never seen during fine-tuning. Given a natural language description of a task, the model performs the task without any examples. This ability is qualitatively different from few-shot learning: instead of inferring the task from input-output pairs, the model interprets a procedural description and executes it. This “instruction following” ability is the foundation of modern AI assistants.

5.9 Instruction Tuning and RLHF

Pretraining on next-token prediction produces a capable but unaligned model: it can complete any text, but it does not reliably follow instructions, produce helpful answers, or avoid harmful outputs. Closing this gap requires alignment techniques that shape model behavior to match human intent.

5.9.1 Supervised Instruction Tuning

The first alignment step is supervised fine-tuning (SFT) on a dataset of high-quality (instruction, response) pairs. This teaches the model to interpret prompts as instructions and produce appropriately structured responses, rather than merely continuing text in the style of the pretraining corpus.

The SFT dataset typically contains tens of thousands to hundreds of thousands of examples spanning diverse task categories: question answering, summarization, code generation, creative writing, and mathematical reasoning. Quality matters more than quantity: a small dataset of expert-written responses often outperforms a large dataset of lower-quality examples.

5.9.2 Reinforcement Learning from Human Feedback

Ouyang et al. [2022] introduced the RLHF pipeline, which further aligns the model through three stages:

1.

Reward model training: Human annotators rank model outputs by quality. These rankings train a reward model \(R_\phi (x, y)\) that predicts human preference scores.

2.

Policy optimization: The language model (“policy”) \(\pi _\theta \) is optimized to maximize the reward while staying close to the SFT model via a KL-divergence penalty: \begin {equation} \max _\theta \; \mathbb {E}_{x \sim \mathcal {D},\, y \sim \pi _\theta (\cdot \mid x)} \bigl [ R_\phi (x, y) - \beta \, D_{\text {KL}}\!\left (\pi _\theta (\cdot \mid x) \,\|\, \pi _{\text {SFT}}(\cdot \mid x)\right ) \bigr ]. \label {eq:rlhf} \end {equation}

3.

Iterative refinement: The process is repeated, collecting new human comparisons on the updated model’s outputs.

The KL penalty in Equation 5.27 is critical: without it, the model would collapse to degenerate outputs that exploit the reward model’s imperfections (reward hacking).

5.10 Key Models: A Comparative View

The LLM landscape has diversified considerably since GPT-1. This section provides a comparative overview of the major model families.

5.10.1 GPT Family (OpenAI)

The GPT family established the decoder-only autoregressive paradigm:

^*Estimated; OpenAI has not published official figures for GPT-4.

5.10.2 BERT and Encoder-Only Models

BERT [Devlin et al., 2019] introduced a fundamentally different approach: bidirectional pre-training using masked language modeling. Instead of predicting the next token (left-to-right), BERT masks random tokens and trains the model to predict them from the full bidirectional context:

\begin {equation} \mathcal {L}_{\text {BERT}} = -\sum _{i \in \mathcal {M}} \log p_\theta (x_i \mid \mathbf {x}_{\backslash \mathcal {M}}). \label {eq:bert-loss} \end {equation}

BERT’s bidirectional attention makes it better suited for understanding tasks (classification, question answering, named entity recognition) but unsuitable for generation tasks, since it cannot generate text autoregressively.

5.10.3 LLaMA and Open-Source Models

LLaMA [Touvron et al., 2023] demonstrated that high-quality, relatively small models trained on more data can match or exceed larger models. Key innovations:

• RMSNorm instead of LayerNorm (faster, equally effective).
• SwiGLU activation in the FFN (replacing ReLU with a gated variant).
• RoPE for positional encoding (enabling length generalization).
• Grouped Query Attention (GQA): Sharing key/value heads across query heads to reduce memory during inference.

5.10.4 Comparison Table

5.11 Limitations of the Transformer Paradigm

The transformer’s success is undeniable. Yet success should not obscure structural limitations that constrain what the paradigm can ultimately achieve. This section identifies five fundamental limitations that motivate the DOGMA architecture.

5.11.1 Quadratic Attention Cost

Self-attention requires \(O(T^2)\) time and memory in the sequence length \(T\). Various approximations exist—sparse attention, linear attention, FlashAttention’s IO-aware tiling [Dao et al., 2022]—but they either sacrifice expressiveness or provide constant-factor improvements without changing the asymptotic scaling. For genomic sequences that may span millions of bases, quadratic attention is impractical.

5.11.2 No Persistent State

A transformer has two forms of “memory”: its frozen parameters (encoding the training distribution) and its context window (a bounded, ephemeral buffer). There is no intermediate state that persists across forward passes, accumulates experience, and can be selectively modified. Every new prompt starts from the same parameter snapshot.

In contrast, biological organisms maintain a genome that persists across the organism’s lifetime, is read selectively in response to environmental signals, and can be modified through somatic mutation and epigenetic regulation. DOGMA’s persistent genomic state fills this gap.

5.11.3 No Native Typed Regions

In a standard transformer, all tokens inhabit the same representational space and are processed by the same attention mechanism. System prompts, user queries, retrieved context, chain-of-thought reasoning, and generated output are all undifferentiated tokens. Any distinction between these roles must be learned implicitly from formatting conventions—a fragile and unverifiable approach.

DOGMA introduces typed genomic bases organized into typed regions (promoter, coding, regulatory, structural), each with distinct interaction semantics, regulatory weights, and compatibility constraints. This is analogous to how a genome contains structurally and functionally distinct regions (exons, introns, promoters, enhancers) rather than a uniform sequence of nucleotides.

5.11.4 Output-First Bias

The next-token prediction objective directly optimizes the quality of generated text. There is no explicit intermediate representation of the model’s “plan” or “intent” before token generation begins. The model must simultaneously decide what to do and how to say it within a single left-to-right pass.

DOGMA separates these concerns across distinct stages: regulation determines which genomic regions are active, synthesis assembles the relevant information, transcription reads it into intermediate representations, expression folds those representations into structured behavior, and only then does realization render the output. This multi-stage pipeline mirrors the Central Dogma of molecular biology (Chapter 1).

5.11.5 No Evolutionary Self-Modification

Once a transformer is trained, its parameters are fixed. Adapting to new tasks requires explicit fine-tuning, prompt engineering, or retrieval augmentation—all external interventions. The model cannot modify its own computational structure in response to experience.

DOGMA’s evolutionary training loop (Chapter 12) enables the system to mutate its own prompt genome, evaluate the fitness of mutations, and select beneficial changes—implementing a continuous evolutionary process analogous to biological adaptation.

5.11.6 Summary: What Transformers Cannot Do That Biology Can

ATCGATCG

5.12 Exercises

5.1.

[Fundamentals] Derive the gradient of the scaled dot-product attention output (Equation 5.2) with respect to \(\mathbf {Q}\). Show how the softmax Jacobian appears in the result and explain why the scaling factor \(1/\sqrt {d_k}\) mitigates the vanishing gradient problem.

5.2.

[Worked Example] Consider a 4-token sequence with \(d_k = 2\). Let the query and key matrices be: \begin {equation} \mathbf {Q} = \begin {pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \\ -1 & 1 \end {pmatrix}, \quad \mathbf {K} = \begin {pmatrix} 1 & 0 \\ 0 & 1 \\ -1 & 0 \\ 0 & -1 \end {pmatrix}. \end {equation} (a) Compute the raw attention scores \(\mathbf {Q}\mathbf {K}^\top \). (b) Apply the scaling factor \(1/\sqrt {d_k}\). (c) Apply a causal mask and compute the softmax. (d) Interpret the attention pattern: which tokens attend most strongly to which other tokens, and why?

5.3.

[Coding] Implement a single-head causal attention layer from scratch (NumPy only). Verify that your implementation produces the same output as the equivalent PyTorch nn.MultiheadAttention layer with num_heads=1 and an appropriate attention mask.

5.4.

[Analysis] Consider a sequence of length \(T = 4096\) processed by a 12-layer transformer with \(d = 768\) and \(h = 12\) attention heads. Calculate: (a) the total number of entries in all attention matrices across all layers and heads, (b) the FLOPs required for the attention computation alone, and (c) the ratio of attention FLOPs to FFN FLOPs.

5.5.

[Theory] The causal mask ensures that position \(i\) cannot attend to position \(j > i\). Prove that this constraint is sufficient to make the autoregressive factorization in Equation 5.19 valid—i.e., that the representation at position \(t\) depends only on tokens \(x_1, \ldots , x_t\) and not on any future tokens.

5.6.

[Positional Encoding] Compute the sinusoidal positional encodings for positions \(t = 0, 1, 2, 3\) with \(d = 8\). (a) Verify that the dot product \(\text {PE}(t) \cdot \text {PE}(t+k)\) depends primarily on \(k\) and not on \(t\). (b) Explain why this property is useful for learning relative position relationships.

5.7.

[Scaling Laws] A research lab has a fixed compute budget of \(C = 10^{22}\) FLOPs. Using the Chinchilla scaling law (Equation 5.24) and the approximation \(C \approx 6ND\), compute: (a) the optimal model size \(N_{\text {opt}}\), (b) the optimal number of training tokens \(D_{\text {opt}}\), and (c) compare with the actual GPT-3 allocation (175B parameters, 300B tokens) and discuss whether GPT-3 was undertrained.

5.8.

[Design] Propose a “biologically inspired attention” mechanism that restricts interaction patterns based on typed regions rather than using dense pairwise attention. Formally specify (a) the type system for tokens, (b) the interaction rules between types, and (c) the resulting computational complexity. Compare with DOGMA’s regulation stage in Chapter 7.

5.9.

[Emergent Abilities] Design an experiment to test whether a specific ability (e.g., multi-digit addition) is truly emergent (discontinuous in model scale) or merely appears so due to the evaluation metric. Propose two evaluation metrics—one binary (exact match) and one continuous (per-digit accuracy)—and predict what each would show as a function of model size.

5.10.

[Research] Read Kaplan et al. [2020a] and Hoffmann et al. [2022a]. Both papers derive compute-optimal training strategies but reach different conclusions about the optimal parameter-to-data ratio. Explain the source of the discrepancy and discuss which finding is more relevant for DOGMA-style architectures with structured inputs.

5.13 Key Takeaways

Suggested Reading

• Vaswani et al. [2017a]: The original transformer paper—required reading.
• Radford et al. [2019]: GPT-2 and the case for large-scale language modeling.
• Brown et al. [2020]: GPT-3 and in-context learning at scale.
• Devlin et al. [2019]: BERT and bidirectional pre-training.
• Touvron et al. [2023]: LLaMA and the power of training smaller models on more data.
• Kaplan et al. [2020a]: Neural scaling laws—empirical foundations for the scaling paradigm.
• Hoffmann et al. [2022a]: Chinchilla—compute-optimal training and the case for more data.
• Ouyang et al. [2022]: InstructGPT and RLHF—aligning LLMs with human preferences.
• Dao et al. [2022]: FlashAttention—IO-aware exact attention for efficient transformers.
• Su et al. [2024]: RoPE—rotary position embeddings for length generalization.