Toward AGI

Scaling Laws and Post-Transformer Architectures

Chapter 1533 min7,255 words

The dominant paradigm in AI since 2020 has been simple: make the model bigger . Scaling laws revealed smooth power-law relationships between model size, training compute, dataset size, and loss. These laws enabled precise predictions of model capability and drove the largest engineering projects in AI history—GPT-4,...

AGI Evidence Ladder

Capability claims rise only as independent evidence accumulates

Move from narrow success toward transfer, retention, calibration, and governed autonomy.

Task

Competence

Beat a meaningful baseline on one defined task.

score > baseline

There is no reason why intelligence should be proportional to model size. Efficiency is a form of intelligence.

— DOGMA Design Manifesto

The dominant paradigm in AI since 2020 has been simple: make the model bigger. Scaling laws revealed smooth power-law relationships between model size, training compute, dataset size, and loss. These laws enabled precise predictions of model capability and drove the largest engineering projects in AI history—GPT-4, Gemini, Claude, and their successors consumed hundreds of millions of dollars in compute.

But scaling alone cannot be the complete story. Quadratic attention costs, energy consumption, and the diminishing returns of brute-force scaling create pressure for architectural innovation—new ways of organizing computation that achieve better capability per parameter. This chapter examines the scaling paradigm in detail, surveys the emerging post-transformer landscape (state space models, linear attention, mixture-of-experts), and situates DOGMA within this evolving ecosystem. We will see that DOGMA’s biology-inspired structure offers a fundamentally different scaling axis: one that draws on the organizational principles that allowed biological genomes to scale from bacteria (\(\sim \)\(10^6\) bases) to humans (\(\sim \)\(3 \times 10^9\) bases) without quadratic complexity.

15.1 Neural Scaling Laws

The empirical discovery of neural scaling laws transformed AI research from art into something closer to engineering. Before scaling laws, predicting the performance of a model at a given size was guesswork. After scaling laws, it became a calculation.

15.1.1 The Kaplan Scaling Laws

Kaplan et al. [2020a] established that cross-entropy loss \(L\) on language modeling follows power-law scaling across three independent dimensions:

\begin {equation} L(N) = \left (\frac {N_c}{N}\right )^{\alpha _N}, \quad L(D) = \left (\frac {D_c}{D}\right )^{\alpha _D}, \quad L(C) = \left (\frac {C_c}{C}\right )^{\alpha _C}, \label {eq:ch15-scaling-laws} \end {equation}

where the variables are defined as follows:

\(N\) is the number of non-embedding parameters in the model.
\(D\) is the number of training tokens (dataset size).
\(C\) is the total compute budget measured in FLOPs.
\(N_c\), \(D_c\), \(C_c\) are critical constants that depend on the architecture and data distribution.
\(\alpha _N \approx 0.076\), \(\alpha _D \approx 0.095\), \(\alpha _C \approx 0.050\) are empirically determined exponents.

Example 15.1 (Reading the Scaling Exponents). Consider the parameter scaling law \(L(N) = (N_c / N)^{0.076}\). Suppose we double the model size from \(N\) to \(2N\). The new loss is: \[ L(2N) = \left (\frac {N_c}{2N}\right )^{0.076} = \frac {1}{2^{0.076}} \cdot L(N) \approx 0.949 \cdot L(N). \] Doubling model size reduces loss by approximately 5.1%. This seemingly modest improvement compounds: a \(100\times \) increase in parameters yields \(L(100N) \approx 0.71 \cdot L(N)\), a 29% reduction. The power-law form means we never reach zero loss—each increment of capability costs exponentially more resources.

The key finding of Kaplan et al. [2020a] was that these power laws are smooth and predictable. There are no sudden jumps or plateaus within an architecture family. This predictability enabled organizations to plan multi-hundred-million-dollar training runs with reasonable confidence in the outcome.

15.1.2 The Chinchilla Correction

Hoffmann et al. [2022a] revisited the Kaplan scaling laws and arrived at a different conclusion about the optimal allocation of a fixed compute budget \(C\) between model size \(N\) and dataset size \(D\). Where Kaplan et al. recommended scaling parameters faster than data, the Chinchilla analysis found that model size and data should be scaled in roughly equal proportion:

\begin {equation} N_{\text {opt}} \propto C^{a}, \quad D_{\text {opt}} \propto C^{b}, \quad \text {with } a \approx b \approx 0.5. \label {eq:ch15-chinchilla} \end {equation}

Definition 15.1 (Compute-Optimal Training). A model is compute-optimal if, for a given compute budget \(C\), the model size \(N\) and dataset size \(D\) are chosen to minimize the loss \(L(N, D)\) subject to the constraint \(C \approx 6ND\) (the approximate FLOP cost of a single training epoch on \(D\) tokens with an \(N\)-parameter model).

The practical implication was dramatic. Many existing models—including the original Chinchilla paper’s comparison target, Gopher (280B parameters)—were over-parameterized and under-trained. A smaller model trained on more data (Chinchilla, 70B parameters) achieved the same or better performance at lower inference cost.

Table 15.1: Comparison of Kaplan vs. Chinchilla scaling recommendations.

Aspect	Kaplan et al. (2020)	Chinchilla (2022)
Optimal \(N\) scaling	\(N \propto C^{0.73}\)	\(N \propto C^{0.50}\)
Optimal \(D\) scaling	\(D \propto C^{0.27}\)	\(D \propto C^{0.50}\)
Key insight	Scale parameters aggressively	Scale parameters and data equally
Practical impact	Very large, undertrained models	Smaller, better-trained models

15.1.3 Visualizing Scaling Laws

The power-law relationships of Equation 15.1 become linear on log-log axes, which is why log-log plots are the standard visualization for scaling behavior.

Figure 15.1: Schematic log-log scaling plot. Dense transformers (solid blue) follow established power-law scaling. State space models (dotted teal) show competitive scaling with lower per-token cost. DOGMA (dashed red) is hypothesized to achieve better loss-per-parameter through biological structural priors. The gold arrow indicates the scaling efficiency gap at large model sizes.

15.1.4 Emergent Abilities

Wei et al. [2022a] documented emergent abilities: capabilities that appear suddenly as models cross certain size thresholds. In-context learning, chain-of-thought reasoning, and instruction following all emerge at specific scale points—typically between \(10^{10}\) and \(10^{11}\) parameters.

Definition 15.2 (Emergent Ability). An ability is emergent if it is not present in smaller models but appears in larger models. Formally, let \(\mathcal {A}(N)\) denote the performance of a model of size \(N\) on a specific task. The ability is emergent if \(\mathcal {A}(N) \approx 0\) for \(N < N^*\) and \(\mathcal {A}(N) > \delta \) for \(N > N^*\), where \(N^*\) is the emergence threshold and \(\delta \) is a meaningful performance level.

Whether these abilities are truly discontinuous or merely appear so due to evaluation metrics remains debated [Schaeffer et al., 2023a]. What is clear is that scale enables qualitatively different behaviors—and that architectural innovations (like DOGMA’s structured genome) might enable similar capabilities at smaller scale by providing the structural scaffolding that dense models must discover from data alone.

Scaling Laws and Architecture

Scaling laws hold within an architecture family. Changing the architecture changes the scaling exponents and constants—the slopes and intercepts of the log-log lines in Figure 15.1. DOGMA’s thesis is that biology-inspired structured architectures may achieve better scaling—higher capability per parameter—by introducing inductive biases that reduce the amount of structure the model must learn from data alone.

15.2 The Limits of Dense Scaling

Despite the impressive trajectory of scaling, several fundamental factors limit the indefinite scaling of dense transformer models. Understanding these limits is essential for appreciating why architectural innovation—not just bigger models—is the path forward.

15.2.1 Quadratic Attention Cost

Self-attention scales as \(O(T^2)\) in sequence length \(T\), where \(T\) is the number of tokens in the input. For a single attention head with query, key, and value matrices \(\mathbf {Q}, \mathbf {K}, \mathbf {V} \in \mathbb {R}^{T \times d}\):

\begin {equation} \text {Attention}(\mathbf {Q}, \mathbf {K}, \mathbf {V}) = \text {softmax}\!\left (\frac {\mathbf {Q}\mathbf {K}^\top }{\sqrt {d}}\right )\mathbf {V}, \label {eq:attention-cost} \end {equation}

the matrix product \(\mathbf {Q}\mathbf {K}^\top \) requires \(O(T^2 d)\) operations and \(O(T^2)\) memory. This creates a hard wall for long-context applications:

Example 15.2 (The Quadratic Wall). Consider processing a human genome (\(T \approx 3 \times 10^9\) base pairs) with standard attention:

The attention matrix would have \((3 \times 10^9)^2 = 9 \times 10^{18}\) entries.
At 2 bytes per entry (FP16), this requires \(\mathbf {18}\) exabytes of memory—more than the total global DRAM production in a year.
Even for a modest context window of \(T = 10^6\) tokens, the attention matrix requires 2 terabytes.

The quadratic wall is not merely an inconvenience; it is a fundamental barrier to processing the long sequences that many real-world tasks demand.

15.2.2 Energy and Environmental Cost

Training a frontier model consumes enormous energy. The training of GPT-4 is estimated to have required on the order of \(10^{25}\) FLOPs, consuming megawatt-hours of electricity over several months. Each successive generation of frontier models approximately doubles or triples the energy requirement while yielding progressively smaller loss improvements (because the power-law exponent \(\alpha _C \approx 0.05\) is small).

15.2.3 Data Exhaustion

High-quality text data is finite. Models are approaching the boundary of available internet text. Villalobos et al. [2022] estimated that high-quality language data may be exhausted by 2026–2028 at current training scales. While synthetic data generation offers a partial solution, it introduces risks of model collapse—where models trained on their own outputs degrade in quality over generations.

15.2.4 Capability Plateaus

Certain capabilities have not shown the same smooth scaling as perplexity:

Robust reasoning: Multi-step logical and mathematical reasoning improves with scale but remains fragile, especially on out-of-distribution problems.
Planning: Long-horizon planning and goal-directed behavior remain weak even in the largest models.
Self-correction: Models struggle to identify and correct their own errors without external scaffolding.
Persistent learning: No amount of scaling gives a transformer persistent memory across sessions.

These plateaus suggest that some capabilities require not just more computation but different computation—architectural structures that provide the right inductive biases for reasoning, planning, and learning.

Remark 15.1. The limits of dense scaling do not imply that scaling is useless. Scaling has been the single most productive research strategy of the past five years. The point is that scaling alone is insufficient—it must be complemented by architectural innovation to overcome the barriers enumerated above.

15.3 State Space Models: From S4 to Mamba

State space models (SSMs) represent the most significant challenge to the transformer’s dominance. They process sequences in \(O(T)\) time by modeling sequence dependencies as linear dynamical systems, achieving competitive performance with transformers on many benchmarks while avoiding the quadratic attention bottleneck.

15.3.1 The Continuous-Time State Space

An SSM maps an input sequence \(u(t) \in \mathbb {R}\) to an output sequence \(y(t) \in \mathbb {R}\) through a latent state \(x(t) \in \mathbb {R}^n\):

\begin {equation} \dot {x}(t) = \mathbf {A} x(t) + \mathbf {B} u(t), \quad y(t) = \mathbf {C} x(t) + D u(t), \label {eq:ssm-continuous} \end {equation}

where \(\mathbf {A} \in \mathbb {R}^{n \times n}\) is the state transition matrix, \(\mathbf {B} \in \mathbb {R}^{n \times 1}\) is the input matrix, \(\mathbf {C} \in \mathbb {R}^{1 \times n}\) is the output matrix, and \(D \in \mathbb {R}\) is a feedthrough scalar. To process discrete sequences (as in language modeling), we discretize using a step size \(\Delta \):

\begin {equation} \bar {\mathbf {A}} = \exp (\Delta \mathbf {A}), \quad \bar {\mathbf {B}} = (\Delta \mathbf {A})^{-1}(\bar {\mathbf {A}} - \mathbf {I}) \cdot \Delta \mathbf {B}. \label {eq:ssm-discrete} \end {equation}

The discrete-time recurrence then becomes:

\begin {equation} x_k = \bar {\mathbf {A}} x_{k-1} + \bar {\mathbf {B}} u_k, \quad y_k = \mathbf {C} x_k + D u_k. \label {eq:ch15-ssm-recurrence} \end {equation}

The SSM Dual View

The SSM admits two equivalent computational views:

1.: Recurrent view: Process tokens one at a time using the recurrence in Equation 15.6. Cost: \(O(T \cdot n)\). Memory: \(O(n)\). Ideal for inference.
2.: Convolutional view: Unroll the recurrence into a convolution \(y = K * u\) where \(K_k = \mathbf {C} \bar {\mathbf {A}}^k \bar {\mathbf {B}}\). Compute via FFT in \(O(T \log T)\). Ideal for training.

This dual view gives SSMs both fast training (parallel convolution) and fast inference (sequential recurrence), a combination that transformers lack.

15.3.2 S4: Structured State Spaces for Sequences

The Structured State Space for Sequences (S4) model [Gu et al., 2022] addressed a key challenge: how to parameterize the state matrix \(\mathbf {A}\) so that the SSM can capture long-range dependencies. The solution was to initialize \(\mathbf {A}\) using the HiPPO (High-Order Polynomial Projection Operator) matrix, which is designed to optimally compress the history of the input sequence into the state vector:

\begin {equation} A_{nk} = -\begin {cases} (2n+1)^{1/2}(2k+1)^{1/2} & \text {if } n > k, \\ n+1 & \text {if } n = k, \\ 0 & \text {if } n < k. \end {cases} \label {eq:hippo} \end {equation}

This initialization ensures that the state \(x_k\) maintains a compressed polynomial approximation of the entire input history up to step \(k\), enabling the model to capture dependencies across thousands of time steps.

Example 15.3 (Worked Example: Processing a Sequence Through a 2-State SSM). Let us trace a concrete 4-element sequence \(u = [1.0, 0.5, 0.8, 0.2]\) through a minimal SSM with state dimension \(n=2\). We use:

\[ \mathbf {A} = \begin {pmatrix} -1 & 0 \\ 0 & -2 \end {pmatrix}, \quad \mathbf {B} = \begin {pmatrix} 1 \\ 1 \end {pmatrix}, \quad \mathbf {C} = \begin {pmatrix} 1 & 0.5 \end {pmatrix}, \quad D = 0. \]

With step size \(\Delta = 0.5\), the discretized matrices are: \[ \bar {\mathbf {A}} = \exp (\Delta \mathbf {A}) = \begin {pmatrix} e^{-0.5} & 0 \\ 0 & e^{-1.0} \end {pmatrix} \approx \begin {pmatrix} 0.607 & 0 \\ 0 & 0.368 \end {pmatrix}. \] For \(\bar {\mathbf {B}}\), using the simplified form \(\bar {\mathbf {B}} \approx \Delta \mathbf {B}\): \[ \bar {\mathbf {B}} \approx 0.5 \cdot \begin {pmatrix} 1 \\ 1 \end {pmatrix} = \begin {pmatrix} 0.5 \\ 0.5 \end {pmatrix}. \]

Step 1 (\(u_1 = 1.0\), start from \(x_0 = [0, 0]^\top \)): \[ x_1 = \bar {\mathbf {A}} x_0 + \bar {\mathbf {B}} u_1 = \begin {pmatrix}0\\0\end {pmatrix} + \begin {pmatrix}0.5\\0.5\end {pmatrix} = \begin {pmatrix}0.500\\0.500\end {pmatrix}, \quad y_1 = \mathbf {C} x_1 = 1.0 \cdot 0.500 + 0.5 \cdot 0.500 = \mathbf {0.750}. \]

Step 2 (\(u_2 = 0.5\)): \[ x_2 = \begin {pmatrix}0.607 \cdot 0.500 + 0.5 \cdot 0.5\\0.368 \cdot 0.500 + 0.5 \cdot 0.5\end {pmatrix} = \begin {pmatrix}0.554\\0.434\end {pmatrix}, \quad y_2 = 0.554 + 0.5 \cdot 0.434 = \mathbf {0.771}. \]

Step 3 (\(u_3 = 0.8\)): \[ x_3 = \begin {pmatrix}0.607 \cdot 0.554 + 0.5 \cdot 0.8\\0.368 \cdot 0.434 + 0.5 \cdot 0.8\end {pmatrix} = \begin {pmatrix}0.736\\0.560\end {pmatrix}, \quad y_3 = 0.736 + 0.5 \cdot 0.560 = \mathbf {1.016}. \]

Step 4 (\(u_4 = 0.2\)): \[ x_4 = \begin {pmatrix}0.607 \cdot 0.736 + 0.5 \cdot 0.2\\0.368 \cdot 0.560 + 0.5 \cdot 0.2\end {pmatrix} = \begin {pmatrix}0.547\\0.306\end {pmatrix}, \quad y_4 = 0.547 + 0.5 \cdot 0.306 = \mathbf {0.700}. \]

The output sequence is \(y = [0.750, 0.771, 1.016, 0.700]\). Notice how the state remembers past inputs: \(y_3 = 1.016\) is higher than \(u_3 = 0.8\) because the state carries forward information from the earlier large input \(u_1 = 1.0\). The exponential decay in \(\bar {\mathbf {A}}\) (\(0.607\) and \(0.368\)) ensures that older information fades, with the second state dimension decaying faster (modeling shorter-range dependencies).

SSMs and Biological Signal Propagation

The state space formulation mirrors how biological neurons propagate signals. The state vector \(x_k\) is analogous to membrane potential: it integrates incoming signals (\(\bar {\mathbf {B}} u_k\)) while decaying toward rest (\(\bar {\mathbf {A}} x_{k-1}\)). The HiPPO initialization ensures the “neuron” remembers a polynomial summary of its recent input history—remarkably similar to how synaptic plasticity encodes temporal patterns of stimulation. DOGMA’s DNA computing layers implement a biological version of this principle: strand displacement cascades provide selective memory analogous to Mamba’s input-dependent gating.

15.3.3 Mamba: Selective State Spaces

Mamba [Gu and Dao, 2023] introduced a crucial innovation: input-dependent (selective) parameterization. In S4, the matrices \(\mathbf {A}\), \(\mathbf {B}\), \(\mathbf {C}\) are fixed across all inputs. In Mamba, \(\mathbf {B}\), \(\mathbf {C}\), and the step size \(\Delta \) are functions of the input:

\begin {equation} \mathbf {B}_k = \text {Linear}_B(u_k), \quad \mathbf {C}_k = \text {Linear}_C(u_k), \quad \Delta _k = \text {softplus}\!\left (\text {Linear}_\Delta (u_k)\right ). \label {eq:mamba-selective} \end {equation}

This selectivity allows Mamba to perform content-based reasoning: the model can choose which information to remember and which to forget at each time step, analogous to the gating mechanisms in LSTMs but operating through the SSM framework.

Example 15.4 (Step-by-Step Selective Scan). Consider processing three tokens \(u_1, u_2, u_3\) through a Mamba layer with state dimension \(n = 2\):

Step 1. Compute input-dependent parameters for \(u_1\): \[ \mathbf {B}_1 = \text {Linear}_B(u_1), \quad \mathbf {C}_1 = \text {Linear}_C(u_1), \quad \Delta _1 = \text {softplus}(\text {Linear}_\Delta (u_1)). \] Discretize: \(\bar {\mathbf {A}}_1 = \exp (\Delta _1 \mathbf {A})\), \(\bar {\mathbf {B}}_1 = f(\Delta _1, \mathbf {A}) \cdot \mathbf {B}_1\).

Step 2. Update state: \[ x_1 = \bar {\mathbf {A}}_1 \cdot x_0 + \bar {\mathbf {B}}_1 \cdot u_1. \] Output: \(y_1 = \mathbf {C}_1 \cdot x_1\).

Step 3. Repeat for \(u_2\): compute \(\mathbf {B}_2, \mathbf {C}_2, \Delta _2\) from \(u_2\); update \(x_2 = \bar {\mathbf {A}}_2 x_1 + \bar {\mathbf {B}}_2 u_2\); output \(y_2 = \mathbf {C}_2 x_2\).

Step 4. Repeat for \(u_3\): analogously.

The critical feature is that \(\Delta _k\) controls the “forget rate.” A large \(\Delta _k\) means \(\bar {\mathbf {A}}_k \approx \mathbf {0}\), effectively resetting the state (forgetting previous context). A small \(\Delta _k\) means \(\bar {\mathbf {A}}_k \approx \mathbf {I}\), preserving the state (remembering context). The model learns when to forget through the input-dependent parameterization.

15.3.4 The Hardware-Aware Scan Algorithm

A naive implementation of Mamba’s selective scan is sequential (each state depends on the previous one), which would be slow on GPUs. Mamba solves this through a parallel scan algorithm:

Proposition 15.1 (Parallel Scan Complexity). The selective scan over a sequence of length \(T\) can be computed in \(O(T \log T)\) work and \(O(\log T)\) depth using the parallel scan (prefix sum) algorithm. On a GPU with \(P\) processors, the wall-clock time is \(O(T \log T / P + \log T)\).

The key insight is that the recurrence \(x_k = \bar {\mathbf {A}}_k x_{k-1} + \bar {\mathbf {B}}_k u_k\) can be reformulated as an associative binary operation, enabling the use of Blelloch’s parallel prefix sum algorithm. Mamba further optimizes this by keeping the state in GPU SRAM (fast on-chip memory) rather than HBM (slow off-chip memory), achieving up to \(3\times \) speedup over standard implementations.

Example 15.5 (Worked Example: Selective Scan with Actual Numbers). Process the same sequence \(u = [1.0, 0.5, 0.8, 0.2]\) through a selective Mamba layer (\(n=2\)). Unlike the fixed SSM above, here \(\mathbf {B}_k\) and \(\Delta _k\) are input-dependent:

\[ \mathbf {B}_k = W_B \cdot u_k, \quad \Delta _k = \text {softplus}(w_\Delta \cdot u_k + b_\Delta ). \]

With \(W_B = [0.6,\; 0.4]^\top \), \(w_\Delta = 1.0\), \(b_\Delta = -0.5\), \(\mathbf {A} = \text {diag}(-1, -2)\), \(\mathbf {C} = [1, 0.5]\):

Step 1 (\(u_1 = 1.0\)): \(\Delta _1 = \text {softplus}(1.0 - 0.5) = \ln (1 + e^{0.5}) \approx 0.974\). \[ \bar {\mathbf {A}}_1 = \begin {pmatrix}e^{-0.974}\\&e^{-1.948}\end {pmatrix} \approx \begin {pmatrix}0.378\\&0.143\end {pmatrix}, \quad \bar {\mathbf {B}}_1 \approx 0.974 \begin {pmatrix}0.6\\0.4\end {pmatrix} = \begin {pmatrix}0.584\\0.390\end {pmatrix}. \] \(x_1 = \bar {\mathbf {B}}_1 \cdot 1.0 = [0.584, 0.390]^\top \), \(y_1 = 0.584 + 0.5 \cdot 0.390 = \mathbf {0.779}\).

Step 2 (\(u_2 = 0.5\)): \(\Delta _2 = \text {softplus}(0.5 - 0.5) = \ln 2 \approx 0.693\). \[ \bar {\mathbf {A}}_2 \approx \begin {pmatrix}0.500\\&0.250\end {pmatrix}, \quad \bar {\mathbf {B}}_2 \approx 0.693 \begin {pmatrix}0.3\\0.2\end {pmatrix} = \begin {pmatrix}0.208\\0.139\end {pmatrix}. \] \(x_2 = [0.500 \cdot 0.584 + 0.208 \cdot 0.5,\; 0.250 \cdot 0.390 + 0.139 \cdot 0.5]^\top = [0.396, 0.167]^\top \), \(y_2 = \mathbf {0.480}\).

Notice how the smaller input \(u_2 = 0.5\) produces a smaller \(\Delta _2 = 0.693 < 0.974\), which makes \(\bar {\mathbf {A}}_2\) larger (\(0.500 > 0.378\)). This means the model remembers more when the input is weak (nothing important to overwrite) and forgets more when the input is strong (make room for new information). This is the essence of selectivity.

Connection to DOGMA’s ParallelScan Path

DOGMA’s Path B (Automaton) in the DNA Computing Block uses the same parallel scan algorithm as Mamba, but with a key difference: the state dimension is biologically motivated (automaton states represent molecular configurations) rather than arbitrary. The scan state \(s_k = \bar {\mathbf {A}}_k s_{k-1} + \bar {\mathbf {B}}_k x_k\) is computed using the same Blelloch prefix sum, giving DOGMA’s automaton path \(O(T \log T)\) complexity identical to Mamba. The biological grounding means each state dimension has an interpretable meaning (e.g., “concentration of species \(i\)”), unlike Mamba’s opaque hidden states.

15.4 RWKV and Linear Attention

While Mamba replaces attention with state space recurrences, another family of models seeks to linearize attention itself, retaining the transformer’s conceptual framework while eliminating the quadratic cost.

15.4.1 The Quadratic Bottleneck in Attention

Standard attention computes: \begin {equation} \text {Attn}(\mathbf {Q}, \mathbf {K}, \mathbf {V}) = \text {softmax}\!\left (\frac {\mathbf {Q}\mathbf {K}^\top }{\sqrt {d}}\right )\mathbf {V}. \end {equation}

The \(\text {softmax}\) over \(\mathbf {Q}\mathbf {K}^\top \in \mathbb {R}^{T \times T}\) forces materialization of the full \(T \times T\) matrix, giving \(O(T^2)\) cost.

15.4.2 Linear Attention via Kernel Decomposition

Linear attention [Katharopoulos et al., 2020] replaces the softmax with a feature map \(\phi \): \begin {equation} \text {LinAttn}(\mathbf {Q}, \mathbf {K}, \mathbf {V}) = \frac {\phi (\mathbf {Q}) \left (\phi (\mathbf {K})^\top \mathbf {V}\right )}{\phi (\mathbf {Q}) \left (\phi (\mathbf {K})^\top \mathbf {1}\right )}, \label {eq:linear-attention} \end {equation}

where the key trick is the order of operations: instead of computing \(\phi (\mathbf {Q})\phi (\mathbf {K})^\top \) (which is \(T \times T\)), we first compute \(\phi (\mathbf {K})^\top \mathbf {V}\) (which is \(d \times d\)), then multiply by \(\phi (\mathbf {Q})\). This reduces the cost from \(O(T^2 d)\) to \(O(T d^2)\)—linear in sequence length.

15.4.3 RWKV: Receptance Weighted Key Value

RWKV [Peng et al., 2023] combines ideas from transformers and RNNs into an architecture that can be trained like a transformer (parallelizable) but runs at inference like an RNN (constant memory per token). The core mechanism is the WKV (Weighted Key Value) operator:

\begin {equation} \text {wkv}_t = \frac {\sum _{i=1}^{t-1} e^{-(t-1-i)w + k_i} v_i + e^{u + k_t} v_t}{\sum _{i=1}^{t-1} e^{-(t-1-i)w + k_i} + e^{u + k_t}}, \label {eq:wkv} \end {equation}

where \(w\) is a learned channel-wise decay vector, \(k_i\) and \(v_i\) are key and value vectors at position \(i\), and \(u\) is a learned bonus for the current token.

Definition 15.3 (WKV Mechanism). The WKV operator computes a weighted average of all value vectors, where the weights decay exponentially with distance (controlled by \(w\)) and are modulated by key-query similarity (controlled by \(k_i\)). The parameter \(u\) gives a “bonus” to the current token, ensuring that the model always has strong access to the present input.

Example 15.6 (WKV as Exponentially Decaying Attention). Consider a sequence of five tokens. At position \(t=5\), the WKV operator assigns weights: \begin {align*} \text {Weight for } t=1 &\propto e^{-3w + k_1} \quad \text {(3 steps ago, strongly decayed)} \\ \text {Weight for } t=2 &\propto e^{-2w + k_2} \quad \text {(2 steps ago)} \\ \text {Weight for } t=3 &\propto e^{-w + k_3} \quad \text {(1 step ago)} \\ \text {Weight for } t=4 &\propto e^{k_4} \quad \text {(previous token, no decay)} \\ \text {Weight for } t=5 &\propto e^{u + k_5} \quad \text {(current token, with bonus)} \end {align*}

The exponential decay creates a natural locality bias while the key values \(k_i\) allow the model to “boost” attention to contextually important tokens regardless of distance.

The WKV operator can be computed recurrently in \(O(1)\) per token using running accumulators, giving \(O(T)\) total cost. During training, the parallel form uses chunkwise computation for GPU efficiency.

Figure 15.2: The recurrent view of a state space model (or RWKV). Each input \(u_k\) updates the hidden state \(x_k\) through an input-dependent transition, and the output \(y_k\) is a linear projection of the state. The state propagates left-to-right, giving \(O(T)\) cost.

15.5 Mixture of Experts (MoE)

Mixture of Experts (MoE) architectures attack the scaling problem from a different angle: instead of reducing the cost per token, they increase the total parameter count while keeping the active parameter count constant. This is a form of conditional computation—only a subset of the model is activated for any given input.

15.5.1 The MoE Architecture

In a standard MoE transformer, each feed-forward network (FFN) layer is replaced by \(E\) parallel “expert” networks \(\{f_1, \ldots , f_E\}\) and a gating (routing) function \(G\):

\begin {equation} \text {MoE}(x) = \sum _{i=1}^{E} G(x)_i \cdot f_i(x), \quad G(x) = \text {TopK}\!\left (\text {softmax}(W_g \cdot x), k\right ), \label {eq:ch15-moe} \end {equation}

where \(\text {TopK}\) selects the \(k\) experts with the highest gating scores and zeros out the rest. Typically \(k = 1\) or \(k = 2\), so only 1–2 of \(E\) experts are active per token.

Definition 15.4 (Sparse Routing). Sparse routing refers to the MoE gating mechanism where each input token is processed by only \(k \ll E\) experts. The active compute per token is \(O(k \cdot d_{\text {expert}})\) rather than \(O(E \cdot d_{\text {expert}})\), enabling models with very large total parameter counts (e.g., 1.8 trillion for Mixtral \(8 \times 7\)B) to have tractable inference costs.

15.5.2 Expert Parallelism and Load Balancing

A key engineering challenge in MoE is load balancing: ensuring that tokens are distributed roughly evenly across experts. Without load balancing, popular experts become bottlenecks while unpopular experts waste compute.

The standard approach is to add an auxiliary loss that encourages balanced routing: \begin {equation} \mathcal {L}_{\text {balance}} = \alpha \cdot E \cdot \sum _{i=1}^{E} f_i \cdot P_i, \label {eq:load-balance} \end {equation} where \(f_i\) is the fraction of tokens routed to expert \(i\), \(P_i\) is the mean gating probability for expert \(i\), and \(\alpha \) is a balancing coefficient. Minimizing \(\mathcal {L}_{\text {balance}}\) pushes toward uniform routing (\(f_i = P_i = 1/E\) for all \(i\)).

15.5.3 Expert Specialization

Despite the pressure toward balanced routing, experts tend to specialize: some experts handle mathematical reasoning, others handle code, others handle natural language generation. This emergent specialization is reminiscent of cell differentiation in biology—a connection that DOGMA exploits architecturally.

MoE and Cell Differentiation

In biology, all cells contain the same genome but express different genes through regulatory mechanisms, producing neurons, muscle cells, and immune cells from the same genetic material. MoE achieves a similar effect: all experts share the same architecture but develop different “specializations” through training. The difference is that biological differentiation is driven by a rich regulatory network operating on a structured genome, while MoE specialization emerges from a simple gating function operating on a flat token representation. DOGMA’s regulation stage provides the biological level of control over specialization.

15.6 How DOGMA Scales

Having surveyed the post-transformer landscape, we now analyze DOGMA’s scaling properties and argue that its biology-inspired structure provides a fundamentally different—and potentially superior—scaling paradigm.

15.6.1 The Four Scaling Axes of DOGMA

Dense transformers scale along essentially one axis: model size (number of parameters). DOGMA provides four distinct scaling axes, each inspired by how biological genomes scale:

1.: Parameter scaling (Genome Size). Like all architectures, DOGMA can scale by adding more genomic bases, increasing embedding dimension, or deepening the pipeline. This is analogous to genome size increase through DNA duplication.
2.: Structural scaling (Genomic Organization). DOGMA can scale by adding new genomic regions, new region types, or deeper regulatory hierarchies—without simply adding parameters uniformly. This is analogous to how biological genomes scale through gene duplication and neofunctionalization: new genes arise by copying existing ones, then diverge to acquire new functions.
3.: Regulatory scaling (Expression Complexity). The regulation stage can increase in complexity independently of the genome size: more regulatory elements, more complex regulatory logic, deeper regulatory cascades. This mirrors the observation that the number of regulatory genes scales faster than the total gene count as organisms become more complex.
4.: Evolutionary scaling (Meta-Learning). The meta-policy memory enables cumulative improvement across training runs. Each generation of evolved genomes starts from a better population, potentially yielding super-linear improvement curves across evolutionary cycles.

Figure 15.3: DOGMA’s four scaling axes compared to the transformer’s single axis. Each DOGMA axis corresponds to a biological scaling mechanism.

15.6.2 Sample Efficiency Through Biological Priors

Biological inductive biases such as typed bases, regulatory sparsity, and local structure may reduce the data needed to learn some relationships, but they can also impose the wrong bias. DOGMA encodes these distinctions architecturally; matched experiments must determine whether that helps.

Proposition 15.2 (Sample Efficiency Hypothesis). Let \(L_{\text {DOGMA}}(D)\) and \(L_{\text {Transformer}}(D)\) denote the loss achieved by DOGMA and a dense transformer, respectively, after training on \(D\) tokens with the same compute budget. If DOGMA’s structural priors correctly capture regularities present in the data, then: \begin {equation} L_{\text {DOGMA}}(D) \leq L_{\text {Transformer}}(\beta D) \quad \text {for some } \beta > 1, \end {equation} meaning DOGMA achieves comparable loss with \(1/\beta \) of the data. The constant \(\beta \) depends on how well the structural priors match the data distribution.

15.6.3 Why Four-Path Diversity Helps Scaling

DOGMA’s four base types create four parallel processing pathways through the architecture. Each type specializes in a different computational role:

Base Type	Role	Scaling Benefit
A (Archival)	Long-term knowledge storage	Scales knowledge capacity without increasing active compute
T (Transient)	Working memory	Scales context processing independently of storage
C (Control)	Regulatory signals	Scales expression complexity without increasing parameters
G (Generative)	Output synthesis	Scales generation quality independently of knowledge

The intended benefit is partially independent scaling of capabilities. Whether the paths specialize as intended, rather than duplicating or interfering with one another, is an open ablation question.

15.6.4 Parameter Allocation: DOGMA vs. Transformer

The following historical design exercise allocates a hypothetical 224.7M parameters. It was not the model measured in Chapter 14 and must not be read as a completed implementation or benchmark.

Table 15.2: Hypothetical matched-size allocation study: proposed DOGMA design (224.7M) versus a transformer design (224M).

Component	DOGMA	Transformer	Notes
Token embedding	0.13M	16.4M	DOGMA: \(257 \times 512\); Trans: \(32{,}000 \times 512\)
TetraMemory	33.8M	—	\(K_4\) graph ops, face propagation, multi-scale
DNA Computing (4 paths)	88.3M	—	Strand displ. + automaton + stochastic + double
Regulation Gate	22.4M	—	Sigmoid gating, analogous to Trans. layer norm
Strand Attention	44.8M	100.7M	Thermo-weighted vs. \(\mathbf {QKV}\) projections
Expression Folder	22.4M	—	LayerNorm + MLP folding
FFN / MLP	—	100.7M	\(4d \times d\) up/down projections
Epigenetic Memory	8.4M	—	Persistent cross-block state
Allosteric Regulation	4.2M	—	Sparse top-\(k\) communication
Output head	0.13M	6.1M	DOGMA: byte-level; Trans: 32K vocab
Total	224.7M	224M

What the Allocation Would Test

The proposed comparison contrasts repeated attention/FFN blocks with functionally named components. Names do not guarantee specialization. A valid study would test:

whether four compute paths outperform one equally sized recurrent or convolutional path;
whether a 257-byte vocabulary’s smaller embedding offsets its longer sequences and additional compute; and
whether local memory plus recurrence improves length extrapolation.

15.6.5 Per-Block FLOP Analysis

For a concrete sense of computational cost, here is the FLOP count for one DOGMA block processing a batch of shape \((B{=}4, T{=}512, d{=}512)\):

Stage	FLOPs per block	Computation
TetraMemory	\(\sim \)0.5B	\(4 \times B \times T \times d\) face ops
DNA Compute (4 paths)	\(\sim \)2.1B	CRN: \(B \times T \times 64 \times 32\); Scan: \(O(T \log T)\); etc.
Regulation Gate	\(\sim \)0.5B	\(B \times T \times 2d^2\) (gate projection)
Strand Attention	\(\sim \)1.1B	\(B \times T^2 \times d / h\) (thermo-weighted)
Expression Folder	\(\sim \)0.5B	\(B \times T \times 4d^2\) (MLP folding)
Total per block	\(\sim \)4.7B
16 blocks	\(\sim \)75B	Comparable to 12-layer Transformer (\(\sim \)72B)

The total per-forward-pass cost is comparable to a same-sized Transformer, but the distribution of compute is fundamentally different: DOGMA spends 45% of FLOPs in the DNA Computing Block (structured computation) while a Transformer spends 67% in FFN layers (unstructured matrix multiplications).

15.6.6 Complexity Analysis

DOGMA’s computational cost per layer for a sequence of length \(T\) is:

\begin {equation} C_{\text {DOGMA}}(T) = O(T \cdot k \cdot d), \label {eq:dogma-complexity} \end {equation}

where \(k\) is the average number of active neighbors in the tetrahedral mesh (typically 4–8) and \(d\) is the embedding dimension. For global information propagation across the sequence, \(O(T^{1/3})\) propagation steps suffice due to the volumetric packing of the tetrahedral mesh, giving total cost:

\begin {equation} C_{\text {DOGMA}}^{\text {total}}(T) = O(T^{4/3} \cdot k \cdot d). \end {equation}

Compare this with the transformer and SSM costs:

Table 15.3: Asymptotic complexity comparison across architectures.

Architecture	Per-Layer Cost	Layers for Global	Total Cost
Dense Transformer	\(O(T^2 d)\)	\(O(1)\)	\(O(T^2 d)\)
Mamba / S4	\(O(T n)\)	\(O(1)\)	\(O(T n)\)
Linear Attention	\(O(T d^2)\)	\(O(1)\)	\(O(T d^2)\)
RWKV	\(O(T d)\)	\(O(1)\)	\(O(T d)\)
DOGMA	\(O(T k d)\)	\(O(T^{1/3})\)	\(O(T^{4/3} k d)\)

Remark 15.2. DOGMA’s \(O(T^{4/3})\) scaling is worse than the \(O(T)\) of Mamba or RWKV in the asymptotic limit. However, the constant factors and the benefits of structured state may compensate at practical sequence lengths. Furthermore, the \(T^{1/3}\) propagation depth is a worst-case bound for unstructured genomes; well-organized genomes with hierarchical regulation may achieve global information flow in \(O(\log T)\) depth using skip connections analogous to chromatin looping in biology.

15.7 Hybrid Architectures

The most promising direction in post-transformer research may not be any single alternative but hybrid architectures that combine the strengths of multiple approaches.

15.7.1 Attention + SSM Hybrids

Several recent architectures interleave attention layers with SSM layers:

Jamba uses a 1:7 ratio of attention to Mamba layers, providing the “needle-in-a-haystack” precision of attention at critical points while using Mamba’s efficiency for the bulk of processing.
Zamba places attention layers at regular intervals in a Mamba backbone, using shared attention parameters across positions to minimize overhead.
Griffin combines gated linear recurrences with local attention windows, achieving strong performance on long-context tasks.

The design principle is consistent: use attention sparingly for tasks that genuinely require global pairwise comparison (retrieval, precise copying), and use linear-cost methods for everything else.

15.7.2 Attention + MoE Hybrids

Combining MoE with attention yields models like Mixtral and DeepSeek-V2, which achieve large effective parameter counts with tractable compute. The MoE component provides capacity scaling while attention provides the computational backbone.

15.7.3 Where DOGMA Fits in the Hybrid Landscape

DOGMA can be viewed as a hybrid architecture that combines:

Structured state (from SSMs): The genomic base store is a persistent structured state that evolves over time.
Conditional computation (from MoE): The regulation stage selectively activates genomic regions, creating sparse, input-dependent computation paths.
Local interaction with global propagation (from graph neural networks): The tetrahedral mesh provides local connectivity with hierarchical global information flow.
Evolutionary search (unique to DOGMA): The ability to modify the architecture itself through mutation and selection.

DOGMA as a Natural Hybrid

DOGMA unifies the key innovations of the post-transformer landscape—structured state, conditional computation, and hierarchical processing—within a single biologically principled framework. Where other hybrids combine components through engineering decisions (“put attention every 7th layer”), DOGMA’s hybrid structure emerges from the biological design pattern of genomic organization, where structured state, selective expression, and evolutionary modification are inherently integrated.

15.8 Comprehensive Architecture Comparison

Table 15.4 provides a detailed comparison of the architectures surveyed in this chapter.

Table 15.4: Comprehensive comparison of transformer, post-transformer, and DOGMA architectures.

Property	Transformer	Mamba	RWKV	MoE	DOGMA
Sequence cost	\(O(T^2)\)	\(O(T)\)	\(O(T)\)	\(O(T^2/k)\)	\(O(T^{4/3})\)
Structured state	No	Yes (linear)	Yes (linear)	No	Yes (genomic)
Evolvable	No	No	No	No	Yes
Conditional comp.	No	Selective	Decay-gated	Expert routing	Regulation
Long-range	Global	Via state	Via decay	Global	Hierarchical
Interpretable	Attention maps	Limited	Limited	Expert routing	Genome audit
Inference mem.	\(O(T)\)	\(O(n)\)	\(O(d)\)	\(O(T)\)	\(O(\|\mathcal {G}\|)\)
Bio-inspired	No	Partially	No	Partially	Fully

No Free Lunch, But Better Trade-offs

No architecture dominates on all dimensions. Transformers provide unmatched global interaction but at quadratic cost. SSMs achieve linear cost but sacrifice direct global pairwise comparison. MoE scales parameters efficiently but adds routing complexity. DOGMA’s contribution is a different set of trade-offs: it accepts higher asymptotic cost than SSMs in exchange for structured, interpretable, evolvable state—properties that may matter more than raw token throughput on the path to general intelligence.

15.9 Worked Example: Comparing Architectures on a Long Sequence

Example 15.7 (Processing a 100K-Token Document). Suppose we want to process a scientific paper of \(T = 100{,}000\) tokens. Let \(d = 1024\) (embedding dimension), \(n = 64\) (SSM state dimension), \(k = 6\) (DOGMA mesh degree), \(E = 8\) experts with top-2 routing.

Dense Transformer (per layer): \[ C_{\text {Trans}} = T^2 \cdot d = (10^5)^2 \times 1024 = 1.024 \times 10^{13} \text { FLOPs} \]

Mamba (per layer): \[ C_{\text {Mamba}} = T \cdot n \cdot d = 10^5 \times 64 \times 1024 = 6.55 \times 10^9 \text { FLOPs} \]

RWKV (per layer): \[ C_{\text {RWKV}} = T \cdot d^2 = 10^5 \times 1024^2 = 1.05 \times 10^{11} \text { FLOPs} \]

MoE Transformer (per layer, top-2 of 8): \[ C_{\text {MoE}} = T^2 \cdot d + T \cdot 2 \cdot d_{\text {expert}} \approx 1.024 \times 10^{13} \text { FLOPs (attention dominates)} \]

DOGMA (total, \(T^{1/3} \approx 46\) layers): \[ C_{\text {DOGMA}} = T^{4/3} \cdot k \cdot d = (10^5)^{4/3} \times 6 \times 1024 \approx 2.9 \times 10^{10} \text { FLOPs} \]

DOGMA is approximately \(350\times \) cheaper than a dense transformer and comparable to Mamba for this sequence length.

15.10 Exercises

15.1.: [Computation] Using Equation 15.1, calculate how much compute is needed to halve the loss from a baseline. How does this compare across the three scaling dimensions (\(N\), \(D\), \(C\))? Show your work.
15.2.: [Analysis] The Chinchilla scaling law (Equation 15.2) suggests \(N_{\text {opt}} \propto C^{0.5}\). A lab has a budget of \(10^{23}\) FLOPs. Using the approximation \(C \approx 6ND\), compute the optimal model size and dataset size. If the lab doubles its budget, how should it allocate the additional compute?
15.3.: [Design] Propose three ways to “scale” a DOGMA architecture without simply increasing the parameter count of existing modules. For each, explain the biological analogue and predict how it would affect the scaling exponent.
15.4.: [Step-by-Step] Trace through Mamba’s selective scan for a four-token sequence. Assume state dimension \(n=2\) and provide explicit numerical values for \(\mathbf {A}\), \(\bar {\mathbf {B}}_k\), and \(\mathbf {C}_k\). Show the state evolution \(x_0 \to x_1 \to x_2 \to x_3 \to x_4\) and final outputs \(y_1, \ldots , y_4\).
15.5.: [Conceptual] Compare MoE routing with DOGMA regulation. Both achieve conditional computation. What structural advantages does DOGMA’s typed genomic organization provide over MoE’s flat expert selection? When might MoE’s simplicity be preferable?
15.6.: [Research] Read Gu and Dao [2023]. How does Mamba’s selective state space mechanism compare to DOGMA’s regulation stage? Both achieve input-dependent computation—compare their mechanisms, inductive biases, and failure modes.
15.7.: [Theory] If DOGMA’s TetraMemory achieves \(O(T \cdot k)\) cost per layer and requires \(O(T^{1/3})\) layers for global information flow, what is the total cost for processing a sequence of length \(T\)? Compare this with a transformer’s \(O(T^2 \cdot L)\) for \(L\) layers. At what sequence length \(T^*\) does DOGMA become cheaper than a 32-layer transformer?
15.8.: [Coding] Implement a minimal SSM layer in Python using PyTorch. Your implementation should support both the recurrent mode (sequential, \(O(T \cdot n)\)) and the convolutional mode (parallel, \(O(T \log T)\) via FFT). Test on a synthetic sequence and verify that both modes produce the same output.
15.9.: [Discussion] The hybrid architecture trend (Section 15.7) suggests that the future may not belong to any single architecture but to combinations. Design a hybrid that combines DOGMA’s genomic regulation with Mamba’s selective scan. Specify which stages use which mechanism and justify your design.
15.10.: [Advanced] Design an experiment to test the DOGMA scaling hypothesis: that structured architectures achieve better loss-per-parameter than dense transformers. Specify the model sizes (at least four sizes spanning two orders of magnitude), datasets, training procedures, and metrics you would use. Discuss potential confounds and how you would control for them.

15.11 Key Takeaways

Key Takeaways

Neural scaling laws (Kaplan, Chinchilla) reveal power-law relationships between model size, data, compute, and loss. These laws hold within an architecture family—changing the architecture changes the scaling exponents.
Dense transformer scaling faces five fundamental limits: quadratic attention cost, energy consumption, data exhaustion, diminishing returns, and capability plateaus in reasoning, planning, and persistent learning.
State space models (S4, Mamba) achieve \(O(T)\) sequence processing by modeling dependencies as linear dynamical systems with input-dependent (selective) parameterization. The dual recurrent/convolutional view enables both fast inference and fast training.
RWKV and linear attention linearize the attention mechanism, replacing the \(T \times T\) attention matrix with \(O(T d^2)\) or \(O(Td)\) computation through kernel decomposition or exponentially decaying recurrence.
Mixture of Experts (MoE) scales total parameters while keeping active compute constant through sparse routing, achieving conditional computation analogous to (but less structured than) DOGMA’s regulation.
DOGMA provides four scaling axes—parameter, structural, regulatory, and evolutionary—compared to the transformer’s single axis of parameter count. The four base types create parallel processing pathways that can be scaled independently.
Hybrid architectures that combine attention, SSMs, MoE, and structured state are the most promising direction. DOGMA can be understood as a biologically principled hybrid that unifies structured state, conditional computation, and evolutionary self-modification.