Machine Learning Foundations for Biological AI

Parts I established that biology encodes, regulates, and evolves information through sophisticated computational mechanisms. This chapter begins Part II by building the machine learning foundations needed to translate those biological principles into working AI systems. We cover neural networks, optimization, representation learning, and evolutionary algorithms—focusing specifically on the concepts that reappear in DOGMA’s architecture.

This is not a comprehensive ML textbook (we recommend Goodfellow et al. [2016] for that). Instead, it is a targeted introduction that equips the reader with the precise ML vocabulary and intuitions needed for the DOGMA chapters that follow.

4.1 The Learning Problem

4.1.1 Supervised Learning as Function Approximation

At its core, machine learning is about finding patterns in data. The simplest formulation is supervised learning: given a dataset of input-output pairs, find a function that maps inputs to outputs.

There are three crucial choices in any supervised learning setup: (1) the model class (what family of functions \(f_\theta \) can we represent?), (2) the loss function (what does “good” mean?), and (3) the optimizer (how do we search for \(\theta ^*\)?). The rest of this chapter develops each of these in detail.

Worked example: DNA GC-content classification. Suppose we want to classify DNA sequences of length 10 into “GC-rich” (\(\geq 60\%\) G/C content) or “GC-poor” (\(< 60\%\)). Our input is a sequence like ATCGGCTAGC, represented as a vector \(\mathbf {x} \in \{0,1\}^{40}\) using one-hot encoding (4 bases \(\times \) 10 positions), and our output is a binary label \(y \in \{0, 1\}\). We seek a function \(f_\theta \) that predicts \(y\) from \(\mathbf {x}\). A simple approach is logistic regression: \(f_\theta (\mathbf {x}) = \sigma (\mathbf {w}^\top \mathbf {x} + b)\), where \(\sigma \) is the sigmoid function.

Let us trace through this concretely. For the sequence ATCGGCTAGC, the one-hot encoding assigns a 4-dimensional vector to each position: A\(\to [1,0,0,0]\), T\(\to [0,0,0,1]\), C\(\to [0,1,0,0]\), G\(\to [0,0,1,0]\). Concatenating all 10 positions yields a 40-dimensional binary vector. The GC content is \(6/10 = 60\%\), so the label is \(y = 1\) (GC-rich). If we initialize \(\mathbf {w}\) so that the weights corresponding to G and C positions are positive (\(+0.3\) each) and A and T positions are negative (\(-0.2\) each), the pre-activation \(z = \mathbf {w}^\top \mathbf {x} + b\) will be positive for GC-rich sequences, and \(\sigma (z)\) will be close to 1. Training adjusts these weights to maximize classification accuracy on the training set.

4.1.2 Loss Functions

The choice of loss function \(\mathcal {L}\) determines what “good prediction” means. The most common loss functions are:

Why cross-entropy and not MSE for classification? For classification tasks, cross-entropy is preferred over MSE for two reasons. First, the gradient of cross-entropy with respect to the logits simplifies to \(\hat {\mathbf {y}} - \mathbf {y}\) (as you will prove in Exercise 4.1), producing a gradient that is large when the prediction is wrong and small when it is right. MSE gradients, by contrast, can become very small near \(\hat {y} = 0\) or \(\hat {y} = 1\) due to the saturating sigmoid, slowing learning. Second, cross-entropy has an information-theoretic interpretation: it measures the extra bits needed to encode the true distribution using the predicted distribution, connecting machine learning to the information theory discussed in Chapter 1.

Worked example: cross-entropy for nucleotide prediction. Suppose our model predicts the next nucleotide in a DNA sequence. The true next base is G, encoded as \(\mathbf {y} = [0, 0, 1, 0]\) (for A, C, G, T). Our model predicts probabilities \(\hat {\mathbf {y}} = [0.1, 0.2, 0.6, 0.1]\). The cross-entropy loss is: \begin {equation} \mathcal {L} = -(0 \cdot \log 0.1 + 0 \cdot \log 0.2 + 1 \cdot \log 0.6 + 0 \cdot \log 0.1) = -\log 0.6 \approx 0.511. \end {equation} A perfect prediction of \(\hat {y}_G = 1.0\) would give \(\mathcal {L} = 0\). A uniform prediction of \(\hat {y}_G = 0.25\) would give \(\mathcal {L} = -\log 0.25 \approx 1.386\). The loss measures how “surprised” the model is by the true answer.

4.1.3 The Bias-Variance Tradeoff

A fundamental tension in machine learning is between models that are too simple (high bias) and too complex (high variance).

Intuition. Imagine fitting a curve to noisy data:

• A straight line (linear model) may miss the true pattern (high bias, low variance).
• A degree-20 polynomial may fit every noise point (low bias, high variance).
• A degree-3 polynomial may capture the true pattern without overfitting (good tradeoff).

Why the tradeoff matters quantitatively. Consider a concrete scenario. Suppose the true function is \(f(x) = \sin (x)\) and we observe data with Gaussian noise \(\epsilon \sim \mathcal {N}(0, 0.1^2)\). A linear model \(\hat {f}(x) = ax + b\) has high bias (it cannot represent the curvature of sine) but low variance (the estimated line changes little between different training samples). A degree-15 polynomial has low bias (it can approximate \(\sin (x)\) well) but high variance: with only 10 training points, the polynomial wiggles wildly between data points, and different training sets produce drastically different fits. A degree-3 polynomial \(\hat {f}(x) = a_0 + a_1 x + a_2 x^2 + a_3 x^3\) provides a good tradeoff: the Taylor expansion of \(\sin (x)\) is \(x - x^3/6 + \cdots \), so degree 3 captures the dominant pattern while having few enough parameters to be stably estimated.

4.1.4 Unsupervised and Self-Supervised Learning

Not all learning requires labeled data. In unsupervised learning, the model discovers structure in unlabeled data (clustering, dimensionality reduction). In self-supervised learning, the model creates its own supervisory signal from the data structure.

The dominant self-supervised paradigm for language models is next-token prediction: \begin {equation} \mathcal {L}_{\text {LM}}(\theta ) = -\sum _{t=1}^{T} \log p_\theta (x_t \mid x_1, \ldots , x_{t-1}). \label {eq:lm-loss} \end {equation}

This objective requires no human-labeled data—the text itself provides the supervision. The same principle applies to genomic sequences: predicting the next nucleotide given preceding context is a natural self-supervised objective for DNA language models (Chapter 6).

Why self-supervised learning is so powerful. The key advantage is data efficiency at the labeling stage. Consider genomic data: we have billions of base pairs of sequenced DNA, but only a tiny fraction has been functionally annotated. Self-supervised pre-training on raw sequence data learns representations that capture sequence grammar, motif structure, and evolutionary constraints—all without a single label. These pre-trained representations can then be fine-tuned with a small number of labeled examples for specific tasks like promoter prediction or variant effect classification. This pre-train then fine-tune paradigm is the foundation of modern biological AI.

Other self-supervised objectives.

• Masked language modeling (MLM): Randomly mask tokens and predict them from surrounding context. Used in BERT [Devlin et al., 2019] and DNABERT [Ji et al., 2021b]. For example, given “ATG[MASK]GC”, the model must predict that the masked token is most likely C or T based on context.
• Contrastive learning: Learn embeddings where similar items are close and dissimilar items are far. Used in protein representation learning [Lin et al., 2023]. The objective pulls together embeddings of augmented views of the same protein while pushing apart embeddings of different proteins.
• Denoising: Corrupt input and train the model to reconstruct the original. Used in denoising autoencoders and diffusion models. For DNA, corruption might involve random base substitutions, and the model learns to “correct” them.

4.2 Neural Networks from First Principles

4.2.1 The Perceptron: A Single Neuron

The simplest neural network is a single neuron (perceptron), which computes a weighted sum of its inputs, adds a bias, and applies a nonlinear activation function:

\begin {equation} y = \phi (\mathbf {w}^\top \mathbf {x} + b) = \phi \!\left (\sum _{i=1}^d w_i x_i + b\right ). \end {equation}

The weights \(\mathbf {w}\) determine how much each input contributes to the output, and the bias \(b\) shifts the decision boundary. Together, \(\mathbf {w}\) and \(b\) define a hyperplane in \(d\)-dimensional space: the neuron outputs different values on each side of this hyperplane.

Worked example. Consider a neuron with 3 inputs (\(d=3\)), weights \(\mathbf {w} = [0.5, -0.3, 0.8]\), bias \(b = 0.1\), and ReLU activation \(\phi (z) = \max (0, z)\). For input \(\mathbf {x} = [1.0, 2.0, 0.5]\): \begin {align} z &= 0.5 \times 1.0 + (-0.3) \times 2.0 + 0.8 \times 0.5 + 0.1 \\ &= 0.5 - 0.6 + 0.4 + 0.1 = 0.4, \\ y &= \max (0, 0.4) = 0.4. \end {align}

If instead the input were \(\mathbf {x} = [0.1, 2.0, 0.5]\), we would get \(z = 0.05 - 0.6 + 0.4 + 0.1 = -0.05\), and \(y = \max (0, -0.05) = 0\). The neuron is “off” for this input—ReLU has completely silenced it. This sparsity (many neurons outputting exactly zero) is a key property of ReLU networks and contributes to computational efficiency.

A single perceptron can only represent linear decision boundaries. It can compute AND and OR but cannot compute XOR—this limitation motivated the development of multi-layer networks.

AND and OR are linearly separable (a single line separates the 0s from the 1s), but XOR is not. This is the historical motivation for deep networks.

4.2.2 Activation Functions

The activation function \(\phi \) introduces nonlinearity, enabling the network to learn complex patterns. Without nonlinearity, a stack of linear layers would collapse into a single linear transformation. To see why, note that if \(f_1(\mathbf {x}) = \mathbf {W}_1 \mathbf {x} + \mathbf {b}_1\) and \(f_2(\mathbf {x}) = \mathbf {W}_2 \mathbf {x} + \mathbf {b}_2\), then \(f_2(f_1(\mathbf {x})) = \mathbf {W}_2(\mathbf {W}_1 \mathbf {x} + \mathbf {b}_1) + \mathbf {b}_2 = (\mathbf {W}_2 \mathbf {W}_1)\mathbf {x} + (\mathbf {W}_2 \mathbf {b}_1 + \mathbf {b}_2)\), which is just another linear function. No matter how many linear layers you stack, the result is equivalent to a single linear layer.

The vanishing gradient problem in detail. Sigmoid and tanh saturate for large \(|z|\): when \(z = 10\), \(\sigma '(10) \approx 0.0000454\). During backpropagation, gradients are multiplied through layers, so if each layer contributes a factor near zero, the gradient shrinks exponentially. For a network with \(L\) layers, the gradient at the first layer is proportional to \(\prod _{\ell =1}^{L} \sigma '(z^{(\ell )})\). If each factor is \(0.25\) (the maximum of \(\sigma '\)), then after just 10 layers: \(0.25^{10} \approx 9.5 \times 10^{-7}\). This makes training deep networks with sigmoid activations extremely difficult.

ReLU solves this by having a derivative of exactly 1 for \(z > 0\), allowing gradients to flow unchanged through active neurons. The cost is the “dead neuron” problem: if a neuron’s pre-activation \(z\) is always negative for all training inputs, the gradient is always zero and the neuron can never recover. Leaky ReLU mitigates this by using a small positive slope \(\alpha \) (typically 0.01) for negative inputs.

4.2.3 Multi-Layer Perceptrons (MLPs)

A multi-layer perceptron (MLP) composes multiple layers of neurons: \begin {equation} \mathbf {h}^{(\ell )} = \phi ^{(\ell )}\bigl (\mathbf {W}^{(\ell )} \mathbf {h}^{(\ell -1)} + \mathbf {b}^{(\ell )}\bigr ), \quad \ell = 1, \ldots , L. \end {equation}

Here, \(\mathbf {h}^{(0)} = \mathbf {x}\) is the input, \(\mathbf {W}^{(\ell )} \in \mathbb {R}^{d_\ell \times d_{\ell -1}}\) is the weight matrix connecting layer \(\ell -1\) to layer \(\ell \), \(\mathbf {b}^{(\ell )} \in \mathbb {R}^{d_\ell }\) is the bias vector, and \(d_\ell \) is the width (number of neurons) of layer \(\ell \). The total number of learnable parameters is \(\sum _{\ell =1}^{L} (d_\ell \times d_{\ell -1} + d_\ell )\).

Worked example: 2-layer MLP for XOR. We solve XOR using a network with 2 inputs, a hidden layer of 2 neurons (ReLU activation), and 1 output neuron (sigmoid activation): \begin {align} \text {Hidden layer:} \quad \mathbf {W}^{(1)} &= \begin {bmatrix} 1 & 1 \\ 1 & 1 \end {bmatrix}, \quad \mathbf {b}^{(1)} = \begin {bmatrix} 0 \\ -1 \end {bmatrix} \\ \text {Output layer:} \quad \mathbf {w}^{(2)} &= \begin {bmatrix} 1 \\ -2 \end {bmatrix}, \quad b^{(2)} = 0 \end {align}

Let us verify all four cases:

The network outputs high values (\(0.73\)) for XOR=1 cases and \(0.5\) for XOR=0 cases. With further weight tuning (e.g., scaling \(\mathbf {w}^{(2)}\) by a factor of 10), the separation becomes sharper. The key insight: the hidden layer creates a new representation in which the XOR problem becomes linearly separable. Neuron \(h_1\) detects “at least one input is 1” and neuron \(h_2\) detects “both inputs are 1”; the output layer then computes “\(h_1\) but not \(h_2\).”

The theorem guarantees existence but says nothing about efficiency. Deep networks achieve the same approximation quality with exponentially fewer parameters than shallow ones for many function classes—this is why depth matters.

Why depth helps: a counting argument. A function that requires \(2^n\) neurons in a single hidden layer may require only \(O(n)\) neurons distributed across \(O(n)\) layers. For example, computing the parity of \(n\) binary inputs requires exponentially many neurons in one layer but only \(n-1\) neurons across \(\log _2 n\) layers of XOR gates.

4.3 Training Neural Networks

4.3.1 Backpropagation: Computing Gradients Efficiently

Parameters are optimized via gradient descent, and the gradient of the loss with respect to each parameter is computed efficiently by backpropagation—application of the chain rule through the computational graph.

The chain rule in action. Consider a simple 2-layer network: \(z_1 = w_1 x\), \(h = \phi (z_1)\), \(z_2 = w_2 h\), \(\hat {y} = \sigma (z_2)\), \(\mathcal {L} = -[y \log \hat {y} + (1-y)\log (1-\hat {y})]\). The gradient with respect to \(w_1\) is: \begin {equation} \frac {\partial \mathcal {L}}{\partial w_1} = \frac {\partial \mathcal {L}}{\partial \hat {y}} \cdot \frac {\partial \hat {y}}{\partial z_2} \cdot \frac {\partial z_2}{\partial h} \cdot \frac {\partial h}{\partial z_1} \cdot \frac {\partial z_1}{\partial w_1}. \end {equation}

Each factor in this chain is a simple local derivative. Backpropagation computes all gradients in a single backward pass through the network, with computational cost proportional to the forward pass.

Step-by-step backpropagation example. Let \(x = 2\), \(w_1 = 0.5\), \(w_2 = 1.0\), \(y = 1\) (true label), and \(\phi = \text {ReLU}\):

1.

Forward pass: \(z_1 = 0.5 \times 2 = 1.0\); \(h = \max (0, 1.0) = 1.0\); \(z_2 = 1.0 \times 1.0 = 1.0\); \(\hat {y} = \sigma (1.0) = 0.731\).

2.

Loss: \(\mathcal {L} = -\log (0.731) = 0.313\).

3.

Backward pass:

• \(\frac {\partial \mathcal {L}}{\partial \hat {y}} = -\frac {y}{\hat {y}} = -\frac {1}{0.731} = -1.368\)
• \(\frac {\partial \hat {y}}{\partial z_2} = \hat {y}(1-\hat {y}) = 0.731 \times 0.269 = 0.197\)
• \(\frac {\partial \mathcal {L}}{\partial z_2} = -1.368 \times 0.197 = -0.269\) (this is \(\hat {y} - y = 0.731 - 1\))
• \(\frac {\partial z_2}{\partial w_2} = h = 1.0\), so \(\frac {\partial \mathcal {L}}{\partial w_2} = -0.269 \times 1.0 = -0.269\)
• \(\frac {\partial z_2}{\partial h} = w_2 = 1.0\), \(\frac {\partial h}{\partial z_1} = 1\) (ReLU derivative for \(z_1 > 0\))
• \(\frac {\partial \mathcal {L}}{\partial w_1} = -0.269 \times 1.0 \times 1 \times x = -0.269 \times 2 = -0.538\)

4.

Update: With learning rate \(\eta = 0.1\): \(w_1 \leftarrow 0.5 - 0.1 \times (-0.538) = 0.554\); \(w_2 \leftarrow 1.0 - 0.1 \times (-0.269) = 1.027\).

Both weights increased, which will push \(\hat {y}\) closer to \(y = 1\) on the next forward pass. Let us verify: with the updated weights, the forward pass gives \(z_1 = 0.554 \times 2 = 1.108\), \(h = 1.108\), \(z_2 = 1.027 \times 1.108 = 1.138\), \(\hat {y} = \sigma (1.138) = 0.757\). Indeed, \(\hat {y}\) has moved from \(0.731\) to \(0.757\), closer to the target \(y=1\), and the loss decreased from \(0.313\) to \(-\log (0.757) = 0.278\).

4.3.2 Gradient Descent Variants

The basic gradient descent update rule is: \begin {equation} \theta \leftarrow \theta - \eta \nabla _\theta \mathcal {L}(\theta ), \end {equation} where \(\eta \) is the learning rate. In practice, several variants improve convergence:

Worked example: gradient descent on a simple function. To build intuition, consider minimizing \(f(\theta ) = (\theta - 3)^2\) starting from \(\theta _0 = 0\) with learning rate \(\eta = 0.2\). The gradient is \(f'(\theta ) = 2(\theta - 3)\).

The parameter \(\theta \) converges toward the minimum at \(\theta ^* = 3\). Notice that each step reduces the loss by a factor of approximately \(0.36 = (1 - 2\eta )^2 = (1 - 0.4)^2\), which is a geometric convergence rate characteristic of gradient descent on quadratic functions. If we set \(\eta = 0.5\), each step would be \(\theta _{t+1} = \theta _t - 2(\theta _t - 3) = 6 - \theta _t\), causing the parameter to oscillate: \(0 \to 6 \to 0 \to 6 \to \cdots \). If \(\eta > 0.5\), the oscillations diverge. This illustrates why the learning rate must be chosen carefully.

Adam [Kingma and Ba, 2015] is the most widely used optimizer in modern deep learning. It maintains exponential moving averages of the gradient (first moment \(\hat {m}\)) and the squared gradient (second moment \(\hat {v}\)), providing adaptive learning rates for each parameter. The full update rule is:

\begin {align} m_t &= \beta _1 m_{t-1} + (1 - \beta _1) g_t, \\ v_t &= \beta _2 v_{t-1} + (1 - \beta _2) g_t^2, \\ \hat {m}_t &= m_t / (1 - \beta _1^t), \quad \hat {v}_t = v_t / (1 - \beta _2^t), \\ \theta _{t+1} &= \theta _t - \eta \, \hat {m}_t / (\sqrt {\hat {v}_t} + \epsilon ), \end {align}

where \(g_t = \nabla _\theta \mathcal {L}(\theta _t)\), \(\beta _1 = 0.9\), \(\beta _2 = 0.999\), and \(\epsilon = 10^{-8}\) are the standard defaults. The bias correction terms (\(\hat {m}_t\) and \(\hat {v}_t\)) are essential in early training when the moving averages are biased toward zero due to initialization at \(m_0 = v_0 = 0\). Most DOGMA foundation model training uses Adam with learning rate warmup and cosine decay.

4.3.3 The Loss Landscape

Understanding optimization in neural networks requires thinking about the loss landscape—the surface defined by the loss function over the parameter space. For a network with \(d\) parameters, this is a surface in \((d+1)\)-dimensional space, but we can visualize 2D slices.

Key features of neural network loss landscapes:

• Local minima are points where the loss is lower than all nearby points but not the global minimum. In high-dimensional spaces, most local minima have loss values close to the global minimum [Goodfellow et al., 2016].
• Saddle points are far more common than local minima in high dimensions. At a saddle point, the gradient is zero, but the Hessian has both positive and negative eigenvalues (loss increases in some directions and decreases in others).
• Plateaus are flat regions where the gradient is near zero, causing slow learning. Momentum helps escape plateaus by accumulating velocity from previous gradients.

4.3.4 Regularization: Preventing Overfitting

A model that memorizes the training data (overfitting) fails to generalize. Regularization techniques constrain the model to prefer simpler solutions:

1.

L2 regularization (weight decay): Add \(\lambda \|\theta \|^2\) to the loss, penalizing large weights: \begin {equation} \mathcal {L}_{\text {reg}} = \mathcal {L}_{\text {data}} + \lambda \sum _i \theta _i^2. \end {equation} This is equivalent to a Gaussian prior on the weights in a Bayesian framework. Concretely, \(\lambda = 0.01\) means we are saying “we believe the weights should be small, with a prior variance of \(1/(2\lambda ) = 50\).” Larger \(\lambda \) imposes a stronger prior, yielding a simpler model.

2.

Dropout [Srivastava et al., 2014]: During training, randomly set each neuron’s output to zero with probability \(p\) (typically \(p = 0.1\) to \(0.5\)). This forces the network to learn redundant representations, preventing co-adaptation of neurons. At test time, all neurons are active but outputs are scaled by \((1-p)\) to compensate. Dropout can be interpreted as training an ensemble of \(2^N\) sub-networks (where \(N\) is the number of neurons) that share parameters.

3.

Early stopping: Monitor performance on a held-out validation set and stop training when validation loss begins to increase, even if training loss continues to decrease. The gap between training loss (decreasing) and validation loss (increasing) is a signature of overfitting.

4.

Data augmentation: Artificially increase the training set by applying transformations. For DNA sequences, augmentations include reverse complementation, random mutation, and subsequence extraction.

5.

Layer normalization [Ba et al., 2016]: Normalize activations within each layer to have zero mean and unit variance, stabilizing the distribution of hidden representations. For a vector \(\mathbf {h}\) of activations: \(\text {LayerNorm}(\mathbf {h}) = \gamma \cdot \frac {\mathbf {h} - \mu }{\sqrt {\sigma ^2 + \epsilon }} + \beta \), where \(\mu \) and \(\sigma ^2\) are the mean and variance of \(\mathbf {h}\), and \(\gamma , \beta \) are learned scale and shift parameters.

4.4 Convolutional Neural Networks

Before transformers dominated, convolutional neural networks (CNNs) were the architecture of choice for structured data. CNNs remain important in biological AI for processing DNA sequences and protein structures.

4.4.1 The Convolution Operation

A 1D convolution slides a learned filter (kernel) \(\mathbf {k} \in \mathbb {R}^w\) across an input sequence \(\mathbf {x} \in \mathbb {R}^T\) to produce a feature map: \begin {equation} (\mathbf {x} * \mathbf {k})_t = \sum _{i=0}^{w-1} k_i \cdot x_{t+i}. \end {equation}

The convolution has two important properties that make it efficient and effective:

• Parameter sharing: The same filter weights are used at every position. A filter with width \(w\) has only \(w\) parameters regardless of sequence length \(T\), compared to \(T \times T\) for a fully connected layer.
• Translation equivariance: If the input shifts by \(k\) positions, the output feature map also shifts by \(k\) positions. A motif detector works identically regardless of where the motif appears in the sequence.

Worked example: motif detection in DNA. Suppose we have a DNA sequence encoded numerically: \(\mathbf {x} = [1, 0, 0, 1, 1, 0, 1, 0]\) (where 1 = purine, 0 = pyrimidine). A filter \(\mathbf {k} = [1, 1, 1]\) (width 3) detects runs of three purines: \begin {align} (\mathbf {x} * \mathbf {k})_1 &= 1 \cdot 1 + 0 \cdot 1 + 0 \cdot 1 = 1 \\ (\mathbf {x} * \mathbf {k})_2 &= 0 \cdot 1 + 0 \cdot 1 + 1 \cdot 1 = 1 \\ (\mathbf {x} * \mathbf {k})_3 &= 0 \cdot 1 + 1 \cdot 1 + 1 \cdot 1 = 2 \\ (\mathbf {x} * \mathbf {k})_4 &= 1 \cdot 1 + 1 \cdot 1 + 0 \cdot 1 = 2 \\ (\mathbf {x} * \mathbf {k})_5 &= 1 \cdot 1 + 0 \cdot 1 + 1 \cdot 1 = 2 \\ (\mathbf {x} * \mathbf {k})_6 &= 0 \cdot 1 + 1 \cdot 1 + 0 \cdot 1 = 1 \end {align}

The highest activations (value 2) occur at positions 3–5, where purine density is highest. In practice, DNA models like DeepSEA [Zhou and Troyanskaya, 2015] use hundreds of learned filters to detect regulatory motifs.

4.4.2 Key CNN Concepts

• Stride: The step size between filter applications. Stride \(>1\) reduces output length (downsampling). With stride \(s\), an input of length \(T\) with filter width \(w\) produces an output of length \(\lfloor (T - w)/s \rfloor + 1\).
• Padding: Adding zeros around the input to control output size. “Same” padding preserves input length; “valid” padding (no padding) reduces length by \(w - 1\).
• Pooling: Aggregating neighboring activations (max pooling or average pooling) to reduce dimensionality and provide translation invariance. Max pooling with window 2 halves the sequence length while keeping the strongest activations.
• Multiple channels: Each input can have multiple channels (e.g., 4 channels for one-hot encoded DNA), and each filter produces one output channel. A CNN layer with \(C_{\text {in}}\) input channels and \(C_{\text {out}}\) filters has \(C_{\text {out}} \times C_{\text {in}} \times w\) learnable weights.
• Dilated convolutions: Inserting gaps between filter elements allows a small filter to cover a large receptive field. A filter with dilation \(d\) and width \(w\) has an effective receptive field of \(w + (w-1)(d-1)\) positions. This is useful for capturing long-range patterns in DNA without increasing parameters.

Receptive field growth. Stacking \(L\) convolutional layers with filter width \(w\) gives a total receptive field of \(L(w-1) + 1\). With dilated convolutions using dilation rates \(1, 2, 4, \ldots , 2^{L-1}\), the receptive field grows exponentially as \(2^L - 1 + w - 1\). For DNA, where regulatory elements can influence expression from thousands of bases away, this exponential growth is essential.

4.5 Representation Learning

4.5.1 Embeddings: Mapping Symbols to Geometry

The core idea of representation learning is to map discrete symbols into continuous vector spaces where geometric relationships capture semantic relationships. An embedding is a learned function \(e : \Sigma \to \mathbb {R}^d\) that assigns each symbol a dense vector.

For a vocabulary \(V\) of size \(|V|\), an embedding layer is a matrix \(\mathbf {E} \in \mathbb {R}^{|V| \times d}\) where row \(i\) is the embedding of token \(i\). The embedding is learned end-to-end through backpropagation.

Why embeddings rather than one-hot encoding? A one-hot vector for a vocabulary of size \(|V|\) is \(|V|\)-dimensional with a single 1 and all other entries 0. This representation has several problems: (1) it is very high-dimensional for large vocabularies (\(|V| = 64\) for DNA 3-mers, \(|V| \approx 50{,}000\) for text), (2) all pairs of symbols are equidistant (cosine similarity is 0 between any two different one-hot vectors), so the representation encodes no similarity structure, and (3) it is extremely sparse. Embeddings solve all three problems: they are low-dimensional (\(d \ll |V|\)), they capture similarity (similar symbols have similar embeddings), and they are dense.

Worked example: DNA k-mer embeddings. Consider a vocabulary of all DNA 3-mers (64 possible): ATG, AAA, CCC, etc. An embedding dimension of \(d = 16\) gives an embedding matrix \(\mathbf {E} \in \mathbb {R}^{64 \times 16}\). After training on genomic data, we might find:

• Start codons (ATG) and other coding-related 3-mers cluster together.
• GC-rich 3-mers (GCG, CGC, GGC) cluster separately from AT-rich 3-mers (AAT, TAT, ATA).
• Reverse complement pairs (e.g., ATG and CAT) have related but not identical embeddings.

The embedding captures both chemical properties (GC content) and functional properties (coding vs. regulatory) without being explicitly told about either.

Worked example: cosine similarity computation. Suppose after training, the embeddings (simplified to \(d = 4\)) are: ATG \(= [0.8, 0.3, -0.1, 0.5]\) and GTG \(= [0.7, 0.4, -0.2, 0.6]\). The cosine similarity is: \begin {align} \mathbf {u} \cdot \mathbf {v} &= 0.8 \times 0.7 + 0.3 \times 0.4 + (-0.1)(-0.2) + 0.5 \times 0.6 = 0.56 + 0.12 + 0.02 + 0.30 = 1.00 \\ \|\mathbf {u}\| &= \sqrt {0.64 + 0.09 + 0.01 + 0.25} = \sqrt {0.99} \approx 0.995 \\ \|\mathbf {v}\| &= \sqrt {0.49 + 0.16 + 0.04 + 0.36} = \sqrt {1.05} \approx 1.025 \\ \text {sim}(\mathbf {u}, \mathbf {v}) &= \frac {1.00}{0.995 \times 1.025} \approx 0.980. \end {align}

The high similarity (\(0.98\)) reflects that ATG and GTG are functionally related—both are alternative start codons in some organisms.

4.5.2 The Geometry of Meaning

Good embeddings exhibit meaningful geometric structure. Classic examples include Word2Vec’s vector arithmetic: \(\text {king} - \text {man} + \text {woman} \approx \text {queen}\).

For biological sequences, protein language models learn embeddings where proteins with similar function cluster together, even without explicit functional labels [Lin et al., 2023]. DNA language models learn embeddings where regulatory elements cluster by function, coding regions by gene family, and promoters by expression level [Ji et al., 2021b].

Measuring embedding quality. Common metrics include:

• Cosine similarity: \(\text {sim}(\mathbf {u}, \mathbf {v}) = \frac {\mathbf {u} \cdot \mathbf {v}}{\|\mathbf {u}\| \|\mathbf {v}\|}\). Values near 1 indicate similar vectors.
• k-nearest neighbors accuracy: What fraction of an embedding’s \(k\) nearest neighbors share its class label?
• Linear probing: Train a linear classifier on frozen embeddings. High accuracy indicates the embeddings capture relevant structure. For example, if a linear probe on DNABERT embeddings can predict whether a sequence is a promoter with 85% accuracy, the embeddings have learned promoter-relevant features even though the model was never trained on promoter labels.

4.5.3 Positional Encoding

Embeddings alone do not capture position—“ATG at position 1” and “ATG at position 100” have identical embeddings. Positional encoding injects position information:

\begin {equation} \text {PE}(pos, 2i) = \sin \!\left (\frac {pos}{10000^{2i/d}}\right ), \quad \text {PE}(pos, 2i+1) = \cos \!\left (\frac {pos}{10000^{2i/d}}\right ). \end {equation}

These sinusoidal functions create unique position-dependent patterns. The frequency decreases with dimension index \(i\), so the first dimensions encode fine-grained position and later dimensions encode coarser position.

Why sinusoidal? The sinusoidal encoding has two useful properties. First, each position gets a unique encoding (no two positions have the same pattern across all dimensions). Second, the encoding supports relative position computation: for any fixed offset \(k\), \(\text {PE}(pos + k)\) can be expressed as a linear function of \(\text {PE}(pos)\), allowing the model to learn attention patterns that depend on relative position. An alternative is learned positional embeddings, which train a separate embedding vector for each position. Learned embeddings are more flexible but cannot extrapolate to positions longer than seen during training.

4.6 Sequence Models: From RNNs to State Spaces

Biological data is inherently sequential: DNA sequences, protein chains, gene expression time series. Sequence models are therefore central to biological AI.

4.6.1 Recurrent Neural Networks

RNNs process sequences by maintaining a hidden state \(\mathbf {h}_t\) that is updated at each time step: \begin {equation} \mathbf {h}_t = \phi (\mathbf {W}_h \mathbf {h}_{t-1} + \mathbf {W}_x \mathbf {x}_t + \mathbf {b}). \end {equation}

The hidden state acts as a “memory” that accumulates information from previous time steps. At each step, the RNN takes the current input \(\mathbf {x}_t\) and the previous hidden state \(\mathbf {h}_{t-1}\), combines them through a linear transformation, applies a nonlinearity, and produces a new hidden state \(\mathbf {h}_t\). This hidden state serves two purposes: it is the representation used for prediction at time \(t\), and it is the memory passed to time \(t+1\).

However, vanilla RNNs struggle with long sequences due to the vanishing gradient problem: gradients shrink exponentially as they propagate backward through many time steps.

Vanishing gradients: a concrete example. If the gradient at each time step is multiplied by a factor \(\alpha < 1\), then after \(T\) steps the gradient is \(\alpha ^T\). For \(\alpha = 0.9\) and \(T = 100\): \(0.9^{100} \approx 2.7 \times 10^{-5}\). The gradient has effectively vanished, making it impossible to learn long-range dependencies. For DNA sequences, which can be hundreds of thousands of bases long, this is catastrophic: an RNN cannot learn that a promoter 10,000 bases upstream affects a gene’s expression.

4.6.2 Long Short-Term Memory (LSTM)

The LSTM architecture [Hochreiter and Schmidhuber, 1997] addresses vanishing gradients by adding a cell state \(\mathbf {c}_t\) that flows through time with minimal transformation, gated by three learned gates:

\begin {align} \mathbf {f}_t &= \sigma (\mathbf {W}_f [\mathbf {h}_{t-1}, \mathbf {x}_t] + \mathbf {b}_f) & \text {(forget gate)} \\ \mathbf {i}_t &= \sigma (\mathbf {W}_i [\mathbf {h}_{t-1}, \mathbf {x}_t] + \mathbf {b}_i) & \text {(input gate)} \\ \tilde {\mathbf {c}}_t &= \tanh (\mathbf {W}_c [\mathbf {h}_{t-1}, \mathbf {x}_t] + \mathbf {b}_c) & \text {(candidate cell state)} \\ \mathbf {c}_t &= \mathbf {f}_t \odot \mathbf {c}_{t-1} + \mathbf {i}_t \odot \tilde {\mathbf {c}}_t & \text {(cell state update)} \\ \mathbf {o}_t &= \sigma (\mathbf {W}_o [\mathbf {h}_{t-1}, \mathbf {x}_t] + \mathbf {b}_o) & \text {(output gate)} \\ \mathbf {h}_t &= \mathbf {o}_t \odot \tanh (\mathbf {c}_t) & \text {(hidden state)} \end {align}

where \(\odot \) denotes element-wise multiplication and \([\cdot , \cdot ]\) denotes concatenation.

The key idea is the cell state \(\mathbf {c}_t\): information flows along the cell state with only multiplicative gating (the forget gate \(\mathbf {f}_t\)) and additive updates (the input gate \(\mathbf {i}_t \odot \tilde {\mathbf {c}}_t\)). If the forget gate is close to 1 and the input gate close to 0, information is preserved indefinitely. This creates a “gradient highway” that allows gradients to flow backward through time without shrinking.

4.6.3 State Space Models

An emerging alternative to attention-based sequence models is the structured state space model (SSM) [Gu et al., 2022, Gu and Dao, 2023]. SSMs process sequences through a linear dynamical system:

\begin {align} \mathbf {h}_t &= \mathbf {A} \mathbf {h}_{t-1} + \mathbf {B} \mathbf {x}_t, \label {eq:ssm-state} \\ \mathbf {y}_t &= \mathbf {C} \mathbf {h}_t + \mathbf {D} \mathbf {x}_t, \label {eq:ssm-output} \end {align}

where \(\mathbf {A}, \mathbf {B}, \mathbf {C}, \mathbf {D}\) are structured matrices. The key innovation of SSMs is that the recurrence in Equation ?? can be computed as a convolution during training (enabling parallelism) and as a recurrence during inference (enabling efficient autoregressive generation).

Why the dual computation matters. To see how the recurrence becomes a convolution, unroll the state equation: \begin {align} \mathbf {h}_0 &= \mathbf {B}\mathbf {x}_0, \\ \mathbf {h}_1 &= \mathbf {A}\mathbf {B}\mathbf {x}_0 + \mathbf {B}\mathbf {x}_1, \\ \mathbf {h}_2 &= \mathbf {A}^2\mathbf {B}\mathbf {x}_0 + \mathbf {A}\mathbf {B}\mathbf {x}_1 + \mathbf {B}\mathbf {x}_2. \end {align}

The output \(\mathbf {y}_t = \mathbf {C}\mathbf {h}_t\) depends on all past inputs through the kernel \(\bar {K}_t = \mathbf {C}\mathbf {A}^t\mathbf {B}\), yielding \(\mathbf {y} = \bar {K} * \mathbf {x}\)—a convolution that can be computed in \(O(T \log T)\) via FFT during training. During inference, only one step of the recurrence is needed per token, giving \(O(1)\) per-token cost.

Complexity comparison.

The Mamba architecture [Gu and Dao, 2023] makes the SSM matrices input-dependent, achieving selective state-space modeling: the model can choose which information to remember or forget based on the input, similar to LSTM gating but with \(O(T)\) complexity.

4.7 Evolutionary Algorithms

Evolutionary algorithms (EAs) are optimization methods inspired by biological evolution [Holland, 1975, De Jong, 2006]. They maintain a population of candidate solutions and iteratively apply selection, mutation, and recombination to improve population fitness.

4.7.1 The Genetic Algorithm: Step by Step

A standard genetic algorithm (GA) proceeds through the following steps:

1.

Initialize: Create a random population of \(N\) candidate solutions (“individuals”), each encoded as a genome (e.g., a bit string, a real-valued vector, or a structured object).

2.

Evaluate: Compute a fitness score \(f(\mathbf {x}_i)\) for each individual \(\mathbf {x}_i\).

3.

Select parents: Choose individuals for reproduction, with probability proportional to fitness. Common selection methods:

• Tournament selection: Randomly pick \(k\) individuals; the fittest wins.
• Roulette wheel: Select with probability \(p_i = f(\mathbf {x}_i) / \sum _j f(\mathbf {x}_j)\).
• Rank selection: Sort by fitness; select by rank rather than raw fitness value.

4.

Crossover (recombination): Combine two parents to produce offspring:

• One-point crossover: Choose a random split point; child gets first part from parent A and second part from parent B.
• Uniform crossover: For each gene, randomly choose from parent A or B.

5.

Mutate: Randomly modify offspring with small probability \(p_m\) (e.g., flip a bit, add Gaussian noise to a real-valued gene).

6.

Replace: Form the next generation. Options include generational replacement (all individuals replaced) or steady-state (only a few replaced per iteration).

7.

Repeat steps 2–6 until convergence or a computational budget is exhausted.

Worked example: evolving a 6-bit binary string. Suppose our fitness function counts the number of 1-bits (the “OneMax” problem). We want to find the string \(111111\).

1.

Generation 0 (initialize): Random population of \(N=4\):

Individual	Genome	Fitness
\(\mathbf {x}_1\)	100110	3
\(\mathbf {x}_2\)	011001	3
\(\mathbf {x}_3\)	110100	3
\(\mathbf {x}_4\)	001011	3

2.

Selection (tournament, \(k=2\)):

• Pair 1: \(\mathbf {x}_1\) (3) vs \(\mathbf {x}_3\) (3) \(\to \) \(\mathbf {x}_3\) wins (tie-break).
• Pair 2: \(\mathbf {x}_2\) (3) vs \(\mathbf {x}_4\) (3) \(\to \) \(\mathbf {x}_2\) wins.

3.

Crossover (one-point at position 3):

• Parent \(\mathbf {x}_3 = \mathbf {110}|100\), Parent \(\mathbf {x}_2 = \mathbf {011}|001\)
• Child 1: \(110|001 = 110001\) (fitness 3)
• Child 2: \(011|100 = 011100\) (fitness 3)

4.

Mutation (\(p_m = 0.1\) per bit): Suppose bit 6 of child 1 flips: \(110001 \to 110011\) (fitness 4). Child 2 has no mutations.

5.

Generation 1: Replace worst two individuals. New population: \(\{110100, 011001, 110011, 011100\}\). Best fitness improved from 3 to 4.

After several more generations, mutation and crossover create individuals with progressively more 1-bits until the optimum \(111111\) is found. Note that no gradient was computed—the search is guided entirely by fitness evaluation and random variation.

Worked example: evolving a DNA promoter score. Suppose we want to find a 10-base DNA sequence that maximizes a promoter strength predictor \(f : \{A,C,G,T\}^{10} \to [0,1]\). We initialize a population of \(N = 50\) random sequences, evaluate each with the predictor, apply tournament selection (\(k=3\)), one-point crossover, and point mutations (each base mutated with probability \(p_m = 0.05\)). After 100 generations, the population converges toward high-scoring promoter sequences—without ever computing a gradient.

4.7.2 Quality-Diversity Algorithms

Standard optimization seeks a single best solution. Quality-diversity (QD) algorithms seek a diverse collection of high-quality solutions that cover the space of possible behaviors [Mouret and Clune, 2015].

Intuition for MAP-Elites. Consider designing DNA aptamers (short DNA sequences that bind to specific targets). The behavioral descriptor might be a 2D space: (GC content, predicted secondary structure stability). MAP-Elites would fill a grid where each cell contains the best-binding aptamer with that particular GC content and stability combination. This gives the designer a menu of high-quality options spanning different properties—far more useful than a single “best” aptamer.

Novelty search [Lehman and Stanley, 2011] abandons explicit fitness objectives entirely, rewarding solutions that are behaviorally different from previously encountered ones. This can discover solutions that objective-driven search misses due to deceptive fitness landscapes.

4.8 Multi-Objective Optimization

Real-world learning problems rarely have a single objective. DOGMA’s training loop simultaneously optimizes accuracy, schema compliance, distillation agreement, efficiency, and latency.

4.8.1 Pareto Optimality

Worked example. Consider three genomes evaluated on accuracy (\(f_1\)) and efficiency (\(f_2\)):

\(\mathcal {B}_A\) and \(\mathcal {B}_B\) are both Pareto optimal—neither dominates the other (\(A\) is better on accuracy, \(B\) on efficiency). \(\mathcal {B}_C\) is dominated by \(\mathcal {B}_B\) (worse on both objectives), so \(C\) is not Pareto optimal. Note that \(\mathcal {B}_A\) does not dominate \(\mathcal {B}_C\) either: although \(A\) is better on accuracy (\(0.95 > 0.80\)) and efficiency (\(0.60 > 0.55\)), it does dominate \(C\) since it is better on both objectives. The Pareto front is therefore \(\{\mathcal {B}_A, \mathcal {B}_B\}\).

4.8.2 Scalarization Methods

The simplest approach to multi-objective optimization is scalarization: combine multiple objectives into a single scalar using weights: \begin {equation} F_{\text {scalar}}(\mathbf {x}) = \sum _{i=1}^m w_i \cdot f_i(\mathbf {x}), \quad \sum _i w_i = 1. \end {equation}

The choice of weights \(w_i\) determines which point on the Pareto front is found. Different weight vectors trace out different points on the front: \(w = [1.0, 0.0]\) finds the solution with maximum accuracy (ignoring efficiency), \(w = [0.0, 1.0]\) maximizes efficiency, and \(w = [0.5, 0.5]\) seeks a balanced compromise.

DOGMA’s multi-objective scoring uses a weighted combination where the weights are dynamically adjusted by the Tera regime state (Chapter 9), creating an adaptive scalarization that shifts emphasis based on the current evolutionary pressure landscape.

\begin {equation} w_i(t) = \text {softmax}\bigl (\mathbf {W}_{\text {tera}} \cdot \mathbf {r}(t) + \mathbf {b}\bigr )_i \label {eq:ml-tera-weights} \end {equation}

where \(\mathbf {r}(t)\) is the Tera regime state vector at time \(t\). When the system is stagnating on accuracy, Tera increases the accuracy weight; when efficiency is lagging, it shifts emphasis to efficiency. This dynamic reweighting is analogous to how biological organisms shift resource allocation under different environmental pressures.

4.9 Bringing It Together: The DOGMA Learning Stack

The ML concepts introduced in this chapter form the foundation of DOGMA’s learning stack. The following table maps each ML concept to its role in DOGMA:

This mapping illustrates DOGMA’s synthetic approach: rather than choosing between gradient-based and evolutionary optimization, between local and global computation, or between structured and unstructured representations, DOGMA combines the best of each paradigm within a biologically grounded framework.

4.10 Exercises

4.1.

[Fundamentals] Derive the gradient of the cross-entropy loss \(\mathcal {L} = -\sum _i y_i \log \hat {y}_i\) with respect to the logits \(\mathbf {z}\), where \(\hat {y}_i = \text {softmax}(\mathbf {z})_i\). Show that the result simplifies to \(\hat {\mathbf {y}} - \mathbf {y}\). Hint: Start by computing \(\partial \hat {y}_i / \partial z_j\) for the cases \(i = j\) and \(i \neq j\) separately, using the quotient rule on the softmax definition.

4.2.

[Coding] Implement a single-layer neural network from scratch (NumPy only) that classifies DNA 3-mers into categories based on GC content (low: 0–1 G/C, medium: 2, high: 3 G/C). Train with SGD and plot the learning curve. Report final accuracy.

4.3.

[Analysis] Compare the LSTM cell state \(\mathbf {c}_t\) with a biological gene’s expression level \(x_i(t)\) governed by Equation 3.2 from Chapter 3. Identify the biological analogues of the forget gate, input gate, and output gate. Are there regulatory mechanisms that have no LSTM analogue?

4.4.

[Backpropagation] Perform the complete forward and backward pass for a 2-layer MLP with weights \(W^{(1)} = \begin {bmatrix} 0.3 & -0.2 \\ 0.4 & 0.1 \end {bmatrix}\), \(\mathbf {b}^{(1)} = [0, 0]\), \(\mathbf {w}^{(2)} = [0.5, -0.3]\), \(b^{(2)} = 0\), input \(\mathbf {x} = [1, 2]\), and true label \(y = 1\). Use ReLU for the hidden layer and sigmoid for the output. Compute all gradients and the updated weights after one step of SGD with \(\eta = 0.1\).

4.5.

[Convolutions] Design a set of 1D convolutional filters (specify the weights) that detect the following DNA motifs: (a) the start codon ATG, (b) any purine-pyrimidine alternating pattern, (c) a CpG dinucleotide. Assume 4-channel one-hot encoded input (A, C, G, T).

4.6.

[Theory] Prove that the Pareto front of a bi-objective optimization problem with convex feasible set is a connected curve. Give a counterexample for non-convex problems.

4.7.

[Design] Design a quality-diversity algorithm for evolving DNA primer sequences. Define the behavioral descriptor space, the quality metric, and the mutation operators. Explain how your design prevents the population from converging prematurely.

4.8.

[Research] Read Hansen and Ostermeier [2001] on CMA-ES. How does CMA-ES’s covariance adaptation relate to DOGMA’s Tera regime state adaptation? Both systems learn about the local landscape to guide search—compare their mechanisms.

4.9.

[Activation Functions] Implement sigmoid, ReLU, GELU, and Swish in Python. For each, plot the function and its derivative on \([-5, 5]\). Which derivative has the largest maximum value? Which never saturates?

4.10.

[Embeddings] Train Word2Vec-style embeddings on DNA 4-mers using the skip-gram objective on a small genome (e.g., E. coli). Visualize the embeddings using t-SNE. Do reverse complement pairs cluster together?

4.11.

[Gradient Descent] Implement gradient descent on \(f(\theta _1, \theta _2) = (\theta _1 - 2)^2 + 10(\theta _2 - \theta _1^2)^2\) (a scaled Rosenbrock function). Starting from \(\theta _0 = (-1, 1)\), compare the trajectories of plain SGD (\(\eta = 0.001\)), SGD with momentum (\(\beta = 0.9\)), and Adam. Plot the trajectories overlaid on a contour plot of \(f\). Which optimizer converges fastest? Why?

4.12.

[Evolutionary Algorithms] Implement a genetic algorithm to find the DNA 8-mer with maximum GC content (trivial optimal: GCGCGCGC). Use tournament selection (\(k=2\)), uniform crossover, and point mutation (\(p_m = 0.1\) per base). Track the population’s average and best fitness over 50 generations. How does increasing the population size from \(N=10\) to \(N=100\) affect convergence speed?

4.11 Key Takeaways

Suggested Reading

• Goodfellow et al. [2016]: Comprehensive deep learning reference—the standard textbook.
• Hochreiter and Schmidhuber [1997]: LSTM—foundational for sequence modeling and gating mechanisms.
• Kingma and Ba [2015]: Adam optimizer—the workhorse of modern deep learning.
• Gu and Dao [2023]: Mamba—selective state spaces as an attention alternative.
• Srivastava et al. [2014]: Dropout regularization—simple but effective.
• Holland [1975]: Genetic algorithms—the original EA framework.
• Mouret and Clune [2015]: MAP-Elites—quality-diversity algorithms.
• Lehman and Stanley [2011]: Novelty search—objectives are not always necessary.
• Ji et al. [2021b]: DNABERT—applying BERT to genomic sequences.
• Zhou and Troyanskaya [2015]: DeepSEA—CNNs for predicting chromatin effects from DNA sequence.