Foundation Models on Genomic Data

Chapter 12 showed how to evolve prompt genomes through gradient-free optimization. But the evolutionary loop assumes an underlying language model capable of processing genomic data. Where does that model come from? How should it tokenize DNA, learn codon structure, and integrate biological sequences with natural language?

This chapter addresses these questions by building foundation models for genomic data from first principles. We begin with the deceptively simple question: what is genomic data? We explain the FASTA format step by step, then tackle the tokenization problem—comparing BPE, \(k\)-mer, and codon-based approaches before showing how DOGMA’s CodonForge tokenizer unifies biological and textual data in a single vocabulary. We introduce curriculum learning as a training strategy, detail the pre-training objectives (masked language modeling and next-token prediction), and explain how the double-helix structure of DNA informs helix-aware training. We cover multi-scale TetraMemory, the full training data pipeline from GenBank to batching, and evaluation benchmarks for genomic foundation models. Chapter 14 then separates the implemented small baseline from the larger design studies in this chapter.

13.1 What Is Genomic Data?

Before building models, we must understand the data. Genomic data is the digital representation of the molecules that encode hereditary information in all living organisms. At its core, DNA (deoxyribonucleic acid) is a polymer of four nucleotide bases: adenine (A), thymine (T), cytosine (C), and guanine (G). These four letters constitute the “alphabet” of life.

13.1.1 From Molecules to Sequences

In a living cell, DNA exists as a double-stranded helix. The two strands are complementary: A always pairs with T, and C always pairs with G. If one strand reads ATGCGA, the opposite strand reads TACGCT (in the reverse direction). This complementarity means that storing one strand is sufficient to reconstruct both.

When biologists sequence DNA—determining the order of bases in a sample—the output is a text string over the alphabet \(\Sigma _{\mathrm {DNA}} = \{\texttt {A}, \texttt {T}, \texttt {C}, \texttt {G}\}\). A single human genome contains approximately 3.2 billion such characters. A bacterial genome might contain 1–10 million characters. A viral genome can be as small as a few thousand characters. All of these are stored, shared, and processed in a standard text format called FASTA.

13.1.2 The FASTA Format: Step by Step

FASTA is the universal standard for representing biological sequences. Understanding it is essential for anyone building genomic models. A FASTA file has a simple structure:

1.

Header line. Begins with > and contains metadata: an identifier, organism name, chromosome, and other annotations. This is free-form text.

2.

Sequence lines. One or more lines of nucleotide characters (A, T, C, G), typically wrapped at 60 or 80 characters per line. No spaces, no numbers—just the raw sequence.

3.

Multiple records. A file can contain many sequences, each starting with its own > header.

>gene_001 | Homo sapiens | insulin gene | chromosome 11
ATGGCCCTGTGGATGCGCCTCCTGCCCCTGCTGGCGCTGCTGGCCCTCTG
GGGACCTGACCCAGCCGCAGCCTTTGTGAACCAACACCTGTGCGGCTCACA
CCTGGTGGAAGCTCTCTACCTAGTGTGCGGGGAACGAGGCTTCTTCTACAC
>gene_002 | Escherichia coli | beta-galactosidase | lacZ
ATGACCATGATTACGCCAAGCTATTTAGGTGACACTATAGAATACTCAAGC
TATGCATCCAACGCGTTGGGAGCTCTCCCATATGGTCGACCTGCAGGCGGC

13.1.3 Beyond DNA: RNA and Protein Sequences

FASTA also stores RNA sequences (using U instead of T) and protein sequences (using the 20-letter amino acid alphabet). A genomic foundation model must handle all three:

13.1.4 Ambiguity Codes and Real-World Messiness

Real genomic data is not always clean. The IUPAC nucleotide code includes ambiguity characters: N means “any base,” R means “A or G” (purine), Y means “C or T” (pyrimidine), and so on. Sequencing errors, gaps, and low-quality regions are common. A robust tokenizer must handle these gracefully.

13.2 Tokenizing Genomic Sequences

Tokenization—the mapping from raw input to discrete units consumed by a model—is often treated as a solved preprocessing step. For natural language, subword tokenizers such as BPE [Sennrich et al., 2016] and SentencePiece [Kudo and Richardson, 2018] perform well. For genomic data, the situation is far more nuanced.

13.2.1 Why Subword Tokenization Fails for Genomic Data

Subword tokenizers learn merge rules from corpus statistics: frequent character pairs are merged into tokens, and the process repeats until a target vocabulary size is reached. This works for natural language because word frequencies follow Zipf’s law—a small vocabulary of common subwords covers most text.

Genomic sequences violate these assumptions in four fundamental ways:

1.

Uniform character distribution. DNA consists of four nucleotides (A, T, C, G) with roughly equal frequency in many organisms. BPE merges are driven by frequency imbalances; when all bigrams occur at similar rates, the merge order becomes arbitrary and biologically meaningless.

2.

Reading frame sensitivity. The biological meaning of a DNA sequence depends on its reading frame—the codon boundaries that partition nucleotides into triplets. A BPE tokenizer that merges AT into a single token will split the codon ATG (start codon) differently depending on context, destroying frame information.

3.

No whitespace delimiters. Natural language has spaces and punctuation that provide natural token boundaries. Genomic sequences are undelimited streams of characters, offering no structural cues to a frequency-based tokenizer.

4.

Reverse complement ambiguity. The same biological information can be represented on either DNA strand. A subword vocabulary learned from one strand orientation will be suboptimal for the reverse complement, creating asymmetric representations of equivalent biological content.

13.2.2 Three Approaches to Genomic Tokenization

We now compare three tokenization strategies for genomic data: BPE applied to DNA, \(k\)-mer tokenization, and codon-based tokenization.

Byte Pair Encoding (BPE) on DNA

BPE can be applied to DNA by treating the sequence as text and learning merge rules. The process is:

1.

Start with character-level tokens: each of A, T, C, G is a token.

2.

Count all adjacent token pairs in the corpus.

3.

Merge the most frequent pair into a new token.

4.

Repeat until the desired vocabulary size is reached.

On DNA, BPE typically learns to merge common dinucleotides first (AT, GC, etc.), then trinucleotides, and so on. The resulting vocabulary has variable-length tokens with no guaranteed alignment to codon boundaries.

\(k\)-mer Tokenization

\(k\)-mer tokenization splits a DNA sequence into overlapping or non-overlapping windows of length \(k\). For example, with \(k = 6\), the sequence ATGCGATCGA produces the 6-mers: ATGCGA, TGCGAT, GCGATC, CGATCG, GATCGA. The vocabulary size is \(4^k\) (4,096 for \(k = 6\)). This approach was popularized by DNABERT [Ji et al., 2021a].

Advantages: Every \(k\)-mer captures local context of length \(k\). The vocabulary is deterministic—no training required. Disadvantages: Overlapping \(k\)-mers create redundancy. The vocabulary grows exponentially with \(k\) (\(4^k\) tokens). Most critically, \(k\)-mer boundaries are arbitrary and do not respect codon structure unless \(k\) is a multiple of 3.

Codon-Based Tokenization

Codon-based tokenization groups nucleotides into non-overlapping triplets aligned to the biological reading frame:

Advantages: Perfectly aligned with biology. Small vocabulary (64 core tokens). Every token carries direct biological meaning (each codon encodes an amino acid). Disadvantages: Requires knowing the reading frame, which is not always available. Cannot handle non-coding DNA, intergenic regions, or introns, where no reading frame exists.

13.2.3 Comparison Table: Tokenization Methods

The following table summarizes the trade-offs:

13.3 DOGMA’s Unified Tokenization: CodonForge

DOGMA’s solution to the tokenization dilemma is CodonForge (Chapter 11), a unified tokenizer that handles both text and DNA within a single vocabulary. This section explains how CodonForge works at the level relevant to foundation model training.

13.3.1 The CodonForge Vocabulary

CodonForge partitions a 328-token vocabulary into four regions:

Note that the 64 codon tokens overlap with specific byte trigrams (e.g., the codon token ATG and the byte sequence 65-84-71 represent the same characters). The key difference is that a codon token is a single token occupying one position in the sequence, while the byte representation uses three positions. CodonForge dynamically chooses which representation to use based on the input modality.

13.3.2 Adaptive Tokenization Logic

When CodonForge processes a FASTA file, it applies different tokenization strategies to different regions:

1.

Header lines (after >): Tokenized as bytes. Natural language text is best served by character-level representation, and the byte vocabulary covers all ASCII characters.

2.

Coding sequences (with known reading frame): Tokenized as codons. Each triplet becomes a single codon token, preceded by <DNA> and followed by </DNA>.

3.

Non-coding sequences (no known reading frame): Tokenized as bytes. Individual nucleotides become byte tokens within <DNA>...</DNA> markers.

4.

Protein sequences: Each amino acid letter becomes a byte token within <PROT>...</PROT> markers.

13.3.3 Why Unified Tokenization Matters

A unified vocabulary allows a single model to process mixed-modality inputs without mode switching. Consider a training example that contains a gene description in English, followed by its DNA sequence, followed by its protein product:

<TEXT>The insulin gene encodes a peptide hormone</TEXT>
<DNA>ATG GCC CTG TGG ATG CGC CTC CTG</DNA>
<PROT>MALWMRLL</PROT>

With CodonForge, this entire example is a single token sequence. The model learns cross-modal relationships—that the text “insulin” co-occurs with specific DNA and protein patterns—through the same next-token prediction objective used for all training.

13.4 Byte-Level Tokenization as Universal Fallback

While CodonForge provides the primary tokenization for DOGMA, byte-level tokenization serves as the universal fallback and the foundation upon which CodonForge is built.

The byte-level approach has three key advantages for genomic foundation models. First, it is modality-agnostic: the same tokenizer handles FASTA headers (English text), nucleotide sequences, amino acid sequences, and metadata without switching modes. Second, it preserves positional granularity: every nucleotide occupies exactly one position, so positional encodings align with biological positions. Third, it is deterministic: there are no merge rules to learn, no vocabulary to tune, and no edge cases where the same sequence tokenizes differently in different contexts.

13.5 The Bigram Baseline

Before building complex transformer models, we establish a minimal baseline: a bigram language model over genomic sequences, implemented with zero external dependencies. This baseline is not a throwaway exercise—it reveals genuine biological structure and provides a critical diagnostic tool for evaluating more complex models.

13.5.1 Character-Level Statistics of Genomic Sequences

A bigram model estimates the probability of each character given only its immediate predecessor:

\begin {equation} P(x_t \mid x_{<t}) \approx P(x_t \mid x_{t-1}) = \frac {C(x_{t-1}, x_t)}{\sum _{c} C(x_{t-1}, c)}, \label {eq:bigram} \end {equation}

where \(C(a, b)\) counts the number of times character \(b\) follows character \(a\) in the training corpus. For a pure DNA corpus over alphabet \(\{\)A, T, C, G\(\}\), this yields a \(4 \times 4\) transition matrix.

13.5.2 What Bigrams Reveal About Biology

Despite their simplicity, bigram statistics expose genuine biological structure. The dinucleotide frequency matrix is non-uniform in real genomes: CpG dinucleotides are suppressed in vertebrate genomes due to methylation-driven deamination, while AA/TT dinucleotides are enriched in nucleosome positioning sequences. A bigram model trained on human genomic DNA will learn these biases automatically, achieving lower perplexity than a uniform model.

13.6 Pre-Training Objectives for Genomic Foundation Models

A foundation model learns from data through its pre-training objective—the loss function that defines what the model should predict. For genomic data, two objectives dominate: masked language modeling (MLM) and next-token prediction (NTP). Each has distinct advantages.

13.6.1 Masked Language Modeling on DNA

Masked language modeling, popularized by BERT [Devlin et al., 2019], randomly masks a fraction of input tokens and trains the model to predict them from context. Applied to DNA:

Why MLM works for DNA: Nucleotides are constrained by their neighbors. A masked G in a coding region can often be predicted because the codon it belongs to is constrained by the amino acid sequence. A masked position in a promoter region can be predicted because promoter motifs (e.g., TATA box) have conserved sequences. MLM forces the model to learn these contextual dependencies from both directions (bidirectional context).

Original:  ATG | CGA | TCG | ATG | TAA
Masked:    ATG | C[M]A | TCG | A[M]G | TAA

13.6.2 Next-Token Prediction on Codons

Next-token prediction (NTP), the objective used by GPT-family models, predicts each token from its left context only:

\begin {equation} \mathcal {L}_{\mathrm {NTP}} = -\frac {1}{T} \sum _{t=1}^{T} \log P_\theta (x_t \mid x_{<t}). \label {eq:ntp-codon} \end {equation}

When applied at the codon level (using CodonForge’s codon tokens), this becomes next-codon prediction: given a sequence of codons, predict the next codon. This objective is particularly natural for coding sequences because the genetic code imposes strong constraints on codon transitions.

13.6.3 Combining MLM and NTP

DOGMA uses both objectives during pre-training, applied to different portions of the training data:

\begin {equation} \boxed { \mathcal {L}_{\text {pretrain}} = \lambda _{\mathrm {NTP}} \cdot \mathcal {L}_{\mathrm {NTP}} + \lambda _{\mathrm {MLM}} \cdot \mathcal {L}_{\mathrm {MLM}} + \lambda _{\mathrm {align}} \cdot \mathcal {L}_{\mathrm {align}} } \label {eq:combined-pretrain} \end {equation}

where \(\lambda _{\mathrm {NTP}} = 1.0\), \(\lambda _{\mathrm {MLM}} = 0.5\), and \(\lambda _{\mathrm {align}} = 0.3\) are mixing coefficients. The alignment loss \(\mathcal {L}_{\mathrm {align}}\) pulls representations of corresponding DNA and protein sequences together in embedding space (see Section 13.12).

13.7 Curriculum Learning for Genomic Data

Curriculum learning presents training examples in order of increasing complexity, allowing the model to build foundational representations before encountering difficult patterns [Bengio et al., 2009]. For genomic data, this strategy is particularly important because the complexity of biological sequences spans many orders of magnitude.

13.7.1 What Is Curriculum Learning?

In standard training, examples are sampled randomly from the dataset. In curriculum learning, examples are presented in a structured order: easy examples first, hard examples later. The intuition is the same as in education: students learn arithmetic before calculus, and letters before essays.

For genomic data, “easy” and “hard” have precise definitions:

• Easy: Short sequences, simple composition (uniform GC content), single-gene coding regions, well-characterized organisms.
• Hard: Long sequences, complex composition (repeat-rich regions, GC-biased genomes), multi-gene regions with overlapping reading frames, metagenomic samples from diverse organisms.

13.7.2 Why Curriculum Learning Helps

Training on random genomic data from the start creates several problems:

1.

Gradient signal confusion. Early in training, the model cannot distinguish coding from non-coding DNA. Random sampling mixes both, providing inconsistent gradient signals.

2.

Wasted capacity on noise. Large genomic datasets contain repetitive elements, sequencing artifacts, and low-complexity regions. Training on these first wastes model updates.

3.

Failure to learn hierarchical structure. Genomic organization is hierarchical: nucleotides form codons, codons form genes, genes form operons, operons form chromosomes. Random sampling jumbles these levels.

Curriculum learning addresses all three problems by ensuring the model first masters the nucleotide alphabet, then codon patterns, then gene-level structure, and finally genome-level organization.

13.7.3 DOGMA’s Four-Stage Curriculum

13.7.4 Stage Transitions and Mixing

Transitions between stages are not sharp boundaries. DOGMA uses a mixing schedule where data from completed stages continues to appear (at reduced frequency) in later stages:

\begin {equation} P(\text {sample from Stage } s \text { at training step } t) = w_s(t) = \frac {\alpha _s \cdot \mathbb {1}[t \geq t_s^{\text {start}}]}{\sum _{s'} \alpha _{s'} \cdot \mathbb {1}[t \geq t_{s'}^{\text {start}}]}, \label {eq:curriculum-mixing} \end {equation}

where \(t_s^{\text {start}}\) is the step at which Stage \(s\) is introduced and \(\alpha _s\) are stage-specific weights that increase with stage number. This ensures that the model never “forgets” earlier patterns while focusing on increasingly complex data.

13.8 Training Data Pipeline: GenBank to Batching

Building a genomic foundation model requires a robust data pipeline that transforms raw database entries into tokenized, curriculum-ordered training batches. This section traces the full pipeline.

13.8.1 Data Sources

The primary data source is GenBank, the NIH genetic sequence database. GenBank contains over 200 million sequences from more than 500,000 organisms. DOGMA’s training pipeline also draws from:

• RefSeq: Curated reference sequences with high-quality annotations.
• UniProt: Protein sequences with functional annotations (for cross-modal training).
• PDB: Protein structures for 3D structural features.
• Ensembl: Genome assemblies with gene annotations, exon/intron boundaries, and regulatory element predictions.

13.8.2 Preprocessing Steps

Raw GenBank entries undergo several preprocessing steps:

1.

Format conversion. GenBank flat files are converted to FASTA format, preserving key annotations in the header.

2.

Quality filtering. Sequences with more than 10% ambiguous characters (N) are removed. Sequences shorter than 100 nucleotides are discarded.

3.

Deduplication. Exact duplicates and near-duplicates (>95% sequence identity) are removed using MinHash-based clustering. This prevents the model from memorizing specific sequences.

4.

Taxonomy annotation. Each sequence is tagged with its taxonomic lineage (kingdom, phylum, class, order, family, genus, species) for curriculum stratification.

5.

Coding region annotation. Open reading frames (ORFs) are identified using existing GenBank annotations or ab initio gene prediction, and reading frames are marked for codon-aware tokenization.

13.8.3 Structured Rendering

After preprocessing, sequences are rendered into structured training formats:

13.8.4 Curriculum Assignment and Batching

Each preprocessed sequence is assigned to a curriculum stage based on its complexity:

• Stage 1: Synthetically generated. Parameters sampled from organism-specific GC content distributions.
• Stage 2: Single CDS (coding sequence) entries from RefSeq with “complete CDS” annotation.
• Stage 3: Genomic contigs containing 2–10 annotated genes with intergenic regions.
• Stage 4: Chromosomal segments of 50kb–500kb containing complex gene clusters.

Batches are constructed with curriculum-aware sampling (Equation 13.8), and each batch contains sequences from a single stage to avoid length-padding waste.

13.9 TinyGPT: Minimal Transformer for Genomic Sequences

With the data pipeline and objectives established, we now construct TinyGPT—a minimal decoder-only transformer adapted for byte-level genomic sequence modeling. TinyGPT is not a production model; it is a pedagogical tool for understanding how transformer architecture interacts with genomic data.

13.9.1 Decoder-Only Architecture Adapted for DNA

TinyGPT follows the standard GPT architecture [Radford et al., 2019] with modifications for genomic data:

The design is deliberately minimal: 1.2 million parameters, trainable on a single GPU in minutes. The goal is not state-of-the-art performance but pedagogical clarity—a model small enough to inspect, debug, and understand completely.

13.9.2 Byte-Level Embedding with Positional Encoding

The input embedding combines a learned byte embedding with sinusoidal positional encoding:

\begin {equation} \mathbf {h}_0^{(t)} = \mathbf {E}[x_t] + \mathbf {PE}(t), \label {eq:tinygpt-embed} \end {equation}

where \(\mathbf {E} \in \mathbb {R}^{256 \times d}\) is the byte embedding matrix and \(\mathbf {PE}(t) \in \mathbb {R}^d\) is the sinusoidal positional encoding at position \(t\). For genomic sequences, the positional encoding carries biological significance: position within a codon (modulo 3), distance from sequence start, and relative position within a gene all affect nucleotide identity.

13.9.3 Training on Mixed Text/Genome Corpora

TinyGPT is trained on a mixed corpus containing both natural language (FASTA headers, gene annotations, biological descriptions) and raw nucleotide sequences. The training objective is standard next-byte prediction:

\begin {equation} \mathcal {L}_{\text {TinyGPT}} = -\frac {1}{T} \sum _{t=1}^{T} \log P_\theta (x_t \mid x_{<t}). \label {eq:tinygpt-loss} \end {equation}

13.10 Helix-Aware Training

TinyGPT treats genomic sequences as flat byte streams, ignoring a fundamental property of DNA: it is a double-stranded helix. The two strands are complementary and antiparallel, meaning the same genetic information can be read in two directions. Helix-aware training exploits this structure.

13.10.1 Why the Double Strand Matters for Training

In a standard language model, a sentence is read left-to-right. DNA is different: the same region can be read on either strand, in opposite directions, and both readings carry biological information. Genes on the forward strand are transcribed left-to-right; genes on the reverse strand are transcribed right-to-left. A model that sees only the forward strand is blind to half the genome’s information.

This has concrete implications for training:

1.

Data augmentation. Every DNA training example can be augmented with its reverse complement, effectively doubling the training data without introducing new sequences.

2.

Representation symmetry. A well-trained model should produce similar representations for a sequence and its reverse complement, since they encode the same biological information.

3.

Regulatory signal detection. Many regulatory elements (transcription factor binding sites, enhancers) are functional on both strands. A helix-aware model can detect these regardless of strand orientation.

13.10.2 Dual-Stream Architecture

The dual-stream architecture processes each input through two parallel pathways (cf. the HelixHash embedding architecture in Section 8.7 of Chapter 8):

\begin {equation} \mathbf {h}_{\text {std}}^{(\ell )} = \text {TransformerBlock}_{\text {std}}^{(\ell )}\bigl (\mathbf {h}_{\text {std}}^{(\ell -1)}\bigr ), \quad \mathbf {h}_{\text {helix}}^{(\ell )} = \text {TransformerBlock}_{\text {helix}}^{(\ell )}\bigl (\mathbf {h}_{\text {helix}}^{(\ell -1)}\bigr ), \label {eq:dual-stream} \end {equation}

where the standard stream receives byte embeddings (Equation 13.9) and the helix stream receives HelixHash motif embeddings (Equation 8.26 from Chapter 8). At designated fusion layers \(\ell \in \mathcal {F}\), the streams exchange information through cross-attention:

\begin {equation} \mathbf {h}_{\text {fused}}^{(\ell )} = \mathbf {h}_{\text {std}}^{(\ell )} + \alpha _{\text {cross}} \cdot \text {CrossAttn}\bigl (\mathbf {h}_{\text {std}}^{(\ell )}, \mathbf {h}_{\text {helix}}^{(\ell )}\bigr ), \label {eq:helix-fusion} \end {equation}

with a learnable gating coefficient \(\alpha _{\text {cross}} \in [0, 1]\) that controls how much structural information flows into the standard stream.

13.10.3 Reverse Complement Consistency Loss

To ensure the model learns strand-symmetric representations, DOGMA adds a reverse complement consistency loss:

\begin {equation} \mathcal {L}_{\mathrm {RC}} = \frac {1}{N} \sum _{i=1}^{N} \bigl \| \mathbf {z}(x_i) - \mathbf {z}(\overline {x_i}) \bigr \|_2^2, \label {eq:rc-loss} \end {equation}

where \(x_i\) is a DNA sequence, \(\overline {x_i}\) is its reverse complement, and \(\mathbf {z}(\cdot )\) is the model’s pooled representation. This loss penalizes the model for producing different embeddings for complementary sequences, encouraging biologically consistent representations.

13.10.4 Motif-Level Feature Channels

The helix stream operates at the motif level rather than the byte level. Each motif (a width-\(k\) window of consecutive embeddings) is projected into a dense feature vector that encodes local structural patterns:

\begin {equation} \mathbf {m}_i = \text {ReLU}\bigl (W_m \cdot \text {concat}(\mathbf {h}_{i}, \mathbf {h}_{i+1}, \ldots , \mathbf {h}_{i+k-1}) + \mathbf {b}_m\bigr ), \end {equation}

where \(W_m \in \mathbb {R}^{d \times (k \cdot d)}\) is the motif projection matrix. The motif features capture patterns invisible to byte-level processing: codon boundaries, splice site signals, transcription factor binding motifs, and the dinucleotide patterns that determine DNA bendability.

13.10.5 Performance Comparison: Standard vs. Helix-Aware

13.11 Multi-Scale TetraMemory During Pretraining

Large genomic sequences exceed the context window of many practical models. A 500kb chromosomal segment contains 500,000 nucleotides—far beyond the 256-byte context of the measured DOGMA baseline. Multi-scale TetraMemory is a proposed mechanism for retaining selected information beyond the immediate window; its benefit must be established through length-extrapolation ablations.

13.11.1 What Is TetraMemory?

TetraMemory (introduced in Chapter 10) is a structured external memory organized into four hierarchical scales:

13.11.2 How TetraMemory Integrates with Training

During pretraining, TetraMemory operates as a sliding context extension. As the model processes a long sequence in chunks:

1.

Each chunk is processed normally through the transformer.

2.

At the end of each chunk, summary representations are written to the appropriate TetraMemory scales.

3.

At the start of the next chunk, the model reads from TetraMemory to initialize its context with information from previous chunks.

4.

The reading is done via cross-attention between the current chunk’s representations and the TetraMemory contents.

\begin {equation} \mathbf {h}_{\text {ctx}}^{(t)} = \mathbf {h}^{(t)} + \sum _{s=1}^{4} \gamma _s \cdot \text {CrossAttn}\bigl (\mathbf {h}^{(t)}, \tau _s\bigr ), \label {eq:tetramem-read} \end {equation}

where \(\gamma _s\) are learned scale-specific attention weights and \(\tau _s\) denotes the contents of memory scale \(s\).

13.11.3 Multi-Scale Attention Patterns

Different TetraMemory scales capture different types of long-range dependency:

13.12 Cross-Modal Training

A genomic foundation model that processes only DNA sequences misses the rich web of connections between biological modalities. Cross-modal training integrates text, DNA, protein sequences, and 3D structural data into a unified representation space.

13.12.1 The Four Modalities

The cross-modal training framework operates over four biological modalities:

1.

Natural language text. Gene descriptions, functional annotations, literature abstracts, and FASTA headers. Provides semantic context for biological entities.

2.

DNA sequences. Nucleotide sequences at the genomic, transcript, and coding levels. The primary modality of genomic foundation models.

3.

Protein sequences. Amino acid sequences derived from DNA through the genetic code. Proteins are the functional products of genes; their sequences constrain and predict DNA sequences.

4.

3D structural data. Protein structures and chromatin conformations, typically represented as distance matrices or coordinate sets. Structure determines function and provides geometric constraints not visible in sequence alone.

13.12.2 Cross-Modal Training Objectives

The cross-modal training loss combines next-token prediction with modality alignment objectives:

\begin {equation} \boxed { \mathcal {L}_{\text {cross}} = \mathcal {L}_{\text {NTP}} + \lambda _{\text {align}} \mathcal {L}_{\text {align}} + \lambda _{\text {trans}} \mathcal {L}_{\text {trans}} } \label {eq:cross-modal-loss} \end {equation}

where:

• \(\mathcal {L}_{\text {NTP}}\) is the standard next-token prediction loss across all modalities.
• \(\mathcal {L}_{\text {align}}\) is a contrastive alignment loss that pulls representations of corresponding entities (e.g., a gene’s DNA sequence and its protein product) closer in embedding space while pushing unrelated entities apart: \begin {equation} \mathcal {L}_{\text {align}} = -\log \frac {\exp (\text {sim}(\mathbf {z}_{\text {DNA}}, \mathbf {z}_{\text {prot}}) / \tau )}{\sum _{j} \exp (\text {sim}(\mathbf {z}_{\text {DNA}}, \mathbf {z}_{\text {prot}}^{(j)}) / \tau )}, \end {equation} where \(\tau \) is a temperature parameter and the sum is over negative samples.
• \(\mathcal {L}_{\text {trans}}\) is a translation loss that trains the model to generate one modality from another (e.g., predict the protein sequence given the DNA coding sequence), mirroring the biological process of translation.

13.13 Evaluation of Genomic Foundation Models

How do we know if a genomic foundation model is any good? This section presents the benchmarks, metrics, and comparison methodology used to evaluate DOGMA’s genomic models.

13.13.1 Intrinsic Metrics

Intrinsic metrics measure the model’s ability to model genomic sequences without reference to downstream tasks:

• Perplexity (PPL): The exponential of average negative log-likelihood. Lower is better. A uniform four-base predictor has PPL 4.0, but comparisons require identical splits and tokenization.
• Bits per nucleotide (BPN): Cross-entropy in bits, related by \(\text {BPN}=\log _2(\text {PPL})\). A uniform four-base predictor uses 2.0 bits.
• Codon prediction accuracy: For coding sequences, the fraction of codons correctly predicted under a declared reading frame. Dataset bias and codon usage make \(1/64\) an incomplete practical baseline.

13.13.2 Downstream Benchmarks

Downstream benchmarks evaluate the model’s learned representations on biologically meaningful tasks:

13.13.3 Required Comparison with Existing Models

The earlier edition placed numerical values in Table 13.5 without a reproducible checkpoint, dataset split, or evaluation artifact. Those values are withdrawn. The table now defines the comparisons a future DOGMA result must run:

13.14 Proposed Large-Scale DOGMA Configuration

This section preserves a proposed large-scale architecture for future experiments. It is not the model measured in Chapter 14. The parameter values and transfer schedule are hypotheses to test, not a completed training report.

13.14.1 From Foundation Model to DOGMA Pipeline

A future larger DOGMA model could use a pretrained genomic model as its core, then add a staged DOGMA pipeline:

1.

CodonForge Embedding (Stage 1): Uses the same CodonForge tokenizer described in Section 13.3, initialized with the pretrained byte and codon embeddings.

2.

Regulation Engine (Stage 2): Context-dependent activation of DNAComputeBlocks, trained via the curriculum learning described in Section 13.7.

3.

Synthesis Controller (Stage 3): Evolutionary modification of prompt genomes, using the evolutionary training loop of Chapter 12.

4.

DNAComputeBlocks (Stages 4–5): Non-transformer local and recurrent computation initialized from a compatible pretrained model.

5.

Realization Head (Stage 6): Output projection, initialized from the pretrained language model head.

13.14.2 Training Configuration Alignment

The proposed configuration would inherit several decisions from the genomic foundation model:

13.14.3 Transfer Learning Strategy

The design study proposes a staged transfer:

1.

Phase 1: Frozen transfer (first 10% of training). Freeze pretrained weights and train only new compatible modules.

2.

Phase 2: Gradual unfreezing (next 30%). Pretrained layers are unfrozen from top to bottom, one layer per 2,000 steps. The learning rate for unfrozen pretrained layers is \(10\times \) lower than for new modules.

3.

Phase 3: Full fine-tuning (remaining 60%). All parameters are trained jointly with a uniform (low) learning rate.

13.15 DOGMA for Multimodal Intelligence

The cross-modal framework of Section 13.12 motivates a design study spanning text, DNA, protein, and structural modalities. This section proposes how a Central Dogma pipeline might process those inputs. It does not report demonstrated cross-modal generation.

Throughout this section, dimensions and paths are hypothetical design parameters. Attention-based paths belong to a historical variant and are excluded from canonical DOGMA. TetraMesh and multimodal realization require separate implementation and evaluation evidence.

13.15.1 TetraMesh: Converting 3D Structures to Byte-Level Tokens

The central challenge of multimodal intelligence is representation: how do we feed a protein crystal structure, molecular geometry, or 3D point cloud into a byte-sequence model? The design proposes TetraMesh, a pipeline intended to convert 3D data into byte-level tokens. Whether it preserves task-relevant geometry is an open empirical question.

The conversion proceeds in three stages:

1.

Point cloud extraction. The raw input—whether a protein structure from the Protein Data Bank, a small molecule from QM9, or a 3D object mesh—is reduced to a set of 3D coordinates. For images, intensity-weighted importance sampling with depth from grayscale produces a point cloud in \(\mathbb {R}^3\). For proteins, alpha-carbon coordinates are extracted from PDB files. For molecules, atomic positions serve directly as point clouds.

2.

Double-helix mesh construction. The point cloud is fed into build_double_helix(), which constructs a DoubleHelixMesh: a pair of linked tetrahedral chains (sense and antisense) that tessellate the 3D space occupied by the input. Each tetrahedron’s four vertices are assigned DNA bases (A, C, G, T) based on geometric properties, producing a base sequence that encodes structural information.

3.

Base sequence serialization. The mesh is serialized as a compact DNA base string by emitting each tetrahedron’s four bases in order (sense chain first, then antisense chain). The result is a sequence like ACGTTAGC… that can be tokenized by the standard byte-level tokenizer.

The serialized mesh is wrapped in a structured training document:

[TETRA_MESH]
modality: protein_structure
source: pdb_1CRN
sense_chain: 28 tetrahedra
bases: A=12 C=8 G=10 T=6
chirality: +0.14
volume: 0.000342
base_sequence: ACGTTAGCACGT...
[/TETRA_MESH]
USER: Describe this protein_structure.
ASSISTANT: Protein 1CRN: Crambin, a small hydrophobic
  protein from Crambe hispanica seeds.

13.15.2 Image to DNA Encoding: A Detailed Pipeline

We now present the complete pipeline for converting a raster image into a DOGMA-compatible DNA base sequence, processing it through the Central Dogma, and decoding the output back to an image. This is the most detailed encoding pipeline we describe, and serves as the template for video and audio encoding in subsequent sections.

Step 1: Image to Point Cloud

Given an input image \(I \in \mathbb {R}^{H \times W \times 3}\) with RGB channels, we construct a 3D point cloud \(\mathcal {P} = \{(x_i, y_i, z_i)\}_{i=1}^{N}\) as follows:

1.

Normalize coordinates. Each pixel at position \((r, c)\) maps to normalized spatial coordinates \(x = c / W\), \(y = 1 - r / H\), both in \([0, 1]\).

2.

Compute depth. The \(z\)-coordinate encodes color information. We compute a weighted luminance: \(z = 0.299 R + 0.587 G + 0.114 B\), normalized to \([0, 1]\). This projects the RGB color space onto a perceptual brightness axis.

3.

Importance sampling. Rather than converting all \(H \times W\) pixels (which could produce millions of points), we perform importance sampling. Pixels with high gradient magnitude \(\|\nabla I\|\) (edges, textures) are sampled with higher probability. Specifically, we compute the sampling weight \(w_{r,c} = \|\nabla I_{r,c}\| + \epsilon \) and draw \(N\) points (typically \(N = 1024\) to \(4096\)) without replacement.

4.

Color encoding. Each sampled point is augmented with its full RGB values as auxiliary attributes: \(\mathcal {P}_{\text {aug}} = \{(x_i, y_i, z_i, R_i, G_i, B_i)\}_{i=1}^{N}\). The RGB values are quantized to 4 bits per channel (16 levels) to control the vocabulary size.

Step 2: Point Cloud to Delaunay Tetrahedralization

The point cloud \(\mathcal {P}\) is tessellated into a tetrahedral mesh using Delaunay tetrahedralization. Given \(N\) points in \(\mathbb {R}^3\), the Delaunay criterion produces a mesh \(\mathcal {M} = \{T_1, T_2, \ldots , T_M\}\) where each tetrahedron \(T_j\) has four vertices, and no point lies inside the circumsphere of any tetrahedron.

The TetraMesh construction enriches this basic Delaunay mesh:

1.

Vertex classification. Each vertex is classified by its geometric role: apex (highest \(z\)), centroid-adjacent (closest to the tetrahedron centroid), ground-face (in the base triangle), or terminal (remaining vertex).

2.

Sense/antisense pairing. Adjacent tetrahedra sharing a triangular face are paired into sense–antisense doublets, mirroring the double-stranded structure of DNA. This pairing is performed greedily by matching tetrahedra with maximal shared-face area.

3.

Shape fitting. The mesh is analyzed against eight shape primitives (helix, sphere, cube, torus, random, spiral, grid, shell) and the best-fitting shape is recorded as metadata. The shape primitive biases the traversal order in the next step.

Step 3: Tetrahedral Mesh to DNA Base Sequence

The mesh \(\mathcal {M}\) is traversed in a double-helix order to produce a base sequence \(\mathbf {s} = (s_1, s_2, \ldots , s_L)\) where each \(s_i \in \{\texttt {A}, \texttt {C}, \texttt {G}, \texttt {T}\}\).

1.

Traversal ordering. Starting from the tetrahedron closest to the centroid of \(\mathcal {P}\), a spiral traversal visits each tetrahedron exactly once. The traversal follows the sense chain first, then the antisense chain, producing two complementary subsequences.

2.

Base assignment. For each tetrahedron \(T_j\), the four vertices are assigned bases according to their classification: \begin {align} \text {Apex} &\mapsto \texttt {A}, & \text {Centroid-adjacent} &\mapsto \texttt {C}, \notag \\ \text {Ground-face} &\mapsto \texttt {G}, & \text {Terminal} &\mapsto \texttt {T}. \notag \end {align}

3.

Color encoding in codons. The quantized RGB values of the four vertices are encoded as additional bases appended after each tetrahedron’s structural bases. Each 4-bit color channel maps to one of \(\binom {4}{1}^4 = 256\) codon patterns, producing 3 additional bases per vertex (one per color channel), for a total of \(4 + 12 = 16\) bases per tetrahedron.

4.

Complementarity enforcement. The antisense chain is generated by Watson–Crick pairing: \(\texttt {A} \leftrightarrow \texttt {T}\), \(\texttt {C} \leftrightarrow \texttt {G}\). This doubles the sequence length but provides built-in error detection.

Step 4: Base Sequence Through the Central Dogma Pipeline

The base sequence \(\mathbf {s}\) is processed through the full DOGMA pipeline:

\begin {equation} \boxed { \mathbf {s} \xrightarrow {\text {tokenize}} \Gamma \xrightarrow {\text {regulate}} \Gamma ' \xrightarrow {\text {transcribe}} \mathcal {T} \xrightarrow {\text {fold}} \Phi \xrightarrow {\text {realize}} \hat {y} } \label {eq:image-pipeline} \end {equation}

1.

Tokenization. The base string is tokenized by the byte-level tokenizer into tokens of length up to 128 (the TetraMesh max token limit).

2.

Regulation. The epigenetic memory (size 256) provides context from previously processed images. Allosteric regulation (top-\(k=16\)) selectively activates the most relevant genomic regions for the current image content.

3.

Transcription. The 16-layer transformer with 512-dimensional embeddings processes the regulated genome through all four DNA Computing paths simultaneously.

4.

Folding. The transcript is folded into a phenotype representation \(\Phi \) that captures the semantic content of the image.

5.

Realization. The phenotype is realized in the requested mode—text (caption), JSON (structured description), or image (reconstructed pixel data).

Step 5: Decoding Back to Image

When the realization mode is set to image, the pipeline reverses the encoding:

1.

Phenotype to base sequence. The phenotype \(\Phi \) is realized as a DNA base sequence \(\hat {\mathbf {s}}\) through the image realization head.

2.

Base sequence to TetraMesh. The base sequence is parsed back into tetrahedra: every 16 bases reconstruct one tetrahedron (4 structural + 12 color bases), with vertex positions recovered from the structural bases and colors from the color codons.

3.

TetraMesh to point cloud. The tetrahedron vertices with their decoded RGB values form a colored point cloud \(\hat {\mathcal {P}}\).

4.

Point cloud to image. The point cloud is rasterized onto a \(H \times W\) grid. Each pixel is assigned the color of the nearest point, with Gaussian splatting (\(\sigma = 1.5\) pixels) for smooth interpolation of gaps.

Figure 13.4 illustrates the complete image-to-DNA-to-image pipeline with example data at each step.

Comparison with Traditional Computer Vision

Table 13.15.2.0 contrasts the DOGMA tetrahedral encoding with conventional convolutional neural network (CNN) approaches to image processing.

13.15.3 Video to DNA Encoding

Video extends image encoding along the temporal dimension. A video \(\mathcal {V} = (I_1, I_2, \ldots , I_F)\) consisting of \(F\) frames is encoded as a sequence of linked TetraMesh structures.

Temporal TetraMesh Construction

Each frame \(I_t\) is independently converted to a point cloud \(\mathcal {P}_t\) and tetrahedralized into a mesh \(\mathcal {M}_t\) using the image pipeline of Section 13.15.2. The temporal linking proceeds as follows:

1.

Shared-face linking. For consecutive frames \(\mathcal {M}_t\) and \(\mathcal {M}_{t+1}\), boundary tetrahedra (those on the convex hull of each mesh) are matched by proximity. Matched pairs share a virtual triangular face, creating a continuous tetrahedral chain across time. The shared face encodes the optical flow between frames.

2.

Temporal base encoding. Inter-frame connections are encoded using a special codon pattern: ATG (the biological start codon) marks the beginning of each frame, and TAA (a stop codon) marks the end. The inter-frame linking bases encode the displacement vectors between matched boundary tetrahedra.

3.

4D point cloud. Alternatively, the entire video can be treated as a single 4D point cloud \(\mathcal {P}_{4D} = \{(x_i, y_i, z_i, t_i)\}\) where \(t_i = t / F\) is the normalized frame index. This 4D cloud is tetrahedralized directly, producing a mesh that captures spatiotemporal structure in a single tessellation.

Temporal Attention via Allosteric Regulation

DOGMA’s allosteric regulation mechanism (top-\(k=16\)) provides a natural mechanism for temporal attention across video frames. The epigenetic memory (size 256) accumulates context from previously processed frames, functioning as a temporal memory buffer:

\begin {equation} \mathcal {E}_{t+1} = \alpha \cdot \mathcal {E}_t + (1 - \alpha ) \cdot \text {TopK}(\Phi _t, k=16) \label {eq:temporal-epigenetic} \end {equation}

where \(\mathcal {E}_t\) is the epigenetic state at frame \(t\), \(\Phi _t\) is the phenotype of frame \(t\), and \(\alpha \) is a learned decay parameter. The top-\(k\) selection ensures that only the most salient features from each frame persist in memory, analogous to how biological epigenetic marks selectively regulate gene expression in response to environmental stimuli.

Comparison with Traditional Video Models

13.15.4 Audio to DNA Encoding

Audio signals are encoded through a spectrogram-based pipeline that maps frequency–time representations to 3D point clouds and then to TetraMesh structures.

Spectrogram to Point Cloud

Given an audio waveform \(a(t)\) sampled at rate \(f_s\), the encoding proceeds:

1.

Short-time Fourier transform. Compute the spectrogram \(S(f, \tau ) = |\text {STFT}(a(t))|^2\) with window size 1024 samples and hop length 256 samples. This produces a time–frequency matrix.

2.

Mel scaling. Apply a Mel filterbank with 128 filters to produce \(S_{\text {mel}} \in \mathbb {R}^{128 \times T}\), compressing the frequency axis to match human auditory perception.

3.

Point cloud construction. Each spectrogram bin \((f_i, \tau _j)\) with energy above a threshold \(\theta \) becomes a 3D point: \begin {align} x &= \tau _j / T, & y &= f_i / 128, & z &= \log (1 + S_{\text {mel}}(f_i, \tau _j)). \notag \end {align}

4.

Importance sampling. As with images, high-energy and high-gradient regions are oversampled, concentrating points on spectral peaks (formants, onsets, harmonics).

Frequency Bands as Tetrahedral Layers

The TetraMesh for audio has a distinctive layered structure that reflects the physics of sound:

• Bass layer (\(y < 0.25\), corresponding to 0–2 kHz): Large tetrahedra capturing low-frequency energy. Mapped to A-rich base sequences (adenine’s double hydrogen bond mirrors the strong, sustained energy of bass frequencies).
• Mid layer (\(0.25 \le y < 0.625\), 2–5 kHz): Medium tetrahedra encoding speech formants and melodic content. Balanced base distribution.
• Treble layer (\(0.625 \le y < 0.875\), 5–11 kHz): Small, dense tetrahedra capturing transients and harmonics. G/C-rich sequences (the triple hydrogen bond of G–C pairs encodes the higher information density of treble frequencies).
• Ultrasonic layer (\(y \ge 0.875\), above 11 kHz): Sparse tetrahedra for high-frequency detail, present only in high-fidelity audio.

Comparison with Traditional Audio Models

13.15.5 Cross-Modal Generation

DOGMA’s unified genomic representation enables generation across modality boundaries. Because all modalities converge to the same genome representation \(\Gamma \) and diverge only at the realization step, cross-modal generation requires no additional architectural components—only a different RealizationMode selection.

Text to Image via DOGMA

Text-to-image generation follows the pipeline: \begin {equation} \text {``a red car''} \xrightarrow {\text {tokenize}} \Gamma _{\text {text}} \xrightarrow {\text {Central Dogma}} \Phi \xrightarrow {\text {realize(image)}} \hat {I} \label {eq:text-to-image} \end {equation}

The text prompt is tokenized into a genome \(\Gamma _{\text {text}}\) using the standard CodonForge tokenizer. The Central Dogma pipeline processes this genome through all 16 layers, producing a phenotype \(\Phi \). The image realization head converts \(\Phi \) into a TetraMesh base sequence, which is then decoded to a point cloud and rasterized. The multi_express method allows simultaneous text and image realization from the same phenotype:

\begin {equation} \Phi \xrightarrow {\text {multi\_express}} \begin {pmatrix} y_{\text {text}} \\ y_{\text {image}} \\ y_{\text {dna}} \end {pmatrix} \label {eq:multi-express-image} \end {equation}

Image to Text via DOGMA

The reverse direction—image captioning—uses the image encoding pipeline of Section 13.15.2 to produce a genome \(\Gamma _{\text {image}}\), processes it through the Central Dogma, and realizes the phenotype in text mode:

\begin {equation} I \xrightarrow {\text {TetraMesh}} \Gamma _{\text {image}} \xrightarrow {\text {Central Dogma}} \Phi \xrightarrow {\text {realize(text)}} \hat {y}_{\text {caption}} \end {equation}

Comparison with Diffusion Models

13.15.6 The Multimodal Corpus Builder

A multimodal DOGMA model requires training data spanning multiple modalities. The multimodal data pipeline downloads and converts data from five open sources:

Every source follows the same pipeline: raw data is converted to a point cloud, passed through TetraMesh, and rendered as a training document with metadata headers and a caption in the USER:/ASSISTANT: format. The resulting corpus is split into train/valid/test sets and stored in the external_corpus_dirs directory referenced by the multimodal configuration.

13.15.7 Multi-Express: One Genome, Many Realizations

The architectural core of DOGMA’s multimodal capability is the CentralDogma.multi_express() method, which implements a principle with a direct biological analogy: one gene can produce many different proteins through alternative splicing. In DOGMA, one genome produces one transcript, one phenotype, but many realizations:

\begin {equation} \boxed { \Gamma \xrightarrow {\text {regulate}} A \xrightarrow {\text {transcribe}} T \xrightarrow {\text {fold}} \Phi \xrightarrow [\text {realize}_2]{\text {realize}_1} \begin {pmatrix} y_{\text {text}} \\ y_{\text {code}} \\ y_{\text {JSON}} \\ y_{\text {action}} \\ y_{\text {dna}} \\ y_{\text {image}} \end {pmatrix} } \label {eq:multi-express} \end {equation}

The pipeline executes the expensive steps (regulation, transcription, folding) exactly once. Only the final realization step branches, producing outputs in as many modes as requested. The PhenotypeRealizer.multi_realize() method takes a single phenotype and a list of RealizationMode values, returning one Realization per mode:

• TEXT: Natural language text, with system and task prompts rendered as paragraphs.
• CODE: Executable code, with each semantic fragment prefixed by a comment header.
• JSON: Structured JSON serialization of the phenotype’s semantic content.
• ACTION: A world-action payload for downstream execution (includes tool calls).
• DNA: A raw DNA base sequence for genomic applications.
• IMAGE: A TetraMesh-encoded base sequence that decodes to a raster image.

Each realization receives a quality score that measures how well the output captures the underlying phenotype, computed from content length, format validity (e.g., valid JSON for the JSON mode), and the phenotype’s folding quality.

13.15.8 Visual Overviews

This section provides three visual overviews of the DOGMA multimodal system: the full multimodal pipeline, a side-by-side comparison with traditional ML, and a TetraMesh encoding visualization.

Full Multimodal Pipeline

Figure 13.5 illustrates how image, video, and audio data all feed into the Central Dogma through the unified TetraMesh encoding.

Side-by-Side: Traditional ML vs. DOGMA

Figure 13.6 contrasts the conventional multi-encoder approach with DOGMA’s unified pipeline.

TetraMesh Encoding Visualization

Figure 13.7 shows a concrete example of how a simple 3D shape is converted to a DNA base sequence.

13.15.9 Use Cases for DOGMA Multimodal Intelligence

The unified encoding pipeline opens several application domains where DOGMA’s approach offers distinct advantages over modality-specific architectures.

Medical Imaging

Medical images—X-rays, CT scans, MRI volumes—are naturally 3D or quasi-3D data that benefit from DOGMA’s volumetric encoding:

• CT scan analysis. A CT volume (e.g., \(512 \times 512 \times 128\) voxels) is directly represented as a 3D point cloud where voxel intensity maps to the \(z\)-coordinate at the voxel’s spatial position. The resulting TetraMesh captures anatomical structures as connected tetrahedral regions, with tissue boundaries appearing as natural mesh interfaces.
• X-ray encoding. A 2D X-ray is encoded using the image pipeline of Section 13.15.2. The depth (\(z\)) coordinate encodes radiographic density, so dense structures (bone) appear as high-\(z\) regions and soft tissue as low-\(z\) regions.
• Diagnostic output. The Central Dogma pipeline processes the encoded scan and realizes the phenotype in multiple modes simultaneously: a text diagnosis, a JSON-structured findings report, and an annotated image highlighting regions of concern.

Protein Structure Prediction

Protein structures are the original domain for TetraMesh encoding, and DOGMA offers a natural bridge between sequence and structure:

• Structure to genome. A protein’s 3D structure (alpha-carbon coordinates from PDB) is tetrahedralized, producing a DNA base sequence that encodes the spatial fold. This creates a “structural genome” that can be compared with the protein’s actual coding sequence.
• Fold prediction. Given a protein’s amino acid sequence (realized as a DNA coding sequence via the codon table), the Central Dogma pipeline predicts a structural genome. Decoding this structural genome through the TetraMesh reverse pipeline produces predicted 3D coordinates.
• Mutation analysis. Point mutations in the input sequence produce corresponding changes in the structural genome, enabling prediction of structural effects of mutations without explicit molecular dynamics simulation.

Autonomous Driving: Sensor Fusion

Autonomous vehicles integrate data from cameras (images), LiDAR (3D point clouds), radar (range-Doppler maps), and microphones (audio). DOGMA provides a natural sensor fusion framework:

• Unified scene representation. Camera images, LiDAR sweeps, and radar returns are each converted to TetraMesh encodings. Because all encodings share the same DNA base format, they can be concatenated into a single genome that represents the complete scene.
• Temporal fusion. Sequential sensor readings are linked via the temporal TetraMesh mechanism (Section 13.15.3), with epigenetic memory maintaining a persistent model of the driving environment across time steps.
• Action realization. The phenotype is realized in action mode to produce steering, acceleration, and braking commands, while simultaneously realizing in text mode for explainable decision logging.

Drug Discovery

Small molecule drug candidates are naturally represented as 3D atomic coordinates, making TetraMesh encoding particularly apt:

• Molecular encoding. A drug molecule’s 3D equilibrium geometry (from quantum chemistry calculations or experimental crystallography) is directly tetrahedralized. Atom types map to base annotations, and bond orders are captured by the tetrahedral connectivity.
• Property prediction. The Central Dogma processes the molecular genome and realizes predictions in JSON mode: binding affinity, solubility, toxicity, and metabolic stability as structured key–value pairs.
• Generative chemistry. By conditioning the Central Dogma on a target property profile (e.g., high binding affinity to a specific protein target), the DNA realization mode produces a molecular genome that can be decoded back to a 3D structure—a de novo drug candidate.

13.16 DOGMA for Conversational AI

The codebase includes a ChatSession interface for testing sustained, multi-turn language generation. An interface is not a capability result. The native measured baseline still produces malformed prose. A separate ATCG-to-language bridge now passes a narrow four-probe evaluation, so this section documents both the mechanism and the strict boundary between those two results.

13.16.1 The ChatSession Architecture

The ChatSession class maintains a turn-based conversation with a trained DOGMA model. Its architecture is straightforward: a conversation history list stores alternating user and assistant messages, and each response is generated by constructing a prompt from the full history, tokenizing it as raw UTF-8 bytes, and generating autoregressively with top-\(k\) sampling.

The response loop follows five steps:

1.

Append the user’s message to the conversation history.

2.

Enrich (optionally) the input through the Central Dogma pipeline.

3.

Build the full prompt from history in the format USER: … ASSISTANT: … USER: … ASSISTANT:, truncated from the beginning to fit the model’s context window.

4.

Generate byte by byte, applying temperature scaling and top-\(k\) filtering at each step. A streaming callback emits each decoded character in real time.

5.

Stop when an end-of-sequence token is generated or a turn marker (USER:, –-) appears in the output, preventing the model from hallucinating the next user turn.

13.16.2 Central Dogma Context Enrichment

The most distinctive feature of DOGMA’s conversational mode is optional Central Dogma enrichment, toggled by the /dogma command. When enabled, each user message is processed through the full Central Dogma pipeline before being passed to the language model:

1.

The user’s text is encoded into a genomic sequence via CentralDogma.from_text(), producing a genome \(\Gamma \) where the original text is encoded as typed genomic bases.

2.

The genome is expressed through the full pipeline: regulation \(\to \) transcription \(\to \) folding \(\to \) realization.

3.

The execution trace—number of bases in the genome, number of bases expressed, and folding quality—is appended to the user’s message as a compact annotation:

What is the structure of insulin?
  [DOGMA: bases=156 expressed=89 quality=0.73]

This enrichment serves two purposes. First, the trace statistics provide the language model with a genomic fingerprint of the query, encoding information about its complexity and structure that is not obvious from the surface text. Second, the regulation step selectively activates genomic regions relevant to the query, potentially priming the model to produce more contextually appropriate responses.

13.16.3 Realization Focus for Conversational Output

The conversational configuration introduces a critical corpus-level optimization: realization_focus. When this flag is set to true, the training corpus is modified to truncate DNA sequences and emphasize the semantic output—the natural language content that follows the genomic metadata.

The rationale is straightforward: a conversational model must prioritize fluent, coherent language generation over genomic fidelity. The genomic substrate remains the computational engine, but the model’s capacity should be allocated to the realization layer—the part the user actually sees. The max_dna_display parameter limits DNA sequences to 40 characters in the training corpus, ensuring the model spends most of its context window on natural language patterns.

The conversational configuration also includes distilled data from a separate datasets/distilled directory, weighted at \(3\times \) the genomic corpus. This distilled data consists of high-quality conversational examples that have been filtered and refined through a teacher model, providing the chat-specific patterns (greetings, question-answering, multi-turn coherence) that raw genomic data cannot supply.

13.16.4 Measured ATCG-to-Language Bridge

The July 2026 experiments introduced an explicit boundary between genomic representation and natural-language realization: \[ q \xrightarrow {C_{\mathrm {boot}}} (g,t) \xrightarrow {R_{\phi }} \hat {y}, \] where \(q\) is the request, \(C_{\mathrm {boot}}\) is a deterministic bootstrap compiler, \(g\in \{A,T,C,G\}^{64}\) is a canonical trace, \(t\) is a compact auditable transcript, and \(R_{\phi }\) is a learned realization model. This decomposition is useful because a failure can be assigned to a stage. It is also scientifically dangerous if the stages are conflated: success by \(R_{\phi }\) does not prove that a native DOGMA generator produced \(g\).

Bootstrap Compiler

The compiler normalizes the request to printable ASCII, keeps a 16-character semantic plan, pads it to fixed width, and encodes each byte as four bases using \[ 00\mapsto A,\qquad 01\mapsto C,\qquad 10\mapsto G,\qquad 11\mapsto T. \] It then enforces a start marker ATGATG and a terminal marker TAATAAAT, yielding exactly 64 canonical bases. The transcript retains up to 28 normalized characters so a human can audit what was passed to the realizer.

This compiler is deterministic. It demonstrates a stable ATCG-first interface, not learned genomic semantics. A learned native compiler must eventually replace it and outperform controls before the genomic bottleneck can be called useful.

Distillation and Realization

The training corpus contains 96 instruction records spanning genomic explanation, scientific reasoning, critique, general language, and DOGMA research contracts. Answers were distilled from Qwen3-4B-Instruct-2507. A LoRA adapter on Qwen3-0.6B learned the realization contract over four epochs and 48 optimizer updates; recorded training loss fell from about 2.0 to 0.32. The adapter occupies approximately 50 MB.

The teacher and student are transformers. Therefore the measured claim is precise: a compact transformer adapter can realize useful prose when conditioned on a request and a canonical ATCG trace. The experiment does not show that non-transformer DOGMA alone has acquired language.

Held-Out Contract Probes

The frozen dogma-genomic-language-v3 suite contains four transfer probes that are excluded from adapter training. All four passed in the retained evaluation:

The evaluator also required four distinct answers, printable output, no replacement characters, and the genome to precede the transcript and answer. This is stronger than checking a regular expression, but still only a smoke suite.

13.16.5 Streaming Generation

Real-time streaming is essential for conversational AI. DOGMA implements streaming through an incremental decode loop: after each token is sampled from the model, the full sequence of generated tokens is decoded to UTF-8, and any newly decoded characters are emitted through a callback function. This approach handles the complication that UTF-8 is a variable-width encoding—a single character may span multiple bytes, and a byte token may not correspond to a complete character until subsequent bytes arrive.

The streaming loop also handles inference optimizations:

• Float16 on CUDA: The model is converted to half-precision for approximately \(2\times \) speedup on GPU.
• torch.compile: On PyTorch 2.0+ with CUDA, the model is compiled for fused kernel execution.
• Context truncation: The prompt is truncated to context_length - 32 bytes, reserving space for generation without exceeding the model’s positional encoding range.

13.17 Exercises

13.1.

[Fundamentals: FASTA parsing] Write a function that reads a FASTA file and returns a list of (header, sequence) pairs. Handle multi-line sequences correctly. Test it on a file containing three records with sequences of different lengths. What edge cases must your parser handle?

13.2.

[Fundamentals: Tokenization] Apply BPE tokenization (with a vocabulary of 1,000 tokens trained on English text) to the DNA sequence ATGATGATGATGATGATG. How many tokens result? Now apply byte-level tokenization to the same sequence. Compare the two representations in terms of: (a) number of tokens, (b) preservation of codon boundaries, and (c) consistency across different occurrences of the same codon.

13.3.

[Analysis: Bigrams] Given the bigram transition matrix for a genomic corpus where \(P(\texttt {G} \mid \texttt {C}) = 0.18\) while the marginal \(P(\texttt {G}) = 0.25\), compute the CpG suppression ratio \(P(\texttt {CG}) / (P(\texttt {C}) \cdot P(\texttt {G}))\). What biological mechanism explains a ratio less than 1.0? How would this ratio differ between vertebrate and bacterial genomes?

13.4.

[Analysis: Tokenization comparison] Tokenize the sequence ATGAAAGCCTTTGGA using (a) byte-level, (b) 3-mer non-overlapping (codon), and (c) 6-mer overlapping. For each method, count the number of tokens produced and determine whether the start codon ATG is preserved as a single unit. Discuss the trade-offs.

13.5.

[Design: Curriculum] Design a curriculum for training a genomic foundation model on viral genomes specifically. How would your curriculum stages differ from the four-stage curriculum in Section 13.7.3? Consider the unique properties of viral genomes: small size, overlapping reading frames, high mutation rates, and host-dependent codon usage.

13.6.

[Implementation] Implement a bigram model over the DNA alphabet \(\{\)A, T, C, G\(\}\) using only Python’s standard library. Train it on a 10,000-nucleotide sequence sampled from any publicly available genome. Report: (a) the \(4 \times 4\) transition matrix, (b) the per-character cross-entropy in bits, (c) a 100-nucleotide sequence generated by sampling from the model. Compare the generated sequence’s GC content to the training data.

13.7.

[Implementation: CodonForge] Write a simplified CodonForge tokenizer that: (a) detects whether a line is a FASTA header or sequence data, (b) tokenizes headers as bytes, (c) tokenizes coding sequences as codons (assuming frame offset 0), and (d) wraps each modality with appropriate boundary tokens. Test on a FASTA file containing both coding and non-coding sequences.

13.8.

[Theory: Dual-stream] In the dual-stream architecture (Equation 13.11), the helix stream operates at motif granularity (width \(k = 6\)) while the standard stream operates at byte granularity. Derive the sequence length relationship: if the standard stream has \(T\) positions, how many positions does the helix stream have? What are the implications for cross-attention alignment between the two streams?

13.9.

[Theory: TetraMemory] The TetraMemory system (Section 13.11) operates at four scales with different resolutions. Calculate the total memory capacity in nucleotides across all four scales. If a model processes a 1Mb (1,000,000 nt) sequence, what fraction of the sequence can be stored in TetraMemory at each scale? What information must be discarded, and what write policy would you use?

13.10.

[Research] The cross-modal training loss (Equation 13.16) includes a translation loss \(\mathcal {L}_{\text {trans}}\) that mimics biological translation (DNA \(\to \) protein). Propose an additional loss term that would encode gene regulation—the context-dependent activation of genes. How would you represent regulatory signals (promoters, enhancers, transcription factor binding sites) in the training data? How would this loss interact with the evolutionary training loop of Chapter 12?

13.11.

[Research: Evaluation] The benchmarks in Table 13.4 focus on individual genomic tasks. Propose a holistic evaluation framework that tests a model’s understanding of the Central Dogma as a complete pipeline: from DNA to RNA to protein to function. What tasks would your benchmark include? How would you measure whether a model truly understands the flow of genetic information versus merely memorizing sequence patterns?

13.12.

[Challenge: RC Consistency] Prove that if a model achieves zero RC consistency loss (Equation 13.13) on all training sequences, its representations must be invariant to strand orientation. Does this imply the model cannot distinguish the two strands? If not, how can strand-specific information be preserved while maintaining consistency? Hint: Consider what information is in the pooled representation \(\mathbf {z}\) versus the per-position representations \(\mathbf {h}^{(t)}\).

13.18 Key Takeaways

Suggested Reading

• Sennrich et al. [2016]: Byte Pair Encoding—the subword tokenization method whose limitations on genomic data motivate byte-level approaches.
• Ji et al. [2021a]: DNABERT—a BERT-based model for DNA sequences using \(k\)-mer tokenization, an alternative to byte-level approaches.
• Dalla-Torre et al. [2024a]: The Nucleotide Transformer—large-scale foundation models for genomics, demonstrating the scaling potential of genomic language models.
• Devlin et al. [2019]: BERT—the masked language modeling objective that inspires MLM-based genomic pretraining.
• Bengio et al. [2009]: Curriculum learning—the theoretical foundation for staged training from simple to complex examples.
• Chapter 11 of this book: CodonForge tokenization and the DNA compiler that the unified vocabulary builds upon.
• Chapter 8 of this book: HelixHash motif embeddings and the dual-stream architecture that helix-aware models build upon.
• Chapter 10 of this book: TetraData rendering pipelines, TetraMemory, and the structured formats used for curriculum construction.
• Chapter 12 of this book: the evolutionary training loop that optimizes prompts for genomic foundation models.
• Chapter 14 of this book: the measured small baseline, its generation failure, and the evaluation program required before scaling claims.