TONN: Networks That Build Their Own Architecture from Data

People choose a neural network's architecture before training. Architecture search automates that choice, but it ranks whole designs by their final loss, and loss cannot tell a part that captures structure from one that fits noise. We present TONN, which builds sequence models from small circuits, such as delays, running sums, products and attention. Inside one training run, TONN keeps a circuit only if switching it off raises the error, in bits, on data the network has not trained on. We call that rise the circuit's worth. On pure noise, growing a network by this rule added nothing, while five other growth rules added 18 to 24 units. On induction, where the next token is whatever followed the last copy of the current token, TONN matched a transformer (0.018 against 0.017 bits per token, 3 seeds). It did so by denoising, meaning it started from a large random network and kept replacing its least useful circuits. Even when no single circuit could do the lookup alone, it composed an induction head, the copying circuit found in trained transformers, from delays, products and attention. From one random start, it kept short-range circuits for proteins and long-range lookups for movie reviews, so the architecture followed the data. At larger scale, how worth is measured decides whether the search helps. On Long Range Arena Text, which classifies movie reviews read as 4,096 raw bytes, a cheap one-pass estimate of worth lost to the random starts (69.7 against 75.5 percent, 3 seeds). Exact worth reached 78.9 percent against 74.9 for its start, in the one seed we ran; a gentler schedule reached 73.5. That network ran as fast as a transformer of similar size on one GPU. Our transformers, trained on the same budget, reached only 58 to 61 percent, against 79.1 published. On proteins the search added 6 points over its random start but trailed designed networks by 2 to 4 points. On web text, a same-size transformer is 0.13 bits per byte ahead.