Torchwright Hand-Compiles a Transformer to Do Exact Math

Torchwright Hand-Compiles a Transformer to Do Exact Math — Zero Training

An independent researcher has built a compiler that writes the weights of a stock transformer directly from a computation graph, with no training involved. The result is an ordinary Phi-3 checkpoint that multiplies numbers up to 12 digits with 100% accuracy — reopening the question of what transformers can actually express.

Key Takeaways

An independent researcher has built a compiler that writes the weights of a stock transformer directly from a computation graph, with no training involved.
The result is an ordinary Phi-3 checkpoint that multiplies numbers up to 12 digits with 100% accuracy — reopening the question of what transformers can actually express.

AI synthesis4 sources6

Transformers are famously bad at arithmetic. A new open-source project argues that the architecture itself is not the bottleneck — the training is.

Robert Porter, an independent researcher, released Torchwright, a compiler that takes a computation graph written in ordinary Python and produces the weights of a stock transformer that executes it. There is no training anywhere in the pipeline: every weight is computed by hand, layer by layer, then written into a standard Phi-3 checkpoint that loads with vanilla Hugging Face code.

The tool grew out of a simple question: can a transformer even express elementary-school arithmetic, or is that beyond the architecture? While transformers are Turing-complete in idealized proofs, Porter wanted to know whether the exact weights for a given algorithm actually exist. Torchwright's answer is a decisive yes: compiled models achieve 100% accuracy across all 3,000,000 supported expressions for up to 12-digit multiplication — no gradient descent, no data.

The compiler builds on a decade of interpretability research. It inherits ideas from RASP, the programming language Weiss, Goldberg and Yahav introduced to describe transformer computations, and from DeepMind's Tracr, which compiled RASP programs into real weights. Torchwright goes further by targeting a modern, off-the-shelf architecture: causal softmax attention, rotary position embeddings, RMSNorm and gated SwiGLU feed-forward layers, exactly like production LLMs.

Under the hood, the compiler treats the residual stream as a scratchpad: it allocates columns for intermediate values, uses the skip connection as an adder, and frees columns by writing a value's negation. Multiplication turns out to be cleaner under SwiGLU than under ReLU — the gate's symmetry makes a*b expressible exactly.

The repository ships twelve example graphs, including three adders, digit sorting, a Caesar cipher, Fibonacci, and four calculators that compute the same expressions in radically different ways — one of them memorizes every answer. The author's next post will quantify what each strategy costs in layers, parameters, and tokens.

The project is a proof of concept — nobody needs a transformer calculator. But as a laboratory for interpretability, it offers something rare: exact knowledge of what every weight in a model does.