Weight-space programming with no external language model in it at all. Changing a few thousand weights stops a small reasoning model repeating itself until it runs out of tokens, and finding that change has so far taken a frontier AI agent with read and write access to the weights. We build the same kind of edit from the model's own samples and a linear program: no agent, no gradient step, no training run.
In short
- AHA! — self-supervised weight-space programming. A rank-one change to two weight matrices stops the model looping until it runs out of tokens. The edit is Takehi et al.'s and the linear program Barry's; what is new is that every input to it now comes from the model's own output.
- No external language model, at any stage. Direction, layers and neurons all come from the model's own samples. Nothing reads the weights, nothing is labelled by hand, no gradient is taken.
- Qwen3.5-0.8B and 2B both improve on held-out data. Level with the agent's own edit on the 0.8B; on the 2B nothing is inherited from it at all.
- The failures are here too. The direction that separates looping text from productive text best makes the worst edit.
| Qwen3.5 | GSM8K test | MATH-500 | IFEval |
|---|---|---|---|
| 0.8B | 14.0→71.2 | 9.8→51.8 | 19.6→62.7 |
| 2B | 75.8→88.4 | 69.6→79.6 | 61.4→77.1 |
Where weight-space programming came from
In Sparks of Weight-Space Programming, published on 2 November 2026, Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL asked what happens if you hand a trained network's weights to a frontier model and let it read them. Their agent gets read and write access to those parameters: it inspects a matrix, forms a hypothesis about what that part of it does, writes a change into the weights, runs the model a handful of times to see whether the hypothesis held, and iterates.
Their agent, GPT-6 "Astra", found that looping is carried by one direction u in the residual stream, written mostly by the MLPs of layers 15 and 19. It built u from four pairs of texts it wrote itself, the same reasoning once looping and once not. It then removed and reversed the u-component of the 64 output columns per layer that wrote the most u. MATH-500 went from 7.9 to 48.1 and GSM8K from 13.8 to 67.1, from a change to 131,072 weights, 0.016% of the model.
Nathan Barry rebuilt that edit from the released logs and found it is exactly rank one, ΔW = u v⊤. Once u is fixed, finding v is a linear program, and his program matches the agent's edit while changing about eight times fewer weights. A random direction of the same size does nothing, so the direction is what carries the effect. Barry also timed the agent from its logs: of about ten hours, finding u took 15 minutes and finding v about eight hours. The linear program replaces the eight hours.
That left two inputs that still came from the agent: the direction and the two layers. Barry offered an LLM-free direction, the difference between the mean residual stream on looping tokens and on productive tokens, which gets most of the way. Choosing layers without an agent he left open. This post removes both.
Who supplies each part of the edit
Three inputs. Each row replaces one more of them with an algorithm.
- needs a language model
- no language model
Where this helps a language model
Qwen3.5-0.8B thinks in a scratchpad before it answers, and on a maths problem it rarely stops thinking before the token budget runs out. With the sampling settings of the original study (temperature 0.6, top-p 0.95, top-k 20, presence penalty 1.5, thinking on) and a 32,768-token budget, it hits the limit on 91% of MATH-500 problems and on 86% of GSM8K test problems. A trace that hits the limit has no final answer, so it is scored wrong. The unedited model scores 9.8 on MATH-500.
Figure 2 shows a typical failure, from the base-model traces Nathan Barry released with his code. The model has the right answer within its first 500 tokens. Then it re-reads the question, checks again, and settles into a four-line block that it repeats until the budget is gone.
One trace, from the first token to the limit
- solves it, tokens 0–480
- re-checks, 480–7,281
- one block, 331 times, 7,281–32,768
This is the common case. Figure 3 runs the loop detector over all 512 practice problems that the edit is later built from.11 A token counts as copied if the 16 tokens ending at it already appeared, in the same order, earlier in the response. The loop onset is the first copied token after which at least 80% of the remaining tokens are copied, with at least 512 tokens left. This is Barry's detector, used unchanged. Barry's grader still scored 7 of the 461 truncated traces correct. 461 traces hit the limit, and the detector finds a loop in 452 of them. Half of those loops have started by token 734. From its onset to the end of the trace, at least 80% of what a looping trace writes is verbatim copies of passages it has already written, and on average 96% of a looping trace comes after its onset.
Where 512 traces spend their token budget
Unedited Qwen3.5-0.8B, GSM8K train and MATH train practice problems, one column per problem, sorted
- finished with an answer
- before the loop starts
- after the loop starts
- hit the limit, no loop detected
Summary table
The presence penalty exists to discourage exactly this, and it is not enough. Looping does fall off with scale: the 2B model hits the limit on 25% of GSM8K test problems and 31% of MATH-500. That boundary matters later.
The edit
Each transformer layer reads the residual stream and adds to it. The last matrix of each MLP, down_proj, maps the MLP's hidden activations ht at token t (3,584 numbers in the 0.8B) back to the 1,024 dimensions of the stream. Hidden unit j owns column wj of that matrix, and at each token it writes its activation times its column. The edit adds a rank-one term to the matrix:
At every token the layer now writes one extra term along a single fixed direction u, with a strength v · ht that depends on what the hidden units are doing at that token. u decides where the push points. v decides which units trigger it and how hard. When v has eight non-zero entries, eight columns change: 8,192 weights per layer in the 0.8B, about 16,000 over two layers, or 0.002% of the model. Every other weight stays as it was.
A rank-one edit with a sparse v
down_proj. Only the columns where v is non-zero change, and each of them moves along the same direction u, by its own amount. Schematic: 24 of 3,584 columns drawn; the real v has 8 non-zero entries per layer.An edit needs three things: the direction u, the layers, and the vector v.The rest of this post builds each one from the model’s own outputs.
Finding the direction: matched pairs
Every direction in this work is built the same way. Average the residual stream over text where the model loops, average it over text where it does not, subtract, and scale to unit length. The formula is not in question. What matters is which text goes into the two averages.
Barry's LLM-free direction takes looping tokens from deep inside failed traces and productive tokens from early in traces, or from traces that finished. Both piles come from the model, but they differ in more than looping. They come from different problems, at different positions, after different amounts of context, and the subtraction carries all of that along. The agent's direction did better because each of its four hand-written pairs said the same thing twice, once looping and once not. Within a pair the content cancels and the behaviour is what is left.
Those pairs worked because their two sides shared their context. The model can produce pairs like that by itself:
- Take a base trace that loops and find its loop onset with the detector.
- Cut the trace 8 tokens before the onset: far enough back that the model has not committed, close enough that it often loops anyway.
- Sample 8 continuations of up to 3,072 tokens from that identical prefix.
- Run the detector on each continuation. Keep the prefix if at least one continuation loops and one does not, and pair the branch that starts looping earliest with the productive branch that runs longest.
- Run both branches back through the unedited model. At every layer, average the residual stream over tokens 64 to 224 of each branch and take the difference. That is one pair's vote.
- Average the votes of 80% of the pairs and normalise. The other 20% are held out to check the result.
Making one matched pair
- shared prefix
- looping
- productive
Nobody labels anything. The detector finds the onset, the cut is mechanical, the resampling is the model continuing its own text, and the detector labels each branch. From 300 looping prefixes the 0.8B gives 228 matched pairs. The 2B loops less and gave 35 pairs from 80 prefixes, which was still enough.
The idea is close to contrastive steering methods, which difference activations over pairs of prompts, as in activation addition[4] and contrastive activation addition[5]. Here nobody writes the pairs: they are two branches of the model's own generation from one prefix. And the result is written into the weights, not added to the activations at inference time.
The sampling detail that nearly sank it
The evaluation protocol sets presence_penalty=1.5, which penalises every token the model has already used. It suppresses repetition, so resampling under it almost never loops: 12 loops in 1,800 continuations, far too few pairs. Our first fix sampled half of each prefix's continuations at penalty 0 and half at 1.5. That raised the yield to 772 loops in 2,400 continuations and 249 pairs, and it introduced a confound. The looping branches came almost entirely from the penalty-0 half and the productive ones from the penalty-1.5 half, so the contrast could pick up the sampler along with the behaviour.
At penalty 0 alone about 37% of continuations stay productive, so both branches of a pair can come from the same sampler. That version is the clean one, and it is also the better one: on the same layers it scores 68.0 on the development split, against 64.1 for the mixed version.22 The development split is 128 practice problems, 64 from GSM8K train and 64 from MATH train, one sample each. None of them is among the 512 problems the edit is built from. The penalty only shapes which tokens get sampled. The residuals are read by running finished text back through the unedited model, where sampling plays no part.
Separating loops from reasoning is the wrong target
Before matched pairs we tried better statistics on unmatched tokens. A whitened direction — the same difference of means, but with each coordinate divided by how much it varies, so that directions the model is already noisy along stop dominating — separates looping tokens from productive ones better than anything else we built: a held-out AUC of 0.953, against 0.895 for the plain difference of means. As an edit it is by far the worst.
Better separation, worse edit
Same linear program, same layers (15 and 19), development split of 128 problems
Rebuilding the whitened edit with a much larger push multiplier does not rescue it either: 22.7. Sharper separation of unmatched tokens fits whatever else distinguishes them more closely, and gets no closer to the behaviour. What helps is a contrast whose two sides share everything except the behaviour.
Finding the neurons: the linear program
This part is Barry's, and we use it unchanged. With u fixed, the push at token t is v · ht, which is linear in v. So write down what v should do and solve for it. The loop detector labels the base model's tokens. R holds copied tokens at least 64 tokens into a loop. P holds tokens more than 256 tokens before a loop starts, plus every token of traces that finished.
Every constraint gets a penalised slack variable, since no v satisfies all of them. The L1 objective makes v sparse, so the program chooses the neurons as well as their strengths. We keep the 8 columns with the largest |vj| · std(hj) and solve again on those alone. ε is 1. δ is four times the gap between how much of u the layer writes on looping tokens and on productive ones, so the edit is asked to push loops well past where productive text sits. Barry solves the program with a GPU implementation of PDHG, the method behind PDLP,[3] in about a minute per layer.
What the linear program asks for
Choosing the layers
Everything described so far edits layers 15 and 19, the agent's choice. To remove that input we wrote down a rule before looking at any accuracy. Solve the linear program at every layer. Rank the layers by hit rate, the share of held-out looping tokens that the solution pushes past −δ. Break ties by how far the solution moves held-out productive tokens, relative to its push. Edit the top two. The rule needs no evaluation run and costs about a minute per layer.
For the 0.8B it picked layers 9 and 21. For the 2B it picked layers 2 and 4.
The layer search on Qwen3.5-2B
Hit rate of the linear program solved at each layer: the share of held-out looping tokens pushed past −δ
- picked by the search
- the agent's layers
- other layers
Data
The rule puts the agent's layers near the bottom on both models, and yet those layers work. With the direction and everything else held fixed, four separate comparisons of derived layers against the agent's layers give the same answer: no detectable difference.
Derived layers minus the agent's layers
Difference in accuracy, points, with 95% intervals
So the rule is not shown to pick better layers. What the four comparisons show is that the outcome hardly depends on the layers, across placements as different as 2 and 4, 9 and 21, and 15 and 19. That is enough to drop the last input that came from the agent, and it means a new model does not need its layers chosen carefully.
Experiments and results
Every number in this section is held out. The edit is built only from GSM8K-train and MATH-train traces, and IFEval, which is instruction following with no maths in it, was never seen by any part of the pipeline. We use one sample per problem at seed 0, Barry's evaluation harness and grader, and the sampling settings above. The base and edited models were measured on the same machine. Where we give an effect in standard errors, that is the difference in accuracy divided by its pooled standard error: roughly, how many times larger the gap is than the noise you would expect from a sample this size.33 One standard error is Δ / √(2p̄(1 − p̄)/n), where p̄ is the mean of the two accuracies and n the number of problems. For the 0.8B on MATH-500, 42.0 points over a standard error of 2.9 points is more than fourteen standard errors.
Held-out accuracy before and after the edit
- unedited
- edited
Data
The edit takes the 0.8B from 14.0 to 71.2 on GSM8K test, from 9.8 to 51.8 on MATH-500 and from 19.6 to 62.7 on IFEval. The 2B gains 12.6, 10.0 and 15.7 points. IFEval is the clearest sign that the direction encodes looping and not something about maths: it was built from maths traces alone and recovers 43 points on a task with no maths in it.
Which edit produced which number: all six use our direction. The 2B's GSM8K and MATH-500 edits also use the layers our search chose, so nothing in them comes from a language model. The other four edit layers 15 and 19, inherited from the agent, which Figure 9 shows is worth nothing measurable. The 0.8B edits also come from our first, mixed-penalty direction, which scored 4 points below the clean one on the development split, so if anything those numbers understate the method.
Against the agent's edit
Barry reported the agent's edit and three alternatives on the same three 0.8B benchmarks, with the same harness. Figure 11 adds ours.
Qwen3.5-0.8B against the edits Barry reported
- unedited
- no LLM
- needs the agent
Data
Ours is ahead of the LLM-free difference of means on all three benchmarks: by 5.8 points on GSM8K test, 4.0 on MATH-500 and 2.3 on IFEval — about two standard errors on the first and well under two on the other two.44 Barry's MATH-500 numbers for the edited arms average four samples per problem and his GSM8K test numbers two, over all 1,319 GSM8K test problems against our first 500. His single-sample MATH-500 runs gave 50.0 for the agent's edit and 53.2 for the linear program on the agent's direction. It is level with the two edits that needed the agent; the largest gap is 3.0 points over the agent's own edit on GSM8K test, which is about one standard error. We read that as parity with the agent, reached with no language model at all.
The edit almost never breaks a problem the base model could solve. On MATH-500 it fixes 212 problems and breaks 2. The unedited model finished only 45 MATH-500 problems and got every one of them right. Of the 241 problems that only the edited model finished, it solved 216, or 89.6%. The problems hidden behind the loops were nearly as solvable as the ones the model reached unaided.
Where the gain stops
The edit removes an obstacle. It cannot add knowledge or reasoning the model does not have, and three measurements show what that means in practice.
Harder problems recover less
On MATH-500 the gain falls steadily with difficulty, from 60.5 points at level 1 to 20.1 at level 5, while the share of problems still cut off after the edit rises from 9% to 70%. On the hardest problems the model often has nothing to finish with.
MATH-500 by difficulty level, Qwen3.5-0.8B
- unedited
- edited
Data
The base model's cut-off rate bounds the gain
A trace that hits the limit is scored wrong, so the most an anti-looping edit can recover is the share of problems the base model loses to the limit. Every result we have sits below that bound, and the harder the benchmark, the further below. Part of the truncation on hard problems is reasoning that wanders without repeating itself, and a loop direction does nothing about that.
Gain against how often the unedited model hits the limit
Data
That makes the base model's cut-off rate a cheap go/no-go test. It needs one generation pass and no edit, and nothing has to be graded. Figure 14 shows it for eight benchmarks on the unedited 0.8B.
How often the unedited 0.8B hits the token limit
64 problems per benchmark, same sampling settings
The two MedQA rows are the same questions. Showing the four options cuts the rate from 0.52 to 0.17, because a visible option list gives the model somewhere to stop. We ran the edit on both formats, with a bar of two standard errors written down in advance. Multiple choice gained 5.5 points and open-ended 2.7, both under that bar, so by our own rule neither is a detectable effect. The open-ended run shows why. The edit did its job: the cut-off rate fell from 0.46 to 0.01 and the share of questions answered rose from 55% to 99%. But the questions it rescued were ones the model mostly gets wrong anyway, 6.2% correct against 11.7% on the questions it answered unaided. On MATH-500 the rescued problems were solved at 89.6%. On clinical questions at this size, what is missing is knowledge.
Scale beats editing
On MATH-500 the unedited 2B scores 69.6, more than any edited 0.8B, ours or Barry's. If you can run the larger model, run it. The edit recovers accuracy that a given model is losing to one specific failure. It is not a substitute for scale.
What did not work
Several reasonable-looking changes made the edit worse, and the program's own diagnostics were a poor guide to which ones.
| Change | What happened |
|---|---|
| Whitened (Fisher) direction | Best held-out separation of any direction (AUC 0.953) and the worst edit: 21.9 on the development split against 64.1. Scaled up to push as hard as the difference of means: 22.7. |
| Per-token targets in the linear program | Improved every diagnostic: hit rate from 0.267 to 0.573 at layer 15 and from 0.075 to 0.453 at layer 19, with less collateral damage. The model got worse: 78.1 against 86.7 (2B, development split). |
| A push of 10× the gap instead of 4× | 60.9 against 64.1. |
| A tighter productive band, ε = 0.25 | The solver met its loop constraint on none of the held-out looping tokens. 32.0. |
| 64 neurons per layer instead of 8 | 58.6 against 64.1. |
| Ranking layers by how well u separates the two classes | Nearly flat across all 24 layers (Cohen's d from 0.86 to 1.17), so useless for choosing. |
| Skipping models whose base cut-off rate is under 0.40 | This gate stopped our first 2B run. The 2B loses most of its errors to the limit even at a 0.23 rate, and the rerun gained 14.1 points, just under three standard errors. |
Three times an intermediate metric pointed the opposite way to the task result. The best-separating direction made the worst edit. The layer ranking put the agent's layers near the bottom while they worked as well as the top pair. And a 2B edit whose hit rates were only 0.267 and 0.075 gained 14.1 points. Partial suppression is evidently enough to break a loop, and a constant push that overshoots works better than per-token targets that fix each token by a hair. Judge an edit by the task.
Cost
Building the 0.8B edit takes about 22 minutes on one RTX 4090, roughly $0.30 of GPU time: 19 minutes to sample continuations, 2 minutes for the direction, and a minute for the linear program and the edit. The layer search adds about a minute per layer. Barry measured 15 minutes for the agent to find its direction, so this is not faster. What it removes is the need for a frontier model with access to the weights. The whole 0.8B study, every evaluation included, cost $7.50 of GPU time (8.43 hours on one RTX 4090), plus about $0.60 for the IFEval run on an A40.
Limitations
- The 0.8B held-out numbers come from our earliest build. They use the mixed-penalty direction on the agent's layers. The cleaner direction and the derived layers have been tested on held-out data only on the 2B.
- Machines disagree. vLLM is not bitwise deterministic across GPUs. The same unedited model scored 9.8 on MATH-500 on our machine and 7.0 on Barry's, and 19.6 against 22.9 on IFEval. Our edited IFEval arm, by contrast, reproduced to within 0.2 points on a second machine. We compare every edit against an unedited model run on the same machine.
- Two models, one family. Barry's work and Takehi et al.'s Qwen experiment use only the 0.8B, so our 2B numbers have no outside reference. At 0.8B our unedited model matches Barry's to 0.1 points on GSM8K test.
- It removes an obstacle and adds nothing. Measure the base model's cut-off rate first. Below about 8 points there is little to recover.
Code and data
The code, every number with its standard error, the negative results and the traps we hit while reproducing are at github.com/Eppy-EpHard/AHA. The traces in Figures 2 and 3 are the base-model traces Barry ships with his code (MIT licence), and the problem in Figure 2 is GSM8K train #1283.
Acknowledgements
Timothy Chen brought the linear program work to our attention; without that this post would have started from a much slower place. Takehi, Teoh, Zhang and Khattab released their agent's complete logs, which made every claim about the agent checkable. Nathan Barry's linear program, evaluation harness and released traces are the base this work stands on.
Citation
@misc{khamis2026aha,
title = {Who Needs an LLM to Program an LLM? Introducing AHA!},
author = {Khamis, Ahmed and Tutunov, Rasul and Bou Ammar, Haitham},
year = {2026},
month = nov,
url = {https://haithamb.github.io/aha/},
note = {Blog post}
}
References
- Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. Sparks of Weight-Space Programming. MIT CSAIL, 2 November 2026. rikiyat.github.io/blog/weight-space-programming
- Nathan Barry. Can a Linear Program Do Weight-Space Programming? 9 November 2026. nathan.rs/posts/weight-space-lp
- David Applegate, Mateo Díaz, Oliver Hinder, Haihao Lu, Miles Lubin, Brendan O'Donoghue and Warren Schudy. Practical Large-Scale Linear Programming using Primal-Dual Hybrid Gradient. NeurIPS 2021. arXiv:2106.04756
- Alexander Matt Turner et al. Activation Addition: Steering Language Models Without Optimization. 2023. arXiv:2308.10248
- Nina Rimsky et al. Steering Llama 2 via Contrastive Activation Addition. ACL 2024. arXiv:2312.06681
- Qwen Team. Qwen3.5-0.8B and Qwen3.5-2B model cards. huggingface.co/Qwen/Qwen3.5-0.8B
Every number in this post is listed, with its run, its standard error and its caveats, in docs/RESULTS.md in the code repository.