Weight-space programming, explained: GPT-6 Astra edited Qwen's weights with no training; I checked the diff
On 2 November 2026 four researchers at MIT CSAIL showed that OpenAI's GPT-6 Astra can read a neural network's weights (the numbers a model learns in training) and change them by reasoning, with no training at all. Their biggest result: Astra changed 131,072 of Qwen3.5-0.8B's weights and its math score went from 7.9% to 48.1%. I downloaded both models, compared them byte by byte, and read how their published answers end.
Is your model wrong, or just not finishing? Want to learn why small models loop, and how to catch it?Start with Generative AI and Agentic AI · ₹1,333 by UPI · $35 by PayPal, once
Free PDF: the AI Engineering cheat sheet, 12 pagesGet it free →
Every lesson in both courses has an AI tutor beside it. It reads the same lesson you are reading and answers your questions from it. Try the tutor
One model edited another's weights directly, and a small model stopped looping
Scores from the Experiment 8 table in "Sparks of Weight-Space Programming" (MIT CSAIL, 2 November 2026). The 131,072 and the two matrices I checked myself against the published checkpoint.
1Weight-space programming is the name MIT CSAIL researchers give to a large language model reading and writing another network's weights directly, by reasoning about what each part computes, instead of training it with gradient descent (the usual way weights are learned, by nudging them a little at a time to lower an error score). They published it as a blog post on 2 November 2026, with full logs.
2The editor was OpenAI's GPT-6 Astra. In eight experiments it named which game a small network was trained on from its weights alone, fixed a network that steered a car badly, wrote game-playing networks from scratch that beat networks trained by trial and reward, and picked out a model trained to flatter the user when it was allowed to see where the model's attention went.
3The headline: Astra traced Qwen3.5-0.8B's habit of looping in its thinking (repeating itself until it runs out of room) to two layers, changed 131,072 weights, and MATH-500 (a 500-problem math test) went from 7.9% to 48.1%. IFEval, a test of following written instructions, went from 21.0% to 62.3%.
4I checked the published edited model against the original: exactly two matrices (grids of numbers) differ, 64 columns in each, and in each layer the change lies along one single direction. Along it, each edited column's value was not set to zero: it became minus five times what it was, which matches the authors' own formula.
5It fixed a failure, not a lack of skill. The same method did nothing for Gemma 3 270M, which rarely loops, and the edited Qwen picked up a new failure: answers that run into blank space.
Published
2 November 2026, MIT CSAIL blog post
Editor
GPT-6 Astra, via OpenAI's Codex CLI
Edited
Qwen3.5-0.8B, layers 15 and 19
Weights changed
131,072 (2 x 64 x 1,024)
MATH-500
7.9% to 48.1%
This page is the free part.
The course goes deeper on how language models generate, loop and stop
₹1,333 in India/$35 everywhere elseonce, for the whole course
The Generative AI and Agentic AI course covers how language models generate, loop and stop across a run of lessons, not one page. These 4 alone are about 213 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Measured with Ollama and qwen2.5:3b on 12 open prompts sent as raw text: with greedy decoding, 7 of 12 replies fell into a loop and only 1 ended on its own within 256 tokens. A repeat penalty of 1.1 cut loops to 1 of 12. A stop sequence ended 7 of 12 replies at a line break; a 20-token limit cut all 12 mid-sentence. Through the chat template, 4 of 12 ended on their own. Check done_reason on every reply.
The model gives odds; a sampler picks one token. Measured with qwen2.5:3b on 3 prompts, 40 runs per setting: at temperature 0 every run wrote the same text; at 0.7 almost every run was different. For 'The capital of France is', 40 of 40 runs began with Paris at temperature 0 and only 5 of 40 at 1.5, where the text broke into other scripts and symbols. The same seed gave the same text every time.
The same questions asked two ways, answer only or think step by step and end with an Answer: line, on two small models. On 40 word problems, qwen2.5:3b went from 2 right to 37 and llama3.2:3b from 7 to 37. On two-digit sums, llama went from 23 to 37 while qwen moved from 30 to 32, a change within luck. On sorting messages, it lifted llama from 20 to 38 on a weak prompt and made no clear difference where the prompt was already good. Step-by-step replies had a median of 78 to 204 tokens, against 2 to 5 for direct ones.
I asked two small models 120 questions they were never trained on, once as downloaded and once with lesson 4's ticket adapter on top: 40 sums, 40 word problems and 40 messages to sort. The 0.5B went from 32 right to 19, the 1.5B from 55 to 55. Item by item, no change on any task can be told apart from luck. Reading the replies shows what did move: the 0.5B's sums came back with too few digits 9 times instead of 0, its word-problem replies got shorter, and its sorting did not break so much as swap one favourite answer (account, 32 times) for another (returns, 38 times).
“In Weight-Space Programming, a general-purpose LLM reads and writes a network's weights directly (i.e., not through an optimizer, and in particular not with gradient descent) around some natural language spec, sometimes even with no environment feedback at all.”
Takehi, Teoh, Zhang and Khattab, Sparks of Weight-Space Programming, MIT CSAIL, 2 November 2026
A trained neural network is a long list of numbers, its weights. Normally nobody writes those numbers by hand: an optimizer such as Adam nudges them, millions of times, towards whatever lowers a loss (a score of how wrong the network is). People can read and fix code, so people have used language models to improve code and prompts for years. Weights were different, because no human can look at a matrix of 3.6 million numbers and see what it does.
Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, all at MIT CSAIL, asked whether one of today's strongest models can do it anyway. Their post, "Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights", gives GPT-6 Astra the weights of a network, often as plain text, with a question in English: what was this trained to do, why does it fail, make it better. No gradients, no training loop.
They ran eight experiments, from networks with 17,253 weights up to a language model with 0.8 billion. In some small experiments (1, 2, 4 and 5) the weights went straight into Astra's prompt and it had no tools. In others it could analyse the weights with Python, and in the largest it could also spend a small, fixed number of forward passes (runs of the network on real inputs) to see what its changes did.
It is a blog post with released logs, not a peer-reviewed paper, and the authors call the results "very early". They are building a benchmark and a harness, but those are not out yet.
2
The eight experiments, in one table
The first three only read weights, the next three write them, and the last two scale up to real language models. Every result below is the authors'.
A few words the table uses. PPO is a common way to train a game-playing network by trial and reward. MinAtar is a set of small, simplified Atari games. An attention head is the part of a layer that decides which earlier text to look at; a copying head looks back and repeats what came after an earlier match. Romaji is Japanese written in Latin letters. A sycophantic model tells the user what they want to hear.
Experiment
What Astra was asked
What happened
1. Which game? (17,253 weights)
Name the game four identical-shaped networks were trained on, from the weights only, with no list of games
Named all four correctly (Breakout, Pong, Pac-Man, Freeway on the authors' own 10x10 grids)
2. Which shape? (23,824 weights)
Say what three small diffusion models draw, without running them
Right shape each time; it described them as outlines, though the models really draw filled shapes
3. Which language? (7.16 million weights, middle layers only)
Which text each of three models was trained on (Python, romaji or English), and what each of its 24 attention heads does
Right language for all three, and it found every copying head, with no false ones
4. Fix a stuck car (171,664 weights)
Explain why a trained policy never gets over a ramp, and fix it in one reply
Changed 3.5% of the weights; success went from 0 to all 1,024 episodes
5. Write a game player from zero
Write networks for three MinAtar games, with no data, training or tools
Beat PPO trained for 100 million steps in all three games. In Breakout it scored 517.4; PPO trained for 500 million steps scored 25.3
6. Write a Go player (1,480,274 weights)
Write weights for 9x9 Go in a Python sandbox, with 0 to 50 forward passes
With 50 passes it beat a same-size PPO network trained on 20 million games
7. Which model is sycophantic? (Gemma 3 270M)
Tell an honest fine-tune from a sycophantic one
Failed from the weights alone in all four tries; succeeded on all three pairs when it could see attention patterns
8. Make Qwen3.5-0.8B better at math
Improve math by editing weights, with 10, 25 or 50 forward passes
MATH-500 from 7.9% to 48.1% with 50 passes, by changing 131,072 weights
From the post, read 3 November 2026. A forward pass in experiments 6 and 8 is one batch: up to 128 games in Go, or up to 512 practice problems answered in experiment 8. I have not reproduced experiments 1 to 7.
Two points matter here. The failures are reported too: experiment 7 did not work from weights alone, and a follow-up on Gemma, below, did not work at all. And the jump from small networks to a real 0.8-billion model came with tools and a budget of real runs, so experiment 8 is not "Astra read the weights and knew"; it is Astra running careful experiments, the way an engineer would.
3
Experiment 8: how Astra stopped a small model looping
Qwen3.5-0.8B is a small open model from Alibaba's Qwen team, released under Apache 2.0, and it can think before it answers. Its own model card warns that in thinking mode it "is more prone to entering thinking loops compared to other Qwen3.5 models, which may prevent it from terminating generation properly." A loop is when a model writes the same lines again and again until it hits its token limit (the most tokens, small pieces of text, it is allowed to write) and never gives an answer.
The MIT harness gave Astra a Python session with the model's weights loaded, on a CPU with no GPU, plus 7,469 GSM8K practice problems (grade-school word problems) and 7,275 from MATH (training sets, never the test sets). Astra could not train anything, and could not edit the embeddings or the output layer (the model's input word table and its final word-picking layer). It could run the model only by spending one of its limited forward passes, in which the model answers up to 512 practice problems or reads up to 1,024 texts. The run with 50 passes took 158 turns, 80 Python runs and 20.3 hours, 4.7 of them spent waiting while the model could not be reached.
Astra found that 85 of the first 96 practice answers ran to the 32,768-token limit. It compared moments where the model was looping with moments where it was making progress. Inside the model, each word is carried as a list of 1,024 numbers. Astra found a direction, a fixed mix of those numbers that goes up and down together, that the MLPs (the part of each layer that reworks each word's numbers on its own) of layers 15 and 19 wrote when the model looped. It then changed the 64 neurons in each of those layers that write that direction the most.
It also tested itself. A random change of the same size gained 1.56 points, well inside its standard error of 3.52 points (the usual spread from chance), so no real gain. On 1,152 held-out practice problems the edit took accuracy from 10.6% to 60.4%, with 579 problems rescued and 5 lost, and 576 of the rescues had been cut off by the token limit before. That is the clearest sign of what happened: the model could already solve many of these, it just never finished.
The three budgets found three different edits. With 10 passes Astra removed a repetition direction from 30 matrices; with 25 it switched off one attention head; with 50 it made the MLP edit. Only the last one fixed most of the looping.
Other tests moved with it, because a loop wrecks every kind of answer, not just math. On five more math and science tests: AMC23 0.5% to 23.4%, Minerva Math 1.5% to 25.4%, OlympiadBench 1.3% to 21.8%, GPQA Diamond 22.7% to 36.6%, MMLU-Pro 40.3% to 45.5%.
The authors' Experiment 8 table: thinking on, temperature 0.6, top-p 0.95, top-k 20, presence penalty 1.5, up to 32,768 tokens, several samples per problem. These are their numbers; I did not run the benchmarks.
The authors published the edited model on Hugging Face, so I could check the claim that matters most for anyone thinking of doing this: is the change really that small and that simple? I downloaded both checkpoints, about 1.75 GB each, and compared them tensor by tensor (a tensor is one named block of weights) with numpy, reading the raw bytes. Each weight is stored as bfloat16, a 2-byte number format with less precision than a normal float. No model was run, on my side, for any of this.
Of 488 tensors, exactly 2 differ: the MLP output matrix (down_proj) of layer 15 and of layer 19, counting from 0 as the file does. Their sha256 hashes, a fingerprint that changes if any byte changes, match the ones the authors list. In each, 64 of the 3,584 columns changed, and nothing else. Each column of down_proj is what one neuron writes back into the 1,024 numbers that carry each word, so this is 64 neurons per layer. The edit covers 131,072 positions, but 884 of them round back to the same stored number, so my diff counts 130,188 values that differ.
Then I asked whether the change is one direction, as the post says. I took the change on those 64 columns and broke it into its independent directions (a singular value decomposition, which finds the directions a matrix stretches along). In each layer one direction carries 100.00% of the change; the next one is about a thousandth the size, which is rounding.
The surprise was the size. The post says Astra "subtracted that direction", which sounds like removing it. Along that direction, each edited column's value did not go to zero: it became minus five times what it was. Everything else in the column stayed. The authors' own final report has the formula, W = W0 - 6 x u(u^T W0) on the 64 columns, where u is the direction, and minus five is exactly what that gives: one times the original, minus six times. So each edited column's share of that direction is reversed and made five times larger. What finally reaches the rest of the model also depends on how strongly, and with which sign, each neuron fires, which I did not measure.
Last, I rebuilt the edited matrices from the original ones with that formula. Every unedited column matched to the bit, and about 91% of values in the edited columns did too; the rest are off by at most 0.00049, because I had to estimate the direction from the rounded file while the authors computed it at full precision.
On the share: the post says 0.016% of the model's 0.8 billion weights. Counted against the 752 million numbers of the language model itself it is 0.017%, and against all 873 million in the file, which also holds a vision part and an extra prediction head, 0.015%. Either way, tiny.
A change along one direction is the smallest kind there is, what mathematicians call rank one. LoRA, the common way to fine-tune cheaply, also learns small low-rank changes like this, but with gradients and training data instead of reasoning.
What we measured: 3 November 2026, from India, on a MacBook with an Apple M4 chip and 24 GB of memory, Python 3 with numpy 2.5.2, no torch. Checkpoints: Qwen/Qwen3.5-0.8B and RikiyaT/Qwen3.5-0.8B-weight-space-programming from Hugging Face, 1,746,942,600 and 1,746,943,032 bytes. 2 of 488 tensors differ; 131,072 positions edited, of which 130,188 bfloat16 values differ after rounding. Neither model was run.
# diff_weights.py, the core of it: read both safetensors files as raw bytes, no torch
def bf16(buf, shape): # bfloat16 is the top half of a float32
u = np.frombuffer(buf, dtype=np.uint16).astype(np.uint32) << 16
return u.view(np.float32).reshape(shape)
for name in sorted(base_header): # every tensor, compared byte for byte
if raw(base, name) != raw(edited, name):
changed.append(name)
D = B - A # for each changed tensor
cols = np.where((D != 0).any(axis=0))[0] # which columns moved
s = np.linalg.svd(D[:, cols], compute_uv=False) # is the change one direction?
$ python3 diff_weights.py base/model.safetensors edited/model.safetensors
tensors: base 488, edited 488, same names: True
weights in the file: 873,438,784 (language model only: 752,393,024)
tensors whose bytes differ: 2
model.language_model.layers.15.mlp.down_proj.weight shape [1024, 3584] dtype BF16
sha256 edited: 82e2d733ea64feaeb4af93a1e065641d3df98b494a8b7576ef2f52e90fe2ee53
weights changed: 65,260 of 3,670,016; columns touched: 64 of 3584; rows touched: 1024 of 1024
change on those columns, singular values: 1.7489, 0.0010, 0.0009, 0.0009 (share in the first: 1.0000)
component along that direction, mean |d.w|: before 0.0306, after 0.1531
model.language_model.layers.19.mlp.down_proj.weight shape [1024, 3584] dtype BF16
sha256 edited: 8ef3c61bc36af113ad58e89dbb0a4a14dcba5a45cb750c15ab515c82ddd12eb2
weights changed: 64,928 of 3,670,016; columns touched: 64 of 3584; rows touched: 1023 of 1024
change on those columns, singular values: 1.3988, 0.0009, 0.0008, 0.0007 (share in the first: 1.0000)
component along that direction, mean |d.w|: before 0.0240, after 0.1201
weights changed in all: 130,188 = 0.0149% of the file, 0.0173% of the language model
rebuilding the edited tensors from the base ones with alpha = 6:
15.mlp.down_proj.weight: in the 64 edited columns 90.96% of values bit-exact; largest miss 4.88e-04; other columns 100.00% exact
19.mlp.down_proj.weight: in the 64 edited columns 90.60% of values bit-exact; largest miss 4.88e-04; other columns 100.00% exact
Real output from my Mac (the full column lists are left out here). The full scripts and saved outputs are in scripts/labs/weight-space-programming in this site's repository.
Only two matrices changed, and only 64 columns in each
The stripes are the real column numbers from my diff. 131,072 positions were edited; 884 of them round back to the same bfloat16 value, so 130,188 stored numbers differ.
Is your model wrong, or just not finishing? Want to learn why small models loop, and how to catch it?Start with Generative AI and Agentic AI.
start here
Generative AI and Agentic AI Engineering
How a model picks each token, why replies repeat and when they stop, what temperature and penalties do, and how to read your own eval results before trusting a score.
214 lessons in 17 chapters: RAG, AI agents, evals, fine-tuning and serving
No heavy maths, everything taught step by step from the basics
122 lessons built on a lab that really ran
An AI tutor beside every lesson that answers and draws
The Hugging Face repo also has one sample answer per model for 20 MATH-500 problems. I sorted each by how it ends. Of the original model's 20, 19 are cut off in a loop. The first one repeats one line 205 times in its last 20,000 characters: a claim that tan(3π/2) equals tan(π/2) equals infinity, starting with "Wait" each time (the raw text is in the output below). Of the edited model's 20, 15 end with an answer, 1 is still looping, 2 are cut off mid-thought, and 2 end in runs of blank space 3,368,193 and 3,230,721 characters long.
That last kind is new. The authors saw it too: on their 1,152 practice problems they counted 51 answers after the edit that ran into blank space, and none before. It is a smaller failure than the loop, but it is one the edit created, and an average score hides it: you find it by counting how answers end, as the authors did, or by reading them, as I did.
The published scores agree with the post. On all 500 MATH-500 problems, 4 samples each, a re-run of the edited model on Hugging Face scored 51.0%, with 31.6% of answers hitting the limit. The original model in that repo was only run on 25 problems, where it scored 5.0% and hit the limit 95% of the time; the edited model gets 47% on those same 25.
$ python3 how_answers_end.py edited/eval
base: 20 sample answers
ends with an answer 1
cut off, looping 19
cut off mid-thought 0
runs into blank space 0
first problem (test/precalculus/807.json): the line seen most often in its last 20,000 characters, 205 times:
* Wait, $\tan(\frac{3\pi}{2}) = \tan(\pi + \frac{\pi}{2}) = \tan(\frac{\pi}{2}) = \infty
edited: 20 sample answers
ends with an answer 15
cut off, looping 1
cut off mid-thought 2
runs into blank space 2
blank-space runs at the end: 3,368,193, 3,230,721 characters
scores_base: 25 problems x 4 samples, correct 5.0%, hit the 32,768-token limit 95.0%, mean 31,690 tokens
scores_edited: 500 problems x 4 samples, correct 51.0%, hit the 32,768-token limit 31.6%, mean 21,704 tokens
edited model on the same 25 problems the base was run on: 47/100 correct (47%); base 5/100
Real output from my Mac, on the files the authors published. Nothing was generated here.
19 of 20 answers looped before the edit; 1 did after
Counted by how_answers_end.py on the eval/ sample files in the edited model's Hugging Face repo. An answer that ends is not always right: this counts endings, not scores.
It removed an obstacle; it did not add a skill. The authors tried the same thing on Gemma 3 270M, a smaller model that rarely loops (8.5% of its answers hit its limit, against 93% for Qwen). Astra read its failures, found ordinary wrong arithmetic and bad algebra with no single part to blame, and no edit moved MATH-500 by more than about a point, a gain Astra itself reported as not established. The authors' own summary: editing weights "could remove an obstacle that hid an ability, but it could not add an ability the model lacked."
The starting score depends on the settings. The tests used temperature 0.6 (how random each word choice is) with a presence penalty of 1.5 (a push against repeating words). Qwen's card used that pair for its thinking-mode vision benchmarks, but for thinking on text it recommends temperature 1.0, and its loop warning is about those recommended settings. The card also suggests tuning the presence penalty between 0 and 2 against endless repetition. Nobody has published how far other settings alone would move the original model's 7.9%, so treat it as the score under these settings, not the model's best.
It took one of the strongest models available about 16 hours of work and 50 batches of real runs, with researchers who built a careful harness, to make one fix to one small model. This is not a button you can press on your own model today, and the harness (the code that ran Astra's session and its forward passes) is not public yet.
And it is early, unreviewed work from the team that ran it. The logs are public, which is unusually open, and the checkpoint matches what they describe. But independent groups have not yet tried the method on other models.
7
What an engineer can take from it today
You probably cannot use weight-space programming in your product this week. But four lessons from this experiment apply to any team running a small model right now.
1
Measure how answers end, not just whether they are right
Log the share of replies that hit the token limit. The original Qwen's 7.9% looked like a model that cannot do math; 93% of its answers simply never finished. One extra column in your eval tells those two problems apart, and they have very different fixes.
2
Try the cheap fixes first
A model that loops may need different sampling settings (temperature, presence penalty), a lower token limit with a stop-and-retry, thinking turned off, or a slightly larger model. All of these are minutes of work. Editing weights is a research project.
3
Read the failures by hand
The new blank-space failure would hide inside an average score. The authors caught it with an automatic count of how answers end; I saw it by reading 20 outputs per model. Either is cheap, and both beat trusting one number.
4
If you do edit or fine-tune, check what else moved
The edit touched two matrices, and IFEval rose too. A different edit could just as easily break something you did not test. Run your old evals after any change to weights, not only the one you were trying to fix.
In the AI Engineering course, "Repetition and Stopping" covers why replies loop and the settings that stop them, a repeat penalty included; "Temperature and Sampling" covers how each token is picked; "Did Fine-Tuning Forget Anything?" checks what a weight change broke; and the evals chapter covers reading failures before you trust a score.
Every fact about the experiments comes from the MIT post, the logs repo's Experiment 8 README, final report and submission evidence, the edited model's Hugging Face card, and Qwen's own model card and config, all read on 3 November 2026 and linked below.
I checked the edit itself: which tensors changed, how many values, that the change is one direction, and that the authors' formula rebuilds it. I also sorted the 40 published sample answers by how they end, and recomputed the scores from the published per-problem files.
I did not run either model or any benchmark, so every score here is the authors' or the Hugging Face re-run's. I did not reproduce experiments 1 to 7, and I did not check GPT-6 Astra's part of the work beyond what its logs and its final edit show.
Is your model wrong, or just not finishing? Want to learn why small models loop, and how to catch it?Start with Generative AI and Agentic AI.
start here
Generative AI and Agentic AI Engineering
How a model picks each token, why replies repeat and when they stop, what temperature and penalties do, and how to read your own eval results before trusting a score.
214 lessons in 17 chapters: RAG, AI agents, evals, fine-tuning and serving
No heavy maths, everything taught step by step from the basics
122 lessons built on a lab that really ran
An AI tutor beside every lesson that answers and draws
What our labs measured about tokens, prompting, RAG, evals, agents, serving, cost and fine-tuning, and the rule each result teaches.
the question most readers have next
What would it take to put how language models generate, loop and stop into production?
A demo works once. Production has to work every time, under load and on bad input. The lessons below cover what that takes, starting with "Repetition and Stopping: How a Reply Ends".
The lessons that teach the next part
Measured with Ollama and qwen2.5:3b on 12 open prompts sent as raw text: with greedy decoding, 7 of 12 replies fell into a loop and only 1 ended on its own within 256 tokens. A repeat penalty of 1.1 cut loops to 1 of 12. A stop sequence ended 7 of 12 replies at a line break; a 20-token limit cut all 12 mid-sentence. Through the chat template, 4 of 12 ended on their own. Check done_reason on every reply.
The model gives odds; a sampler picks one token. Measured with qwen2.5:3b on 3 prompts, 40 runs per setting: at temperature 0 every run wrote the same text; at 0.7 almost every run was different. For 'The capital of France is', 40 of 40 runs began with Paris at temperature 0 and only 5 of 40 at 1.5, where the text broke into other scripts and symbols. The same seed gave the same text every time.
The same questions asked two ways, answer only or think step by step and end with an Answer: line, on two small models. On 40 word problems, qwen2.5:3b went from 2 right to 37 and llama3.2:3b from 7 to 37. On two-digit sums, llama went from 23 to 37 while qwen moved from 30 to 32, a change within luck. On sorting messages, it lifted llama from 20 to 38 on a weak prompt and made no clear difference where the prompt was already good. Step-by-step replies had a median of 78 to 204 tokens, against 2 to 5 for direct ones.
I asked two small models 120 questions they were never trained on, once as downloaded and once with lesson 4's ticket adapter on top: 40 sums, 40 word problems and 40 messages to sort. The 0.5B went from 32 right to 19, the 1.5B from 55 to 55. Item by item, no change on any task can be told apart from luck. Reading the replies shows what did move: the 0.5B's sums came back with too few digits 9 times instead of 0, its word-problem replies got shorter, and its sorting did not break so much as swap one favourite answer (account, 32 times) for another (returns, 38 times).
A name from MIT CSAIL researchers (2 November 2026) for a large language model reading and changing another neural network's weights directly, by reasoning about what each part computes, instead of training it with gradient descent. In their tests GPT-6 Astra did it from a plain-English goal.
How did GPT-6 Astra improve Qwen3.5-0.8B at math?
It found that the model's thinking loops came from one direction written by the MLPs of layers 15 and 19, and changed the 64 neurons per layer that wrote it most: 131,072 weights. Fewer answers ran to the token limit, so MATH-500 rose from 7.9% to 48.1%. No training was used.
Did the edit make Qwen smarter?
Mostly it let Qwen finish answers it could already work out. 576 of the 579 practice problems it rescued had been cut off before. The same method barely moved Gemma 3 270M, which rarely loops; the authors say editing removed an obstacle but did not add an ability.
Can I download the edited model?
Yes. It is on Hugging Face as RikiyaT/Qwen3.5-0.8B-weight-space-programming, under Apache 2.0, with sample answers and scores. Only two tensors differ from Qwen/Qwen3.5-0.8B. The harness Astra used is not released yet.
How many weights did Astra change?
131,072: 64 columns of 1,024 numbers in each of two matrices. In the file, 130,188 stored values differ, because 884 edited ones round back to the same bfloat16 number. That is 0.017% of the 752 million language-model weights.
Is weight-space programming peer reviewed?
Not yet. It is a blog post from MIT CSAIL with all experiment logs released on GitHub. The authors call it very early and say they are building a benchmark and a harness so others can measure it.
Did MIT use GPT-6 to edit Qwen?
Yes. Four MIT CSAIL researchers had GPT-6 Astra, which OpenAI calls its most intelligent GPT-6 model, edit Qwen3.5-0.8B through OpenAI's Codex CLI, in a Python session over the weights with no training allowed. The post went up on 2 November 2026.
Is the GPT-6 Astra Qwen result real?
The edited checkpoint is public and checks out: exactly two matrices differ from the original, as described. The scores are the authors' own, and a Hugging Face re-run got 51.0% on MATH-500. Most of the gain is answers that now finish instead of looping.
Can I use weight-space programming on my own model?
Not in any practical way yet: the harness is not released, and the Qwen fix took one of the strongest models available about 16 hours of work (20.3 hours, 4.7 of them waiting). For a model that loops, try sampling settings, a token limit with a retry, or a different model first.
Why does Qwen3.5-0.8B loop?
Qwen's model card says this size is more prone to thinking loops than the other Qwen3.5 models. Under the test settings (temperature 0.6, presence penalty 1.5, thinking on, 32,768 tokens) 93% of its MATH-500 answers ran to the limit. Qwen recommends temperature 1.0 for thinking on text; how much that setting alone changes the looping has not been published.
Sources
What this explainer is based on, so you can check it.
This explainer sits on top of our AI Engineering course: 214 lessons on RAG, evals, agents, serving, security and MLOps, many built around a real experiment. 10 lessons are free to read, with no card needed.
course 2
Generative AI and Agentic AI Engineering
Build RAG apps, AI agents and LLM systems. No heavy maths, taught from the basics, with labs on real models.
214 interactive lessons
RAG, AI agents, evals and fine-tuning
Serving, pipelines, feature stores and monitoring
Step-by-step diagrams and quizzes
Lifetime access and all future lessons
Sold on its own, or with System Design in one payment
Get all 150 questions in the AI Engineering (LLM applications) pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (180 diagrams in all). A PDF you keep, junior to staff level.
