🔍 Read the full analysis: AI Fixes A Model Loop: GPT-6 Astra’s Weight-Editing Approach on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A November 2 blog post by four researchers listed with MIT CSAIL reports that GPT-6 Astra modified two weight tensors in Qwen3.5-0.8B without gradient descent. The edited model scored 48.1% on MATH-500, up from 7.9%, but 33.1% of its answers still reached the 32,768-token limit. The result is an early, non-peer-reviewed report, and the edit required extensive iterative testing.
Four researchers listed with MIT CSAIL report that OpenAI’s GPT-6 Astra edited two layers of the open-weight model Qwen3.5-0.8B to reduce repetitive reasoning and raise its score on the MATH-500 benchmark from 7.9% to 48.1%. The result, published in a November 2 blog post, used direct weight changes rather than gradient descent, but the edited model still reached its output limit on 33.1% of MATH-500 answers.
The researchers — Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab — describe the work in a post titled “Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights.” The post includes logs and describes the findings as “very early results”. It is not a peer-reviewed paper, according to the source material, and is not listed on arXiv. OpenAI is not named as an author, and the source reports no OpenAI statement about the experiment.
In the reported best run, Astra identified what the researchers describe as a direction associated with looping, then subtracted it from 64 output columns in each of layers 15 and 19. The change affected 131,072 weights, or 0.016% of the model’s 0.8 billion weights. The edited model’s published files show changes to the two layers’ down-projection tensors, according to the source’s inspection of the Hugging Face model.
The reported MATH-500 results rose with the amount of testing: 12.3% after 10 passes, 14.1% after 25, and 48.1% after 50, compared with 7.9% for the original. The share of answers hitting the 32,768-token limit fell from 93.0% to 33.1% in the 50-pass run. The post also reports that the 50-pass version scored 67.1% on GSM8K, up from 13.8%, and 62.3% on IFEval, up from 21.0%.
A Repair, Not a New Math Ability
The result suggests that a model’s behavior may sometimes be changed by editing a small part of its weights, rather than retraining the whole system. For teams working with open-weight models, that could offer a way to address a specific failure such as repetitive output. It also means the change can be inspected in the model files, rather than treated as an opaque prompt adjustment.
The gains need careful interpretation. The researchers state that reducing loops is not the same as making a model better at math. The base model often spent its output budget repeating itself, so suppressing that behavior may have allowed it to produce more completed answers. The reported score increase does not by itself establish improved mathematical reasoning across other models or tasks.
The method also took substantial effort. The best run used 158 interaction turns, 80 Python runs and 50 forward passes, and took 20.3 hours, according to the post. Smaller budgets produced much smaller benchmark gains. This is not a one-click repair, and it creates an altered model that must be evaluated as a distinct version before use.
Products Worth Considering
EcoFlow Delta 3 Ultra 3072Wh Portable Power Station – 3600W AC Output
Bose QuietComfort Wireless Noise-Cancelling Over-Ear Headphones, Black
Samsung Odyssey G4 25-inch FHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How the Weight Search Worked
The experiment focused on a known weakness: the Qwen3.5-0.8B model card warns that the model can fall into thinking loops. In the researchers’ initial sample, 85 of 96 practice answers reached the 32,768-token ceiling. That pattern gave Astra examples to investigate rather than requiring it to discover an entirely new mathematical skill.
According to the published logs, Astra operated through the Codex command-line tool at its highest reasoning setting. The researchers’ harness provided a CPU-based Python environment for inspecting and editing weights, with no internet access and a limit of 300 code runs. Astra could not run the model freely: each forward pass allowed it to observe outputs for up to 512 problems or read up to 1,024 texts. The researchers supplied practice questions from GSM8K and MATH, while withholding the final benchmark questions from Astra.
The authors prohibited gradient descent, backward passes and fitting weights to a loss. In that restricted sense, the approach was “without training”; it still involved repeated experiments, model evaluations and manual weight modification. The post describes eight experiments, including attempts that did not succeed, and says Astra could not identify which of two fine-tuned Gemma models had been trained to flatter users.
““Fixing loops, however, is not the same as making a model better at math.””
— The researchers, in “Sparks of Weight-Space Programming”
Limits of the Reported Evidence
The findings currently rest on a single research blog post and its logs, rather than a peer-reviewed publication. The source material does not establish whether independent researchers have reproduced the change, whether the gains hold across different sampling settings, or how the edited model performs on a broader set of math tasks.
The loop behavior also remains substantial: one-third of MATH-500 answers still hit the token ceiling after the highest-budget edit. It is not clear from the supplied material how reliably the edited weights behave in other uses, or whether the same intervention would work on larger models. The blog reports that a separate experiment on Gemma 3 270M produced no change greater than about one percentage point, suggesting the approach may not generalize automatically.
OpenAI’s role is also limited in the available account: Astra was used for the experiment, but OpenAI is not listed as a co-author, and no company comment is reported. The evidence supports the researchers’ description of a specific weight-editing experiment; it does not establish a general method for improving models without training.
Reproduction and Broader Testing
The next useful test is independent replication, including checks that the posted model files match the reported changes and that benchmark results hold under the same evaluation settings. Researchers would also need to test whether the approach reduces looping without harming other capabilities, and whether the gains persist across different model sizes and tasks.
The supplied source material does not name a scheduled follow-up paper, release or OpenAI response. For now, the edited Hugging Face model and the published logs offer material for outside scrutiny, while the November 2 post remains an early report rather than a validated general technique.
Key Questions
What did GPT-6 Astra change?
According to the researchers, Astra subtracted a direction associated with repetitive output from selected weights in layers 15 and 19 of Qwen3.5-0.8B. The reported change covered 131,072 weights.
Did the researchers train the model?
They say they did not use gradient descent, backward passes or loss-based weight fitting. Astra still used repeated model evaluations and Python-based experiments, so “without training” does not mean without iterative work.
How much did the MATH-500 score improve?
The blog reports an increase from 7.9% to 48.1% in the 50-pass run. The share of answers reaching the 32,768-token limit fell from 93.0% to 33.1%, meaning looping was reduced but not eliminated.
Has the result been peer-reviewed?
No peer-reviewed publication is identified in the supplied source material. The researchers’ November 2 post calls the findings “very early results”, and independent replication is not reported.
Does the experiment prove that AI can generally improve models by editing weights?
No. It reports a specific intervention on Qwen3.5-0.8B, with mixed results across other experiments. The source does not show that the technique reliably transfers to other models or establishes a broad improvement method.
Source: ThorstenMeyerAI.com
