Not making any claims about a specific model's capabilities, but all these "can interpret model weights" posts brings to mind the line in Shakespeare's Henry IV
GLENDOWE: I can call spirits from the vasty deep.
HOTSPUR: Why, so can I, or so can any man, but will they come when you do call for them?
I can interpret weights too, but will that interpretation be correct?
> I’m not ready to declare that the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways.
That sounds a bit like biology; maybe AI will become like that.
It's a blog post which received feedback from collaborators.
It hasn't gone through the academic paper lifecycle ... the authors themselves refer to it as a post and as preliminary.
I feel like I'm taking crazy pills, but can we be precise when talking about work of this form? It started with referring to arXiv pre-prints as papers, and now we're at blog posts with an academic template as papers.
A paper is a fucking paper: something which has been vetted by the scientific community, with at least some form (often, a few rounds) of anonymous review from people outside your circle of collaborators
This is not there yet, and clearly so due to the various feedback from people (including the two pushback posts on HN) critiquing the lack of discussion about the existence of these techniques in mech interp, etc?
Like you said, in the AI space a pre-print on arXiv is massively more likely than anything that's been through peer-review.
I called it a paper here because it's by people with an academic affiliation and is of comparable quality (if not better than) many of the other "papers" we see.
I also like it when a paper isn't a PDF. PDFs are a bad way of communicating information.
(I will note that the authors themselves didn't call it a paper, their own BibTeX reference calls it a "blog post":)
@misc{takehi2026weightspace,
title = {{Sparks of Weight-Space Programming: LLMs can natively read \& write neural network weights}},
author = {Takehi, Rikiya and Teoh, Jayden and Zhang, Alex and Khattab, Omar},
year = {2026},
month = nov,
howpublished = {\url{https://rikiyat.github.io/blog/weight-space-programming/}},
note = {Blog post}
}
But for (1, 2), just to point out, neither of your criteria would pass muster in the scientific communities who are ultimately responsible for vetting this work, right?
The ideal is: we don't accept papers at NeurIPS (or wherever) because of affiliation, and the badge of "quality" is bestowed by peer review, from other scientists.
> is of comparable quality (if not better than)
But the posts that you refer to in your own post point out several ways in which the quality is not good ...
But, in any case, I find it a bizarre choice to call it a paper when the authors themselves do not refer to it as such. We can agree to disagree, but just offering feedback.
Why would you want to change weights directly versus running through a new training cycle? For example, I have collected a bunch of data I would like to train weights that I use local inference for. I figured a LoRa would be the way to go, perhaps be more cost effective and generalized too.
Because text-to-weights is a learnable function. Everything else seems to be. So, why not? Why train weights if you could just prompt for them. If we managed to train image edit models, then why can't we train model-edit models?
I wouldn't be surprised at all if 3.8-27B could tweak 3.5-0.8B the same way Astra did, but it wouldn't be a big deal either because that part of the paper has nothing to do with directly tweaking the weights like the first part of the paper, but is instead a fairly straightforward abliteration method on a model that is known for a specific pathological behavior (ending up stuck in a thinking loop).
isn't consciousness just that, a thinking loop with interruptions for sleep until we break beyond repair? and, just maybe, if we managed to figure out the meaning of life, our thinking would satisfyingly stop?
What I want to see someone try and succeed at would be to get the really highly quantized models to loop less. Many people with only 16GB VRAM are running the IQ3_XXS or IQ3_S quants of Qwen3.8-27B. If it could figure out how to improve those quants, that'd actually be pretty substantial. That's all they did for those improved math results is removed the looping. Why not? Please someone. Please.
> It’s a cool paper, but some of the wider commentary I’ve seen around it does not feel justified. I’m not ready to declare that the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways.
The age of source code is definitely not going to end anytime soon if ever. But the future will involve incomprehensible machines editing other models in a "fairly certain" way, but we are going to need a lot of tooling to keep up with the process and not just take the model updates as "trust me bro".
I did exactly the same thing with Qwen 3.8 27B but i also asked it something else: a couple years ago i wrote a simple upscaler[0] when i was learning about neural networks. The network is dead simple: it takes a 25 inputs for a 5x5 grid around a pixel and outputs the 2x2 pixel grid that the input pixel should be replaced for a (width2)x(height2) upscale. I trained it on a bunch of random images (in hindsight i should have trained it on game screenshots but w/e) and it looked quite nice.
So i asked Qwen 3.8 27B two things:
1. What it is this network all about? I only gave it the ann.c code that implemented a simple artificial neural network library in C and the setup code that specified the network's shape but not its purpose.
Qwen 3.8 27B tried a bunch of stuff, taking the layers one by one, proding numbers through it, etc and finally figured out it is some sort of 5x5 to 2x2 filter, most likely working with pixels (i never mentioned pixels or graphics), though it wasn't certain about the exact purpose. It did rule out a bunch of uses like being a downscaler or classifier, from its behavior.
2. After i revealed what it was, i asked it if it can improve it but keep it 100% compatible with the existing program that used the weights file. And sure enough, it did improve it. I gave it the original training C code and dataset. It made a few experiments (and wrote its own training C code because it found mine too slow :-P) and ended up improving the weights. The difference[1] is small, but it basically eliminated some small noise that was always there and made some transitions less "halo-y". Of course for the purpose i wanted it (use it to upscale games in realtime) it isn't that great (too heavy) but it was learning material anyway. Though one bonus thing i noticed is that it upscales my random doodles much better than any Krita filter[2] :-P.
The above isn't Qwen 3.8 27B observing the weights directly and making sense out of them (ok actually it did seem to make some sense of the weights in the "#1" but i didn't dump the entire network to it - though thinking about it, it is only around 16384 floats so it might work). It just used a bunch of scripts it wrote in Python (and later C) but i still found it interesting to watch it go at it.
Not making any claims about a specific model's capabilities, but all these "can interpret model weights" posts brings to mind the line in Shakespeare's Henry IV
GLENDOWE: I can call spirits from the vasty deep.
HOTSPUR: Why, so can I, or so can any man, but will they come when you do call for them?
I can interpret weights too, but will that interpretation be correct?
And this reminds me of the king in the little prince, who had the power to order the sun to rise or go down. But only once a day at a certain time.
They did test for correct interpretation, so what point are you making?
> I’m not ready to declare that the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways.
That sounds a bit like biology; maybe AI will become like that.
Superior as mammals are to plants, so may AI become to us mammals.
Nope squishy is where it's at.
[dead]
Hopefully we get a new benchmark: How well can a model tweak a set of weights to produce pelican on a bike?
Why do people keep calling this a paper?
It's a blog post which received feedback from collaborators.
It hasn't gone through the academic paper lifecycle ... the authors themselves refer to it as a post and as preliminary.
I feel like I'm taking crazy pills, but can we be precise when talking about work of this form? It started with referring to arXiv pre-prints as papers, and now we're at blog posts with an academic template as papers.
A paper is a fucking paper: something which has been vetted by the scientific community, with at least some form (often, a few rounds) of anonymous review from people outside your circle of collaborators
This is not there yet, and clearly so due to the various feedback from people (including the two pushback posts on HN) critiquing the lack of discussion about the existence of these techniques in mech interp, etc?
Like you said, in the AI space a pre-print on arXiv is massively more likely than anything that's been through peer-review.
I called it a paper here because it's by people with an academic affiliation and is of comparable quality (if not better than) many of the other "papers" we see.
I also like it when a paper isn't a PDF. PDFs are a bad way of communicating information.
(I will note that the authors themselves didn't call it a paper, their own BibTeX reference calls it a "blog post":)
Sure, I agree with point (3).
But for (1, 2), just to point out, neither of your criteria would pass muster in the scientific communities who are ultimately responsible for vetting this work, right?
The ideal is: we don't accept papers at NeurIPS (or wherever) because of affiliation, and the badge of "quality" is bestowed by peer review, from other scientists.
> is of comparable quality (if not better than)
But the posts that you refer to in your own post point out several ways in which the quality is not good ...
But, in any case, I find it a bizarre choice to call it a paper when the authors themselves do not refer to it as such. We can agree to disagree, but just offering feedback.
It's not considered a paper until it's vetted by others?
Yes: before that stage, it is often called a "pre-print" ... if it is not at the state of a "pre-print", then "blog post" ... :)
In most fields of science, it's not a paper until it's accepted for publication in a journal. That normally includes peer review of some sort.
Why would you want to change weights directly versus running through a new training cycle? For example, I have collected a bunch of data I would like to train weights that I use local inference for. I figured a LoRa would be the way to go, perhaps be more cost effective and generalized too.
[delayed]
Because text-to-weights is a learnable function. Everything else seems to be. So, why not? Why train weights if you could just prompt for them. If we managed to train image edit models, then why can't we train model-edit models?
Something someone else brought up in one of the previous threads was to identify the refusal routes and remove them.
I wonder how far this can be taken. Could Qwen 3.8 can interpret and make improvements to Qwen 3.5 0.8B weights, or even its own weights?
It can, as long as you can give it benchmarks. What it can't do is optimize general intelligence without any external validation.
You can optimize a model by randomly changing its weights and validating results on your task. A monkey can type a book given enough attempts.
I wouldn't be surprised at all if 3.8-27B could tweak 3.5-0.8B the same way Astra did, but it wouldn't be a big deal either because that part of the paper has nothing to do with directly tweaking the weights like the first part of the paper, but is instead a fairly straightforward abliteration method on a model that is known for a specific pathological behavior (ending up stuck in a thinking loop).
isn't consciousness just that, a thinking loop with interruptions for sleep until we break beyond repair? and, just maybe, if we managed to figure out the meaning of life, our thinking would satisfyingly stop?
What I want to see someone try and succeed at would be to get the really highly quantized models to loop less. Many people with only 16GB VRAM are running the IQ3_XXS or IQ3_S quants of Qwen3.8-27B. If it could figure out how to improve those quants, that'd actually be pretty substantial. That's all they did for those improved math results is removed the looping. Why not? Please someone. Please.
Is RSI here? No.
Still amazing to me that a model that good, Qwen3.8:27b, runs on a laptop.
Wasn’t this one of the main marketing points for the Inkling model as well?
> It’s a cool paper, but some of the wider commentary I’ve seen around it does not feel justified. I’m not ready to declare that the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways.
The age of source code is definitely not going to end anytime soon if ever. But the future will involve incomprehensible machines editing other models in a "fairly certain" way, but we are going to need a lot of tooling to keep up with the process and not just take the model updates as "trust me bro".
I did exactly the same thing with Qwen 3.8 27B but i also asked it something else: a couple years ago i wrote a simple upscaler[0] when i was learning about neural networks. The network is dead simple: it takes a 25 inputs for a 5x5 grid around a pixel and outputs the 2x2 pixel grid that the input pixel should be replaced for a (width2)x(height2) upscale. I trained it on a bunch of random images (in hindsight i should have trained it on game screenshots but w/e) and it looked quite nice.
So i asked Qwen 3.8 27B two things:
1. What it is this network all about? I only gave it the ann.c code that implemented a simple artificial neural network library in C and the setup code that specified the network's shape but not its purpose.
Qwen 3.8 27B tried a bunch of stuff, taking the layers one by one, proding numbers through it, etc and finally figured out it is some sort of 5x5 to 2x2 filter, most likely working with pixels (i never mentioned pixels or graphics), though it wasn't certain about the exact purpose. It did rule out a bunch of uses like being a downscaler or classifier, from its behavior.
2. After i revealed what it was, i asked it if it can improve it but keep it 100% compatible with the existing program that used the weights file. And sure enough, it did improve it. I gave it the original training C code and dataset. It made a few experiments (and wrote its own training C code because it found mine too slow :-P) and ended up improving the weights. The difference[1] is small, but it basically eliminated some small noise that was always there and made some transitions less "halo-y". Of course for the purpose i wanted it (use it to upscale games in realtime) it isn't that great (too heavy) but it was learning material anyway. Though one bonus thing i noticed is that it upscales my random doodles much better than any Krita filter[2] :-P.
The above isn't Qwen 3.8 27B observing the weights directly and making sense out of them (ok actually it did seem to make some sense of the weights in the "#1" but i didn't dump the entire network to it - though thinking about it, it is only around 16384 floats so it might work). It just used a bunch of scripts it wrote in Python (and later C) but i still found it interesting to watch it go at it.
[0] https://mastodon.gamedev.place/@badsector/113765818432603610
[1] http://runtimeterror.com/pages/iv/images/a3d94ea369ff95ecf05...
[2] http://runtimeterror.com/pages/iv/images/4e29d027a8aa6e89db5...
Which coding agent harness did you use with Qwen for that?