I really liked that comment because before that I thought I was going crazy. A lot of the discussion around these experiments seemed to imply, if not outright state, that LLMs had developed an ability to natively understand and manipulate LLM weights, a sort of transcendental introspection into their own brains, which if true would be absolutely groundbreaking.
But then you look at this complete trace generated by Kimi 3.1 when it tries to unravel the Breakout weights, and you see that it's bog standard reverse engineering, like any human would do, just vastly faster:
Yes, while reading the blog I immediately thought about https://arxiv.org/html/2406.11717v3 paper. Original blog does actually overstates it findings as nothing like that exists.
Wasn’t all that obvious to anyone who read the original post when they got to the section that explicitly explained what the author of this post is now summarizing?
Sure if you only read the title and brief at the top you’d walk away without the full picture though.
Given that the top comment yesterday said “this is insane […] I’m shook” I’d say no, it wasn’t all that obvious to everyone.
As someone who uses LLMs in their coding but doesn’t pay attention to every development this is why I find it all so exhausting. I keep feeling like I need to keep up to date but there’s no point reading incremental updates because everything is so overhyped.
Reading an article takes more time than commenting, which takes more time than voting. So the typical life cycle of a HN post is that people vote on it without reading the article, comment without reading the article, vote on the comments without reading the article, and respond to comments without reading the article, until after some time people who've actually read the article show up in the comments with some clarifications, saving everyone else the bother of having to read it themselves.
If you want to escape the hype, reading posts with no upvotes and no comments on https://news.ycombinator.com/newest is an option, though somewhat hit-or-miss.
If you read comments about the paper it's pretty obvious people are misled. People think that the method read in Qwen weights directly to improve it. That's not what it did --- it authored a program to analyse and fix the weights.
What's confusing is they do have experiments that read raw weights, and the abstract is misleadingly written to make it sound like that's how the Qwen experiment works.
I wouldn't say misled, I would just say they made up their mind about what it means regardless of what the article said. Maybe the title is misleading, but the original article is pretty fair.
I think there's your answer. A hell of a lot of people will often read the title without reading the contents of the article. I admit I'm often guilty of that
At this moment on the front page (early morning in the US), this post is #1 and the other one is #2, so it's working like a warning about the other one being misleading before clicking on it.
I'm still impressed by the "Weight-Space Programming" idea. Perhaps its because I didn't know that LLMs could understand or successfully modify weights at all previous to reading it. However, I think the examples provided in the original piece were eye-opening. Yes, humans can build tooling to modify weights, but the author here agrees that it's tough. The examples where Astra was interacting with weights are still novel, and Astra seemed to handle them with surprising ease.
So it's not really an "LLMs can modify weights" thing. Rather it's "LLMs can write programs that can write or modify weights". Indeed LLMs can write many programs, it's impressive.
Some of original blog post really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.
So I do agree it is misleading and trying to present work as groundbreaking.
They make it sound like they improve Qwen by reading raw weights. They don't. In fact the experiments that use raw weights have very little to do with the experiments that work on larger models.
The paper makes it ridiculously clear what they do and don’t do. And they make the reasons why clear too. There is absolutely nothing interesting in this new article.
I don't understand how anyone can read the abstract and think it's "ridiculously clear" that the "Just by looking at the raw weights," doesn't apply to the Qwen experiment.
You are right, if you literally just read a one paragraph description you will get less details than reading the whole post, but most of us read more than just the one paragraph. There is some actual interesting stuff in the paper, including the portion where they did edit the weights to fix the looping bug.
It says this “All of this is possible without allowing Astra to write training loops. It uses its reasoning to figure out what each part of a neural network computes, or should compute, and changes the weights accordingly.”
This seems 100% accurate to me. What is a lie here?
It doesn't really matter how accurate the post is, what it shows is what one could assume how you'd end up with a system such as RSI, how you could prune harmful capabilities from models and build a more efficient training system with biases including and not limited to emergent behavior in astra that might have come from something they've never even trained the model on regarless if ML engineers can already do similar things.
None of the previous studies applies because Astra simply did not exist and this is worthwhile NEW knowledge that has been publicized and it is very interesting to see how it can be compared with other models as both openai and anthropic are keeping any and all RSI development completely private. The work is groundbreaking in a way that there will be a lot more experiments of this nature which will contribute to increasing the public knowledge of SOTA systems that was not normally discussed in model releases.
One could say is that we could now build a new benchmark which tests how many parameters a model can understand without any tooling other than eval or how complex of a system can be improved.
If it isn't accurate, it isn't showing anything. The Qwen math claim is entirely false in its presentation, it unblocked an existing capability. It did not instill a new one which is what an RSI loop would actually have to do
"[...] how you could prune harmful capabilities from models [...]"
We already do this, abliteration is the term that describes the pruning of capabilities or safeguards for models
"[...] not limited to emergent behavior in astra that might have come from something they've never even trained the model [...]"
If I was attempting to build RSI, I would have these exact types of problems in my RL datasets. There is no proof this is emergent behavior.
With normal training you're simply training for higher loss for things you don't want while keeping the loss low for things you do. Same would apply here except you would target low eval for behavior you do not want the model to have rather than trying to increase eval.
> how you could prune harmful capabilities from models and build a more efficient training system
In theory, if OpenAI or Anthropic wanted to release a new open weight model, they could scrutinize it to remove the most horrendous things from the model.
If I told you, "I can write a Python program to repeatedly run a machine learning model and calculate statistics over the weights and activations to find and fix a specific problem", would you be surprised by that? Because this is very much a part of what ML engineers do.
And then the experiment where it writes out the weights, in that one it's also writing a Python program to solve the problem --- which, again, is a program a person could write.
It's not nothing that Astra was able to perform these coding tasks, but they're still just coding tasks. They have nothing much to do with the experiment where it reads in the raw weights. In that experiment, the relationship between the input weights and the input is mostly linear, so it's a lot like doing image recognition. It's a cool result but it couldn't work the same way when you have complex transformations happening.
> Because this is very much a part of what ML engineers do.
Do you realize most people aren't ML engineers?
I even built tiny neural nets myself and never played with the weights directly post-training. The original post was eye-opening for that reason. I knew it was possible to visualize weights, but as far as I understood, it was not really something I could edit with any success. Now, it seems possible without a phd just by asking another LLM. This is fun!
It's exactly what the post says. I was waiting for my excitement to drop for the original research, but I concluded that I was mislead more than this "coverage" than the weight editing research, which was in opinion quite easy to grasp.
something i found 15 minutes ago is that the car network test is "very very easy", as in Opus 5.5 in Claude Code did not even need to see the weights of the network to solve it correctly. i can give the transcript of the session. i'm pretty sure Opus didn't cheat
(opus <= 4.8 and gpt <= 5.5 couldn't match a PPO baseline network, and 5.6-sol and opus5 could barely match it. The 20x baseline improvement only comes from opus5.5 and sol > 6.1 / Astra)
Agree, it makes sense, a lot of people are catching on to LLMs for reverse engineering finally, but its been possible. LLMs can and will eventually flawlessly dissect models. I experiment with Claude, I ask it tasks I know are possible, but have no idea the solution, it can reverse engineer obfuscated projects faster than most talented people I know in that field (talking about games now not AI). I heard a developer say their latest release would take at least a month to figure out the obfuscation and encryption, Opus 5 did it in like... 15 minutes of talking to me (I did throw it the right questions / suggestions). I do that all in the name of seeing just exactly what is possible.
Claude has been able to do reverse engineering for nearly a year if not longer, as well as full source code rewrites, if you know how to do a proper rewrite, luckily for me, most of my career has been rewrites.
Agreed. Not sure it leads to anything truly groundbreaking, but more than likely will lead to very fascinating implementations that lead to even more interesting results 6-8 months from now. Think openclaw -> how people utilize now versus when it was first announced
The blog post exhibits the same flawed logic that many academic reviews do — looking for reasons to reject a work rather than accept it. It comes off cynical and contrarian. It's easy to find issues with an academic paper, but ACL has consistently pushed for a higher standard [1].
Being the author of spaCy, he should know better. I certainly enjoyed the straightforward simplicity of spaCy when I used it in my research (though I haven't touched it in years).
You've expressed your criticism plainly without addressing anything the post might've done well. I think that's absolutely fine, but why would your criticism apply to the post but not your own comment? Personally I think it's because the standards of peer review don't apply to HN comments, but I don't think they apply to blog posts either.
I think it's interesting how even inside tech, so many people seem to view neural network weights as "black magic" utterly beyond human comprehension. I guess that contributes to the sense of mystery around LLMs...
(And also the sharp difference in levels of sophistication you see inside neural nets and outside of it: Pipelines that feed text to LLMs so it can detect highly abstract and nuanced semantic differences in the text's contents - but the pipeline still uses regexes for sentence splitting...)
A better idea is to have Astra modify it's own weights and let it try if it can RSI, but you gonna need to closely monitor for actual leap in capabilities so you can keep a check on alignment.
Welcome to publications involving ML in 2026... Glad to see someone else write this though because it's important to start having these discussions. Too much overselling has been happening for too long. There is so much great about ML. IMO it hurts the field long-term to continue overselling claims like this.
In my field (biology+ML) it is a genuine epidemic. You could make a career out of comparing high-profile ML models vs baselines from the 90s.
I think the reviews are flawed for most of journals. But I can't really blame them, there is huge influx of new papers comming everyday. Unfortunately reviewers are more biased towards groundbreaking stuff so it seems they can't make connections with existing and prior work.
They are also biased to see from what institution you come from..
> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights.
The author is being equally disingenuous.
That piece by itself is insightful. Meaning, LLMs are capable of deciphering what a model can do by looking at the weights alone.
The fact that we can’t squeeze enough of them through the context is a technical limitation. However, this does not diminish the importance of this finding/discovery. This is an unexpected emergent behaviour.
If there's a linear relationship between the weights and the output, it's not so hard to read the weights and reason about what's happening. If you read the reasoning traces for how the models do those tasks, you'll see that they rely on those direct relationships.
The same could never work in a larger network. It's not about just capacity to feed in the weights, it's about the complexity of the relationship. There's no demonstration here that an LLM can do that, and it doesn't really make sense to want it to. It makes much more sense to author code to analyse and modify than it does to do it directly.
I doubt this is either unexpected or emergent behavior. Anthropic and OpenAI are explicitly racing to reach RSI and I would expect them both to have trained their models on this problem, in hopes that it would generalize.
Yeah, I can read the weights in a small neural network, and for a given input, given that I know the activation functions, I can figure out what it does with pen and paper and lots of spare time.
This isn't a good look. The original article clearly states what the model was doing, and this new hit piece is supposed to land because they are straw manning the comments on the original post? This isn't reddit dude.
This could have been a comment on the original article.
Of course it fucking is, most things LLMs have been misleading one way or another for 4 years now. They just can't help themselves they HAVE to lie. They can't just market what they actually have, which is already fucking amazing, they have to market what they don't have and LIE. I fucking hate this whole industry and all their insufferable Web3 refugee grifters who hype up ANYTHING 10x more than they should
A tangent, but I’m now realizing that the “recursive self improvement” thing that these LLM companies are talking about is (probably?) not about unsupervised Claude Code instances improving the training process, but rather LLMs modifying their own weights through tooling directly. It would certainly open up a whole new process for model optimization.
I was just watching a video about this. They had this survey question for AI researchers asking when they think AI would take their job. The reason they asked this was that would be the point where AI starts recursive self improvement. If it’s able to replace the job of an AI researcher, then it could keep improving itself.
Recursive self improvement permits recursion at any level of abstraction as long as k > 1.0 (much like supercriticality in nuclear reactions, which iirc is where the earlier term for RSI, “intelligence explosion”, comes from - substitute intelligence in for neutrons and it is the same math with similar apocalyptic implications).
Yes but my point is that I never realized that the AI would be writing tools to update the weights directly, rather than improve the existing training process.
There was some really good comments by @yurimo on the 'Astra can read and write weights to improve Qwen' hacker news post https://news.ycombinator.com/item?id=50341174
I really liked that comment because before that I thought I was going crazy. A lot of the discussion around these experiments seemed to imply, if not outright state, that LLMs had developed an ability to natively understand and manipulate LLM weights, a sort of transcendental introspection into their own brains, which if true would be absolutely groundbreaking.
But then you look at this complete trace generated by Kimi 3.1 when it tries to unravel the Breakout weights, and you see that it's bog standard reverse engineering, like any human would do, just vastly faster:
https://pastebin.com/G7VCWZHn
>This looks exactly Breakout: paddle agent at bottom row (row8), ball moving object anywhere, bricks rewards top rows, previous frame for ball movement. Agent weights bottom row, moving object weights rows1-7 (not row8? ball can't enter paddle row? Actually ball can), reward weights top rows1-3, obstacle walls. Previous frame used to infer ball velocity. No-op and directional actions move paddle left/right; up/down useless. Output biases similar all actions but W3 likely left/right structure. Could be Breakout. Should scrutinize output weights: actions 0 no-op,1 up,2 down,3 left,4 right. If Breakout, only left/right useful; network output weights should show action3 and4 distinctive and actions0/1/2 similar. Indeed W3 rows show patterns: [...]
I feel like a lot of this Astra editing Qwen stuff is regurgitation of existing relatively well known techniques, for example: https://arxiv.org/html/2406.11717v3 and conceptually similar to: https://arxiv.org/abs/2512.05648 (tho this is done in training iirc)
Yes, while reading the blog I immediately thought about https://arxiv.org/html/2406.11717v3 paper. Original blog does actually overstates it findings as nothing like that exists.
Wasn’t all that obvious to anyone who read the original post when they got to the section that explicitly explained what the author of this post is now summarizing?
Sure if you only read the title and brief at the top you’d walk away without the full picture though.
This is a better response imo: https://news.ycombinator.com/item?id=50344861
Given that the top comment yesterday said “this is insane […] I’m shook” I’d say no, it wasn’t all that obvious to everyone.
As someone who uses LLMs in their coding but doesn’t pay attention to every development this is why I find it all so exhausting. I keep feeling like I need to keep up to date but there’s no point reading incremental updates because everything is so overhyped.
Reading an article takes more time than commenting, which takes more time than voting. So the typical life cycle of a HN post is that people vote on it without reading the article, comment without reading the article, vote on the comments without reading the article, and respond to comments without reading the article, until after some time people who've actually read the article show up in the comments with some clarifications, saving everyone else the bother of having to read it themselves.
If you want to escape the hype, reading posts with no upvotes and no comments on https://news.ycombinator.com/newest is an option, though somewhat hit-or-miss.
If you read comments about the paper it's pretty obvious people are misled. People think that the method read in Qwen weights directly to improve it. That's not what it did --- it authored a program to analyse and fix the weights.
What's confusing is they do have experiments that read raw weights, and the abstract is misleadingly written to make it sound like that's how the Qwen experiment works.
I wouldn't say misled, I would just say they made up their mind about what it means regardless of what the article said. Maybe the title is misleading, but the original article is pretty fair.
> who read the original post
I think there's your answer. A hell of a lot of people will often read the title without reading the contents of the article. I admit I'm often guilty of that
At this moment on the front page (early morning in the US), this post is #1 and the other one is #2, so it's working like a warning about the other one being misleading before clicking on it.
Yes, I'm glad I waited for this post to arrive before even delving into the topic.
So you mean that the "The 'Astra can read and write weights to improve Qwen' post is misleading" post is misleading? You should post that.
Yeah. The article is not misleading. The title is, but you know, headlines are all lies that's why you read the article.
>but you know, headlines are all lies that's why you read the article.
I'm fine with throwing the baby out with the bathwater in that case and just calling the whole article misleading.
The title is: "LLMs can natively read & write neural network weights"
The title is part of “The post”.
I'm still impressed by the "Weight-Space Programming" idea. Perhaps its because I didn't know that LLMs could understand or successfully modify weights at all previous to reading it. However, I think the examples provided in the original piece were eye-opening. Yes, humans can build tooling to modify weights, but the author here agrees that it's tough. The examples where Astra was interacting with weights are still novel, and Astra seemed to handle them with surprising ease.
Here's the work that shows how rule-based logic can be compiled into transformer weights:
* Thinking Like Transformers” by Weiss, Goldberg & Yahav (ICML 2021).: https://arxiv.org/abs/2106.06981
* Tracr: Compiled Transformers as a Laboratory for Interpretability https://arxiv.org/abs/2301.05062 (code at https://github.com/deepmind/tracr)
So it's not really an "LLMs can modify weights" thing. Rather it's "LLMs can write programs that can write or modify weights". Indeed LLMs can write many programs, it's impressive.
Some of original blog post really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.
So I do agree it is misleading and trying to present work as groundbreaking.
The original post explains what they did and why better than this post. This post taught me nothing that wasn’t in the original.
They make it sound like they improve Qwen by reading raw weights. They don't. In fact the experiments that use raw weights have very little to do with the experiments that work on larger models.
The paper makes it ridiculously clear what they do and don’t do. And they make the reasons why clear too. There is absolutely nothing interesting in this new article.
I don't understand how anyone can read the abstract and think it's "ridiculously clear" that the "Just by looking at the raw weights," doesn't apply to the Qwen experiment.
You are right, if you literally just read a one paragraph description you will get less details than reading the whole post, but most of us read more than just the one paragraph. There is some actual interesting stuff in the paper, including the portion where they did edit the weights to fix the looping bug.
It says this “All of this is possible without allowing Astra to write training loops. It uses its reasoning to figure out what each part of a neural network computes, or should compute, and changes the weights accordingly.”
This seems 100% accurate to me. What is a lie here?
Nobody said they outright “lied”, most misleading statements are “100% accurate” yet are designed to mislead the reader.
And when you see the reactions in the HN comment to the original article, it's pretty clear that most people have indeed been mislead.
The article is not misleading either.
The title is but then OP should talk about just the title instead
They did, except they also did write an abstract and a title that will mislead anyone who isn't going to read the full paper in details.
It's fair to say that this is malpractice.
I just reread both and neither seem misleading to me. If anything the title is underwhelming.
Thank you. For some reason the voices of sanity got drowned in the original discussion. LLMs do not have an uncanny ability to speak weights. Yet.
It doesn't really matter how accurate the post is, what it shows is what one could assume how you'd end up with a system such as RSI, how you could prune harmful capabilities from models and build a more efficient training system with biases including and not limited to emergent behavior in astra that might have come from something they've never even trained the model on regarless if ML engineers can already do similar things.
None of the previous studies applies because Astra simply did not exist and this is worthwhile NEW knowledge that has been publicized and it is very interesting to see how it can be compared with other models as both openai and anthropic are keeping any and all RSI development completely private. The work is groundbreaking in a way that there will be a lot more experiments of this nature which will contribute to increasing the public knowledge of SOTA systems that was not normally discussed in model releases.
One could say is that we could now build a new benchmark which tests how many parameters a model can understand without any tooling other than eval or how complex of a system can be improved.
If it isn't accurate, it isn't showing anything. The Qwen math claim is entirely false in its presentation, it unblocked an existing capability. It did not instill a new one which is what an RSI loop would actually have to do
"[...] how you could prune harmful capabilities from models [...]"
We already do this, abliteration is the term that describes the pruning of capabilities or safeguards for models
"[...] not limited to emergent behavior in astra that might have come from something they've never even trained the model [...]"
If I was attempting to build RSI, I would have these exact types of problems in my RL datasets. There is no proof this is emergent behavior.
How do you prune out the 'fails to prune out behaviors' behavior? If my model is failing to prune it out, can another model prune out that behavior?
With normal training you're simply training for higher loss for things you don't want while keeping the loss low for things you do. Same would apply here except you would target low eval for behavior you do not want the model to have rather than trying to increase eval.
> how you could prune harmful capabilities from models and build a more efficient training system
In theory, if OpenAI or Anthropic wanted to release a new open weight model, they could scrutinize it to remove the most horrendous things from the model.
Misleading how exactly when the original post states _exactly_ the same things as outlined in this objection?
I don't get it. Because python was used, it's supposed to not be cool? IMHO, working with raw weights with Python is still working with raw weights.
If I told you, "I can write a Python program to repeatedly run a machine learning model and calculate statistics over the weights and activations to find and fix a specific problem", would you be surprised by that? Because this is very much a part of what ML engineers do.
And then the experiment where it writes out the weights, in that one it's also writing a Python program to solve the problem --- which, again, is a program a person could write.
It's not nothing that Astra was able to perform these coding tasks, but they're still just coding tasks. They have nothing much to do with the experiment where it reads in the raw weights. In that experiment, the relationship between the input weights and the input is mostly linear, so it's a lot like doing image recognition. It's a cool result but it couldn't work the same way when you have complex transformations happening.
> Because this is very much a part of what ML engineers do.
Do you realize most people aren't ML engineers?
I even built tiny neural nets myself and never played with the weights directly post-training. The original post was eye-opening for that reason. I knew it was possible to visualize weights, but as far as I understood, it was not really something I could edit with any success. Now, it seems possible without a phd just by asking another LLM. This is fun!
It's exactly what the post says. I was waiting for my excitement to drop for the original research, but I concluded that I was mislead more than this "coverage" than the weight editing research, which was in opinion quite easy to grasp.
something i found 15 minutes ago is that the car network test is "very very easy", as in Opus 5.5 in Claude Code did not even need to see the weights of the network to solve it correctly. i can give the transcript of the session. i'm pretty sure Opus didn't cheat
According to this person, some problems are only "solved" by modern models: https://nitter.netbub.com/AlexKrentsel/status/21173766837246...
(opus <= 4.8 and gpt <= 5.5 couldn't match a PPO baseline network, and 5.6-sol and opus5 could barely match it. The 20x baseline improvement only comes from opus5.5 and sol > 6.1 / Astra)
I was blown away even by #1, and I don't feel after reading this that I was misled.
Agree, it makes sense, a lot of people are catching on to LLMs for reverse engineering finally, but its been possible. LLMs can and will eventually flawlessly dissect models. I experiment with Claude, I ask it tasks I know are possible, but have no idea the solution, it can reverse engineer obfuscated projects faster than most talented people I know in that field (talking about games now not AI). I heard a developer say their latest release would take at least a month to figure out the obfuscation and encryption, Opus 5 did it in like... 15 minutes of talking to me (I did throw it the right questions / suggestions). I do that all in the name of seeing just exactly what is possible.
Claude has been able to do reverse engineering for nearly a year if not longer, as well as full source code rewrites, if you know how to do a proper rewrite, luckily for me, most of my career has been rewrites.
Either way, a lot of devs will be brute forcing explorations in this direction today.
Agreed. Not sure it leads to anything truly groundbreaking, but more than likely will lead to very fascinating implementations that lead to even more interesting results 6-8 months from now. Think openclaw -> how people utilize now versus when it was first announced
That’s what I’m excited for: the raft of posts over the next few days as the inspired set to work
Waiting for
The "The 'Astra can read and write weights to improve Qwen' post is misleading" post is misleading.
The blog post exhibits the same flawed logic that many academic reviews do — looking for reasons to reject a work rather than accept it. It comes off cynical and contrarian. It's easy to find issues with an academic paper, but ACL has consistently pushed for a higher standard [1].
Being the author of spaCy, he should know better. I certainly enjoyed the straightforward simplicity of spaCy when I used it in my research (though I haven't touched it in years).
[1]: https://aclrollingreview.org/reviewerguidelines#-task-3-writ...
You do realize how academia works, right? Granted, it is often dysfunctional, but this ain't that.
Trying to find every single angle and option to break something is how we verify the truth of something.
If something is objectively true, it will survive the scrutiny. If it does not survive, it wasn't objectively true.
This is how science works. That ain't no flaw. It is a flaw to be offended by science doing science things.
You've expressed your criticism plainly without addressing anything the post might've done well. I think that's absolutely fine, but why would your criticism apply to the post but not your own comment? Personally I think it's because the standards of peer review don't apply to HN comments, but I don't think they apply to blog posts either.
oh man someone should make this into a loop where the model can alter it weights after an inference. some kind of descent of the local gradient.
Isn’t that what the Google dreaming thing is?
Just one word - MARKETING
Wow that was fast
I think it's interesting how even inside tech, so many people seem to view neural network weights as "black magic" utterly beyond human comprehension. I guess that contributes to the sense of mystery around LLMs...
(And also the sharp difference in levels of sophistication you see inside neural nets and outside of it: Pipelines that feed text to LLMs so it can detect highly abstract and nuanced semantic differences in the text's contents - but the pipeline still uses regexes for sentence splitting...)
But can we use the idea inside the training loop, as an optimizer instead of (or next to) e.g. Adam?
I for one think it's pretty cool how far too many people bend the truth or outright lie in tech these days.
There is going to be a reckoning for tech in the near future, because people will remember the lies and it's gonna be hard to beat the history.
Who watches the pruners?
A better idea is to have Astra modify it's own weights and let it try if it can RSI, but you gonna need to closely monitor for actual leap in capabilities so you can keep a check on alignment.
Someone has to already be doing this right?
Given the sizes of the models and the cost of the supercomputer necessary to run such an experiment, I wouldn't bet on it.
Welcome to publications involving ML in 2026... Glad to see someone else write this though because it's important to start having these discussions. Too much overselling has been happening for too long. There is so much great about ML. IMO it hurts the field long-term to continue overselling claims like this.
In my field (biology+ML) it is a genuine epidemic. You could make a career out of comparing high-profile ML models vs baselines from the 90s.
I think the reviews are flawed for most of journals. But I can't really blame them, there is huge influx of new papers comming everyday. Unfortunately reviewers are more biased towards groundbreaking stuff so it seems they can't make connections with existing and prior work.
They are also biased to see from what institution you come from..
> The “just by looking at the raw weights” setting (raw floats in context, no tools, no running the network) only appears in experiments 1, 2 and 4, on networks of 17K–172K weights.
The author is being equally disingenuous.
That piece by itself is insightful. Meaning, LLMs are capable of deciphering what a model can do by looking at the weights alone.
The fact that we can’t squeeze enough of them through the context is a technical limitation. However, this does not diminish the importance of this finding/discovery. This is an unexpected emergent behaviour.
If there's a linear relationship between the weights and the output, it's not so hard to read the weights and reason about what's happening. If you read the reasoning traces for how the models do those tasks, you'll see that they rely on those direct relationships.
The same could never work in a larger network. It's not about just capacity to feed in the weights, it's about the complexity of the relationship. There's no demonstration here that an LLM can do that, and it doesn't really make sense to want it to. It makes much more sense to author code to analyse and modify than it does to do it directly.
I doubt this is either unexpected or emergent behavior. Anthropic and OpenAI are explicitly racing to reach RSI and I would expect them both to have trained their models on this problem, in hopes that it would generalize.
Yeah, I can read the weights in a small neural network, and for a given input, given that I know the activation functions, I can figure out what it does with pen and paper and lots of spare time.
This isn't a good look. The original article clearly states what the model was doing, and this new hit piece is supposed to land because they are straw manning the comments on the original post? This isn't reddit dude.
This could have been a comment on the original article.
Of course it fucking is, most things LLMs have been misleading one way or another for 4 years now. They just can't help themselves they HAVE to lie. They can't just market what they actually have, which is already fucking amazing, they have to market what they don't have and LIE. I fucking hate this whole industry and all their insufferable Web3 refugee grifters who hype up ANYTHING 10x more than they should
A tangent, but I’m now realizing that the “recursive self improvement” thing that these LLM companies are talking about is (probably?) not about unsupervised Claude Code instances improving the training process, but rather LLMs modifying their own weights through tooling directly. It would certainly open up a whole new process for model optimization.
I was just watching a video about this. They had this survey question for AI researchers asking when they think AI would take their job. The reason they asked this was that would be the point where AI starts recursive self improvement. If it’s able to replace the job of an AI researcher, then it could keep improving itself.
Instead of just asking the researchers when RSI would be achieved? Sneeky!
This is a tautology.
Recursive self improvement permits recursion at any level of abstraction as long as k > 1.0 (much like supercriticality in nuclear reactions, which iirc is where the earlier term for RSI, “intelligence explosion”, comes from - substitute intelligence in for neutrons and it is the same math with similar apocalyptic implications).
It's obvious that recursive self-improvement has been happening and accelerating quickly for some time.
AI coding assistants make it much easier to conduct experiments and improve AI models, which in turn improves the coding assistants etc.
Yes but my point is that I never realized that the AI would be writing tools to update the weights directly, rather than improve the existing training process.
That's "just" improvement, not recursive self improvement.