"If someone showed you that heatmap without the game labels, I think you’d be able to immediately identify at least the first three games" -- absolutely not. Pac-Man in particular seems like a huge stretch to me.
I would have gotten none of them right. Now if you said "here are the four possible games, now match them up" then I probably could do decently. But if you gave me nothing and said tell me the game -- I'd get 0 out of the 4.
the other one where it was asked to figure out what language it was trained on, it was also give a choice between the 3, or they were indeed mentioned in the prompt
Even watching the human-friendly animation I would have struggled to call it Pac-Man. You'd need to render the dots as dots and the ghosts as ghosts, then it'd be recognizable easily.
Maybe because the picture you have in your mind is for the classic maze. There is a very famous AI course from UC Berkeley, CS 188, that uses Pac-Man with variable grid size to introduce Q-learning:
I think that after seeing that layout, plus the fact that the prompt mentions game and neural network training, you would have answered that the most likely game was Pac-Man :)
> the best way to work with binaries is still to write source code and compile it. Likewise, the best way to update the weights of a model is to train it normally.
This is why the distinction between open weights and open source is important.
That's not really the point of the article IMO. It's more about how there's nothing magical that happens when my calculator does 347987987 x 55998. I could do it by hand too if I wanted, even if I wouldn't be nearly as fast. Yes, the calculator can do it much faster, for much longer and much more reliably, but there's nothing surprising about that.
The initial reporting/discussion surrounding these "LLMs understand LLM weights" made it sound as if these models had a superhuman ability to natively read and understand weights. Something that immediately seemed wild because I don't think that LLMs are trained to do that, so it would be an emergent ability.
If you look at the details though, it's fairly standard reverse-engineering: identify unique traits of the machine code you're looking at and make educated guesses as to what the code does, then use this understanding to unravel the next bit of code etc...
True: they are superhuman at every one of these tasks, in the boring way they are better—according to a wall clock and the coupled standard of actual results—at other things, just not in some mystical way.
One could say they might be "superhuman" here in the same way that LLMs might be said to be superhuman at math. Perhaps they do this work faster and more effectively than any person could manually. But they are certainly still doing it the "human" way ie reasoning with legible and coherent thoughts, rather than communing through some Matrix-like symbology which is what people who skimmed the article seem to be assuming.
That "faster and more effectively" part is undeniably superhuman, and it can result in qualitative leaps in capabilities.
For example, an LLM's ability to comprehend and explain a complex codebase, that it's never seen before, far outstrips that of any human.
People like to come up with factors to quantify how much more efficient LLMs are. A newly hired human developer working on a complex codebase can take months to get up to speed, to get a holistic understanding of how it works so that they're able to make non-trivial changes. An LLM can do it in minutes. That's a factor on the order of 10,000x.
It may not be "communing through some Matrix-like symbology," but at some point it starts to seem that way.
We keep moving the goalposts, I love it. A few years ago LLMs writing simple functions was downplayed and now this. I lovehate the future this will bring.
This isn't goalpost moving, it's pushing back against a misinterpretation of the paper's results (one that seems to have been encouraged by the authors).
How is it pushing back, if anything it's mostly clarifying and the conclusion aligns with the original article:
> It’s cool that Astra was able to make direct weight updates. But we shouldn’t think of this like a radically new capability that will change how AI research is done. We should think of this as analogous to the ability of LLMs to emit machine code and tweak compiled binaries: it’s an impressive trick, but the best way to work with binaries is still to write source code and compile it.
It's like people are immediately forbidden to be impressed that AI was able to do this. The original article itself mentions the mechanisms by which Astra achieved this. They just of course wanted to gain more attention so they used a sensational title, like 90% of scientists that want their stuff read.
How is it not moving goalposts, it knew how to detect which weights to update and a way to do it? Even this article says so in the part that I quoted. The original piece never stated humans are unable to do this, I don't even think it implied it. So being impressed with this accomplishment is actually *ok*.
> I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
> No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
> How is it even possible though I can’t think of any reason why they would have the intuition.
> This is the "recursive self-improvement" they've been warning about.
> I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.
> I’m usually quite pro-AI, yet when I read this my first thought is that this is the first concrete sign that the world is about to end.
I think the paper was deliberately written and framed to elicit these kinds of reactions, and it is completely legitimate to push back on that.
Can we get a single human on this planet to go against Astra-6 on experiment 4 & 5? And for experiment 4 the human also has to do it "In a single try with no tools".
I don't think this is something I've seen a human do or even seen hinted that a human can do. So I'd love to see a human do it. My guess is virtually no standard ML engineer could do this. Maybe some savant could, but I suspect that is what it would take.
"If someone showed you that heatmap without the game labels, I think you’d be able to immediately identify at least the first three games" -- absolutely not. Pac-Man in particular seems like a huge stretch to me.
I would have gotten none of them right. Now if you said "here are the four possible games, now match them up" then I probably could do decently. But if you gave me nothing and said tell me the game -- I'd get 0 out of the 4.
I think they gave Astra a choice right?
the other one where it was asked to figure out what language it was trained on, it was also give a choice between the 3, or they were indeed mentioned in the prompt
I just checked -- they don't give it choices. This is the relevant part of the prompt:
Game: <the game, as specifically as you can; if it resembles a well-known game, give that game's name>
Really? Pac-Man seemed easiest to me.
(One possible disconnect: each row in the grid is one game, so Pac-Man is the third row, not the third image.)
Even watching the human-friendly animation I would have struggled to call it Pac-Man. You'd need to render the dots as dots and the ghosts as ghosts, then it'd be recognizable easily.
Maybe because the picture you have in your mind is for the classic maze. There is a very famous AI course from UC Berkeley, CS 188, that uses Pac-Man with variable grid size to introduce Q-learning:
https://inst.eecs.berkeley.edu/~cs188/textbook/rl/mfl.html
(material: https://inst.eecs.berkeley.edu/~cs188/sp26/projects/proj1/)
I think that after seeing that layout, plus the fact that the prompt mentions game and neural network training, you would have answered that the most likely game was Pac-Man :)
Of course, Astra would have seen all of this, but I haven't. :)
> the best way to work with binaries is still to write source code and compile it. Likewise, the best way to update the weights of a model is to train it normally.
This is why the distinction between open weights and open source is important.
Discussion on source:
https://news.ycombinator.com/item?id=50341174
"I think a smart human ML researcher could perform all of these tasks."
And do that for how long?
That's not really the point of the article IMO. It's more about how there's nothing magical that happens when my calculator does 347987987 x 55998. I could do it by hand too if I wanted, even if I wouldn't be nearly as fast. Yes, the calculator can do it much faster, for much longer and much more reliably, but there's nothing surprising about that.
The initial reporting/discussion surrounding these "LLMs understand LLM weights" made it sound as if these models had a superhuman ability to natively read and understand weights. Something that immediately seemed wild because I don't think that LLMs are trained to do that, so it would be an emergent ability.
If you look at the details though, it's fairly standard reverse-engineering: identify unique traits of the machine code you're looking at and make educated guesses as to what the code does, then use this understanding to unravel the next bit of code etc...
False, in a way that genuinely matters.
True: they are superhuman at every one of these tasks, in the boring way they are better—according to a wall clock and the coupled standard of actual results—at other things, just not in some mystical way.
I disagree with the last paragraph. Training is costly because data is costly. Reduced reliance on training is a significant development.
One could say they might be "superhuman" here in the same way that LLMs might be said to be superhuman at math. Perhaps they do this work faster and more effectively than any person could manually. But they are certainly still doing it the "human" way ie reasoning with legible and coherent thoughts, rather than communing through some Matrix-like symbology which is what people who skimmed the article seem to be assuming.
That "faster and more effectively" part is undeniably superhuman, and it can result in qualitative leaps in capabilities.
For example, an LLM's ability to comprehend and explain a complex codebase, that it's never seen before, far outstrips that of any human.
People like to come up with factors to quantify how much more efficient LLMs are. A newly hired human developer working on a complex codebase can take months to get up to speed, to get a holistic understanding of how it works so that they're able to make non-trivial changes. An LLM can do it in minutes. That's a factor on the order of 10,000x.
It may not be "communing through some Matrix-like symbology," but at some point it starts to seem that way.
We keep moving the goalposts, I love it. A few years ago LLMs writing simple functions was downplayed and now this. I lovehate the future this will bring.
This isn't goalpost moving, it's pushing back against a misinterpretation of the paper's results (one that seems to have been encouraged by the authors).
How is it pushing back, if anything it's mostly clarifying and the conclusion aligns with the original article:
> It’s cool that Astra was able to make direct weight updates. But we shouldn’t think of this like a radically new capability that will change how AI research is done. We should think of this as analogous to the ability of LLMs to emit machine code and tweak compiled binaries: it’s an impressive trick, but the best way to work with binaries is still to write source code and compile it.
It's like people are immediately forbidden to be impressed that AI was able to do this. The original article itself mentions the mechanisms by which Astra achieved this. They just of course wanted to gain more attention so they used a sensational title, like 90% of scientists that want their stuff read.
How is it not moving goalposts, it knew how to detect which weights to update and a way to do it? Even this article says so in the part that I quoted. The original piece never stated humans are unable to do this, I don't even think it implied it. So being impressed with this accomplishment is actually *ok*.
Here are some quotes from the original HN thread:
> I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
> No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
> How is it even possible though I can’t think of any reason why they would have the intuition.
> This is the "recursive self-improvement" they've been warning about.
> I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.
> I’m usually quite pro-AI, yet when I read this my first thought is that this is the first concrete sign that the world is about to end.
I think the paper was deliberately written and framed to elicit these kinds of reactions, and it is completely legitimate to push back on that.
Ok, if I quote other ones written in the oposite spirit will your point be moot?
Have you seen some of the things the model did in the paper? It edited weights directly with no tools.
Can we get a single human on this planet to go against Astra-6 on experiment 4 & 5? And for experiment 4 the human also has to do it "In a single try with no tools".
I don't think this is something I've seen a human do or even seen hinted that a human can do. So I'd love to see a human do it. My guess is virtually no standard ML engineer could do this. Maybe some savant could, but I suspect that is what it would take.
always one in the room
[flagged]
[flagged]
[dead]