This feels to me more like incremental belief building than "thinking ahead of time" (which is not what the paper is claiming).
The model only has partial observability of the program it is working on (whatever tool call outputs are present in the context), as well as the trajectory of actions it has taken, and from this is building up some internal beliefs about the program - the probes used were looking for pretty crude things like "is this program well-formed" and "is this program correct (will it pass tests)".
The paper says that these program "properties" (beliefs) predict future state of the program up to 25 "steps" ahead, but given the setup this seems to be expected. An agent is trying to fix a program and/or maintain it in a working state, so it doesn't seem surprising that current well-formedness and correctness persist into the future, or that the model is correctly "optimistic" about the outcome of the next action it is planning/predicting.
This incremental belief building from partial observability reminds me of the ability of LLMs to predict valid chess moves when only given a truncated history of the games moves so far (e.g. last 20 moves, not all moves back to start of the game).
Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.
It possibly goes a little further, though, since we’re talking about multi-step agentic loops here. So in addition to next token prediction there are of iterations of collecting input from outside sources.
I’m still hesitant to interpret this as “thinking ahead” without at least seeing some more back-and-forth in the literature first, though. This just seems like one of those spots where it makes sense to give other researchers some time to come up with additional hypotheses to explain the observations instead of focusing on the first one anyone proposes in isolation.
This is all obvious and not news, but there's a lot of people doing fighting retreats against LLM intelligence where the degree of obviousness matters.
I think that's reading a little too much into it. The paper shows the hidden states contain signals about whether the code is good right now, and whether the run is probably going to work out. That's interesting, for sure. But it doesn't mean the model has some detailed idea of what it's going to write 25 steps later.
A lot of that signal could be much simpler stuff. This task is hard. The agent seems stuck. The tests are getting better. The current approach looks promising. All of those things make future success easier to predict without the model actually "knowing what comes next" in any strong sense.
Also, their 25 steps are agent turns, not 25 code edits. The median run had something like 52 steps but only two edits, and the program label stays the same between edits. So "25 steps ahead" may sometimes just mean basically the same codebase, with a bunch of reading and test output in between.
So yeah, I'd say it's consistent with Sutskever's view. But "consistent with" and "confirmatory of" are doing very different amounts of work here.
It's been known for several years that LLM activations encode future tokens ahead of time (e.g. https://arxiv.org/abs/2404.00859).
But this has only been shown on simple tasks, so I think this paper is still quite neat. The interesting thing is that they show "future horizon length" varies across models.
Thank you for sharing. The way I reasoned about it myself: to make better predictions, we should know what type of outcomes are likely. We can express these outcomes by doing computations in some of the layers, and the training signal adjusts them so our model becomes more correct.
Of course, an interesting question what part of this internal computation is modeling for the future compared to guessing based on the given context (the past).
This is true of truly skillful people in their craft as well.
I know people, who initialize all required variables and write the logic after. which used to feel bonkers to me until I realized, they've done enough practice and memorization to be able to figure what they would need 10 steps down the line.
this does show that, models have a better model of the task and the expected end state.
It makes intuitive sense. How else could you write a 500-line script top-to-bottom with no backspace key and no arrow keys and get all the imports etc. right upfront?
...by inferring both the imports and the script body from the same context? I think you're suggesting there's some kind of information flow from the anticipated body of the script back up to the imports, but I don't see why that would be necessary. Infer imports from context, infer body from context + imports. All strictly causal.
Sure, try it. It's harder than you think. It's not just imports, it's the entire program.
> you're suggesting there's some kind of information flow from the anticipated body of the script back up to the imports
Yes, I am suggesting this. I don't think it is possible to write programs without either anticipating what you're going to write down below before you get there, or else being able to go back and edit what you already wrote.
Of course agent harnesses allow the latter, but raw models without a harness can still do an exceptionally, superhumanly, good job of straight-line programming with no editing.
> Infer imports from context, infer body from context + imports. All strictly causal.
Of course it's causal, that's kind of a reductive way to look at it.
Just infer the entire program from context and then type what you inferred.
I used to ask my coding agent to present two alternatives to choose from while implementing a task, and include a tokens required for each alternative to implement. (So that I can choose one which needs less token vs one which needs more rigour depending on task)
Finally there is evidence that the model kinda actually knows the correct token spend on each method.
>> Probes trained to predict the outcome of future edits (before they are materialized and written on disk) achieve performance above chance up to roughly 25 steps in advance.
Are these probes effectively run in parallel? The way this reads is more about predicting a future outcome than keeping the current token relevant based on past tokens.
In other words, since the next semantic prediction for forecasting the future is built on the training dataset, it's hard for anything truly new to emerge.
Then how do humans create something 'creative'—something that didn't exist before? I think it might be because the process of simplifying the complex system of nature differs between individuals. The data being learned now is all labeled by humans and simplified through human cognition. Within that kind of information, creativity seems hard to emerge.
Ultimately, with data that already contains interpretation, no matter how much you repeat the learning, it just becomes an encyclopedia that only explores within human knowledge, repeating predictions within human interpretation. So I wonder if we actually need a different encoder that interprets raw data—not based on human interpretation.
In reality, what changed Newton's absolute time to Einstein's relativity was a conclusion derived simply from observing the world. Newton's interpretation was supported by a lot of evidence in its time. If an AI studied all the medieval data from Newton's era, could it actually come up with the theory of relativity?
I'm always curious about this. I think AI is already very good at coding and will soon become better than humans. Logical structures are ultimately human interpretations, and reasoning within that framework is something AI can probably do more logically than humans. In other words, once humans create the framework, stacking the logical Jenga blocks within it—AI will be better at that.
But true creativity lies in breaking the framework itself, and I'm skeptical about whether AI can do that. The encoder also seems insufficient. There will likely be limits. I might be trapped in my own biases.
But the limitations of the current approach seem too clear to ignore.
When I look at the approach of these papers, it feels like an argument that adding shadows that imitate the world will eventually make them become the objects themselves.
I think the text, code, images, papers, and conversations that humans leave behind are not the world itself, but rather shadows of the world that have passed through human cognition and language. No matter how much you learn from those shadows, whether that leads to the ability to actually engage with the objects themselves seems like a separate issue.
I feel like something different is needed. But I'm not intellectually sharp enough to reason this through logically.this is just my intuition
Why not? I think there’s fairly strong evidence that there is something that convincingly looks like reasoning. I think anthropic has done some nice circuit tracing and mechanistic interpretability work on this for instance.
This feels to me more like incremental belief building than "thinking ahead of time" (which is not what the paper is claiming).
The model only has partial observability of the program it is working on (whatever tool call outputs are present in the context), as well as the trajectory of actions it has taken, and from this is building up some internal beliefs about the program - the probes used were looking for pretty crude things like "is this program well-formed" and "is this program correct (will it pass tests)".
The paper says that these program "properties" (beliefs) predict future state of the program up to 25 "steps" ahead, but given the setup this seems to be expected. An agent is trying to fix a program and/or maintain it in a working state, so it doesn't seem surprising that current well-formedness and correctness persist into the future, or that the model is correctly "optimistic" about the outcome of the next action it is planning/predicting.
This incremental belief building from partial observability reminds me of the ability of LLMs to predict valid chess moves when only given a truncated history of the games moves so far (e.g. last 20 moves, not all moves back to start of the game).
Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.
Isn't that what "Attention is all you need" was about anyway? Does not sound like news to me.
It possibly goes a little further, though, since we’re talking about multi-step agentic loops here. So in addition to next token prediction there are of iterations of collecting input from outside sources.
I’m still hesitant to interpret this as “thinking ahead” without at least seeing some more back-and-forth in the literature first, though. This just seems like one of those spots where it makes sense to give other researchers some time to come up with additional hypotheses to explain the observations instead of focusing on the first one anyone proposes in isolation.
This is all obvious and not news, but there's a lot of people doing fighting retreats against LLM intelligence where the degree of obviousness matters.
> To effectively predict the next token it needs a good idea of what comes after the next token.
And that's all it needs. Not reasoning.
At some level of performance reasoning becomes the most effective method to predict the next token
[delayed]
Save us from the reasoning / sentience / consciousness / thinking semantic quicksand.
Babbage’s Analytical Engine didn’t actually analyze anything, and terminology hadn’t gotten any more clear-cut since.
What’s the difference?
How do you define reasoning in a measurable way?
I keep asking the same question, and I think the steelman version would be "has metacognitive patterns similar to humans"
I think that's reading a little too much into it. The paper shows the hidden states contain signals about whether the code is good right now, and whether the run is probably going to work out. That's interesting, for sure. But it doesn't mean the model has some detailed idea of what it's going to write 25 steps later.
A lot of that signal could be much simpler stuff. This task is hard. The agent seems stuck. The tests are getting better. The current approach looks promising. All of those things make future success easier to predict without the model actually "knowing what comes next" in any strong sense.
Also, their 25 steps are agent turns, not 25 code edits. The median run had something like 52 steps but only two edits, and the program label stays the same between edits. So "25 steps ahead" may sometimes just mean basically the same codebase, with a bunch of reading and test output in between.
So yeah, I'd say it's consistent with Sutskever's view. But "consistent with" and "confirmatory of" are doing very different amounts of work here.
It's been known for several years that LLM activations encode future tokens ahead of time (e.g. https://arxiv.org/abs/2404.00859).
But this has only been shown on simple tasks, so I think this paper is still quite neat. The interesting thing is that they show "future horizon length" varies across models.
Thank you for sharing. The way I reasoned about it myself: to make better predictions, we should know what type of outcomes are likely. We can express these outcomes by doing computations in some of the layers, and the training signal adjusts them so our model becomes more correct.
Of course, an interesting question what part of this internal computation is modeling for the future compared to guessing based on the given context (the past).
This is true of truly skillful people in their craft as well.
I know people, who initialize all required variables and write the logic after. which used to feel bonkers to me until I realized, they've done enough practice and memorization to be able to figure what they would need 10 steps down the line.
this does show that, models have a better model of the task and the expected end state.
It makes intuitive sense. How else could you write a 500-line script top-to-bottom with no backspace key and no arrow keys and get all the imports etc. right upfront?
...by inferring both the imports and the script body from the same context? I think you're suggesting there's some kind of information flow from the anticipated body of the script back up to the imports, but I don't see why that would be necessary. Infer imports from context, infer body from context + imports. All strictly causal.
Sure, try it. It's harder than you think. It's not just imports, it's the entire program.
> you're suggesting there's some kind of information flow from the anticipated body of the script back up to the imports
Yes, I am suggesting this. I don't think it is possible to write programs without either anticipating what you're going to write down below before you get there, or else being able to go back and edit what you already wrote.
Of course agent harnesses allow the latter, but raw models without a harness can still do an exceptionally, superhumanly, good job of straight-line programming with no editing.
> Infer imports from context, infer body from context + imports. All strictly causal.
Of course it's causal, that's kind of a reductive way to look at it.
Just infer the entire program from context and then type what you inferred.
I used to ask my coding agent to present two alternatives to choose from while implementing a task, and include a tokens required for each alternative to implement. (So that I can choose one which needs less token vs one which needs more rigour depending on task)
Finally there is evidence that the model kinda actually knows the correct token spend on each method.
How is this news? isn't it an obvious fact from the Transformer architecture?
> isn't it an obvious fact
Just below your question is a very confidently incorrect take about "parroting"... So, not obvious at all, at least for some people :)
It's a mechanistic interpretability tool. Useful even if it is not surprising.
>> Probes trained to predict the outcome of future edits (before they are materialized and written on disk) achieve performance above chance up to roughly 25 steps in advance.
Are these probes effectively run in parallel? The way this reads is more about predicting a future outcome than keeping the current token relevant based on past tokens.
In other words, since the next semantic prediction for forecasting the future is built on the training dataset, it's hard for anything truly new to emerge.
Then how do humans create something 'creative'—something that didn't exist before? I think it might be because the process of simplifying the complex system of nature differs between individuals. The data being learned now is all labeled by humans and simplified through human cognition. Within that kind of information, creativity seems hard to emerge.
Ultimately, with data that already contains interpretation, no matter how much you repeat the learning, it just becomes an encyclopedia that only explores within human knowledge, repeating predictions within human interpretation. So I wonder if we actually need a different encoder that interprets raw data—not based on human interpretation.
In reality, what changed Newton's absolute time to Einstein's relativity was a conclusion derived simply from observing the world. Newton's interpretation was supported by a lot of evidence in its time. If an AI studied all the medieval data from Newton's era, could it actually come up with the theory of relativity?
I'm always curious about this. I think AI is already very good at coding and will soon become better than humans. Logical structures are ultimately human interpretations, and reasoning within that framework is something AI can probably do more logically than humans. In other words, once humans create the framework, stacking the logical Jenga blocks within it—AI will be better at that.
But true creativity lies in breaking the framework itself, and I'm skeptical about whether AI can do that. The encoder also seems insufficient. There will likely be limits. I might be trapped in my own biases.
But the limitations of the current approach seem too clear to ignore.
When I look at the approach of these papers, it feels like an argument that adding shadows that imitate the world will eventually make them become the objects themselves.
I think the text, code, images, papers, and conversations that humans leave behind are not the world itself, but rather shadows of the world that have passed through human cognition and language. No matter how much you learn from those shadows, whether that leads to the ability to actually engage with the objects themselves seems like a separate issue.
I feel like something different is needed. But I'm not intellectually sharp enough to reason this through logically.this is just my intuition
> A coding agent solving a software-engineering task spends dozens of steps reasoning
No. That's simple PR hype. Parrotry is not reasoning.
Why not? I think there’s fairly strong evidence that there is something that convincingly looks like reasoning. I think anthropic has done some nice circuit tracing and mechanistic interpretability work on this for instance.
There is a certain amount of irony in your comment that I hope you appreciate.
What exactly do you consider reasoning?