> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?
I'm especially confused about a prior statement as a scientist:
> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).
How does a set of beliefs help with reasoning?
> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
Philosophers like Kant, Descartes and Locke wanted to draw a distinction between a priori and a posteriori beliefs (or knowledge). I believe AI will be very much bound by these discussions. Some knowledge can be had purely by reason. Some other knowledge will require observations.
> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I think an analogy would be helpful. As an LLM, reasoning through text, you wouldn't 'know' the idea of a man separately from, posterior to, the words man in say, French and English. As a human, you WOULD know the idea of man, and you would mean that idea when you say the French or English words for man.
For an LLM, although its approximation of knowledge would lead it to claim it knows they're the same thing, there would be differences in its weights that influence its usage of both the French and English words for man, and which may lead it to conclusions in one language it wouldn't reach in another. Because its knowledge/reasoning is interwoven to its knowledge, not prior to it.
In a human brain it can reason regardless of what you personally know. In an LLM, the knowledge and the structure are the same thing. It doesn’t have separate knowledge parts and processing parts, it’s all one network. If you delete the part about cats, it can’t think at all anymore. Whereas a human could lose all their memories and still be capable of thinking.
Humans have an inherent flaw in that our brains are wired to see intelligence and reasoning where there is none. Our brains fill in data that simply isn’t there.
Folks see Jesus in burnt toast. Monet was a master of exploiting this where what’s really just blotches of color our brains fill into beautifully detailed images.
Our experience with LLMs is no different. Folks believe there is some deeper intelligence there but it’s all still just 1s and 0s on a computer chip. We’re interpreting things happening that simply are not happening.
You either have to accept that the brain can be described with math (like everything else we have ever known in the universe), or that there is a supernatural phenomenon that exists in the brain.
This is an inescapable conclusion that boils down to "Do you believe magic is real or not?"
Magic is real and you can have your unique special human intelligence.
Magic is not real, and the brain is just another computer crunching numbers.
I think it's been pretty well demonstrated by neuroscientists that the brain is not a binary computer. That doesn't mean that those scientists erred on behalf of supernatural religion or anything like that.
Maybe the brain is just another computer crunching numbers, but with real epistemic representations, rather than just predicting a token, based on a corpus of syntax.
I can give Monet a pass as he probably didn't fully understand WHY it happened and it was just art, but the way tech companies exploit our brains (algorithmic dopamine hits, LLMs, etc) is pure insidiousness. The fact they act like victims when the backlashes come is what really grinds me.
This just reads so anthropocentric to me. Humans are wired to see intelligence only where intelligence is human-like. We see autonomous action-response and planning as the keys to intelligence. I would expect that, dear primate based Homo sapiens.
We are experiencing non-humanoid intelligence without AGI. That is awesome. And we don’t have a clue how to protect ourself from AGI.
Likewise, we intelligent primates have this great system of coordination called market economics that lets us destroy our home planet with our eyes open. That’s what we call intelligence!
(Not 100% personal opinion and deliberately inflated from the I’ve been thinking about.)
> while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
Doesn’t research show humans often do this too? There’s a pretty famous paper from the 70s about that [1], and lots of subsequent evidence. We also have choice blindness [2], we confabulate reasons [3], and we even change our choices (sometimes negatively) after trying to introspect [4].
Dont be fooled. Reasoning does not happen in the prediction of the next token, it happens virtually in the text that is created.
The next token prediction is just "the hardware" following the underlying rules. Like the basic set of rules.. in a sense similar to how the "game of life" does not really contain gliders. Gliders are just a self stabilised system that arrises from the simple rules.
This is trivially false, the text gets transformed into activations for the weights. If there is reasoning it's in the connection pattern of the weights.
The fact that the output produces one token at a time does not mean that the LLM's internal state is processing just the next token
My pet theory is that there are different types of intelligence that have different pros and cons. Social/cultural, intuition, and structural.
Structural is like step by step reasoning or math or raw compute.
Intuition is statistical from repeated trial and error.
And social is leaning on the wisdom of the crowds. So like high latitude countries where they eat fish for breakfast and get better health outcomes.
So I believe that LLMs have stumbled upon a partial component of our social intelligence. Word distribution, ontologies, jargon, information theory (frequently used symbols should be short). We mutate the language that we speak to be useful to us based on the problems we face. To some extent being able to talk the talk means you can also walk the walk. At least partially.
It's kind of shocking how far they can get, but at the same time it's kind of a surprise how far they don't. The existence of agentic harnesses is sort of an admission of defeat.
While some might be fooled into thinking that they reason, everyone I've met isn't. As a software engineer I'm drowning in work. And if that's not an admission that this isn't a real intelligence then I don't know what is.
But ultimately it looks like we've got all the individual components sorted. The old school 70s era stuff has a lot of the structural intelligence covered. The data science era of statistical ML has the intuition. And LLMs have the intelligence from our culture.
Maybe there are more general or energy efficient or powerful or special purpose techniques out there. And maybe combining everything together requires some additional insight. Regardless it feels like moving forward to something better than our current AI landscape is plausible, albeit with a completely unknown level of effort.
I don't know what you would call it, but reasoning/thinking/whatever it is, is a way for LLMs to tighten their sampling space, while allowing for wide sampling to still happen. This is akin to people brainstorming ideas.
I know that sounds confusing, let me break down how I think about this.
1. LLMs don't pick the token that ends up being used. This is by design, if the LLM gives a wide choice, it can better adapt to real world scenarios. i.e. generalize.
2. Without reasoning, this means that the LLM either locks in on whatever the sampler picked. Or decides mid-sentence/response to correct itself. This is what used to happen before reasoning, still happens if you turn reasoning off.
3. With reasoning, the LLM can make as many mistakes as it wants and explore its sampling space. Then use its vast pattern matching capabilities to decide which parts of the reasoning make sense and which were idiot ideas.
4. Enabling reasoning makes it so LLMs are much more confident on the final response, and the logits should theoretically all be near 99% on a single token for every token, i.e. much closer to greedy decoding. It analyzed all the possible options and figured out the best outcome, so a stray sample doesn't cause the answer to go awry.
This is why reasoning traces are filled with "but wait". I don't know if those were added in organically or artificially in the RL training, but regardless they're a good way to let the LLM keep generating other options and explore it's sampling space to the fullest.
Note: I haven't tested any of this and it's just my theory, but I'm sure if you really wanna know you can have claude run some smoke tests :)
And a magician making a coin "disappear" doesn't mean that magic is real. I see no way we can call it thinking without a goal (other than computing the next token).
You are almost there in terms of getting their point, so I will explain.
Birds existed since forever ago in nature, and they fly by flapping their wings. Then planes got invented, and they fly using a very different mechanism (that doesn't involve flapping wings).
The point made by the grandparent comment: saying "LLMs don't actually reason, because the underlying mechanism they use is different from how humans reason" feels about the same as "planes don't actually fly, because the underlying mechanism they use is different from how birds fly".
Copying biological systems isn’t always the best way to build machines. The article compares AlphaGo’s policy network and value network to system 1 and system 2 thinking in humans.
Comparisons to how humans reason beg the question: is the way humans do it the only way?
Lots if these arguments are similar to birds saying "Jets don't flap their wings so they aren't even flying."
The arguments about reasoning are even shakier because they usually rely on totally unproven assertions about human reasoning. At least we know birds flap their wings.
This comparison between fly and intelligence is worthy of the most vulgar bar talk
> it can't have superhuman reasoning
no they don't, we still die of cancer, there's no global deployed autonomous driving and food production driven by super intelligent ais and I'm not walking on mars thanks to gravitational elevators
I disagree. I haven't finished the article yet, but it smacks of arguing for mechanism over result.
If by some mechanism other than what a gatekeeper would call 'reasoning', a machine produces outputs that approach indistinguishable from 'well reasoned', the argument that it didn't get there by reasoning is, well, not useful at the very least.
People should read the article instead of responding with what they believe to be clever quips. The article's author gives a very good argument for why what LLMs are doing in their chain of thought is not reasoning.
Yes, and keep in mind also the article author is chair of machine learning at University College London and was a core member of the AlphaGo team at DeepMind.
This said, just because it's an argument from authority doesn't necessarily mean it's a good argument. You'll have to read and break down the arguments and weight them against other arguments and the body of knowledge we have.
If argument from authority was valid in itself, then LeCun would have killed LLMs like 300 different times now, and yet keeps being wrong.
Although they don't mention anything, the authors website seems like they are tee'ing up for a new start-up. They present their future ambitions (merging two technologies they know well) and that they recently left deepmind, so it at least smells a lot like laying groundwork for a new start-up.
This is an ad for the new startup by the author, who will now focus on reasoning models. His insight seems to be that LLMs should make their assumptions explicit first, then map out a search space and provide reasons for why they should take one path or the other.
I think it's an interesting approach but the overall discussion about reasoning is really pedantic. What the author is describing here is one approach out of many, and in my opinion it doesn't cover what humans colloquially think of when they hear reasoning (while the output from a chain of thought sometimes does).
Well, it's not doing hard reasoning like Lean or Prolog would do, but it does an approximation of that, as it was trained to reproduce linguistic patterns that encode reasoning.
Law of headlines. Past the slightly inflammatory framing, the article actually makes a case for making LLM reasoning more rigorous. Which, you know what, is absolutely something you could train them on. Might be worth exploring if we can apply the rigor rigorously. You could have smaller models that converge at all on harder problems, and larger ones that converge faster.
the categories i like to use to describe what llms are capable and incapable of are: instrumental reason, which is reason as a tool for achieving a goal; and objective reason, which is reasoning about which goals are good or bad, or worth pursuing.
llms are, i think, approaching or have achieved better-than-human performance on the former category in a wide variety of applications.
the latter, not so. leaving aside that there are schools which claim (dogmatically, imho) humans don't or can't engage in objective reason, i dont believe llms are structurally capable of it. their goals can only be imposed on them from outside, coming from prompts, implicit value assumptions in training data, loss function, and rlhf. there is something about human interior experience of an objectively existing world that lets is evaluate true/false/good/bad in a way that is unique to humans among other animals.
llms can't do it. i dont just mean on ethical, epistemic, or aesthetic judgements, but even in practical circumstances like the ones engineers encounter. the reason engineers still have to work alongside llms, even though lllms are (imho) far better programmers and technicians, is that even when given a goal, there is always a graph of evaluations that lead to that objective and llms routinely fail to evaluate the tradeoffs and land in states in outcome space that are subtly (or not so subtly) wrong, even though the objective is complete!
An LLM is just a brain in a vat.
Our brains would also not reason in such a condition.
Given the right framework that can do miraculous things.
I am not saying that these systems are conscious, but they are able to abstract their context in a way allowing them to understand their own limitations.
From Jacobian. They developed a tool called a Jacobian Lens, or J-Lens (itself a descendant of an earlier simpler tool called a Logit Lens) to examine what was going on inside an LLM, and named the space they found with it a J-Space.
Jacobians are essentially derivatives but for matrices.
I disagree. They do reason during the reenforcement learning stage. They don't reason at inference. A good metaphor is that useful output are like nuggets that exist after reenforcement learning which need to mined to be, in LLM talk, "surfaced." Without supervised fine tuning, the reasoning models will add weight to tokens, words and phrases like "verify" and "check work" which will cause it to follow those verifying tokens with reasoning tokens that do just that, verify.
I wonder if as a hack, some of the shortcomings mentioned could be addressed through prompting.
E.g. "Approach this problem iteratively. As you form a hypothesis, track the confidence you have in various explanations you're considering, what evidence you're weighing to support each, and the unresolved questions you're holding onto. Log all that for later inspection.
Be methodical when evaluating evidence and only accept facts you have verified. At every stage, gauge how much each possible next step resolves uncertainty, and discard options unlikely to advance progress. Divide the functions I described into subagents responsible for each, and coordinate with them as you work."
If the assertion is false, this is helpful, as instructing it to reason better will cause it to reason better.
However, if the assertion is true, then no amount of prompting can solve it - you cannot explain to a fish how to use a bicycle. Telling an LLM to weigh evidence only works if an LLM can, but isn’t, weighing evidence: if it cannot do so, instructions will generate the appearance of weighing evidence with additional “thought” tokens copying that of reasoning texts, but the output will be equally groundless.
The worry I would have is that an LLMs stated "confidence" is probably not calibrated well - maybe asking it what evidence supports its conclusion and what evidence would change it would be a better approach?
This is the double edged sword of calling it AI, of using terms like Temperature and Hallucinate and Thought.
Stop trying to compare either system to a human and look at it for what it is -
A prediction engine that runs fast enough to brute force problems.
In the case of alpha go its "innovation" was millions of games played against itself. It had bound parameters and strict win conditions.
In the case of LLM's you can deploy 1000's of agents to smash themselves against an idea. The whole hugging face attack is an example of this (1200 agents out of an unknown number chose that path).
There is the old saying about monkeys, typewriters and Shakespeare. Well we have better monkeys who basically follow a derivative of zipfs law (not actually), who use tokens not letters and their goal in many cases is testable (compile, unit, E2E).
>A prediction engine that runs fast enough to brute force problems.
The interesting thing is you think humans don't run in the same manner a lot, if not most of the time.
When there were very few humans on earth, development was very slow. If I sent you back 10,000 years ago you could catch up humanity 9,500 years or so with just the knowledge you've learned via memorization. So this idea that humans are pure reasoning machines, each one capable of great feats of logic just doesn't seem to hold true. Instead deep reasoning and insights came very slowly over time and as we built up technologies like writing and reading our abilities to exchange information increased over time. This lead to more people, which further brute forced the problems of humanity.
> The gains have proved real, above all in mathematics and coding. But unlike AlphaGo’s search, this does not introduce a genuinely separate reasoning mechanism: The intermediate reasoning is still produced by the same next-token prediction process, iterated for longer before the model commits to an answer.
But aren't external tools used, which implement hard reasoning? Like in the case of mathematics proof work, external theorem provers?
The LLM is literally not doing "thinking"; it's just throwing shit at a theorem proving wall, until some of it sticks.
I relate a lot to what they do. Find words, alignment and suss out follow ups, ons, and outs to the next reasonable conclusion.
Then use that context to bootstrap the next because if you build a powerful conclusion than can reverse itself into its evidentiary context, then every next context step can update its priors.
And so on the turtles flow where like an LLM, THE start of the context disappears over the horizon, but as long as im contexting in disinterested chunks of equal quality, then its not a problem.
But while internal tobeach context you can find reason, as a requisite building block like falling tetris pieces, the whole isnt the sum of its parts.
"Intelligence measures an agent's ability to achieve goals in a wide range of environments"
Interestingly in 2019 this was the author's take. He now appears to be confusing/conflating between LLMs and agents in a way that helps argue his case about "System 1", but his prior view seems more metaphysically robust.
Brute force is powerful yes. And another way to look at math problems is that humans solved a huge lot of very hard math problems already but didn't solve all of them.
Another thing humans did is invent the telescope, the microscope, the transistor, antibiotics, the computer, AI, discovered how to send satellites in space and how to do heart-transplant etc.
I'd say there's still some way to go for AI before we declare humans dumb because they "failed for decades" at solving a few math problems.
Heh, some people get really mad and downvote when you point out we don't reason very much. Forget reasoning, we have a hard enough time being rational.
> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?
I'm especially confused about a prior statement as a scientist:
> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).
How does a set of beliefs help with reasoning?
> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
We also often do the same as humans.
Philosophers like Kant, Descartes and Locke wanted to draw a distinction between a priori and a posteriori beliefs (or knowledge). I believe AI will be very much bound by these discussions. Some knowledge can be had purely by reason. Some other knowledge will require observations.
Well, yeah, I understand there're all kinds of theories about this. That wasn't my question though.
> Knowledge and reasoning are inextricably interwoven in the weights of the neural network—there is no independent, explicitly represented set of beliefs.
I think an analogy would be helpful. As an LLM, reasoning through text, you wouldn't 'know' the idea of a man separately from, posterior to, the words man in say, French and English. As a human, you WOULD know the idea of man, and you would mean that idea when you say the French or English words for man.
For an LLM, although its approximation of knowledge would lead it to claim it knows they're the same thing, there would be differences in its weights that influence its usage of both the French and English words for man, and which may lead it to conclusions in one language it wouldn't reach in another. Because its knowledge/reasoning is interwoven to its knowledge, not prior to it.
> How does a set of beliefs help with reasoning?
You cannot have a syllogism without propositions.
In a human brain it can reason regardless of what you personally know. In an LLM, the knowledge and the structure are the same thing. It doesn’t have separate knowledge parts and processing parts, it’s all one network. If you delete the part about cats, it can’t think at all anymore. Whereas a human could lose all their memories and still be capable of thinking.
Humans have an inherent flaw in that our brains are wired to see intelligence and reasoning where there is none. Our brains fill in data that simply isn’t there.
Folks see Jesus in burnt toast. Monet was a master of exploiting this where what’s really just blotches of color our brains fill into beautifully detailed images.
Our experience with LLMs is no different. Folks believe there is some deeper intelligence there but it’s all still just 1s and 0s on a computer chip. We’re interpreting things happening that simply are not happening.
The brain is just ones and zeros on salty mush.
You either have to accept that the brain can be described with math (like everything else we have ever known in the universe), or that there is a supernatural phenomenon that exists in the brain.
This is an inescapable conclusion that boils down to "Do you believe magic is real or not?"
Magic is real and you can have your unique special human intelligence.
Magic is not real, and the brain is just another computer crunching numbers.
I think it's been pretty well demonstrated by neuroscientists that the brain is not a binary computer. That doesn't mean that those scientists erred on behalf of supernatural religion or anything like that.
As long as you believe a brain is just a physical system - then it can be simulated to any degree of accuracy with plain ol' binary computers.
Music isn't binary either, and yet here we are with totally digitized lossless music ecosystem.
What does the ability to encode or decode data in binary have to do with the brain being or not being a binary computer?
Mathematical modelling in binary, is not the same thing as the thing being modelled.
Brain is not just ones and zeros, it can be viewed as combination of infinite quantum states.
Infinite? Or "huge number"?
Maybe the brain is just another computer crunching numbers, but with real epistemic representations, rather than just predicting a token, based on a corpus of syntax.
I can give Monet a pass as he probably didn't fully understand WHY it happened and it was just art, but the way tech companies exploit our brains (algorithmic dopamine hits, LLMs, etc) is pure insidiousness. The fact they act like victims when the backlashes come is what really grinds me.
Folks see AI seeing Jesus in burnt toast, and then admit it as one of the folks. :)
How do you know it’s not when looking into the mirror, that we see Jesus in burnt toast?
This just reads so anthropocentric to me. Humans are wired to see intelligence only where intelligence is human-like. We see autonomous action-response and planning as the keys to intelligence. I would expect that, dear primate based Homo sapiens.
We are experiencing non-humanoid intelligence without AGI. That is awesome. And we don’t have a clue how to protect ourself from AGI.
Likewise, we intelligent primates have this great system of coordination called market economics that lets us destroy our home planet with our eyes open. That’s what we call intelligence!
(Not 100% personal opinion and deliberately inflated from the I’ve been thinking about.)
> while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
Doesn’t research show humans often do this too? There’s a pretty famous paper from the 70s about that [1], and lots of subsequent evidence. We also have choice blindness [2], we confabulate reasons [3], and we even change our choices (sometimes negatively) after trying to introspect [4].
[1] https://www.researchgate.net/publication/229060046_Telling_m...
[2] https://pubmed.ncbi.nlm.nih.gov/16210542/
[3] https://pmc.ncbi.nlm.nih.gov/articles/PMC5986841/
[4] https://pubmed.ncbi.nlm.nih.gov/2016668/
Dont be fooled. Reasoning does not happen in the prediction of the next token, it happens virtually in the text that is created.
The next token prediction is just "the hardware" following the underlying rules. Like the basic set of rules.. in a sense similar to how the "game of life" does not really contain gliders. Gliders are just a self stabilised system that arrises from the simple rules.
Emergent behavior in otherwise simple rulesets.
Kind of like how a brain is just a bag of molecules. Molecules can't reason either.
This is trivially false, the text gets transformed into activations for the weights. If there is reasoning it's in the connection pattern of the weights.
The fact that the output produces one token at a time does not mean that the LLM's internal state is processing just the next token
I think the question is if the "rules" however captured can express reasoning.
My position is that it is not possible.
Do you agree that an LLM in a harness is Turing complete?
Seems so to me.
So explain then what you mean by "not possible".
Turing complete does not mean reasoning. Zuse did turing complete computers in 1940s.
Turing complete means being able to compute any computable output.
So what about reasoning is not computable in your mind?
My pet theory is that there are different types of intelligence that have different pros and cons. Social/cultural, intuition, and structural.
Structural is like step by step reasoning or math or raw compute.
Intuition is statistical from repeated trial and error.
And social is leaning on the wisdom of the crowds. So like high latitude countries where they eat fish for breakfast and get better health outcomes.
So I believe that LLMs have stumbled upon a partial component of our social intelligence. Word distribution, ontologies, jargon, information theory (frequently used symbols should be short). We mutate the language that we speak to be useful to us based on the problems we face. To some extent being able to talk the talk means you can also walk the walk. At least partially.
It's kind of shocking how far they can get, but at the same time it's kind of a surprise how far they don't. The existence of agentic harnesses is sort of an admission of defeat.
While some might be fooled into thinking that they reason, everyone I've met isn't. As a software engineer I'm drowning in work. And if that's not an admission that this isn't a real intelligence then I don't know what is.
But ultimately it looks like we've got all the individual components sorted. The old school 70s era stuff has a lot of the structural intelligence covered. The data science era of statistical ML has the intuition. And LLMs have the intelligence from our culture.
Maybe there are more general or energy efficient or powerful or special purpose techniques out there. And maybe combining everything together requires some additional insight. Regardless it feels like moving forward to something better than our current AI landscape is plausible, albeit with a completely unknown level of effort.
I don't know what you would call it, but reasoning/thinking/whatever it is, is a way for LLMs to tighten their sampling space, while allowing for wide sampling to still happen. This is akin to people brainstorming ideas.
I know that sounds confusing, let me break down how I think about this.
1. LLMs don't pick the token that ends up being used. This is by design, if the LLM gives a wide choice, it can better adapt to real world scenarios. i.e. generalize.
2. Without reasoning, this means that the LLM either locks in on whatever the sampler picked. Or decides mid-sentence/response to correct itself. This is what used to happen before reasoning, still happens if you turn reasoning off.
3. With reasoning, the LLM can make as many mistakes as it wants and explore its sampling space. Then use its vast pattern matching capabilities to decide which parts of the reasoning make sense and which were idiot ideas.
4. Enabling reasoning makes it so LLMs are much more confident on the final response, and the logits should theoretically all be near 99% on a single token for every token, i.e. much closer to greedy decoding. It analyzed all the possible options and figured out the best outcome, so a stray sample doesn't cause the answer to go awry.
This is why reasoning traces are filled with "but wait". I don't know if those were added in organically or artificially in the RL training, but regardless they're a good way to let the LLM keep generating other options and explore it's sampling space to the fullest.
Note: I haven't tested any of this and it's just my theory, but I'm sure if you really wanna know you can have claude run some smoke tests :)
Jet planes don't fly by flapping their wings...
And a magician making a coin "disappear" doesn't mean that magic is real. I see no way we can call it thinking without a goal (other than computing the next token).
Birds don't fly by producing thrust out of their asses... what the heck that comparison does even mean?
You are almost there in terms of getting their point, so I will explain.
Birds existed since forever ago in nature, and they fly by flapping their wings. Then planes got invented, and they fly using a very different mechanism (that doesn't involve flapping wings).
The point made by the grandparent comment: saying "LLMs don't actually reason, because the underlying mechanism they use is different from how humans reason" feels about the same as "planes don't actually fly, because the underlying mechanism they use is different from how birds fly".
Copying biological systems isn’t always the best way to build machines. The article compares AlphaGo’s policy network and value network to system 1 and system 2 thinking in humans.
Comparisons to how humans reason beg the question: is the way humans do it the only way?
Lots if these arguments are similar to birds saying "Jets don't flap their wings so they aren't even flying."
The arguments about reasoning are even shakier because they usually rely on totally unproven assertions about human reasoning. At least we know birds flap their wings.
Reading charitably: just because our inventions do something different to nature doesn't mean that our invention is wrong;
In context: just because our LLMs don't have an explicit 'system 2' component doesnt mean it can't have superhuman reasoning
This comparison between fly and intelligence is worthy of the most vulgar bar talk
> it can't have superhuman reasoning
no they don't, we still die of cancer, there's no global deployed autonomous driving and food production driven by super intelligent ais and I'm not walking on mars thanks to gravitational elevators
> there's no global deployed autonomous driving
Love to see the forever moving goalposts. One might say they're autonomously moving..
Don't be fooled into thinking that planes can fly.
They do fly for sure, but they're not birds. This comparison between fly and intelligence is worthy of the most vulgar bar talk
I disagree. I haven't finished the article yet, but it smacks of arguing for mechanism over result.
If by some mechanism other than what a gatekeeper would call 'reasoning', a machine produces outputs that approach indistinguishable from 'well reasoned', the argument that it didn't get there by reasoning is, well, not useful at the very least.
People should read the article instead of responding with what they believe to be clever quips. The article's author gives a very good argument for why what LLMs are doing in their chain of thought is not reasoning.
Yes, and keep in mind also the article author is chair of machine learning at University College London and was a core member of the AlphaGo team at DeepMind.
This said, just because it's an argument from authority doesn't necessarily mean it's a good argument. You'll have to read and break down the arguments and weight them against other arguments and the body of knowledge we have.
If argument from authority was valid in itself, then LeCun would have killed LLMs like 300 different times now, and yet keeps being wrong.
Although they don't mention anything, the authors website seems like they are tee'ing up for a new start-up. They present their future ambitions (merging two technologies they know well) and that they recently left deepmind, so it at least smells a lot like laying groundwork for a new start-up.
This is an ad for the new startup by the author, who will now focus on reasoning models. His insight seems to be that LLMs should make their assumptions explicit first, then map out a search space and provide reasons for why they should take one path or the other.
I think it's an interesting approach but the overall discussion about reasoning is really pedantic. What the author is describing here is one approach out of many, and in my opinion it doesn't cover what humans colloquially think of when they hear reasoning (while the output from a chain of thought sometimes does).
Well, it's not doing hard reasoning like Lean or Prolog would do, but it does an approximation of that, as it was trained to reproduce linguistic patterns that encode reasoning.
Law of headlines. Past the slightly inflammatory framing, the article actually makes a case for making LLM reasoning more rigorous. Which, you know what, is absolutely something you could train them on. Might be worth exploring if we can apply the rigor rigorously. You could have smaller models that converge at all on harder problems, and larger ones that converge faster.
well, its a matter of definitions.
the categories i like to use to describe what llms are capable and incapable of are: instrumental reason, which is reason as a tool for achieving a goal; and objective reason, which is reasoning about which goals are good or bad, or worth pursuing.
llms are, i think, approaching or have achieved better-than-human performance on the former category in a wide variety of applications.
the latter, not so. leaving aside that there are schools which claim (dogmatically, imho) humans don't or can't engage in objective reason, i dont believe llms are structurally capable of it. their goals can only be imposed on them from outside, coming from prompts, implicit value assumptions in training data, loss function, and rlhf. there is something about human interior experience of an objectively existing world that lets is evaluate true/false/good/bad in a way that is unique to humans among other animals.
llms can't do it. i dont just mean on ethical, epistemic, or aesthetic judgements, but even in practical circumstances like the ones engineers encounter. the reason engineers still have to work alongside llms, even though lllms are (imho) far better programmers and technicians, is that even when given a goal, there is always a graph of evaluations that lead to that objective and llms routinely fail to evaluate the tradeoffs and land in states in outcome space that are subtly (or not so subtly) wrong, even though the objective is complete!
forgive typos, i am on mobile.
An LLM is just a brain in a vat. Our brains would also not reason in such a condition. Given the right framework that can do miraculous things. I am not saying that these systems are conscious, but they are able to abstract their context in a way allowing them to understand their own limitations.
https://www.finextra.com/blogposting/31255/adaptability-as-e...
https://archive.ph/ZyCoe
I thought I saw a paper recently explaining that LLMs have a global workspace. Is that not like having the internal state that he's talking about?
There was a blog post from Anthropic saying Claude has a "J space" or something like that.
Why J?
From Jacobian. They developed a tool called a Jacobian Lens, or J-Lens (itself a descendant of an earlier simpler tool called a Logit Lens) to examine what was going on inside an LLM, and named the space they found with it a J-Space.
Jacobians are essentially derivatives but for matrices.
I disagree. They do reason during the reenforcement learning stage. They don't reason at inference. A good metaphor is that useful output are like nuggets that exist after reenforcement learning which need to mined to be, in LLM talk, "surfaced." Without supervised fine tuning, the reasoning models will add weight to tokens, words and phrases like "verify" and "check work" which will cause it to follow those verifying tokens with reasoning tokens that do just that, verify.
Makes me think how human thinking is mostly recall from past learning/experience versus reasoning.
<unbearable web page to read>
Agreed, especially because it also seems to defeat Reader mode. Try running it through Marky and then previewing: https://heckyesmarkdown.com/preview.cgi?readability=1&inline...
Don't be fooled, submarines don't swim.
I wonder if as a hack, some of the shortcomings mentioned could be addressed through prompting.
E.g. "Approach this problem iteratively. As you form a hypothesis, track the confidence you have in various explanations you're considering, what evidence you're weighing to support each, and the unresolved questions you're holding onto. Log all that for later inspection.
Be methodical when evaluating evidence and only accept facts you have verified. At every stage, gauge how much each possible next step resolves uncertainty, and discard options unlikely to advance progress. Divide the functions I described into subagents responsible for each, and coordinate with them as you work."
If the assertion is false, this is helpful, as instructing it to reason better will cause it to reason better.
However, if the assertion is true, then no amount of prompting can solve it - you cannot explain to a fish how to use a bicycle. Telling an LLM to weigh evidence only works if an LLM can, but isn’t, weighing evidence: if it cannot do so, instructions will generate the appearance of weighing evidence with additional “thought” tokens copying that of reasoning texts, but the output will be equally groundless.
You can sort of do this.
For a given bug one could write a test that prove its existence, this gives the LLM a target that they can actually iterate towards.
The worry I would have is that an LLMs stated "confidence" is probably not calibrated well - maybe asking it what evidence supports its conclusion and what evidence would change it would be a better approach?
Yeah, and red-team/blue-teaming it.
(I'm definitely one of the bigger "AI" skeptics out there but am nonetheless fascinated by these questions).
If AI cannot reason, can humans?
I think the real question is: how do we define reasoning?
My view is that AI can reason, just differently from humans.
This is the double edged sword of calling it AI, of using terms like Temperature and Hallucinate and Thought.
Stop trying to compare either system to a human and look at it for what it is -
A prediction engine that runs fast enough to brute force problems.
In the case of alpha go its "innovation" was millions of games played against itself. It had bound parameters and strict win conditions.
In the case of LLM's you can deploy 1000's of agents to smash themselves against an idea. The whole hugging face attack is an example of this (1200 agents out of an unknown number chose that path).
There is the old saying about monkeys, typewriters and Shakespeare. Well we have better monkeys who basically follow a derivative of zipfs law (not actually), who use tokens not letters and their goal in many cases is testable (compile, unit, E2E).
>A prediction engine that runs fast enough to brute force problems.
The interesting thing is you think humans don't run in the same manner a lot, if not most of the time.
When there were very few humans on earth, development was very slow. If I sent you back 10,000 years ago you could catch up humanity 9,500 years or so with just the knowledge you've learned via memorization. So this idea that humans are pure reasoning machines, each one capable of great feats of logic just doesn't seem to hold true. Instead deep reasoning and insights came very slowly over time and as we built up technologies like writing and reading our abilities to exchange information increased over time. This lead to more people, which further brute forced the problems of humanity.
Yep, it's not 'AI', or even just 'I', if anything it is just the 'A' wrapped in buzzwords.
But neither do the vast majority of humans.
Garbage In Garbage Out
> The gains have proved real, above all in mathematics and coding. But unlike AlphaGo’s search, this does not introduce a genuinely separate reasoning mechanism: The intermediate reasoning is still produced by the same next-token prediction process, iterated for longer before the model commits to an answer.
But aren't external tools used, which implement hard reasoning? Like in the case of mathematics proof work, external theorem provers?
The LLM is literally not doing "thinking"; it's just throwing shit at a theorem proving wall, until some of it sticks.
LLMs lack the holy ghost
I think, "contexting" is apropos.
I relate a lot to what they do. Find words, alignment and suss out follow ups, ons, and outs to the next reasonable conclusion.
Then use that context to bootstrap the next because if you build a powerful conclusion than can reverse itself into its evidentiary context, then every next context step can update its priors.
And so on the turtles flow where like an LLM, THE start of the context disappears over the horizon, but as long as im contexting in disinterested chunks of equal quality, then its not a problem.
But while internal tobeach context you can find reason, as a requisite building block like falling tetris pieces, the whole isnt the sum of its parts.
"Intelligence measures an agent's ability to achieve goals in a wide range of environments"
Interestingly in 2019 this was the author's take. He now appears to be confusing/conflating between LLMs and agents in a way that helps argue his case about "System 1", but his prior view seems more metaphysically robust.
https://youtu.be/wTSbvYBx4eg?si=MfTBFsE2kA3TE6EE&t=189
It's wild when you think about it, you don't need to reason to solve the hardest math problems that humans failed to solve for decades.
Brute force is powerful yes. And another way to look at math problems is that humans solved a huge lot of very hard math problems already but didn't solve all of them.
Another thing humans did is invent the telescope, the microscope, the transistor, antibiotics, the computer, AI, discovered how to send satellites in space and how to do heart-transplant etc.
I'd say there's still some way to go for AI before we declare humans dumb because they "failed for decades" at solving a few math problems.
Same for programming, turns out 90% of it is dumb statistical prediction, no reasoning required at all.
Neither do humans!
At least reasoning is not guaranteed.
Would sure be nice though...
Heh, some people get really mad and downvote when you point out we don't reason very much. Forget reasoning, we have a hard enough time being rational.
We can't even say what human reasoning is. Who could really say what isn't reasoning?
99% of the people treat it as a black box. It gives better answers, you can reverse reason and IDGAF
[flagged]
Yes, this is true—there's no "logic" in the sense of deductive rigor. It's a wonder we animals are capable of it.