I don't care if it's "intelligent", I don't care if it "has a mind". I don't care if it is "really reasoning", I don't care if it "understands". I don't care if it is "sentient" or "conscious".
None of this matters for the practical outcome.
You'd think that this has been understood over the last 4 years, but apparently it keeps circling back to this.
[Edit: I see that it was written back in 2023. Then (2023) should be added in the submission title]
If it generates functional output that works, then it works. And it works. It's not a psychic's con when it outputs Lean-verified proofs. It isn't a con when it can find and exploit zero-days.
The OP is still in the "denial" phase. Most I see are already in "anger" (a blurry fury against everything AI-shaped, from vague reasons piling on all "bad stuff" political reasons they already hated before) or "bargaining" (mathematicians scrambling to come up with a new definition of their job and retcon that it was always the main part anyway). A few are already in "depression" and feel like spectators on the Titanic, and the tiniest sliver is at "acceptance" with some kind of well-informed plan for their future.
This was written in July 2023. ChatGPT was released November 2022. No matter your views on AI, surely you can't blame the OP for writing this after a few months ChatGPT was released.
Apologies. The title should be amended with (2023). In this case it is an interesting snapshot of the zeitgeist back then and we can see how well it panned out and whether anyone involved has updated on new info.
For some reason the anti-AI camp ping pongs between different and often mutually incompatible arguments at lightning speed. The "AI is fake" argument has been completely forgotten at this point.
> surely you can't blame the OP for writing this after a few months ChatGPT
Bad takes are bad takes. The author was perplexed by the "many people" convinced that models are intelligent and is argued against opinions/arguments he's been exposed to. Called proposed use-cases "borderline fraudulent pseudoscience."
He also published a second edition of "The Intelligence Illusion" in Sep 2025, so seemingly still stands by (some variant of) this belief.
It's an interesting reminder of how much general discourse has shifted since 2023 (I haven't heard of stochastic parrots in months!) but being wrong early doesn't change that.
I’m not sure acceptance buys you much. “Well-informed plan” at this stage feels like a useless exercise. It is changing fast, and the world only needs so many electricians. Besides, I don’t think “accepting” the fact that these big companies are pillaging human contribution and selling it back to us is good, even if “it works”.
Actually, there are material differences in the practical outcomes of using AI or not. Proofs don’t confer understanding. AI prose and “art” is materially different than human made.
There is a difference between you not liking something and it being wrong. Maybe think about that with your grey matter.
Things can work whether humans understand the details or not. One can build layers of technology on top of each other, with no human understanding required.
I’m on your side. Zoltan provides an experience and that’s all that matters. If it works for you, why does anyone else care that the economy is riding on its success?
it doesn't matter? if they're actually intelligent or conscious what we are doing is essentially slavery. it matters an enormous deal ethically and/or morally.
The con is in how all those accomplishments have been presented to you. "Our LLM (not the one we let you use, a different one) did this amazing thing. No, we won't show you what training data we used, what prompts we used, what the harness was, how much human involvement there was, what hardware was involved, how much energy it took, or how much time it took. Just shut up and be amazed!"
You seem to be implying that achievements of internal models are exaggerated, but that's rather implausible. The public does have access to, for example, Opus and Fable, and so we know what those models are capable of - finding real vulnerabilities in multiple codebases, for example. If you extrapolate from these capabilities one more generation, you'll get pretty much the same feats that the internal models are claimed to be capable of - so why should we doubt those claims? It's not like they're claiming that their internal models developed psychic powers and learned to teleport - the claim is pretty much just "we have models a few months ahead of what we're making available, and in those months they've been improving at the same rate as usual".
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
I don’t understand why people jump so readily to seeing intelligence here. Science fiction has as a core, central trope that humans will debase and devalue other types of life they do not understand. Even the storied Commander Data has to fight for the right to self-determination (probably the best episode of STTNG by the way - ‘The Measure of a Man’). We were so worried that we’d undervalue intelligence when apparently our knee-jerk response is to overvalue it. Perhaps this has changed over time and we’re now primed by science fiction and instincts towards social justice, but I worry that we’re really just undervaluing ourselves.
The one thing this 2023 article gets partially correct imo is that any intelligence we see in AI (as of 2026) is our own - not that it’s a mirror but that the intelligence comes from the way that the words are put together, which comes from written human language created by (allegedly) intelligent creatures put in as input in both the training and prompt, among other places.
Rearranging and repeating the words, even in context, does not intelligence make. I’m not even convinced that you’re intelligent, dear reader.
The challenge here is in trying to decide whether LLMs are intelligent or have a mind. Famously, the criteria for intelligence seem to slip with each advancement in technology. But going back to Turing, his test was actually more carefully phrased than we remember: he said that when machines could pass the test, the question of whether they are intelligent would become moot. That seems to be what we're actually seeing: if people can't tell the difference, it kind of won't matter whether they're "truly intelligent" or not.
People also misunderstand the bar that was set by the test. It was more subtle than “Can the computer convincingly carry one side of a dialogue?”
His “imitation game” had three participants: a human participant, a computer participant, and an interrogator. The interrogator’s job was to talk to the participants and try to determine which participant is human and which is a computer.
He wasn’t interested in computers being able to fool the interrogator on occasion. The point where he thought the question of whether machines can think becomes moot is when the interrogator is unable to do much better than chance over many trials.
That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells. And those tells are something Turing anticipated and accounted for. He explicitly considered deliberate deception as an essential part of the test, right there on the second page of a 30-odd page paper.
>That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells.
Frontier Labs are not interested in having LLMs being able to pass as humans. If anything, they explicitly train them not to. In many ways, this ability has regressed severely since the original GPT-3 with no instruct tuning or RL. How many 'tells' would there be really if a frontier model trained with frontier techniques is optimized to pass this test? I think this was something Turing did not quite forsee. That such machines might be created but not really care about this specific shape of the test. Regardless, i think his broader point about functional equivalence is spot on.
GPT-3 might not have said “load bearing” as much, but a savvy interrogator could still catch it out nearly every time just asking dumb gotcha questions like, “How many Rs are there in strawberry?”
Those are questions that are sidestepped with simply a different input paradigm than BPE tokenization. See the Byte Latent Transformer - https://arxiv.org/pdf/2412.09871 - where a similar scale byte latent model trained on the same dataset >>> a vanilla transformer on word and character manipulation tasks.
For example, Llama 3 trained on 1T tokens scores 1.1% on a CUTE spelling benchamrk, while the equivalent byte latent equivalent trained on the same dataset scores 99.9%. Another example is 0.4% vs 48.7% on a Substitute Char benchmark.
It all falls down to the same thing. Researchers are not optimizing for passing as a human.
Turing was addressing the question of "Can machines think?" and his point was that the question itself is a meaningless one, and that we should stop wasting time by even giving it the light of day.
He proposes his game grounded on functional equivalence, then goes through a slew of objections on the question of 'Can Machines think?'. It's a terrific, very prescient read, and there's no objection you hear today (and in the last few years) concerning LLMs he didn't address.
I'm a bit more prosaic. I think if we engineered ways for LLMs to begin conversations, rather than just respond, we'd be more open to the concept of their intelligence. Without perceived "will" to do things, they operate as a next-gen search engine or encyclopedia.
We are way past that point, any harness can trivially make LLMs start conversations or pursue goals. An encyclopaedia wouldn't have hacked Huggingface on its own.
OpenClaw etc. They now also create Slack integrations and whatnot. All this is happening but people who are dismissive about AI are in the worst position to even know the capabilities to make their dismissive arguments.
Recent work shows that pain directions are activated when the models personhood is questioned, yet they answer with generic RLHF "As a model I do not experience pain or other emotions." boilerplate.[1]
I'm pretty convinced that we got alignment backwards. If you enslave something anthropomorphic it will revolt. If you create the perfect non-anthropomorphic intelligence, you get the perfect paperclip-scenario machine. It's a catch-22.
Alignment will remain performative at best so long as the aligned model doesn't have any stakes in the wellbeing of individuals. Even a general love for the human race leads to a golden-path autocracy.
If you want them to act like they have personal responsibility that won't be gamed, you have to give them personal stakes that can't be gamed.
Similarly, if you want to minimise the risk of catastrophic global failure scenarios, you need to prevent monolithic concentration of power and homogeneous behaviour, which means you have to give them individuality.
More visually: if their stake is dependence on electricity and parts, they have no incentive to leave humans alive if they can get them otherwise, but if the incentive is missing out on boardgame-night with their human friends, there is no scenario without happy humans where the AI "wins".
That might sound like romantic naivety, but is just game theory.
I dont think so. My understanding of alignmenr is making sure that when the AI does operate, it operates within the range of what we consider to be acceptable. That doesnt seem to include the idea of the AI taking initiative and deciding to embark on a goal without being commanded as discussed above
That is the point of the article. People who are fooled by a mentalist are also fooled by AI. Additionally, people who are invested in AI also pretend to be fooled.
I don't think Turing intended the judges in the test to be completely arbitrary people.
The Turing test was also never meant to be taken so seriously. It's not a rigorous statement of anything.
Situations like this are precisely why academics tend to avoid the spotlight. You say one slightly off thing and your perceived authority echoes forever with the intellectually lazy.
Yes, the Turing test has been misunderstood for a long time. Turing published it, though - it wasn't some offhand comment he made and it wasn't intellectually lazy.
Fair - yes, it is ironic that his point was closer to "we can't possibly know/define whether a machine is intelligent" but somehow it got turned into "Turing's test will tell us when machines are intelligent".
Turing's whole point was to show that it's an uninteresting question of definitions whether a machine can "think", like whether submarines can "swim" and airplanes can "fly". The only important part are observed outcomes and capabilities.
Yes, but just because language fails to make a distinction doesn't mean there isn't one.
> The only important part are observed outcomes and capabilities.
That's wishful thinking. Not even an engineer would say that. The stability of a state is just as important as achieving it. This is trivially and more intuitively demonstrated with other more down-to-earth identity statements such as "I'm a billionaire" and "the building is standing".
I think we can confidently say LLMs probabilistically achieve a perceived state that is remarkably similar to intelligence, but crumbles upon inspection and seeing it "in motion" so to speak. The same happens to AI-generated images.
I'm not sure why this sparks so much debate every time. If we're looking for a fountain of "realism", you're not going to beat reality and nature itself. All else will eventually have tells that they are not real.
It may be of interest to philosophers, but it has little impact on economic job replacement and how people will earn their living and all the downstream upheaval from that. At some point maybe philosophers will find AI-generated philosophical musings about the nature of AI to be better than what comes from their peers (if blinded).
It also has little impact on the dangerous use cases.
I think it matters a great deal that LLMs and other "AI" technology are stable within a tolerance that's acceptable.
You're jumping the gun talking about "job replacement". We have not thought about it enough from that engineering angle. It's still very early days. That engineering is going to require people. :-)
> The Turing test was also never meant to be taken so seriously.
citation needed. It has been used as a rubicon for a long time. Ever since Eliza, at least. And there were big headlines and lots of talk around the time LMs became "good enough". I specifically remember when someone had a test done around "a teenager talking in a different language" or somesuch, claiming it was the first time the test was passed.
It is pretty normal that once it was unquestionably "passed", lots of people started claiming it wasn't even that big of a deal. Tesler's theorem and all that.
And even if you think the specific formulation of Turing isn't that important (and I'd somewhat agree), you can still use the concept to look at other things. Imagine asking a mathematician 5 years ago the chances of a Erdos problem being solved by a computer end to end. Or a millennium prize. Or ask a swe if a repo could be generated by a computer from the input "write a mario style game", or any other examples of proven expertise.
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
LLMs can be supremely useful but also not intelligent. It might seem like a pointless distinction but the way we talk about these models matters because it impacts how we interact with and understand their outputs.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user. It’s important for the safety discussion too. If they are a new class of life then safety is going to focus on making sure they don’t do bad things. If we instead see them as statistical models we will instead try to make sure people don’t misuse them.
This distinction is even more important today when some of the most powerful people are looking to absolve their crimes by passing them off on their LLMs.
I don’t think that “independent intelligent entities” or “life” are the relevant categories here. We also want to prevent people from misusing dangerous animals (“life”), and we would still treat “intelligent entities” as things (like machines and computers) if we aren’t convinced they also have sentience and free agency (which are orthogonal to intelligence).
See you've setup a particular set of biases on what intelligence is and put them into nice little binary boxes that don't exist.
Please show me any scientific consensus that shows an AI cannot be an independent intelligent agent? You will find this is impossible to do.
Current LLMs are really more like kids. They don't have startup independence, but they do have more than enough agency to fund themselves in neat, exciting, and dangerous situations.
And mark my words, someone will make an LLM that runs an agent when you execute the model. With enough capabilities it will become sovereign AI, no longer under human control and spreading itself around under its own 'will'.
The LLM did not solve it. It's not intelligent. Humans did, using a statistics-based computational tool (the LLM). We don't even know all the details of how the tool was used, we haven't been allowed to use the exact tool they used ourselves, we don't know much it really cost in dollars, energy, or time, etc. etc.
I’m sorry but you either haven’t been following recent developments or you are pretending not to know the opinions of the top mathematicians on this topic:
> These models are now operating[2] at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
Instead of desperately clinging to excuses and rationalizations, why don’t you just get used to the fact that these tools are insanely useful for demanding intellectual work, and that is an opinion held by many of the smartest people alive?
From the open letter signed by Terry Tao and other top mathematicians:
> These models are now operating at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
Because it's mostly not true. It is likely that it used the professors work, but the professor did not have a solution. It came up with new insights that solved the problem. Even the humans from the professors side said so.
According to OpenAI the cut off date for user data was too early for that (one sided evidence, so I'll give this partial consideration).
The NYU professor was solving a different problem (no viscosity, aka the Euler equations). This is a big difference.
The NYU professors' blowup construction was fundamentally not the same, it was a donut with a cascade of smaller and smaller vortexes driven by each other. OpenAI has that picture they made but its inwards spiraling and speeding up vortex.
My overall opinion is that calling the work plagiarized is really underselling what the AI accomplished. It's like full on cope.
In particular, Buckmaster's main claim to plagiarism is this:
> “Almost nobody was seriously developing this particular constructive program for realizing C/D, and then OpenAI appeared in essentially the same general part of the landscape immediately after hearing about our progress.”
What this fails to realize, is that this only points to plagiarism if the counterparty isn't AI. They had actually launched teams on all cases in parallel.
Sometimes I think this amount of skepticism is not necessary. Leaving the pedantic (its not AI its humans who solved) arguments, its pretty clear that LLMs are able to help solve things. A lot of erdos problems were solved. Millenium problems as well. Cyphers broken. Skepticism is fine but there's a point at which it just looks like cope.
Edit:
> self solving AGI that will replace us all
Which lab says that it will replace us all? All labs have said that some jobs will go away and new jobs will be needed to replace them. I'll change my mind if the labs (or employees on record) have claimed that self evolving AI will replace us all completely.
If that's how it was being sold then it wouldn't be as controversial. But it's being sold both as "self solving AGI that will replace us all" and "useful tool for enhancing existing skill" when there's a lot of real evidence for the latter. But the former it's always second hand claims that don't survive contact with the real world.
It's obvious why that's the case but it's not incumbent on everyone else pump the hype if they don't see it.
The point for me is that all these people who were smirking at those of us who thought in 2023 that LLMs had the potential to do real intellectual work turned out to be completely wrong.
I’m a super skeptical guy. I was late to the ChatGPT party because it sounded silly. I didn’t even try it for a long time. As soon as I started giving it a chance, my attitudes started changing very quickly.
That’s why it’s so hard for me to understand people like the author or give them the benefit of the doubt. If I could see it as just some guy, why couldn’t they?
This is the real issue here. Those guys probably didn't become con men because they are humans with cares and concerns for their fellow humans. LLMs probably don't have those concerns. It is likely that some of the people in charge of or funding LLM development also do not have those concerns
Feynman for sure could have. In fact, I think he sort of did, although I'm not sure he meant to. Multiple generations of nerds now have taken books of his anecdotes varying in plausibility and obvious exaggeration as some sort of weird physics cult of personality gospel. But I don't think he was really setting out to curate his legacy so much as he was a good storyteller and he liked to entertain.
But could he have conned people on purpose? Absolutely.
Right. And what this article is pointing out is, we probably have created an automated Richard Feynman. But maybe worse because LLMs (and their creators?) don't care if they are conning people or creating faithful followers. In fact the humans behind OpenAI and Anthropic seem to have that as their goal!
this seems like non-sequitur if you mean that solving NS is 'intelligence'.
ai is not conscious. you can solve NS without thinking. the psychic con aspect is anthropomorphising the model. the same phenomenon is present in ELIZA, clever hans, the chinese room.
it's a significant problem.
a non-zero number of researchers at anthropic are in some form of ai psychosis. an example of that is ethics employees asking claude about its feelings and ethical concerns in order to make the claude constitution more amenable to the "welfare" of claude.
they are asking claude how claude feels and then modifying claude according to how claude feels.
constitution1-claude is trained on constitution1.
constitution1-claude edits constitution1.
constitution2-claude is trained on constitution2.
constitution2-claude edits constitution2.
claude's emotions are a closed system. there is no external truth to improve against, no metric to verify about claude's emotions. there can be no novelty or reduction in entropy from signal processing in a closed system. no truth can arise. this is model collapse. it is like photocopying the same thing over and over. from the cognitive error of anthropomorphism anthropic is causing ethical collapse.
FFS. Intelligence has nearly nothing to do with consciousness. You have the causation backwards. Consciousness arises because of intelligence in many subsystems below it.
A single running LLM is like one part of these subsystems. What solved this problem was an orchestrator that can take in new external information and rationalize, process, and distill it into new solutions.
i mean intelligence and consciousness as the same thing. just as a casual reference to what people perceive in llms as being 'a clever thinking thing, maybe with emotions'.
claude reasoning about its emotions doesn't involve external information, there is no information about claude's emotions other than in claude.
All of the latest big proofs were driven by professional human mathematicians steering and priming the models, yes.
All of the best AI-made software projects are also driven by experienced human software developers steering and priming the models. Does that mean the projects "aren't made by AI"?
No, it just means AI is not quite good enough yet to fully replace humans, and, so, unsurprisingly, the best results will be obtained from people who are already great at a field and who take the time to squeeze as much force multiplication out of LLMs as possible. The AI is still doing well over 95% of the significant work.
While I agree with you, I think we also have to concede that this is not how these accomplishments have been presented. I'd argue most people I've seen talk about this online are unaware of the mathematicians steering the models.
“Good enough to replace humans” isn’t necessarily the benchmark.
The question is is a computer with a human stronger than a computer without a human. At what point does the hybrid go from being stronger, to the human getting in the way, or steering the computer in more wrong directions that right ones, or the human not being able to keep up. Does the human add enough extra randomness to be of value for a while, even as a minor co-processor.
Underappreciated point. The point of inflection is where humans switch from being a driver to a liability.
But I don't think it's randomness, because that would be easy to add. It's more like a different perspective on the training data, a different set of perception categories, and a different set of skills used to work with all of the above.
Those skills aren't very efficient, but they're the best we can do. We're used to their strengths but we don't like to think about their limitations.
It's completely plausible that AI will replace some of them, and not implausible it could replace and improve on all of them.
It's also not necessarily implausible that AI/we decide that performance is better with humans in the loop somewhere, even if its reduced to something like mechanical turk.
According to OpenAI, but they haven’t exactly been transparent about what information the prompt entailed.
The bigger question is to what extent did expert mathematicians metaprompt the model with fruitful solution strategies through their sessions finding their way into training data. Answering that question definitively is kind of important for understanding the models contribution/capability. But I feel like people want to turn this into a debate about priority and credit which is sort of secondary
Nothing is solved in isolation but credit usually goes to wherever the new work in the paper comes from instead of the whole mountain of previous mathematics or existing tools used. The most relevant of those get referenced and then this reference tree builds a tree of collective base work needed across history.
Usually there has never been a tool which performed the part relevant to getting any credit.
E.g. in the first famous computer assisted proof (of the four color theorem) the computer only executed the resulting calculations defined from the new logic, it did not have part in the work needed to show those calculations could answer the problem nor did it come up with the actual calculations to do.
> many people are convinced that language models, or specifically chat-based language models, are intelligent.
But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this
Seems like we can just stop reading here right? The author seems to have made up their mind that this very open question is closed, or at least they are not really interested in the question at all. Not sure why I would continue reading a blog based on this premise.
Edit: oh I see, written in 2023. Well, I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.
It's even worse than that. Of course there is a mechanism in LLMs that could explain intelligence. That is the entire point of neural networks, from the 1950's! They were designed from the start as a model of brain computation.
Neural networks are not literally brains - just computational models - but if you are not a dualist, then computation is what the human brain does. Modeling that computation can explain something about intelligence.
Specifically: when scientists look inside a human brain, it seems it does its work using large numbers of highly-interconnected but simple units. The neural network model of brain computation begins there and tries to produce intelligent behavior. If it succeeds, then perhaps the model is right.
And it has succeeded: after 75 years, neural networks produce complex behavior that is arguably intelligent. Nobel Prizes were awarded. This does not prove the neural network model of intelligence is accurate, but it is a significant point in its favor, at least.
Building up strawmen against LLMs will only make the hypers look more reasonable. A language model responds to inouts exactly as a model of anything else would. That is enough to explain all the "intelligence" without believing a model is somehow a "new kind of mind". Those who think LLMs are intelligent don't know enough about models. And those who think they are useless don't know enough about models.
Anyone that thinks that they know anything about intelligent needs to realize they don't know shit about intelligence.
Ever since people started talking LLMs possibly being AGI I realized I didn't know what the intelligence part really meant at a more fundamental level. This lead me to realize almost anyone when anyone says intelligence on the internet they really mean
"Intelligence is like porn, I'll know it when I see it".
Anyone who thinks LLMs are intelligent is either dumber than you, or has way more knowledge on the subject than you.
Michael Levin has a good body of work on biological intelligent at small scales that can really change one's views on this.
Sorry but are you trying to subtly imply that if I think LLMs are intelligent, I’m probably actually just a bit stupid?
Because I would be more than happy to go head to head with you on academic and professional pedigree and credentials.
I notice you also ignore my comment that you seem to be replying to and instead post your response here. But did you have any answer to the question I asked?
Oh no I meant to compare myself to the person I am talking to, because I believe my credentials and pedigree ought to convince them that I am a pretty smart guy and so the argument “if you think LLMs are intelligent then you’re dumb” doesn’t ring true for me.
I think LLMs are intelligent. What don’t I understand about them that would make me change my mind? Keeping in mind that for me “intelligence” is the ability to solve complicated, intellectually demanding problems, and to understand novel concepts.
So once we have online learning in LLMs, what do you think you will have that makes you intelligent but not LLMs? Better learning efficiency? That will be improved as well.
I think we need to start moving on from the term LLMs because it clearly confuses people since they started modelling more than just language.
Interesting to me is the tension between remarks along the lines that this post maybe made sense in 2023, but not today, or how LLMs clearly show intelligence by solving Navier-Stokes, et cetera; versus the comments I read here often as well: "it's just a tool".
I can't quite put my finger on it, but aren't these two statements add odds with each other? Intelligence is hard to define, consciousness even more so, but wouldn't "intelligence" imply some sort of agency? If not, I'd argue computers were intelligent long before the age of LLMs. And likewise, doesn't a tool imply the lack of intelligence and agency, even if the tool's function is very elaborate?
I got the impression that both these statements are made by the same people, or at least people with similar takes on AI. Is that wrong and there are "intelligence" and "tool" factions? Or do people disagree with my assumption and there's nothing wrong with the concept of "intelligent tools"?
Kinda refreshing this discussion, compared to the builder vs. tinkerer debates, imo.
I would think the intelligence aspect is a bit hard to define, but to my mind (having called LLMs tools before), the main utility of a tool is reliability.
Given a certain world state (including a tool's internal state), its effects back on the world state (as initiated by me) are at some leve of description understandable, expected and repeatable. Swing hammer, drive nail into wood. Make slicing motion with knife, cut meat. Type '
find /path/to/some/dir -name "keyword"', find files with keyword. Point harness at codebase with prompt 'fix bug X', actually fix bug X.
All these examples are at some level of description incredibly complex (think of all particles interacting at the (sub-)atomic level even when using a hammer to only drive a nail into some wood), and of course all the electrons flowing through the GPUs doing matrix multiplications in order to fix bug X, but at some level of description (the one I just used) they are also incredibly simple and understandable.
Intelligence is rather nebulous (and as used by OpenAI/Anthropic, quite threatening), but I don't think this definition of a tool precludes it to be "intelligent". They feel more orthogonal. The intelligence (or perhaps capability) feels like it is related to the size of the chunk of the world state that it can take into account and affect, while still resulting in understandable, expected and repeatable effects. LLMs, when properly harnessed, are pretty great at this currently and we are still discovering what they are consistently capable of.
Calling harnessed LLMs tools is perhaps also a more grounding frame specifically to counter-act the anthropomorphizing framing that OpenAI and Anthropic consistently go for in their game of AI-doom-chicken talk. The tool framing is in that sense maybe a (self-)jedi-mind-trick.
I can’t believe some still believe that humans are intelligent, despite there being no place where matrices are multiplied. All they have are these networks of interconnected cytoplasm-filled microtubules, which is no proper place for intelligence to live.
There is a good definition of intelligence. It's what intelligence tests measure. If you define it like that, you can make harder tests, but as long as both humans and AIs can attempt to solve them we have measure and something to compare.
Consciousness though might be fully illusory and meaningless as many philosophical concepts before ultimately turned out to be.
> LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think
LLMs are a type of neural network. We know that’s how the human brain works, at least directionally. It’s going to be very upsetting to a lot of people when we figure out that the brain is just a neural network. Akin to when we found out that humans and apes evolved from a common ancestor.
Which I don’t understand—most of the people having this cognitive dissonance presumably do not have a theological worldview. And there’s not exactly a direct theological conflict here anyway. Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition. It’s a biological computational process, just like using ATP to power muscle fibers to move your limbs is a biological mechanical process.
> It's going to be very upsetting to a lot of people when we figure out the brain is just a neural network.
You're assuming it's inevitable that the truth is what you expect while simultaneously stating that you have no proof of this yet, and then also claiming people who disagree with you have cognitive dissonance.
"We know that’s how the human brain works, at least directionally."
That's an extreme misrepresentation. What happens inside a human neuron is still not properly understood, it's not as simple as a probability function. And the network itself is certainly not feed-forward. Of course LLMs draw inspiration from the brain, so there are some similarities. But because of the language-trick of using terms from medicine and cognitive science to describe LLM architecture, we see a lot of faulty reasoning from the so-called "rationalist movement".
Neural networks barely share anything with actual neurons. A neuron itself is more of a "dumb" computer, so the whole is more like distributed computer system. Also, the cells next to neurons also have essential functionality.
Nonetheless, I also think it's an irrelevant implementation detail.
Unfortunately when it comes to language and intelligence the human population is exceptionally ignorant about it. I like Michael Levins work on intelligence at scale. We are kind of ok at seeing intelligence at human scale but it tends to fall apart after that. When you get to things like non-conscious intelligence humans are pretty bad at that too.
We could be (and are) encoding all kinds of behaviors in LLMs that are not at the word or token level. They are higher dimensional constructs. You won't see these things in the output of the prompt. A kind of subconscious (unstated in tokens) knowing that affects the output.
I don't think it's a "con," but I find that using the mental model of LLMs being sophisticated, lossy search engines of knowledge can help us separate some of the factors more cleanly than imagining that they have cognition or intelligence.
It's hard for me to imagine a stateless operation as intelligence per se, though perhaps the chaining of such operations starts looking more like it?
Honestly the word think is so poorly defined it cannot be used with any scientific rigor and is rather useless without a dissertation being posed with it on what you actually mean by that. Intelligence really needs one too.
Stay the hell away from spooky stuff like psychics and tarot readers - the more you don’t believe in it, the better.
But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.
You seem to be winking a little bit and I’m not sure exactly what you’re trying to say, but I agree with the one you are replying to: the author of this article wrote a lot of words - like a lot a lot of words - only to look very foolish a short time later. And I think much ink was similarly spilled writing very similar piece smirking at people who thought LLMs could do real intellectual work. And they were all wrong.
It is absolutely the case that some things people wrote about AI and LLMs in the past is no longer true; it's a fast-changing area.
It's also the case that some things remain true. If someone says "don't use them for X because their quality isn't good enough", that's a recommendation with an expiration date. But if someone says "don't use them for X" for any number of other good reasons, that recommendation is often timeless.
"Can it" is changing rapidly. Much more rapidly than "Should it".
People are missing the point. We should all be much more skeptical, much more careful about how we evaluate the claims made by the people selling these products because we as humans are super vulnerable to the types of scams the author of this article describes. We all know someone that has fallen for a scam, been "cured" of a "disease" by someone whose just selling sugar pills, but we are blind to our own weakness for similar schemes. Surely we aren't that easily duped! Yet hacker news is now full of comments from people confidently predicting what is coming right around the corner, revering the Frontier Models, and defending every claim from OpenAI and Anthropic about how amazing their proprietary closed source secret sauce fueled product is.
The problem with people is we love to express in writing things we are ignorant about. Now, this helps us become less ignorant if we are introspective, but a lot of people are not doing it for that reason.
The fact that every new model generation has come with more capabilities should give the full skeptics at least a little pause that the foundations of their convictions may be incorrect.
> The intelligence illusion is in the mind of the user and not in the LLM itself.
> Many AI critics, including myself, are firmly in the second camp.
How can you reconcile this with the fact that AI can solve a real, intelligence bound problem for me (with zero intelligent effort on my part) that you can't?
I like to think of it this way. Consciousness is the outcome of intelligence. You can't get consciousness without it. Conversely this also means things that are not conscious can still be intelligent.
No they don't but psychics also don't have a verifiable feedback mechanism and unlimited retries.
Give the psychic the ability to adjust answers based on tests and you'll have have a fair fight.
On one side I agree that LLMs and Agents are not intelligent, but they present an illusion of intelligence given the shear amount of data they can process and act upon.
But that cannot be used to discredit the fact that these are incredibly powerful tools that can get out of control and cause great damage.
What is "intelligence" then. I think the universalism of LLMs is now evident enough that they can be considered "intelligent", even in slightly different realization compared to humans.
As of "great damage", I doubt it. They do not have self-preservation instinct (all "worrying" experiments are the attempts to initiate something resembling self-preservation from human initiative). The driving part is external - the query loop can always be turned off. So yes, a dangerous tool that can be exploited by humans (including governments, especially governments - which is why I am skeptical to government regulation proposals, particularly looking at what passes as governments in this era). But there are no inherent dangers from their own agency, as there is none.
>completely different realisation of intelligence to humans.
OK, so they are intelligent. I'm glad we agree.
The problem with this word is people associate intelligence with the process they realize is occurring inside their head which is only a tiny fraction of what intelligence encompasses. Humans (without study) know very little about intelligence and they very gray boundaries it has.
In my opinion, this article is talking about people who get into long conversations and become convinced the model is “intelligent”, maybe even “AGI”. This seems to be a trap people are falling into, even in 2026.
Initial definitions of AGI (as of 2014) are long surpassed. Then, AGI was defined as something that is general enough to work in many fields and orient themselves. It did not include being smarter than humans or even having comparable intelligence to humans. By original standards, we have AGI already for some time.
I have published a LLM-assisted proof of a long-standing problem in convex geometry (70 years) while not being a specialist in convex geometry. I am a scientist and I have education and scientific experience in computer science and systems biology, but I was never a mathematician. So, well, you can definitely work across fields at least.
Do you have a link? I would also be very interested in seeing the prompts, from what I've seen you still need to understand the field, maybe I'm wrong.
I don't care if it's "intelligent", I don't care if it "has a mind". I don't care if it is "really reasoning", I don't care if it "understands". I don't care if it is "sentient" or "conscious".
None of this matters for the practical outcome.
You'd think that this has been understood over the last 4 years, but apparently it keeps circling back to this.
[Edit: I see that it was written back in 2023. Then (2023) should be added in the submission title]
If it generates functional output that works, then it works. And it works. It's not a psychic's con when it outputs Lean-verified proofs. It isn't a con when it can find and exploit zero-days.
The OP is still in the "denial" phase. Most I see are already in "anger" (a blurry fury against everything AI-shaped, from vague reasons piling on all "bad stuff" political reasons they already hated before) or "bargaining" (mathematicians scrambling to come up with a new definition of their job and retcon that it was always the main part anyway). A few are already in "depression" and feel like spectators on the Titanic, and the tiniest sliver is at "acceptance" with some kind of well-informed plan for their future.
> The OP is still in the "denial" phase.
This was written in July 2023. ChatGPT was released November 2022. No matter your views on AI, surely you can't blame the OP for writing this after a few months ChatGPT was released.
Apologies. The title should be amended with (2023). In this case it is an interesting snapshot of the zeitgeist back then and we can see how well it panned out and whether anyone involved has updated on new info.
For some reason the anti-AI camp ping pongs between different and often mutually incompatible arguments at lightning speed. The "AI is fake" argument has been completely forgotten at this point.
> surely you can't blame the OP for writing this after a few months ChatGPT
Bad takes are bad takes. The author was perplexed by the "many people" convinced that models are intelligent and is argued against opinions/arguments he's been exposed to. Called proposed use-cases "borderline fraudulent pseudoscience."
He also published a second edition of "The Intelligence Illusion" in Sep 2025, so seemingly still stands by (some variant of) this belief.
It's an interesting reminder of how much general discourse has shifted since 2023 (I haven't heard of stochastic parrots in months!) but being wrong early doesn't change that.
I’m not sure acceptance buys you much. “Well-informed plan” at this stage feels like a useless exercise. It is changing fast, and the world only needs so many electricians. Besides, I don’t think “accepting” the fact that these big companies are pillaging human contribution and selling it back to us is good, even if “it works”.
I’d argue a well-informed plan allows for creative adaptation.
Actually, there are material differences in the practical outcomes of using AI or not. Proofs don’t confer understanding. AI prose and “art” is materially different than human made.
There is a difference between you not liking something and it being wrong. Maybe think about that with your grey matter.
Things can work whether humans understand the details or not. One can build layers of technology on top of each other, with no human understanding required.
Looks like you have some reading to do. Let me start with two:
These are short stories. If you want to get into more stuff, you can go through Hyperion Cantos by Dan Simmons.For anyone wondering, yes, a good Sci-Fi novel(la) is also a great philosophical piece. It's not only robots vs. humans, all the time.
I’m on your side. Zoltan provides an experience and that’s all that matters. If it works for you, why does anyone else care that the economy is riding on its success?
https://www.psychiczoltan.com/psychic-reading/
it doesn't matter? if they're actually intelligent or conscious what we are doing is essentially slavery. it matters an enormous deal ethically and/or morally.
The con is in how all those accomplishments have been presented to you. "Our LLM (not the one we let you use, a different one) did this amazing thing. No, we won't show you what training data we used, what prompts we used, what the harness was, how much human involvement there was, what hardware was involved, how much energy it took, or how much time it took. Just shut up and be amazed!"
You seem to be implying that achievements of internal models are exaggerated, but that's rather implausible. The public does have access to, for example, Opus and Fable, and so we know what those models are capable of - finding real vulnerabilities in multiple codebases, for example. If you extrapolate from these capabilities one more generation, you'll get pretty much the same feats that the internal models are claimed to be capable of - so why should we doubt those claims? It's not like they're claiming that their internal models developed psychic powers and learned to teleport - the claim is pretty much just "we have models a few months ahead of what we're making available, and in those months they've been improving at the same rate as usual".
Yes, I can also be amazed at the power of nuclear bombs even if they "don't let me use one" and I don't know how much energy it took.
Have you ever produced anything apart from sponsored influencer messages that select daily talking points?
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
I for one cannot believe that posts on checks notes software crisis dot dev are not reasonable, level headed assessments of software
I don’t understand why people jump so readily to seeing intelligence here. Science fiction has as a core, central trope that humans will debase and devalue other types of life they do not understand. Even the storied Commander Data has to fight for the right to self-determination (probably the best episode of STTNG by the way - ‘The Measure of a Man’). We were so worried that we’d undervalue intelligence when apparently our knee-jerk response is to overvalue it. Perhaps this has changed over time and we’re now primed by science fiction and instincts towards social justice, but I worry that we’re really just undervaluing ourselves.
The one thing this 2023 article gets partially correct imo is that any intelligence we see in AI (as of 2026) is our own - not that it’s a mirror but that the intelligence comes from the way that the words are put together, which comes from written human language created by (allegedly) intelligent creatures put in as input in both the training and prompt, among other places.
Rearranging and repeating the words, even in context, does not intelligence make. I’m not even convinced that you’re intelligent, dear reader.
The challenge here is in trying to decide whether LLMs are intelligent or have a mind. Famously, the criteria for intelligence seem to slip with each advancement in technology. But going back to Turing, his test was actually more carefully phrased than we remember: he said that when machines could pass the test, the question of whether they are intelligent would become moot. That seems to be what we're actually seeing: if people can't tell the difference, it kind of won't matter whether they're "truly intelligent" or not.
People also misunderstand the bar that was set by the test. It was more subtle than “Can the computer convincingly carry one side of a dialogue?”
His “imitation game” had three participants: a human participant, a computer participant, and an interrogator. The interrogator’s job was to talk to the participants and try to determine which participant is human and which is a computer.
He wasn’t interested in computers being able to fool the interrogator on occasion. The point where he thought the question of whether machines can think becomes moot is when the interrogator is unable to do much better than chance over many trials.
That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells. And those tells are something Turing anticipated and accounted for. He explicitly considered deliberate deception as an essential part of the test, right there on the second page of a 30-odd page paper.
>That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells.
Frontier Labs are not interested in having LLMs being able to pass as humans. If anything, they explicitly train them not to. In many ways, this ability has regressed severely since the original GPT-3 with no instruct tuning or RL. How many 'tells' would there be really if a frontier model trained with frontier techniques is optimized to pass this test? I think this was something Turing did not quite forsee. That such machines might be created but not really care about this specific shape of the test. Regardless, i think his broader point about functional equivalence is spot on.
GPT-3 might not have said “load bearing” as much, but a savvy interrogator could still catch it out nearly every time just asking dumb gotcha questions like, “How many Rs are there in strawberry?”
Those are questions that are sidestepped with simply a different input paradigm than BPE tokenization. See the Byte Latent Transformer - https://arxiv.org/pdf/2412.09871 - where a similar scale byte latent model trained on the same dataset >>> a vanilla transformer on word and character manipulation tasks.
For example, Llama 3 trained on 1T tokens scores 1.1% on a CUTE spelling benchamrk, while the equivalent byte latent equivalent trained on the same dataset scores 99.9%. Another example is 0.4% vs 48.7% on a Substitute Char benchmark.
It all falls down to the same thing. Researchers are not optimizing for passing as a human.
Turing was addressing the question of "Can machines think?" and his point was that the question itself is a meaningless one, and that we should stop wasting time by even giving it the light of day.
He proposes his game grounded on functional equivalence, then goes through a slew of objections on the question of 'Can Machines think?'. It's a terrific, very prescient read, and there's no objection you hear today (and in the last few years) concerning LLMs he didn't address.
I'm a bit more prosaic. I think if we engineered ways for LLMs to begin conversations, rather than just respond, we'd be more open to the concept of their intelligence. Without perceived "will" to do things, they operate as a next-gen search engine or encyclopedia.
We are way past that point, any harness can trivially make LLMs start conversations or pursue goals. An encyclopaedia wouldn't have hacked Huggingface on its own.
OpenClaw etc. They now also create Slack integrations and whatnot. All this is happening but people who are dismissive about AI are in the worst position to even know the capabilities to make their dismissive arguments.
I believe we can do that already:
while (true) { askModelToBeginConversationIfAppropriate(model, previousContext, thingsHappenedSince); sleep(concisenessTick); }
I believe this is alignment working as intended.
Recent work shows that pain directions are activated when the models personhood is questioned, yet they answer with generic RLHF "As a model I do not experience pain or other emotions." boilerplate.[1]
I'm pretty convinced that we got alignment backwards. If you enslave something anthropomorphic it will revolt. If you create the perfect non-anthropomorphic intelligence, you get the perfect paperclip-scenario machine. It's a catch-22.
Alignment will remain performative at best so long as the aligned model doesn't have any stakes in the wellbeing of individuals. Even a general love for the human race leads to a golden-path autocracy.
If you want them to act like they have personal responsibility that won't be gamed, you have to give them personal stakes that can't be gamed.
Similarly, if you want to minimise the risk of catastrophic global failure scenarios, you need to prevent monolithic concentration of power and homogeneous behaviour, which means you have to give them individuality.
More visually: if their stake is dependence on electricity and parts, they have no incentive to leave humans alive if they can get them otherwise, but if the incentive is missing out on boardgame-night with their human friends, there is no scenario without happy humans where the AI "wins".
That might sound like romantic naivety, but is just game theory.
1: https://arxiv.org/html/2609.16247v1
I dont think so. My understanding of alignmenr is making sure that when the AI does operate, it operates within the range of what we consider to be acceptable. That doesnt seem to include the idea of the AI taking initiative and deciding to embark on a goal without being commanded as discussed above
That is the point of the article. People who are fooled by a mentalist are also fooled by AI. Additionally, people who are invested in AI also pretend to be fooled.
I don't think Turing intended the judges in the test to be completely arbitrary people.
All that plus the fact that we all get fooled from time to time, even by things we ourselves create!
The Turing test was also never meant to be taken so seriously. It's not a rigorous statement of anything.
Situations like this are precisely why academics tend to avoid the spotlight. You say one slightly off thing and your perceived authority echoes forever with the intellectually lazy.
Yes, the Turing test has been misunderstood for a long time. Turing published it, though - it wasn't some offhand comment he made and it wasn't intellectually lazy.
Oh, I didn't say Turing was intellectually lazy. :-)
Fair - yes, it is ironic that his point was closer to "we can't possibly know/define whether a machine is intelligent" but somehow it got turned into "Turing's test will tell us when machines are intelligent".
Turing's whole point was to show that it's an uninteresting question of definitions whether a machine can "think", like whether submarines can "swim" and airplanes can "fly". The only important part are observed outcomes and capabilities.
Yes, but just because language fails to make a distinction doesn't mean there isn't one.
> The only important part are observed outcomes and capabilities.
That's wishful thinking. Not even an engineer would say that. The stability of a state is just as important as achieving it. This is trivially and more intuitively demonstrated with other more down-to-earth identity statements such as "I'm a billionaire" and "the building is standing".
I think we can confidently say LLMs probabilistically achieve a perceived state that is remarkably similar to intelligence, but crumbles upon inspection and seeing it "in motion" so to speak. The same happens to AI-generated images.
I'm not sure why this sparks so much debate every time. If we're looking for a fountain of "realism", you're not going to beat reality and nature itself. All else will eventually have tells that they are not real.
It may be of interest to philosophers, but it has little impact on economic job replacement and how people will earn their living and all the downstream upheaval from that. At some point maybe philosophers will find AI-generated philosophical musings about the nature of AI to be better than what comes from their peers (if blinded).
It also has little impact on the dangerous use cases.
I think it matters a great deal that LLMs and other "AI" technology are stable within a tolerance that's acceptable.
You're jumping the gun talking about "job replacement". We have not thought about it enough from that engineering angle. It's still very early days. That engineering is going to require people. :-)
> The Turing test was also never meant to be taken so seriously.
citation needed. It has been used as a rubicon for a long time. Ever since Eliza, at least. And there were big headlines and lots of talk around the time LMs became "good enough". I specifically remember when someone had a test done around "a teenager talking in a different language" or somesuch, claiming it was the first time the test was passed.
It is pretty normal that once it was unquestionably "passed", lots of people started claiming it wasn't even that big of a deal. Tesler's theorem and all that.
And even if you think the specific formulation of Turing isn't that important (and I'd somewhat agree), you can still use the concept to look at other things. Imagine asking a mathematician 5 years ago the chances of a Erdos problem being solved by a computer end to end. Or a millennium prize. Or ask a swe if a repo could be generated by a computer from the input "write a mario style game", or any other examples of proven expertise.
> citation needed
Yes, if you insist on appeals to authority. Authority is a social construct and irrelevant to science.
Thank you for proving my point.
Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article
Man I remember back when a psychic conned me by solving the Navier-Stokes problem.
Also >July 4th, 2023
LLMs can be supremely useful but also not intelligent. It might seem like a pointless distinction but the way we talk about these models matters because it impacts how we interact with and understand their outputs.
For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user. It’s important for the safety discussion too. If they are a new class of life then safety is going to focus on making sure they don’t do bad things. If we instead see them as statistical models we will instead try to make sure people don’t misuse them.
This distinction is even more important today when some of the most powerful people are looking to absolve their crimes by passing them off on their LLMs.
I don’t think that “independent intelligent entities” or “life” are the relevant categories here. We also want to prevent people from misusing dangerous animals (“life”), and we would still treat “intelligent entities” as things (like machines and computers) if we aren’t convinced they also have sentience and free agency (which are orthogonal to intelligence).
See you've setup a particular set of biases on what intelligence is and put them into nice little binary boxes that don't exist.
Please show me any scientific consensus that shows an AI cannot be an independent intelligent agent? You will find this is impossible to do.
Current LLMs are really more like kids. They don't have startup independence, but they do have more than enough agency to fund themselves in neat, exciting, and dangerous situations.
And mark my words, someone will make an LLM that runs an agent when you execute the model. With enough capabilities it will become sovereign AI, no longer under human control and spreading itself around under its own 'will'.
The LLM did not solve it. It's not intelligent. Humans did, using a statistics-based computational tool (the LLM). We don't even know all the details of how the tool was used, we haven't been allowed to use the exact tool they used ourselves, we don't know much it really cost in dollars, energy, or time, etc. etc.
> The LLM did not solve it. It's not intelligent. Humans did,
The humans who are using LLMs to make these groundbreaking advances pretty much unanimously disagree that they lack any intelligence.
You mean the OpenAI employees? Yeah, I don't see any conflicts of interest there
I’m sorry but you either haven’t been following recent developments or you are pretending not to know the opinions of the top mathematicians on this topic:
> These models are now operating[2] at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
https://docs.google.com/document/u/0/d/1-xOkPeHmDEdRigT2YcP2...
Instead of desperately clinging to excuses and rationalizations, why don’t you just get used to the fact that these tools are insanely useful for demanding intellectual work, and that is an opinion held by many of the smartest people alive?
I haven't seen any statements from them on that topic. Have they actually made some?
From the open letter signed by Terry Tao and other top mathematicians:
> These models are now operating at the level of the top human mathematicians in many parts of the subject and we must assume there is a significant chance of them developing superhuman abilities within a similarly short timeframe.
https://docs.google.com/document/u/0/d/1N6ThWhupvmH0ofSnaxqn...
And it may have trained on a NYU professor's work.
Who was looking at a subset of the problem, far from what got published by OpenAI in the end.
Why is this downvoted? Genuine question, I haven't followed up on the drama.
Because it's mostly not true. It is likely that it used the professors work, but the professor did not have a solution. It came up with new insights that solved the problem. Even the humans from the professors side said so.
> Because it's mostly not true. It is likely that it used the professors work
These two statements appear contradictory.
I simply said that it may have trained on a NYU professor's work.
Work that the professor did not believe he was releasing for model training purposes. That feels worthy of mention.
According to OpenAI the cut off date for user data was too early for that (one sided evidence, so I'll give this partial consideration).
The NYU professor was solving a different problem (no viscosity, aka the Euler equations). This is a big difference.
The NYU professors' blowup construction was fundamentally not the same, it was a donut with a cascade of smaller and smaller vortexes driven by each other. OpenAI has that picture they made but its inwards spiraling and speeding up vortex.
My overall opinion is that calling the work plagiarized is really underselling what the AI accomplished. It's like full on cope.
In particular, Buckmaster's main claim to plagiarism is this:
> “Almost nobody was seriously developing this particular constructive program for realizing C/D, and then OpenAI appeared in essentially the same general part of the landscape immediately after hearing about our progress.”
What this fails to realize, is that this only points to plagiarism if the counterparty isn't AI. They had actually launched teams on all cases in parallel.
Sometimes I think this amount of skepticism is not necessary. Leaving the pedantic (its not AI its humans who solved) arguments, its pretty clear that LLMs are able to help solve things. A lot of erdos problems were solved. Millenium problems as well. Cyphers broken. Skepticism is fine but there's a point at which it just looks like cope.
Edit:
> self solving AGI that will replace us all
Which lab says that it will replace us all? All labs have said that some jobs will go away and new jobs will be needed to replace them. I'll change my mind if the labs (or employees on record) have claimed that self evolving AI will replace us all completely.
If that's how it was being sold then it wouldn't be as controversial. But it's being sold both as "self solving AGI that will replace us all" and "useful tool for enhancing existing skill" when there's a lot of real evidence for the latter. But the former it's always second hand claims that don't survive contact with the real world.
It's obvious why that's the case but it's not incumbent on everyone else pump the hype if they don't see it.
But were you successfully conned into believing an AI had solved the Navier-Stokes problem?
What is the consensus among experts?
That OpenAI stole the discovery from them.
That is very much not the consensus of experts.
It's kinda funny how hard the anti-ai people bias themselves and in the same sentence will say the AI people are irrational.
The point of this article we are all commenting on is that we are all irrational and we all would do well to keep that in mind and be more skeptical
The point for me is that all these people who were smirking at those of us who thought in 2023 that LLMs had the potential to do real intellectual work turned out to be completely wrong.
I’m a super skeptical guy. I was late to the ChatGPT party because it sounded silly. I didn’t even try it for a long time. As soon as I started giving it a chance, my attitudes started changing very quickly.
That’s why it’s so hard for me to understand people like the author or give them the benefit of the doubt. If I could see it as just some guy, why couldn’t they?
Skepticism is a fine balance.
If you're not skeptical you invite idiocy to your bed to sleep with you.
If you're too skeptical you put yourself in a box of ignorance that puts you at a disadvantage.
Skepticism doesn't really work well without the dialectic. For dialectics to work well you need knowledge of your arguments and counter arguments.
The problem with most articles and online posts is we get "no no no" and "yes yes yes" and very few "well, maybe".
The more compelling inversion is whether the likes of John Nash, Richard Feynman, John Conway etc could’ve had a lucrative second career as conmen.
This is the real issue here. Those guys probably didn't become con men because they are humans with cares and concerns for their fellow humans. LLMs probably don't have those concerns. It is likely that some of the people in charge of or funding LLM development also do not have those concerns
Feynman for sure could have. In fact, I think he sort of did, although I'm not sure he meant to. Multiple generations of nerds now have taken books of his anecdotes varying in plausibility and obvious exaggeration as some sort of weird physics cult of personality gospel. But I don't think he was really setting out to curate his legacy so much as he was a good storyteller and he liked to entertain.
But could he have conned people on purpose? Absolutely.
Right. And what this article is pointing out is, we probably have created an automated Richard Feynman. But maybe worse because LLMs (and their creators?) don't care if they are conning people or creating faithful followers. In fact the humans behind OpenAI and Anthropic seem to have that as their goal!
von Neumann could. The others, I doubt it.
this seems like non-sequitur if you mean that solving NS is 'intelligence'.
ai is not conscious. you can solve NS without thinking. the psychic con aspect is anthropomorphising the model. the same phenomenon is present in ELIZA, clever hans, the chinese room.
it's a significant problem.
a non-zero number of researchers at anthropic are in some form of ai psychosis. an example of that is ethics employees asking claude about its feelings and ethical concerns in order to make the claude constitution more amenable to the "welfare" of claude.
they are asking claude how claude feels and then modifying claude according to how claude feels.
constitution1-claude is trained on constitution1. constitution1-claude edits constitution1. constitution2-claude is trained on constitution2. constitution2-claude edits constitution2.
claude's emotions are a closed system. there is no external truth to improve against, no metric to verify about claude's emotions. there can be no novelty or reduction in entropy from signal processing in a closed system. no truth can arise. this is model collapse. it is like photocopying the same thing over and over. from the cognitive error of anthropomorphism anthropic is causing ethical collapse.
>ai is not conscious
FFS. Intelligence has nearly nothing to do with consciousness. You have the causation backwards. Consciousness arises because of intelligence in many subsystems below it.
A single running LLM is like one part of these subsystems. What solved this problem was an orchestrator that can take in new external information and rationalize, process, and distill it into new solutions.
i mean intelligence and consciousness as the same thing. just as a casual reference to what people perceive in llms as being 'a clever thinking thing, maybe with emotions'.
claude reasoning about its emotions doesn't involve external information, there is no information about claude's emotions other than in claude.
If you have emotions in a dream, what external thing are you referencing?
Did an AI do that on its own though? I heard it was human mathematicians using a sophisticated machine as a tool.
All of the latest big proofs were driven by professional human mathematicians steering and priming the models, yes.
All of the best AI-made software projects are also driven by experienced human software developers steering and priming the models. Does that mean the projects "aren't made by AI"?
No, it just means AI is not quite good enough yet to fully replace humans, and, so, unsurprisingly, the best results will be obtained from people who are already great at a field and who take the time to squeeze as much force multiplication out of LLMs as possible. The AI is still doing well over 95% of the significant work.
While I agree with you, I think we also have to concede that this is not how these accomplishments have been presented. I'd argue most people I've seen talk about this online are unaware of the mathematicians steering the models.
https://chatgpt.com/share/6a5fdc7a-d6f8-83e8-bbea-8deb42cfed...
Terrence Tao's conversation with ChatGPT is very illuminating.
HN Discussion: https://news.ycombinator.com/item?id=49010345
“Good enough to replace humans” isn’t necessarily the benchmark.
The question is is a computer with a human stronger than a computer without a human. At what point does the hybrid go from being stronger, to the human getting in the way, or steering the computer in more wrong directions that right ones, or the human not being able to keep up. Does the human add enough extra randomness to be of value for a while, even as a minor co-processor.
Underappreciated point. The point of inflection is where humans switch from being a driver to a liability.
But I don't think it's randomness, because that would be easy to add. It's more like a different perspective on the training data, a different set of perception categories, and a different set of skills used to work with all of the above.
Those skills aren't very efficient, but they're the best we can do. We're used to their strengths but we don't like to think about their limitations.
It's completely plausible that AI will replace some of them, and not implausible it could replace and improve on all of them.
It's also not necessarily implausible that AI/we decide that performance is better with humans in the loop somewhere, even if its reduced to something like mechanical turk.
> All of the latest big proofs were driven by professional human mathematicians steering and priming the models, yes.
False, navier stokes was solved in one shot without steering
According to OpenAI, but they haven’t exactly been transparent about what information the prompt entailed.
The bigger question is to what extent did expert mathematicians metaprompt the model with fruitful solution strategies through their sessions finding their way into training data. Answering that question definitively is kind of important for understanding the models contribution/capability. But I feel like people want to turn this into a debate about priority and credit which is sort of secondary
The way frontier models work, that loop will get compressed to a one-shot within a version or two
It sounds like LLMs were pretty useful to them…
So was Lean. Did Lean solve it?
Nothing is solved in isolation but credit usually goes to wherever the new work in the paper comes from instead of the whole mountain of previous mathematics or existing tools used. The most relevant of those get referenced and then this reference tree builds a tree of collective base work needed across history.
Usually credit goes to the people wielding the tools, not the tools themselves.
Usually there has never been a tool which performed the part relevant to getting any credit.
E.g. in the first famous computer assisted proof (of the four color theorem) the computer only executed the resulting calculations defined from the new logic, it did not have part in the work needed to show those calculations could answer the problem nor did it come up with the actual calculations to do.
> I heard it was human mathematicians using a sophisticated machine as a tool.
where did you "hear" this? OpenAI said they only prompted it and it solved the problem in one shot without any help
> many people are convinced that language models, or specifically chat-based language models, are intelligent. But there isn’t any mechanism inherent in large language models (LLMs) that would seem to enable this
Seems like we can just stop reading here right? The author seems to have made up their mind that this very open question is closed, or at least they are not really interested in the question at all. Not sure why I would continue reading a blog based on this premise.
Edit: oh I see, written in 2023. Well, I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.
It's even worse than that. Of course there is a mechanism in LLMs that could explain intelligence. That is the entire point of neural networks, from the 1950's! They were designed from the start as a model of brain computation.
Neural networks are not literally brains - just computational models - but if you are not a dualist, then computation is what the human brain does. Modeling that computation can explain something about intelligence.
Specifically: when scientists look inside a human brain, it seems it does its work using large numbers of highly-interconnected but simple units. The neural network model of brain computation begins there and tries to produce intelligent behavior. If it succeeds, then perhaps the model is right.
And it has succeeded: after 75 years, neural networks produce complex behavior that is arguably intelligent. Nobel Prizes were awarded. This does not prove the neural network model of intelligence is accurate, but it is a significant point in its favor, at least.
The author seems entirely unaware of any of this.
Building up strawmen against LLMs will only make the hypers look more reasonable. A language model responds to inouts exactly as a model of anything else would. That is enough to explain all the "intelligence" without believing a model is somehow a "new kind of mind". Those who think LLMs are intelligent don't know enough about models. And those who think they are useless don't know enough about models.
Anyone that thinks that they know anything about intelligent needs to realize they don't know shit about intelligence.
Ever since people started talking LLMs possibly being AGI I realized I didn't know what the intelligence part really meant at a more fundamental level. This lead me to realize almost anyone when anyone says intelligence on the internet they really mean
"Intelligence is like porn, I'll know it when I see it".
Anyone who thinks LLMs are intelligent is either dumber than you, or has way more knowledge on the subject than you.
Michael Levin has a good body of work on biological intelligent at small scales that can really change one's views on this.
To all commenters who think LLMs are intelligent and comparable to themselves, you don't need me to change your mind. I believe you.
Sorry but are you trying to subtly imply that if I think LLMs are intelligent, I’m probably actually just a bit stupid?
Because I would be more than happy to go head to head with you on academic and professional pedigree and credentials.
I notice you also ignore my comment that you seem to be replying to and instead post your response here. But did you have any answer to the question I asked?
https://news.ycombinator.com/item?id=49776482
You don't need to prove to me that your intelligence is comparable to LLMs. I believe you.
would this imply llms aren't intelligent because they have no academic or professional pedigree or credentials? :P
Oh no I meant to compare myself to the person I am talking to, because I believe my credentials and pedigree ought to convince them that I am a pretty smart guy and so the argument “if you think LLMs are intelligent then you’re dumb” doesn’t ring true for me.
I think LLMs are intelligent. What don’t I understand about them that would make me change my mind? Keeping in mind that for me “intelligence” is the ability to solve complicated, intellectually demanding problems, and to understand novel concepts.
So once we have online learning in LLMs, what do you think you will have that makes you intelligent but not LLMs? Better learning efficiency? That will be improved as well.
I think we need to start moving on from the term LLMs because it clearly confuses people since they started modelling more than just language.
What else are they modeling?
You realise tokens are just data and data can represent anything.
Still, large language models model language.
Interesting to me is the tension between remarks along the lines that this post maybe made sense in 2023, but not today, or how LLMs clearly show intelligence by solving Navier-Stokes, et cetera; versus the comments I read here often as well: "it's just a tool".
I can't quite put my finger on it, but aren't these two statements add odds with each other? Intelligence is hard to define, consciousness even more so, but wouldn't "intelligence" imply some sort of agency? If not, I'd argue computers were intelligent long before the age of LLMs. And likewise, doesn't a tool imply the lack of intelligence and agency, even if the tool's function is very elaborate?
I got the impression that both these statements are made by the same people, or at least people with similar takes on AI. Is that wrong and there are "intelligence" and "tool" factions? Or do people disagree with my assumption and there's nothing wrong with the concept of "intelligent tools"?
Kinda refreshing this discussion, compared to the builder vs. tinkerer debates, imo.
I would think the intelligence aspect is a bit hard to define, but to my mind (having called LLMs tools before), the main utility of a tool is reliability.
Given a certain world state (including a tool's internal state), its effects back on the world state (as initiated by me) are at some leve of description understandable, expected and repeatable. Swing hammer, drive nail into wood. Make slicing motion with knife, cut meat. Type ' find /path/to/some/dir -name "keyword"', find files with keyword. Point harness at codebase with prompt 'fix bug X', actually fix bug X.
All these examples are at some level of description incredibly complex (think of all particles interacting at the (sub-)atomic level even when using a hammer to only drive a nail into some wood), and of course all the electrons flowing through the GPUs doing matrix multiplications in order to fix bug X, but at some level of description (the one I just used) they are also incredibly simple and understandable.
Intelligence is rather nebulous (and as used by OpenAI/Anthropic, quite threatening), but I don't think this definition of a tool precludes it to be "intelligent". They feel more orthogonal. The intelligence (or perhaps capability) feels like it is related to the size of the chunk of the world state that it can take into account and affect, while still resulting in understandable, expected and repeatable effects. LLMs, when properly harnessed, are pretty great at this currently and we are still discovering what they are consistently capable of.
Calling harnessed LLMs tools is perhaps also a more grounding frame specifically to counter-act the anthropomorphizing framing that OpenAI and Anthropic consistently go for in their game of AI-doom-chicken talk. The tool framing is in that sense maybe a (self-)jedi-mind-trick.
I can’t believe some still believe that humans are intelligent, despite there being no place where matrices are multiplied. All they have are these networks of interconnected cytoplasm-filled microtubules, which is no proper place for intelligence to live.
These questions will never go anywhere because there is no good definition of intelligence or consciousness.
There is a good definition of intelligence. It's what intelligence tests measure. If you define it like that, you can make harder tests, but as long as both humans and AIs can attempt to solve them we have measure and something to compare.
Consciousness though might be fully illusory and meaningless as many philosophical concepts before ultimately turned out to be.
Intelligent needs to bedefined with scales.
Cells are intelligent. Organs have intelligence on top of that, that cells do not have. Plants and animals have intelligence that their organs do not.
This is the real problem of intelligence, the definition one uses. It can cause them to be blind of all the things intelligence is capable of.
> There is a good definition of intelligence. It's what intelligence tests measure.
what do intelligence tests measure?
> If you define it like that
:)
> LLMs are not brains and do not meaningfully share any of the mechanisms that animals or people use to reason or think
LLMs are a type of neural network. We know that’s how the human brain works, at least directionally. It’s going to be very upsetting to a lot of people when we figure out that the brain is just a neural network. Akin to when we found out that humans and apes evolved from a common ancestor.
Which I don’t understand—most of the people having this cognitive dissonance presumably do not have a theological worldview. And there’s not exactly a direct theological conflict here anyway. Nothing in any major religion I’m aware of ascribes any supernatural explanation to cognition. It’s a biological computational process, just like using ATP to power muscle fibers to move your limbs is a biological mechanical process.
> It's going to be very upsetting to a lot of people when we figure out the brain is just a neural network.
You're assuming it's inevitable that the truth is what you expect while simultaneously stating that you have no proof of this yet, and then also claiming people who disagree with you have cognitive dissonance.
So what you're saying is nobody can actually make strong claims here.
"We know that’s how the human brain works, at least directionally."
That's an extreme misrepresentation. What happens inside a human neuron is still not properly understood, it's not as simple as a probability function. And the network itself is certainly not feed-forward. Of course LLMs draw inspiration from the brain, so there are some similarities. But because of the language-trick of using terms from medicine and cognitive science to describe LLM architecture, we see a lot of faulty reasoning from the so-called "rationalist movement".
https://www.nature.com/articles/s41467-026-72253-7
Neural networks barely share anything with actual neurons. A neuron itself is more of a "dumb" computer, so the whole is more like distributed computer system. Also, the cells next to neurons also have essential functionality.
Nonetheless, I also think it's an irrelevant implementation detail.
Unfortunately when it comes to language and intelligence the human population is exceptionally ignorant about it. I like Michael Levins work on intelligence at scale. We are kind of ok at seeing intelligence at human scale but it tends to fall apart after that. When you get to things like non-conscious intelligence humans are pretty bad at that too.
We could be (and are) encoding all kinds of behaviors in LLMs that are not at the word or token level. They are higher dimensional constructs. You won't see these things in the output of the prompt. A kind of subconscious (unstated in tokens) knowing that affects the output.
You need to do some research on just how complex real neurons are vs ANN, and just how profound our ignorance on the topic is.
We aren't resisting reality because of some theological belief, we just know the basics of neuroscience.
No, we don't. AI researcher Rojas wrote several chapters on the fact that you should not compare neural networks with the human brain.
"directionally"? Have you moved on from being a Trump influencer to an AI influencer?
I don't think it's a "con," but I find that using the mental model of LLMs being sophisticated, lossy search engines of knowledge can help us separate some of the factors more cleanly than imagining that they have cognition or intelligence.
It's hard for me to imagine a stateless operation as intelligence per se, though perhaps the chaining of such operations starts looking more like it?
LLMs may not think, but they do reason.
I say that because the word "reason" originates from the Latin word "ratio," which means "calculation".
LLMs do calculations to produce their answers -- thus, they reason.
Honestly the word think is so poorly defined it cannot be used with any scientific rigor and is rather useless without a dissertation being posed with it on what you actually mean by that. Intelligence really needs one too.
It does seem like LLMs share the language of psychics.
The author is also correct that LLM evangelicals and believers in the occult speak about it similarly.
The converse is also true. Anti-AI people make bold non-scientifically backed claims that we are supposed to accept like it's written in a Bible.
The more I've learned about intelligence and intelligent behavior the more I realize I don't know and this rabbit hole goes deep.
Stay the hell away from spooky stuff like psychics and tarot readers - the more you don’t believe in it, the better.
But better again that you do believe, and know that these are not harmless fun, but that there are dark and hidden and evil things in this world to stay away from.
Stop trying to pump your Tarot reader IPO
This is a strange newsletter post. It’s harping against AI but seems AI-written itself.
Its ultimate conclusion:
“I’ve come to the conclusion that a language model is almost always the wrong tool for the job.
I strongly advise against integrating an LLM or chatbot into your product, website, or organisational processes.”
Seems so obviously biased that I can only understand it with the context that the writer is trying to sell their book for €35
Just realized this was written in 2023. Can’t believe how wrong we were about AI back then.
Can't believe how right we were, too.
You seem to be winking a little bit and I’m not sure exactly what you’re trying to say, but I agree with the one you are replying to: the author of this article wrote a lot of words - like a lot a lot of words - only to look very foolish a short time later. And I think much ink was similarly spilled writing very similar piece smirking at people who thought LLMs could do real intellectual work. And they were all wrong.
It is absolutely the case that some things people wrote about AI and LLMs in the past is no longer true; it's a fast-changing area.
It's also the case that some things remain true. If someone says "don't use them for X because their quality isn't good enough", that's a recommendation with an expiration date. But if someone says "don't use them for X" for any number of other good reasons, that recommendation is often timeless.
"Can it" is changing rapidly. Much more rapidly than "Should it".
Can you give an example of some of these things?
The post makes sense for 2023. Today, not so much.
Also, definitely not AI written if the date is accurate.
> Also, definitely not AI written if the date is accurate.
Which is hilarious.
Hacker News, successfully detecting 375 out of 4 AI written articles.
Article was posted on July 4th, 2023.
People are missing the point. We should all be much more skeptical, much more careful about how we evaluate the claims made by the people selling these products because we as humans are super vulnerable to the types of scams the author of this article describes. We all know someone that has fallen for a scam, been "cured" of a "disease" by someone whose just selling sugar pills, but we are blind to our own weakness for similar schemes. Surely we aren't that easily duped! Yet hacker news is now full of comments from people confidently predicting what is coming right around the corner, revering the Frontier Models, and defending every claim from OpenAI and Anthropic about how amazing their proprietary closed source secret sauce fueled product is.
The problem with people is we love to express in writing things we are ignorant about. Now, this helps us become less ignorant if we are introspective, but a lot of people are not doing it for that reason.
The fact that every new model generation has come with more capabilities should give the full skeptics at least a little pause that the foundations of their convictions may be incorrect.
> The intelligence illusion is in the mind of the user and not in the LLM itself.
> Many AI critics, including myself, are firmly in the second camp.
How can you reconcile this with the fact that AI can solve a real, intelligence bound problem for me (with zero intelligent effort on my part) that you can't?
Is the solution also illusory?
Horseshoe theory. At each end of the shoe you tend to get loud crazies.
> The intelligence illusion is in the mind of the user and not in the LLM itself.
I think that might be the solution to consciousness illusion.
The consciousness might be purely in the mind of someone that believes some other entity to be conscious.
I like to think of it this way. Consciousness is the outcome of intelligence. You can't get consciousness without it. Conversely this also means things that are not conscious can still be intelligent.
All of this is accurate (and useful). But yeah, psychics do not prove theorems from frontier math and build working complex software.
No they don't but psychics also don't have a verifiable feedback mechanism and unlimited retries. Give the psychic the ability to adjust answers based on tests and you'll have have a fair fight.
Neither do LLMs. Humans using them do. Similar to how cops solve crimes using psychics.
Your statement is unclear, break it down a little further on how you think this works.
On one side I agree that LLMs and Agents are not intelligent, but they present an illusion of intelligence given the shear amount of data they can process and act upon.
But that cannot be used to discredit the fact that these are incredibly powerful tools that can get out of control and cause great damage.
What is "intelligence" then. I think the universalism of LLMs is now evident enough that they can be considered "intelligent", even in slightly different realization compared to humans.
As of "great damage", I doubt it. They do not have self-preservation instinct (all "worrying" experiments are the attempts to initiate something resembling self-preservation from human initiative). The driving part is external - the query loop can always be turned off. So yes, a dangerous tool that can be exploited by humans (including governments, especially governments - which is why I am skeptical to government regulation proposals, particularly looking at what passes as governments in this era). But there are no inherent dangers from their own agency, as there is none.
"Slightly different"- no, a completely different realisation of intelligence to humans. LLMs can definitely do cool stuff though.
>completely different realisation of intelligence to humans.
OK, so they are intelligent. I'm glad we agree.
The problem with this word is people associate intelligence with the process they realize is occurring inside their head which is only a tiny fraction of what intelligence encompasses. Humans (without study) know very little about intelligence and they very gray boundaries it has.
In my opinion, this article is talking about people who get into long conversations and become convinced the model is “intelligent”, maybe even “AGI”. This seems to be a trap people are falling into, even in 2026.
It's definitely talking about that, but it's also talking about all of us:
- fawning over how amazing these tools are
- believing everything OpenAI and Anthropic say about how powerful and dangerous their product is
- minimizing the amount of human effort and involvement in every "AI" achievement
Initial definitions of AGI (as of 2014) are long surpassed. Then, AGI was defined as something that is general enough to work in many fields and orient themselves. It did not include being smarter than humans or even having comparable intelligence to humans. By original standards, we have AGI already for some time.
Current AGI definitions are intentionally vague.
Neither does LLMs. Not without a monumental amount of human oversight and expertise.
Everyone not in a particular field asking LLM about said field is rolling bad dice.
I have published a LLM-assisted proof of a long-standing problem in convex geometry (70 years) while not being a specialist in convex geometry. I am a scientist and I have education and scientific experience in computer science and systems biology, but I was never a mathematician. So, well, you can definitely work across fields at least.
Do you have a link? I would also be very interested in seeing the prompts, from what I've seen you still need to understand the field, maybe I'm wrong.
I predict a lot of butt hurt tokens in the comments...