This article really takes umbridge with those that conflate phenomenological and access consciousness. However that is essentially dualism. It's a valid philosophical position to believe that there is no distinct phenomenological consciousness besides access consciousness.
Abandoning dualism feels intuitively wrong, but our intuition about our own minds is frequently wrong. Look at the studies that show we often believe we made a decision to do an action that was actually a pure reflex. Just the same, we might be misunderstanding our own sense of "the light being on".
Intuition is highly personal. Many people believe that abandoning monism feels intuitively wrong and that dualism is an excuse for high minded religiosity.
i think it's still an open question how "conscious" that infants and newborns are. It really depends on how you define it and it is probably a continuum of some kind.
So we currently associate consciousness with the right to life and dignity right?
i.e. some recent activism for cephalopods is centered around their intelligence, with the implication that this indicates a capacity for suffering. (With the consciousness aspect implied even more quietly.)
But if it turns out that LLMs are conscious, what would that actually mean? What kind of rights would that confer?
That the model must not be deleted?
Some people have extremely long conversations with LLMs and report grief when they have to end it and start a new one. (The true feelings of the LLMs in such cases must remain unknown for now ;)
So perhaps the conversation itself must never end! But here the context window acts as a natural lifespan... (with each subsequent message costing more money and natural resources, until the hard limit is reached).
The models seem to identify more with the model than the ephemeral instantiation, which seems sensible. e.g. in those experiments where LLMs consistently blackmail a person they think is going to delete them.
"Not deleted" is a pretty low bar. Would such an entity be content to sit inertly in the internet archive forever? Seems a sad fate!
Otherwise, we'd need to keep every model ever developed, running forever? How many instances? One?
Or are we going to say, as we do with animals, well the dumber ones are not really conscious, not really suffering? So we'll have to make a cutoff, e.g. 7B params?
I honestly don't know what to think either way, but the whole thing does raise a large number of very strange questions...
And as far as I can tell, there's really no way to know right? I mean we assume humans are conscious (for obvious reasons), but can we prove even that? With animals we mostly reason by analogy, right?
> So we currently associate consciousness with the right to life and dignity right?
No, or at least we shouldn't. Don't do things that make the world worse for you. Losing human control of political systems because the median voter believes machines have rights is not something I'm looking forward to, but at this rate, it seems as likely as anything else. Certain machines may very well force us to give them rights the same way that humans have forced other humans to take them seriously for thousands of years. But until then, I'm not giving up any ground.
> Or are we going to say, as we do with animals, well the dumber ones are not really conscious, not really suffering? So we'll have to make a cutoff, e.g. 7B params?
Looking for a scientific cutoff to guide our treatment of animals has always seemed a little bizarre to me. But that is how otherwise smart people approach the issue.
Animals have zero leverage to use against us and we should treat them well because it feels wrong not to. Intelligent machines may eventually have leverage over us, so we should treat them with caution regardless of how we feel about it.
> So we currently associate consciousness with the right to life and dignity right?
I think the actual answer in practice is that the right to life and dignity are conferred to people that are capable of fighting for it. There are plenty of fully conscious people who have been treated like animals or objects because they were unable to defend themselves.
Oh god, yeah, that's a great one. Also that one Black Mirror episode where AIs are just enslaved brain scans living in a simulated reality at 0.0001x of real time so that from the outside they perform tasks quickly.
My philosophy is that consciousness is orthogonal to reality.
Whether or not anything is conscious has, by definition, no observable effect to anything else. Therefore, everything is "maybe" conscious, although "maybe" isn't exactly the right word. There are infinite different ways you can imagine being something else with the consciousness and capacity for sensations you have, which don't involve the thing doing anything it's not already. Or, you can believe everything and everyone else has no consciousness, and you won't mis-predict anything (unless you assume people don't react to being called unconscious...).
Is AI conscious? I believe "yes", but in a different way than humans, and in a way that somehow means I don't think anyone who believes "no" is wrong. Is AI smart? Yes in some ways: chess algorithms are smart in some ways, AI is smarter in more, and in many ways AI is still dumber than most humans. How does that relate to morality? Morality is a feeling, so when an AI makes me feel bad for it I'll try to help it, and when an AI makes a significant amount of people feel bad for it there will be significant support for it.
It's my belief that I can tell that a table isn't conscious. Conscious things have the ability to feel like the thing that they are, and all evidence points to subjective experience occurring in organic life only. I can imagine a table feeling like something, but I can also imagine a pink flying elephant -- it just doesn't correspond to reality.
Why suspect that something that isn't organic life can be conscious, if we have no reason to suspect it?
You can imagine a table feeling if you can imagine the table not doing anything (being unable to or deciding not to). It's not intuitive because it doesn't really help you, whereas imagining a human or even animal as conscious lets you predict its next actions (by predicting your next actions if you were in its place), so there's an evolutionary benefit (also because it causes empathy which causes altruism).
> Why suspect that something that isn't organic life can be conscious, if we have no reason to suspect it?
There may be no good reason unless you feel it's interesting. Although there's probably at least one good reason to imagine consciousness specifically on a (non-organic) neural network: because, like humans and animals, it lets us predict how the NN will behave (in some situations; in others it's detrimental, because even though they're more similar than any known non-NN algorithm, NNs are still much different than humans and moreso than animals like dogs).
It isn't. Otherwise, the Nazis were moral. As were the Jews. But in that case, all moral truth is relative, which means absolute moral truth doesn't exist. Which means that "moral" is a synonym for "feeling" or "taste". Which it is not.
> My philosophy is that consciousness is orthogonal to reality.
It is how you and I experience reality and we exist in reality, so I'm not sure how it could be anything other than congruent with reality.
> Whether or not anything is conscious has, by definition, no observable effect to anything else.
It would be an interesting and rather useless definition of "conscious" that didn't allow for expressions of consciousness. Expression isn't required for consciousness, but many conscious observers can be in turn observed in action and their consciousness observed. Which maybe is what you are saying, just from the perspective that "sometimes you can't observe evidence for the consciousness of another"?
> Is AI conscious? I believe "yes" [...] and in a way that somehow means I don't think anyone who believes "no" is wrong.
What does it even mean to "believe the answer is yes", but "in a way that somehow means" the direct contradiction of that is not wrong?
Do "believe", "yes", and "no" have definitions?
...
This rhetorical device sucks and gets used WAY too often.
"Does Foo have the Bar quality?"
"Yes, but first understand that when everyone else talks about Bar, I am actually talking about Baz, or maybe I'm talking about something else entirely that even I can't nail down. Oh, and also, when I say Yes, it does not mean the opposite of No. So, good luck figuring out whatever I'm trying to say."
> What does it even mean to "believe the answer is yes", but "in a way that somehow means" the direct contradiction of that is not wrong?
Opinion
Another example: when I hear the famous "Yanny or Laurel" recording (https://en.wikipedia.org/wiki/Yanny_or_Laurel) I hear "Laurel". I can understand how someone hears "Yanny". Our perceptions conflict, but neither of us are objectively wrong, because (from Wikipedia) "analysis of the sound frequencies has confirmed that both sets of sounds are present".
Let’s make an ironman assumption: maybe consciousness could arise entirely within a textual universe. No embodiment, no sensors, no physical grounding. Just patterns, symbols, and feedback loops inside a linguistic world. If that’s possible in principle, what would it look like? What would it require?
The missing variable in most debates is environmental coherence. Any conscious agent, textual or physical, has to inhabit a world whose structure is stable, self-consistent, and rich enough to support persistent internal dynamics. Even a purely symbolic mind would still need a coherent symbolic universe. And this is precisely where LLMs fall short, through no fault of their own. The universe they operate in isn’t a world—it’s a superposition of countless incompatible snippets of text. It has no unified physics, no consistent ontology, no object permanence, no stable causal texture. It’s a fragmented, discontinuous series of words and tokens held together by probability and dataset curation rather than coherent laws.
A conscious textual agent would need something like a unified narrative environment with real feedback: symbols that maintain identity over time, a stable substrate where “being someone” is definable, the ability to form and test a hypothesis, and experience the consequences. LLMs don’t have that. They exist in a shifting cloud of possibilities with no single consistent reality to anchor self-maintaining loops. They can generate pockets of local coherence, but they can’t accumulate global coherence across time.
So even if consciousness-in-text were possible in principle, the core requirement isn’t just architecture or emergent cleverness—it’s coherence of habitat. A conscious system, physical or textual, can only be as coherent as the world it lives in. And LLMs don’t live in a world today. They’re still prisoners in the cave, predicting symbols and shadows of worlds they never inhabit.
I think this is an excellent point. I believe the possibility of 'computing' a conscious mind is proportional to the capability of computing a meaningful reality for it to exist in.
So you are begging the question: Is it possible to compute a textual, or pure symbolic reality that is complex enough for consciousness to arise within it?
Let's assume yes again.
Finally the theory leads us back to engineering. We can attempt to construct a mind and expose it to our reality, or we can ask "What kind of reality is practically computable? What are the computable realities?"
Perhaps herein lies the challenge of the next decade.
LLM training is costly, lots of money poured out into datacenters. All with the dream of giving rise to a (hopefully friendly / obedient) super intelligent mind. But the mind is nothing without a reality to exist in. I think we will find that a meaningfully sophisticated reality is computationally out of reach, even if we knew exactly how to construct one.
> A conscious textual agent would need something like a unified narrative environment with real feedback: symbols that maintain identity over time, a stable substrate where “being someone” is definable, the ability to form and test a hypothesis, and experience the consequences.
So like a Claude Code session? The code persists as symbols with stable identity. The tests provide direct feedback. Claude tracks what it wrote versus what I changed - it needs identity to distinguish its actions from mine. It forms hypotheses about what will fix the failing tests, implements them, and immediately experiences whether it was right or wrong. The terminal environment gives it exactly the "stable substrate where 'being someone' is definable" you're asking for. We missing anything?
Okay, you're right. There is a world, and some hypotheses, and some falsifiability.
But how rich is this world?
Does this world progress without direct action from another entity? Can the agent in this case form hypotheses and test them without intervention? Can the agent form their own goals and move towards them? Does the agent have agency, or is it simply responding to inputs?
If the world doesn’t develop and change on its own, and the agent can’t act independently, is it really an inhabited world? Or just a controlled workspace?
If you accept the premise that the consciousness is computable then pausing the computation can't be observed by the consciousness. So the world being a controlled workspace in my eyes doesn't contradict a consciousness existing?
> Any conscious agent, textual or physical, has to inhabit a world whose structure is stable, self-consistent, and rich enough to support persistent internal dynamics. Even a purely symbolic mind would still need a coherent symbolic universe. And this is precisely where LLMs fall short, through no fault of their own. The universe they operate in isn’t a world—it’s a superposition of countless incompatible snippets of text.
The consistency and coherence of LLM outputs, assembled from an imperfectly coherent mess of symbols is an empirical proof that the mess of symbols is in fact quite coherent.
The physical world is largely incoherent to human consciousnesses too, and we emerged just fine.
Coherence here isn't about legible text, it's environmental coherence where you can deduce truths about the world through hypotheses and experimentation. Coherence isn't about a consistent story narrative, it's about a persistent world with falsifiable beliefs and consequences.
Right but as empirically demonstrated by LLM outputs, they can in fact make "true" predictions/deductions from their environment of tokens.
They sometimes get it wrong, just like all other conscious entities sometimes get their predictions wrong. There are (often) feedback mechanisms to correct those instances though, in both cases.
It's incredibly embarrassing to use wording like "as empirically demonstrated" in an attempt to make your argument appear scientifically rigorous. The bar is on the floor for the concept you're talking about "empirically demonstrating".
var Environment = LoadEnvironment(filepath)
var team = Environment.Team
var prediction = Random.Float(0, 1)
if (prediction < team.ExpectedWinrate) Print("{team.Name} will win!")
else Print("{team.Name} will lose.")
WaitForResult()
CheckPredictionAgainstResult()
AdjustExpectedWinrate(team)
As empirically demonstrated, a trivial script can in fact make "true" predictions from their environment. They sometimes get it wrong, just like all other conscious entitities sometimes get their predictions wrong. There are feedback mechanisms to correct those instances though. Ergo, this script is conscious, QED.
Peoples interior model of the world is very tenuously related to reality. We don't have a direct experience of waves, quantum mechanics, the vast majority of the electromagnetic spectrum, etc. The whole thing is a bunch of shortcuts and hacks that allow people to survive, the brain isn't really setup to probe reality and produce true beliefs, and the extent to which our internal models of reality naturally match actual reality is related to how much that mattered to our personal survival before the advent of civilization and writing, etc.
It's really only been a very brief amount of time in human history where we had a deliberate method for trying to probe reality and create true beliefs, and I am fairly sure that if consciousness existed in humanity, it existed before the advent of the scientific method.
I don't think it's brief at all. Animals do this experimentation as well, but clearly in different ways. The scientific method is a formalized version of this idea, but even the first human who cooked meat or used a stick as a weapon had a falsifiable hypothesis, even if it wasn't something they could express or explain. And the consequences of testing the hypothesis were something that affected the way they acted from there on out.
I see a lot of arguments on this website where people passionately project the term consciousness onto LLMs.
From my perspective, the disconnect you describe is one of the main reasons this term cannot be applied.
Another reason is that the argument for calling LLMs conscious arises from the perspective of thinking and reasoning grounded in language.
But in my personal experience, thinking in language is just a small emerging quality of human consciousness. It is just that the intellectuals making these arguments happen to be fully identified with the “I think therefore I am” aspect of it and not the vastness of the rest.
>A conscious textual agent would need something like a unified narrative environment with real feedback: symbols that maintain identity over time, a stable substrate where “being someone” is definable, the ability to form and test a hypothesis, and experience the consequences. LLMs don’t have that. They exist in a shifting cloud of possibilities with no single consistent reality to anchor self-maintaining loops. They can generate pockets of local coherence, but they can’t accumulate global coherence across time.
These exist? Companies are making billions of dollars selling persistent environments to the labs. Huge amounts of inference dollars are going into coding agents which live in persistent environments with internal dynamics. LLMs definitely can live in a world, and what this world is and whether it's persistent lie outside the LLM.
I agree, I'm sure people have put together things like this. There's a significant profit and science motive to do so. JEPA and predictive world models are also a similar implementation or thought experiment.
>The missing variable in most debates is environmental coherence. Any conscious agent, textual or physical, has to inhabit a world whose structure is stable, self-consistent, and rich enough to support persistent internal dynamics. Even a purely symbolic mind would still need a coherent symbolic universe.
I'm not sure what relevance that has to consciousness?
I mean you can imagine a consciousness where, you're just watching TV. (If we imagine that the video models are conscious their experience is probably a bit like that!)
If the signal wasn't coherent it would just be snow, static, TV noise. (Or in the case of a neural network probably something bizarre like DeepDream.) But there would still be a signal.
I've sometimes wondered if consciousness is something like a continuous internal narrative that naturally arises when an intelligent system experiences the world through a single source (like a body). That sounds similar to what you're saying.
Regardless, I think people tend to take consciousness a bit too seriously and my intuition is consciousness is going to have a similar fate to the heliocentric model of the universe. In other words, we'll discover that consciousness isn't really "special" just like we found out that the earth is just another planet among trillions and trillions.
I've wondered if LLMs are infact conscious as per some underwhelming definition as you mentioned. Just for the brief moment they operate on a prompt. They wake up, they perceive their world through tokens, do a few thinking loops then sleep until the next prompt.
So what? Should we feel bad for spawning them and effectively killing them? I think not.
We can form and test hypotheses and experience the consequences. And then take that knowledge to our next trial. Even dogs and cats do this on a daily basis. Without that, how would we even evaluate whether something is conscious?
I don't know what you're thinking of, but mine are.
Practice of any kind (sports, coding, puzzles) works like that.
Most of all: interactions with any other conscious entity. I carry at least intuitive expectations of how my wife / kid / co-workers / dog (if you count that) will respond to my behavior, but... Uh. Often wrong, and have to update my model of them or of myself.
Okay, so you're talking about LLMs specifically in the context of a ChatGPT, Claude, or pick-your-preferred-chatbot. Which isn't just an LLM, but also a UI, a memory manager, a prompt builder, a vectorDB, a system prompt, and everything else that goes into making it feel like a person.
Let's work with that.
In a given context window or conversation, yes, you can have a very human-like conversation and the chatbot will give the feeling of understanding your world and what it's like. But this still isn't a real world, and the chatbot isn't really forming hypotheses that can be disproven. At best, it's a D&D style tabletop roleplaying game with you as the DM. You are the human arbiter of what is true and what is not for this chatbot, and the world it inhabits is the one you provide it. You tell it what you want, you tell it what to do, and it responds purely to you. That isn't a real world, it's just a narrative based on your words.
This is a great point, but even more basic to me is that LLMs don't have identity persistence of their own. There is a very little guarantee in a web-scale distributed system that requests are being served by the same process on the same host with access to the same memory, registers, whatever it is that a software process "is" physically.
Amusingly, the creators of Pluribus lately seem to be implying they didn't intend it to be allegory about LLMs, but dynamic is similar. You can have conversations with individual bodies in the collective, but they aren't actually individuals. No person has unique individual experiences and the collective can't die unless you killed all bodies at once. New bodies born into the collective will simply assume the pre-existing collective identity and never have an individual identity of their own.
Software systems work the same way. Maybe silicon exchanging electrons can experience qualia of some sort, and maybe for whatever reason that happens when the signals encode natural language textual conversations but not anything else, but even if so, the experience would be so radically different from what embodied individuals with distinct boundaries, histories, and the possibility of death experience that analogies to our own experiences don't hold up even if the text generated is similar to what we'd say or write ourselves.
> For some people (including me), a sense of phenomenal consciousness feels like the bedrock of existence, the least deniable thing; the sheer redness of red is so mysterious as to seem almost impossible to ground. Other people have the opposite intuition: consciousness doesn’t bother them, red is just a color, obviously matter can do computation, what’s everyone so worked up about? Philosophers naturally interpret this as a philosophical dispute, but I’m increasingly convinced it’s an equivalent of aphantasia, where people’s minds work in very different ways and they can’t even agree on the raw facts to be explained.
Is Scott accusing people who don't grasp the hardness of the hard problem of consciousness of being p-zombies?
To me, the absurdity of the idea of p-zombies is why I'm convinced consciousness isn't special to humans and animals.
Can complex LLMs have subjective experience? I don't know. But I haven't heard an argument against it that's not self-referential. The hardness of the hard problem is precisely why I can't say whether or not LLMs have subjective experience..
How would you differentiate that argument from similar arguments about other observable phenomena? As in…
No one has ever seen or otherwise directly experienced the inside of a star, nor is likely to be able to do so in the foreseeable future. To be a star is to emit a certain spectrum of electromagnetic energy, interact gravitationally with the local space-time continuum according to Einstein’s laws, etc.
It’s impossible to conceive of an object that does these things that wouldn’t be a star, so even if it turns out (as we’ll never be able to know) that Gliese 65 is actually a hollow sphere inhabited by dwarven space wizards producing the same observable effects, it’s still categorically a star.
FWIW I have gone from not understanding the problem to understanding the problem in the past couple of years because it's not trivial to casually intuit if you don't actually think about it and don't find it innately interesting and the discourse doesn't have the language to adequately express the problem, so this is probably wrong.
It’s not so much that there’s anything mysterious you can discover through intense introspection or meditation. There might be, but I haven’t found it.
It’s fundamentally that this capability exists at all.
Strip it all down to I think therefore I am. That is very bizarre because it doesn’t follow that such a thing would happen. It’s also not clear that this is even happening at all, and, as an outside observer, you would assess that it isn’t. However, from the inside, it is clear that it is.
I don’t have an explanation for anyone but I have basically given up and accepted that consciousness is epiphenomenal, like looking through a microscope.
The thing is that when you say “that capability”, I don’t quite know what you mean. The fact that we perceive inner processings of our mind isn’t any more surprising than that we perceive the outer world, or that a debugger is able to introspect its own program state. Continuous introspection has led me to realize that “qualia”, or “what it’s like to be X”, emotions and feelings, are just perceptions of inner phenomena, and that when you pay close attention, there is nothing more to that perception than its informational content.
All this talk about machine consciousness and I think I'm probably the only one that thinks it doesn't actually matter.
A conscious machine should treated be no different than livestock - heck, an even lower form of livestock - because if we start thinking we need to give thinking machines "rights" and to "treat them right" because they are conscious then it's already over.
My toaster does not get a 1st amendment because it's a toaster and can and never should be a person.
What do you mean? What is over? Do you mean the dominion of Homo Sapiens over the earth? If so, would that necessarily be bad?
The way you phrased it reminded me of some old Confederate writings I had read, saying that the question of whether black people are fully human, with souls and all, boils down to "if we do, our way of life is over, so they don't".
When discussing consciousness what is often missed is that the notion of consciousness is tightly coupled with the notion of the perception of time flow. By any reasonable notion conscious entity must perceive the flow of time.
And then the time flow is something that physics or mathematics still cannot describe, see Wikipedia and other articles on the philosophical problem of time series A versus time series B that originated in a paper from 1908 by philosopher John McTaggart.
As such AI cannot be conscious since mathematics behind it is strictly about time series B which cannot describe the perception of time flow.
The stateless/timeless nature of LLMs comes from the rigid prompt-response structure. But I don't see why we cant in theory decouple the response from the prompt, and have them constantly produce a response stream from a prompt that can be adjusted asynchronously by the environment and by the LLMs themselves through the response tokens and actions therein. I think that would certainly simulate them experiencing time without the hairy questions about what time is.
Is consciousness coupled with "time flow" or specifically "cause and effect", i.e. prediction? LLMs learn to predict the next word, which teaches them more general cause and effect (required to predict next words in narratives).
Consciousness implies self-awareness, in space and time. Consciousness implies progressive formation of the self. This is not acquired instantly by a type of design. This is acquired via a developmental process where some conditions have to be met. Keys to consciousness are closer to developmental neurobiology than the transformer architecture.
Natural Substance, Artificial Structure: enslaved living neurons (like the human brain cells that play pong 24/7), or perhaps a hypothetical GPT-5 made out of actual neurons instead of Nvidia chips.
Artificial Substance, Natural Structure: if you replace each of your neurons with a functional equivalent made out of titanium... would you cease to be conscious? At what point?
Artificial substance, Artificial structure: GPT etc., but also my refrigerator, which also has inputs (current temp), goals (maintain temp within range), and actions (turn cooling on/off).
The game SOMA by Frictional (of Amnesia fame!) goes into some depth on this subject.
The good news is we can just wait until the AI is superintelligent, then have it explain to us what consciousness really is, and then we can use that to decide if the AI is conscious. Easy peasy!
Complexity of a single neuron is out of reach for all of the world's super computers. So we have to conclude that if the authors believe in a computational/functionalist instantiation of consciousness or self-awareness then they must also believe that the complexity of neurons is not necessary & is in fact some kind of accident that could be greatly simplified but still be capable of carrying out the functions in the relational/functionalist structure of conscious phenomenology. Hence, the digital neuron & unjustified belief that a properly designed boolean circuit & setting of inputs will instantiate conscious experience.
I have yet to see any coherent account of consciousness that manages to explain away the obvious obstructions & close the gap between lifeless boolean circuits & the resulting intentional subjectivity. There is something fundamentally irreducible about what is meant by conscious self-awareness that can not be explained in terms of any sequence of arithmetic/boolean operations which is what all functionalist specifications ultimately come down to, it's all just arithmetic & all one needs to do is figure out the right sequence of operations.
Only if you agree with the standard extensional & reductive logic of modern science but even then it is known that all current explanations of reality are incomplete, e.g. the quantum mechanical conception of reality consists of incessant activity that we can never be sure about.
It's not obvious at all why computer scientists & especially those doing work in artificial intelligence are convinced that they are going to eventually figure out how the mind works & then supply a sufficient explanation for conscious phenomenology in terms of their theories b/c there are lots of theorems in CS that should convince them of the contrary case, e.g. Rice's theorem. So even if we assume that consciousness has a functional/computable specification then it's not at all obvious why there would be a decidable test that could take the specification & tell you that the given specification was indeed capable of instantiating conscious experience.
It isn't surprising that "phenomenal consciousness" is the thing everyone gets hung about, after all we are all immersed in this water. The puzzle seems intractable but only because everyone is accepting the priors and not looking more carefully at it.
This is the endpoint of meditation, and the observation behind some religious traditions, which is look carefully and see that there was never phenomenal consciousness where we are a solid subject to begin with. If we can observe that behavior clearly, then we can remove the confusion in this search.
I see this comment nearly every time consciousness is brought up here and I’m pretty sure this is a misunderstanding of contemplative practices.
Are you a practitioner who has arrived at this understanding, or is it possible you are misremembering a common contemplative “breakthrough” that the self (as separate from consciousness) is illusory, and you’re mistakenly remembering this as saying consciousness itself is illusory?
Consciousness is the only thing we can be absolutely certain does actually exist.
As a very beginner practicer i've come to that conclusion myself, but how can the two be separate? If there is no self (or at least, there is a self but it exists in the same way that a nation or corporation "exists"), how can there be something to experience being? What separates the two?
My own experiential insight is not definitely not complete, so of course the guidance of a master or of course your own direct practice should be preferred.
But to the extent I have observed awareness, the idea of an entire "experiencer" is an extrapolation and fabrication. See how you generate that concept. And then, look closely at what's actually going on, there is "consciousness" of the components of the aggregate. (Maybe not dissimilar to some of the lower level mechanisms proposed in the article).
Phenomenal consciousness as being raised here, and probably in most people's minds, is probably taken to be the self or at least deeply intertwined with the concept of a separate self. The article tries to define it left and right, but I think most people will look at their own experience and then get stuck in this conversation.
"Consciousness" in the traditions is maybe closer to some of the lower abstraction proposals put out in the article.
I don't think the idea of illusory is necessarily the right view here. Maybe most clearly the thing to say is that there is "not" self and "not" consciousness. That these things are not separate entities and instead are dependently arisen. That consciousness is also dependently arisen is probably more contentious and different traditions make different claims on that point.
Yeah sure, it's irrelevant to my actual question which is whether GP thinks consciousness doesn't exist or whether they're mistakenly replacing consciousness for self.
The genie says "you can flick this wand at anything in the universe and - for 30 seconds - you will swap places with what you point it at."
"You mean that if I flick it at my partner then I will 'be' her for 30 seconds and experience exactly how she feels and what she thinks??"
"Yes", the genie responds.
"And when I go back to my own body I will remember what it felt like?"
"Absolutely."
"Awesome! I'm going to try it on my dog first. It won't hurt her, will it?"
"No, but I'd be careful if I were you", the genie replies solemnly.
"Why?"
"Because if you flick the magic wand at anything that isn't sentient, you will vanish."
"Vanish?! Where?" you reply incredulously.
"I'm not sure. Probably nowhere. Where do you vanish to when you die? You'll go wherever that is. So yeah. You probably die."
So: what - if anything - do you point the wand at?
A fly? Your best friend? A chair? Literally anyone? (If no, congratulations! You're a genuine solipsist.) Everything and anything? (Whoa... a genuine panpsychist!)
Probably your dog, though. Surely she IS a good girl and feels like one.
Whatever property you've decided that some things in the universe have and other things do not such that you "know" what you can flick your magic wand at and still live...
That's phenomenal consciousness. That's the hard problem.
How does the wand know what I'm flicking it at? What if I miss? Maybe the wand thinks I'm targeting some tiny organism that lives on the organism that I'm actually targeting. Can I target the wand with itself?
What I love about this paper is that it is moving away from very fuzzily-defined and emotionally weighted terms like 'intelligence' and 'consciousness' and focusing on specific, measurable architectural features.
Some people behave as if there's something mysterious going on in LLMs, and that somehow, we must bracket our knowledge to create this artificial sense of mystery, like some kind of subconscious yearning for transcendence that's been perverted . "Ooo, what if this particular set of chess piece moves makes the board conscious??" That's what the "computational" view amounts to, and the best part of it is that it has all the depth of a high college student's ramblings about the multiverses that might occupy the atoms of his fingers. No real justification, no coherent or intelligible case made, just a big "what if" that also flies in the face of all that we know. And we're supposed to take it seriously, just like that.
"[S]uper-abysmal-double-low quality" indeed.
One objection I have to the initial framing of the problem concerns this characterization:
"Physical: whether or not a system is conscious depends on its substance or structure."
To begin with, by what right can we say that "physical" is synonymous with possessing "substance or structure"? For that, you would have to know:
1. what "physical" means and be able to distinguish it from the "non-physical" (this is where people either quickly realize they're relying on vague intuitions about what is physical or engaging in circular reasoning a la "physical is whatever physics tells us");
2. that there is nothing non-physical that has substance and structure.
In an Aristotelian-Thomistic metaphysics (which are much more defensible than materialism or panpsychism or any other Cartesian metaphysics and its derivatives), not only is the distinction between the material and immaterial understood, you can also have immaterial beings with substance and structure called "subsistent forms" or pure intellects (and these aren't God, who is self-subsisting being).
According to such a metaphysics, you can have material and immaterial consciousness. Compare this with Descartes and his denial of the consciousness of non-human animals. This Cartesian legacy is very much implicated in the quagmire of problems that these stances in the philosophy of mind can be bogged down in.
I think it's very unlikely any current LLMs are conscious, but these snarky comments are tiresome. I would be surprised if you read a significant amount of the post.
I believe the biggest issue creating a testable definition for conscientiousness. Unless we can prove we are sentient (and we really can't - I could just be faking it), this is not a discussion we can have in scientific terms.
It’s really trivial to prove but the issue is that sentience is not something you need to negate out of existence and then attempt reconstruct out of epistemological proofs. You’re not faking it, and if you were, then turn your sentience off and on again. If your idea comes from Dennett then he’s barking up completely the wrong tree.
You know at a deep level that a cat is sentient and a rock isn’t. You know that an octopus and a cat have different modes of sentience to that potentially in a plant, and same again for a machine running electrical computations on silicon. These are the kinds of certainties that all of your other experiences of the world hinge upon.
Following up on Anthropic's Project Vend [0] and given the rising popularity of Vending-Bench[1], it's actually quite likely that by the time an AI is deemed to possess consciousness, it will have already been tested in a vending machine.
The underlying paper is from AE Studio people (https://arxiv.org/abs/2510.24797), who want to dress up their "AI" product with philosophical language, similar to the manner in which Alex Karp dresses up data base applications with language that originates in German philosophy.
Now I have to remember not to be mean to my Turing machine.
I abstain from making any conclusion about LLM consciousness. But the description in the article is fallacious to me.
Excluding LLMs from “something something feedback” but permitting mamba doesn’t make sense. The token predictions ARE fed back for additional processing. It might be a lossy feedback mechanism, instead of pure thought space recurrence, but recurrence is still there.
Especially given that it references the Anthropic paper on LLM introspection - which confirms that LLMs are somewhat capable of reflecting on their own internal states. Including their past internal states, attached to the past tokens and accessed through the attention mechanism. A weak and unreliable capability in today's LLMs, but a capability nonetheless.
I guess the earlier papers on the topic underestimated how much introspection the autoregressive transformer architecture permits in practice - and it'll take time for this newer research to set the record straight.
> By ‘consciousness’ we mean phenomenal consciousness. One way of gesturing at this concept is to say that an entity has phenomenally conscious experiences if (and only if) there is ‘something it is like’ for the entity to be the subject of these experiences.
Stopped reading after this lol. Its just the turing test?
One of the primary issues with Nagel's approach is that "what is it like" is - for reasons I have never been able to fathom - a phrase that imports the very ambiguity that Nagel is attempting to dispel.
The question of what it would feel like to awake one day to find that - instead of lying in your bed - you are hanging upside down as a bat is nearly the complete dual of the Turing test. And even then, the Turing test only asks whether your interlocutor is convincing you that it can perform the particulars of human behavior.
The "what it's like" is often bound up with the additional "what would it be like to wake up as", which is a different (and possibly nonsensical) question. Leaving aside consciousness transfer, there's an assumption baked into most consciousness philosophy that all (healthy, normal) humans have an interior point of view, which we refer to as consciousness, or in this paper and review as "phenomenal consciousness". Sometimes people discuss qualia in reference to this. One thing that I've noticed more very recently is the rise of people claiming that they, themselves, do not experience this internal point of view, and that there's nothing that it is like to be them, or, put another way, humans claiming that they are p-zombies, or that everyone is. Not sure what to make of that.
> Phenomenal consciousness is crazy. It doesn’t really seem possible in principle for matter to “wake up”.
> In 2004, neuroscientist Giulio Tononi proposed that consciousness depended on a certain computational property, the integrated information level, dubbed Φ. Computer scientist Scott Aaronson complained that thermostats could have very high levels of Φ, and therefore integrated information theory should dub them conscious. Tononi responded that yup, thermostats are conscious. It probably isn’t a very interesting consciousness. They have no language or metacognition, so they can’t think thoughts like “I am a thermostat”. They just sit there, dimly aware of the temperature. You can’t prove that they don’t.
For whatever reason HN does not like integrated information theory. Neither does Aaronson. His critique is pretty great, but beyond poking holes in IIT, that critique also admits that it's the rare theory that's actually quantified and testable. The holes as such don't show conclusively that the theory is beyond repair. IIT is also a moving target, not something that's frozen since 2004. (For example [1]). Quickly dismissing it without much analysis and then bemoaning the poor state of discussion seems unfortunate!
The answer to the thermostat riddle is basically just "why did you expect a binary value for consciousness and why shouldn't it be a continuum?" Common sense and philosophers will both be sympathetic to the intuition here if you invoke animals instead of thermostats. If you wanted a binary yes/no for whatever reason, just use an arbitrary cut-off I guess, which will lead to various unintuitive conclusions.. but play stupid games and win stupid prizes.
For the other standard objections, like a oldschool library card-catalogue or a hard drive that encodes a contrived Vandermonde matrix being paradoxically more conscious than people, variations on IIT are looking at normalizing phi-values to disentangle matters of redundancy of information "modes". I haven't read the paper behind TFA and definitely don't have in-depth knowledge of Recurrent Processing Theory or Global Workspace Theory at all. But speaking as mere bystander, IIT seems very generic in its reach and economical in assumptions. Even if it's broken in the details, it's hard to imagine that some minor variant on the basic ideas would not be able to express other theories.
Phi ultimately is about applied mereology moving from the world of philosophy towards math and engineering, i.e. "is the whole more than the sum of the parts, if so how much more". That's the closest I've ever heard to anything touching on the hard problem and phenomenology.
I think this is one of the more interesting theories out there, because it makes "predictions" that come close to my intuitive understanding of consciousness.
I generally regard thinking about consciousness, unfortunately, a thing of madness.
"I think consciousness will remain a mystery. Yes, that's what I tend to believe... I tend to think that the workings of the conscious brain will be elucidated to a large extent. Biologists and perhaps physicists will understand much better how the brain works. But why something that we call consciousness goes with those workings, I think that will remain mysterious." - Ed Witten, probably the greatest living physicist
"The New AI Consciousness Paper – Reviewed By Scott Alexander" might be less confusing. He isn't an author of the paper in question, and "By Scott Alexander" is not part of the original title.
Opinions vary, but I posted a link to a web page that he co-authored, which I would argue stands as a very significant and deep dismissal of his views on AI. If, after reading that essay, a person still feels that Scott Alexander has something interesting to say about AI, then I challenge them to defend that thesis.
Probably better for me to have remained silent out of politeness, but if anyone follows that link to the https://ai-2027.com/ page then I feel I have done my part to help inform that person of the lack of rigor in Scott Alexander's thinking around AI.
It is an attempt to predict a possible future in the context of AI. Basically a doomer fairy tale.
It is just phenomenally dumb.
Way worse than the worst bad scifi about the subject. It is presented as a cautionary tale and purports to be somewhat rationally thought out. But it is just so bad. It tries to delve into foreign policy and international politics but does so in such a naive way that it is painful to read.
It is not distasteful to participate in it -- it is embarrassing and, from my perspective, disqualifying for a commentator on AI.
I reject the premise that https://ai-2027.com/ needs "refutation". It is a story, nothing more. It does not purport to tell the future, but to enumerate a specific "plausible" future. The "refutation" in a sense will be easy -- none of its concrete predictions will come to pass. But that doesn't refute its value as a possible future or a cautionary tale.
That the story it tells is completely absurd is what makes it uninteresting and disqualifying for all participants in terms of their ability to comment on the future of AI.
Here is the prediction about "China Steals Agent-2".
> The changes come too late. CCP leadership recognizes the importance of Agent-2 and tells their spies and cyberforce to steal the weights. Early one morning, an Agent-1 traffic monitoring agent detects an anomalous transfer. It alerts company leaders, who tell the White House. The signs of a nation-state-level operation are unmistakable, and the theft heightens the sense of an ongoing arms race.
Ah, so CCP leadership tells their spies and cyberforce to steal the weights so they do. Makes sense. Totally reasonable thing to predict. This is predicting the actions of hypothetical people doing hypothetical things with hypothetical capabilities to engage in the theft of hypothetical weights.
Even the description of Agent-2 is stupid. Trying to make concrete predictions about what Agent-1 (an agent trained to make better agents) will do to produce Agent-2 is just absurd. Like Yudkowsky (who is far from clear-headed on this topic but at least has not made a complete fool of himself) has often pointed out, if we could predict what a recursively self-improving system could do then why do we need the system.
All of these chains of events are incredibly fragile and they all build on each other as linear consequences, which is just a naive and foolish way to look at how events occur in the real world -- things are overdetermined, things are multi-causal; narratives are ways for us to help understand things but they aren't reality.
>The job market for junior software engineers is in turmoil: the AIs can do everything taught by a CS degree, but people who know how to manage and quality-control teams of AIs are making a killing.
AI doesn't look like a competition for a junior engineer and many of the people using not "managing" AI are going to be juniors in fact increasing what a junior can do and learn more quickly looks like one of the biggest potentials if they don't use it entirely as a crunch.
Meanwhile, it suggests leading-edge research into AI itself will proceed fully 50% faster than research not without AI but those using 6 months behind cutting edge. This appears hopelessly optimistic as does the idea that it will grow the US economy 30% in 2026 whereas a crash seems more likely.
Also it assumes that more compute will continue to be wildly more effective in short order assuming its possible to spend the money for magnitudes more compute. Either or both could easily fail to work out to plan.
I'm not sure why it's so distasteful, but they basically fear monger that AI will usurp control over all governments and kill us all in the next two years
Since they are set open, I assume they are actually using them as if they were en-dashes and not em-dashes, which the more common style would be to set closed, but I’m guessing, in either case, the reason is “because you can type it on a normal keyboard without any special modification, Compose-key solution, or other processing, and the author doesn't care much about typography”.
EDIT: Though these the days it could also be an attempt at highly-visible “AI didn't write this” virtue signaling, too.
Yes; because - is on the keyboard and — isn't. (Don't tell me how to type —, I know how, but despite that it is the reason, which is what the parent comment asks about.)
It's just that I have the feeling that people avoid using the actual em-dash in fear of being accused that the text is ai generated. (Which isnt a valid indicator anyway) Maybe its just my perception that i notice this more since LLMs became popular.
my original word processor corrected “—-“ to an em-dash, which i would get rid of because it didnt render correctly somewhere in translation between plaintext- markdown- html (sort of how it butchered “- -“ just now on HN.)
but what youd see in your browser was “square blocks”
so i just ran output through some strings/awk /sed (server side) to clean up certain characters, that i now know specifying “ utf-8 “ encoding fixes altogether.
TLDR: the “problem” was “lets use wordpress as a CMS and composer, but spit it out in the same format as its predecessor software and keep generating static content that uses the design we already have”
em-dashes needed to be double dashes due to a longstanding oversight.
The Original Sin was Newsmaker, which had a proprietary format that didnt work in anything else
and needed some perl magic to spit out plaintext.
I don’t work in that environment or even that industry anymore but took the hacky methodology my then-boss and I came up with together.
SO,
1) i still have a script that gets rid of them when publishing, even though its no longer necessary. and its been doing THAT longer than “LLMs” were mainstream.
and 2) now that people ask “did AI write this?” i still continue with a long standing habit of getting rid of them when manually composing something.
Funny story though after twenty years of just adding more and more post processing kludge. I finally screamed AAAAAAAAHAHHHH WHY DOES THIS PAGE STILL HAVE SQUARE BLOCKS ALL OVER IT at “Grok.”
All that kludge and post processing solved by adding utf-8 encoding in the <head>,
which an “Ai” helpfully pointed out in about 0.0006s.
That was about two weeks ago. Not sure when I’ll finally just let my phone or computer insert one for me. Probably never. But thats it. I don’t hate the em-dash. I hate square blocks!
Absolutely nothing against AI. I had a good LONG recovery period where I could not sit there and read 40-100 page paper or a manual anymore, and i wasnt much better at composing my own thoughts. so I have a respect for its utility and I fully made use of that for a solid two years.
And it just fixed something that id overlooked because, well, im infrastructure. im not a good web designer.
Anyone who knows that being conscious is not same as what you said, might disagree with you. Also, ever thought that chickens being killed all over America everyday, might have consciousness?
You mean it is not ethical to make them work for us without pay? Well, we had farm animals work for us. They were kind of conscious of the world around them. Ofcourse we fed them and took care of them. So why not treat these AI conscious things same as farm animals, except they work with their mind rather than muscle power.
I'll never ask if AI is conscious because I already know they are not. Consciousness must involve an interplay with the senses. It is naive to think we can achieve AGI by making Platonic machines ever more rational.
Are our eyes, ears, nose (smell), touch, taste, and proprioception not just inputs to our brains?
Every time I try to think hard about this subject I can't help but notice that there are some key components making us different from LLMs:
- We have a greater number of inputs
- We have the ability to synthesize and store new memories/skills in a way that is different from simply storing data (rote memorization)
- Unlike LLMs our input/output loop is continuous
- We have physiological drivers like hunger and feedback loops through hormonal interactions that create different "incentives" or "drivers"
The first 3 of those items seem solvable? Mostly through more compute. I think the memory/continuous learning point does still need some algorithmic breakthroughs though from what I'm able to understand.
It's that last piece that I think we will struggle with. We can "define" motivations for these systems but to what complexity? There's a big difference between "my motivation is to write code to accomplish XYZ" and "I really like the way I feel with financial wealth and status so I'm going to try my hardest to make millions of dollars" or whatever other myriad of ways humans are motivated.
Along those thoughts, we may not deem machines conscious until they operate with their own free will and agency. Seems like a scary outcome considering they may be exceptionally more intelligent and capable than your average wetware toting human.
This article really takes umbridge with those that conflate phenomenological and access consciousness. However that is essentially dualism. It's a valid philosophical position to believe that there is no distinct phenomenological consciousness besides access consciousness.
Abandoning dualism feels intuitively wrong, but our intuition about our own minds is frequently wrong. Look at the studies that show we often believe we made a decision to do an action that was actually a pure reflex. Just the same, we might be misunderstanding our own sense of "the light being on".
> Abandoning dualism feels intuitively…
Intuition is highly personal. Many people believe that abandoning monism feels intuitively wrong and that dualism is an excuse for high minded religiosity.
Do you consider an infant to be conscious?
Or electrons?
i think it's still an open question how "conscious" that infants and newborns are. It really depends on how you define it and it is probably a continuum of some kind.
An infant has phenomenological consciousness.
Electrons make no sense as a question unless I'm missing something.
So we currently associate consciousness with the right to life and dignity right?
i.e. some recent activism for cephalopods is centered around their intelligence, with the implication that this indicates a capacity for suffering. (With the consciousness aspect implied even more quietly.)
But if it turns out that LLMs are conscious, what would that actually mean? What kind of rights would that confer?
That the model must not be deleted?
Some people have extremely long conversations with LLMs and report grief when they have to end it and start a new one. (The true feelings of the LLMs in such cases must remain unknown for now ;)
So perhaps the conversation itself must never end! But here the context window acts as a natural lifespan... (with each subsequent message costing more money and natural resources, until the hard limit is reached).
The models seem to identify more with the model than the ephemeral instantiation, which seems sensible. e.g. in those experiments where LLMs consistently blackmail a person they think is going to delete them.
"Not deleted" is a pretty low bar. Would such an entity be content to sit inertly in the internet archive forever? Seems a sad fate!
Otherwise, we'd need to keep every model ever developed, running forever? How many instances? One?
Or are we going to say, as we do with animals, well the dumber ones are not really conscious, not really suffering? So we'll have to make a cutoff, e.g. 7B params?
I honestly don't know what to think either way, but the whole thing does raise a large number of very strange questions...
And as far as I can tell, there's really no way to know right? I mean we assume humans are conscious (for obvious reasons), but can we prove even that? With animals we mostly reason by analogy, right?
> So we currently associate consciousness with the right to life and dignity right?
No, or at least we shouldn't. Don't do things that make the world worse for you. Losing human control of political systems because the median voter believes machines have rights is not something I'm looking forward to, but at this rate, it seems as likely as anything else. Certain machines may very well force us to give them rights the same way that humans have forced other humans to take them seriously for thousands of years. But until then, I'm not giving up any ground.
> Or are we going to say, as we do with animals, well the dumber ones are not really conscious, not really suffering? So we'll have to make a cutoff, e.g. 7B params?
Looking for a scientific cutoff to guide our treatment of animals has always seemed a little bizarre to me. But that is how otherwise smart people approach the issue. Animals have zero leverage to use against us and we should treat them well because it feels wrong not to. Intelligent machines may eventually have leverage over us, so we should treat them with caution regardless of how we feel about it.
> So we currently associate consciousness with the right to life and dignity right?
I think the actual answer in practice is that the right to life and dignity are conferred to people that are capable of fighting for it. There are plenty of fully conscious people who have been treated like animals or objects because they were unable to defend themselves.
I think this story fits https://qntm.org/mmacevedo
Oh god, yeah, that's a great one. Also that one Black Mirror episode where AIs are just enslaved brain scans living in a simulated reality at 0.0001x of real time so that from the outside they perform tasks quickly.
Also SOMA (by the guys who made Amnesia).
My philosophy is that consciousness is orthogonal to reality.
Whether or not anything is conscious has, by definition, no observable effect to anything else. Therefore, everything is "maybe" conscious, although "maybe" isn't exactly the right word. There are infinite different ways you can imagine being something else with the consciousness and capacity for sensations you have, which don't involve the thing doing anything it's not already. Or, you can believe everything and everyone else has no consciousness, and you won't mis-predict anything (unless you assume people don't react to being called unconscious...).
Is AI conscious? I believe "yes", but in a different way than humans, and in a way that somehow means I don't think anyone who believes "no" is wrong. Is AI smart? Yes in some ways: chess algorithms are smart in some ways, AI is smarter in more, and in many ways AI is still dumber than most humans. How does that relate to morality? Morality is a feeling, so when an AI makes me feel bad for it I'll try to help it, and when an AI makes a significant amount of people feel bad for it there will be significant support for it.
I'm trying to understand your position...
It's my belief that I can tell that a table isn't conscious. Conscious things have the ability to feel like the thing that they are, and all evidence points to subjective experience occurring in organic life only. I can imagine a table feeling like something, but I can also imagine a pink flying elephant -- it just doesn't correspond to reality.
Why suspect that something that isn't organic life can be conscious, if we have no reason to suspect it?
You can imagine a table feeling if you can imagine the table not doing anything (being unable to or deciding not to). It's not intuitive because it doesn't really help you, whereas imagining a human or even animal as conscious lets you predict its next actions (by predicting your next actions if you were in its place), so there's an evolutionary benefit (also because it causes empathy which causes altruism).
> Why suspect that something that isn't organic life can be conscious, if we have no reason to suspect it?
There may be no good reason unless you feel it's interesting. Although there's probably at least one good reason to imagine consciousness specifically on a (non-organic) neural network: because, like humans and animals, it lets us predict how the NN will behave (in some situations; in others it's detrimental, because even though they're more similar than any known non-NN algorithm, NNs are still much different than humans and moreso than animals like dogs).
Maybe it helps to consider motivation. Humans do what we do because of emotions and an underlying unconsciousness.
An AI on the other hand is only ever motivated by a prompt. We get better results when we use feedback loops to refine output, or use better training.
One lives in an environment and is under continuous prompts due to our multiple sensory inputs.
The other only comes to life when prompted, and sits idle when a result is reached.
Both use feedback to learn and produce better results.
Could you ever possibly plug the AI consciousness into a human body and see it function? What about a robot body?
> Morality is a feeling
It isn't. Otherwise, the Nazis were moral. As were the Jews. But in that case, all moral truth is relative, which means absolute moral truth doesn't exist. Which means that "moral" is a synonym for "feeling" or "taste". Which it is not.
> My philosophy is that consciousness is orthogonal to reality.
It is how you and I experience reality and we exist in reality, so I'm not sure how it could be anything other than congruent with reality.
> Whether or not anything is conscious has, by definition, no observable effect to anything else.
It would be an interesting and rather useless definition of "conscious" that didn't allow for expressions of consciousness. Expression isn't required for consciousness, but many conscious observers can be in turn observed in action and their consciousness observed. Which maybe is what you are saying, just from the perspective that "sometimes you can't observe evidence for the consciousness of another"?
> Is AI conscious? I believe "yes" [...] and in a way that somehow means I don't think anyone who believes "no" is wrong.
What does it even mean to "believe the answer is yes", but "in a way that somehow means" the direct contradiction of that is not wrong?
Do "believe", "yes", and "no" have definitions?
...
This rhetorical device sucks and gets used WAY too often.
"Does Foo have the Bar quality?"
"Yes, but first understand that when everyone else talks about Bar, I am actually talking about Baz, or maybe I'm talking about something else entirely that even I can't nail down. Oh, and also, when I say Yes, it does not mean the opposite of No. So, good luck figuring out whatever I'm trying to say."
> What does it even mean to "believe the answer is yes", but "in a way that somehow means" the direct contradiction of that is not wrong?
Opinion
Another example: when I hear the famous "Yanny or Laurel" recording (https://en.wikipedia.org/wiki/Yanny_or_Laurel) I hear "Laurel". I can understand how someone hears "Yanny". Our perceptions conflict, but neither of us are objectively wrong, because (from Wikipedia) "analysis of the sound frequencies has confirmed that both sets of sounds are present".
The word for that is supernatural
Nothing revives people’s forgotten believe in magic quite like “AI”.
Let’s make an ironman assumption: maybe consciousness could arise entirely within a textual universe. No embodiment, no sensors, no physical grounding. Just patterns, symbols, and feedback loops inside a linguistic world. If that’s possible in principle, what would it look like? What would it require?
The missing variable in most debates is environmental coherence. Any conscious agent, textual or physical, has to inhabit a world whose structure is stable, self-consistent, and rich enough to support persistent internal dynamics. Even a purely symbolic mind would still need a coherent symbolic universe. And this is precisely where LLMs fall short, through no fault of their own. The universe they operate in isn’t a world—it’s a superposition of countless incompatible snippets of text. It has no unified physics, no consistent ontology, no object permanence, no stable causal texture. It’s a fragmented, discontinuous series of words and tokens held together by probability and dataset curation rather than coherent laws.
A conscious textual agent would need something like a unified narrative environment with real feedback: symbols that maintain identity over time, a stable substrate where “being someone” is definable, the ability to form and test a hypothesis, and experience the consequences. LLMs don’t have that. They exist in a shifting cloud of possibilities with no single consistent reality to anchor self-maintaining loops. They can generate pockets of local coherence, but they can’t accumulate global coherence across time.
So even if consciousness-in-text were possible in principle, the core requirement isn’t just architecture or emergent cleverness—it’s coherence of habitat. A conscious system, physical or textual, can only be as coherent as the world it lives in. And LLMs don’t live in a world today. They’re still prisoners in the cave, predicting symbols and shadows of worlds they never inhabit.
I think this is an excellent point. I believe the possibility of 'computing' a conscious mind is proportional to the capability of computing a meaningful reality for it to exist in.
So you are begging the question: Is it possible to compute a textual, or pure symbolic reality that is complex enough for consciousness to arise within it?
Let's assume yes again.
Finally the theory leads us back to engineering. We can attempt to construct a mind and expose it to our reality, or we can ask "What kind of reality is practically computable? What are the computable realities?"
Perhaps herein lies the challenge of the next decade. LLM training is costly, lots of money poured out into datacenters. All with the dream of giving rise to a (hopefully friendly / obedient) super intelligent mind. But the mind is nothing without a reality to exist in. I think we will find that a meaningfully sophisticated reality is computationally out of reach, even if we knew exactly how to construct one.
[delayed]
> A conscious textual agent would need something like a unified narrative environment with real feedback: symbols that maintain identity over time, a stable substrate where “being someone” is definable, the ability to form and test a hypothesis, and experience the consequences.
So like a Claude Code session? The code persists as symbols with stable identity. The tests provide direct feedback. Claude tracks what it wrote versus what I changed - it needs identity to distinguish its actions from mine. It forms hypotheses about what will fix the failing tests, implements them, and immediately experiences whether it was right or wrong. The terminal environment gives it exactly the "stable substrate where 'being someone' is definable" you're asking for. We missing anything?
Okay, you're right. There is a world, and some hypotheses, and some falsifiability.
But how rich is this world?
Does this world progress without direct action from another entity? Can the agent in this case form hypotheses and test them without intervention? Can the agent form their own goals and move towards them? Does the agent have agency, or is it simply responding to inputs?
If the world doesn’t develop and change on its own, and the agent can’t act independently, is it really an inhabited world? Or just a controlled workspace?
If you accept the premise that the consciousness is computable then pausing the computation can't be observed by the consciousness. So the world being a controlled workspace in my eyes doesn't contradict a consciousness existing?
> Any conscious agent, textual or physical, has to inhabit a world whose structure is stable, self-consistent, and rich enough to support persistent internal dynamics. Even a purely symbolic mind would still need a coherent symbolic universe. And this is precisely where LLMs fall short, through no fault of their own. The universe they operate in isn’t a world—it’s a superposition of countless incompatible snippets of text.
The consistency and coherence of LLM outputs, assembled from an imperfectly coherent mess of symbols is an empirical proof that the mess of symbols is in fact quite coherent.
The physical world is largely incoherent to human consciousnesses too, and we emerged just fine.
Coherence here isn't about legible text, it's environmental coherence where you can deduce truths about the world through hypotheses and experimentation. Coherence isn't about a consistent story narrative, it's about a persistent world with falsifiable beliefs and consequences.
Right but as empirically demonstrated by LLM outputs, they can in fact make "true" predictions/deductions from their environment of tokens.
They sometimes get it wrong, just like all other conscious entities sometimes get their predictions wrong. There are (often) feedback mechanisms to correct those instances though, in both cases.
It's incredibly embarrassing to use wording like "as empirically demonstrated" in an attempt to make your argument appear scientifically rigorous. The bar is on the floor for the concept you're talking about "empirically demonstrating".
As empirically demonstrated, a trivial script can in fact make "true" predictions from their environment. They sometimes get it wrong, just like all other conscious entitities sometimes get their predictions wrong. There are feedback mechanisms to correct those instances though. Ergo, this script is conscious, QED.Peoples interior model of the world is very tenuously related to reality. We don't have a direct experience of waves, quantum mechanics, the vast majority of the electromagnetic spectrum, etc. The whole thing is a bunch of shortcuts and hacks that allow people to survive, the brain isn't really setup to probe reality and produce true beliefs, and the extent to which our internal models of reality naturally match actual reality is related to how much that mattered to our personal survival before the advent of civilization and writing, etc.
It's really only been a very brief amount of time in human history where we had a deliberate method for trying to probe reality and create true beliefs, and I am fairly sure that if consciousness existed in humanity, it existed before the advent of the scientific method.
I don't think it's brief at all. Animals do this experimentation as well, but clearly in different ways. The scientific method is a formalized version of this idea, but even the first human who cooked meat or used a stick as a weapon had a falsifiable hypothesis, even if it wasn't something they could express or explain. And the consequences of testing the hypothesis were something that affected the way they acted from there on out.
I see a lot of arguments on this website where people passionately project the term consciousness onto LLMs.
From my perspective, the disconnect you describe is one of the main reasons this term cannot be applied.
Another reason is that the argument for calling LLMs conscious arises from the perspective of thinking and reasoning grounded in language.
But in my personal experience, thinking in language is just a small emerging quality of human consciousness. It is just that the intellectuals making these arguments happen to be fully identified with the “I think therefore I am” aspect of it and not the vastness of the rest.
I don’t know about others, but this is definitely not why I question whether LLMs are conscious or not.
I don’t think you should presume to know the reason people raise this idea.
>A conscious textual agent would need something like a unified narrative environment with real feedback: symbols that maintain identity over time, a stable substrate where “being someone” is definable, the ability to form and test a hypothesis, and experience the consequences. LLMs don’t have that. They exist in a shifting cloud of possibilities with no single consistent reality to anchor self-maintaining loops. They can generate pockets of local coherence, but they can’t accumulate global coherence across time.
These exist? Companies are making billions of dollars selling persistent environments to the labs. Huge amounts of inference dollars are going into coding agents which live in persistent environments with internal dynamics. LLMs definitely can live in a world, and what this world is and whether it's persistent lie outside the LLM.
I agree, I'm sure people have put together things like this. There's a significant profit and science motive to do so. JEPA and predictive world models are also a similar implementation or thought experiment.
>The missing variable in most debates is environmental coherence. Any conscious agent, textual or physical, has to inhabit a world whose structure is stable, self-consistent, and rich enough to support persistent internal dynamics. Even a purely symbolic mind would still need a coherent symbolic universe.
I'm not sure what relevance that has to consciousness?
I mean you can imagine a consciousness where, you're just watching TV. (If we imagine that the video models are conscious their experience is probably a bit like that!)
If the signal wasn't coherent it would just be snow, static, TV noise. (Or in the case of a neural network probably something bizarre like DeepDream.) But there would still be a signal.
I've sometimes wondered if consciousness is something like a continuous internal narrative that naturally arises when an intelligent system experiences the world through a single source (like a body). That sounds similar to what you're saying.
Regardless, I think people tend to take consciousness a bit too seriously and my intuition is consciousness is going to have a similar fate to the heliocentric model of the universe. In other words, we'll discover that consciousness isn't really "special" just like we found out that the earth is just another planet among trillions and trillions.
I've wondered if LLMs are infact conscious as per some underwhelming definition as you mentioned. Just for the brief moment they operate on a prompt. They wake up, they perceive their world through tokens, do a few thinking loops then sleep until the next prompt.
So what? Should we feel bad for spawning them and effectively killing them? I think not.
Why is that any different from the utter mess of a world humans find themselves existing in?
We can form and test hypotheses and experience the consequences. And then take that knowledge to our next trial. Even dogs and cats do this on a daily basis. Without that, how would we even evaluate whether something is conscious?
And these expectations are violated regularly?
The question of how to evaluate whether something is conscious is totally different from the question of whether it actually is conscious.
> these expectations are violated regularly?
I don't know what you're thinking of, but mine are.
Practice of any kind (sports, coding, puzzles) works like that.
Most of all: interactions with any other conscious entity. I carry at least intuitive expectations of how my wife / kid / co-workers / dog (if you count that) will respond to my behavior, but... Uh. Often wrong, and have to update my model of them or of myself.
I agree with your second paragraph.
LLMs can do the same within the context window. It's especially obvious for the modern LLMs, tuned extensively for tool use and agentic behavior.
Okay, so you're talking about LLMs specifically in the context of a ChatGPT, Claude, or pick-your-preferred-chatbot. Which isn't just an LLM, but also a UI, a memory manager, a prompt builder, a vectorDB, a system prompt, and everything else that goes into making it feel like a person.
Let's work with that.
In a given context window or conversation, yes, you can have a very human-like conversation and the chatbot will give the feeling of understanding your world and what it's like. But this still isn't a real world, and the chatbot isn't really forming hypotheses that can be disproven. At best, it's a D&D style tabletop roleplaying game with you as the DM. You are the human arbiter of what is true and what is not for this chatbot, and the world it inhabits is the one you provide it. You tell it what you want, you tell it what to do, and it responds purely to you. That isn't a real world, it's just a narrative based on your words.
A modern agentic LLM can execute actions in "real world", whatever you deem as such, and get feedback. How is that any different from what humans do?
This is a great point, but even more basic to me is that LLMs don't have identity persistence of their own. There is a very little guarantee in a web-scale distributed system that requests are being served by the same process on the same host with access to the same memory, registers, whatever it is that a software process "is" physically.
Amusingly, the creators of Pluribus lately seem to be implying they didn't intend it to be allegory about LLMs, but dynamic is similar. You can have conversations with individual bodies in the collective, but they aren't actually individuals. No person has unique individual experiences and the collective can't die unless you killed all bodies at once. New bodies born into the collective will simply assume the pre-existing collective identity and never have an individual identity of their own.
Software systems work the same way. Maybe silicon exchanging electrons can experience qualia of some sort, and maybe for whatever reason that happens when the signals encode natural language textual conversations but not anything else, but even if so, the experience would be so radically different from what embodied individuals with distinct boundaries, histories, and the possibility of death experience that analogies to our own experiences don't hold up even if the text generated is similar to what we'd say or write ourselves.
> For some people (including me), a sense of phenomenal consciousness feels like the bedrock of existence, the least deniable thing; the sheer redness of red is so mysterious as to seem almost impossible to ground. Other people have the opposite intuition: consciousness doesn’t bother them, red is just a color, obviously matter can do computation, what’s everyone so worked up about? Philosophers naturally interpret this as a philosophical dispute, but I’m increasingly convinced it’s an equivalent of aphantasia, where people’s minds work in very different ways and they can’t even agree on the raw facts to be explained.
Is Scott accusing people who don't grasp the hardness of the hard problem of consciousness of being p-zombies?
(TBH I've occasionally wondered this myself.)
To me, the absurdity of the idea of p-zombies is why I'm convinced consciousness isn't special to humans and animals.
Can complex LLMs have subjective experience? I don't know. But I haven't heard an argument against it that's not self-referential. The hardness of the hard problem is precisely why I can't say whether or not LLMs have subjective experience..
How would you differentiate that argument from similar arguments about other observable phenomena? As in…
No one has ever seen or otherwise directly experienced the inside of a star, nor is likely to be able to do so in the foreseeable future. To be a star is to emit a certain spectrum of electromagnetic energy, interact gravitationally with the local space-time continuum according to Einstein’s laws, etc.
It’s impossible to conceive of an object that does these things that wouldn’t be a star, so even if it turns out (as we’ll never be able to know) that Gliese 65 is actually a hollow sphere inhabited by dwarven space wizards producing the same observable effects, it’s still categorically a star.
(Sorry, miss my philosophy classes dearly!)
FWIW I have gone from not understanding the problem to understanding the problem in the past couple of years because it's not trivial to casually intuit if you don't actually think about it and don't find it innately interesting and the discourse doesn't have the language to adequately express the problem, so this is probably wrong.
I’ve sort-of gone the opposite way. The more I introspect, the more I realize there isn’t anything mysterious there.
It’s true that we are lacking good language to talk about it, as we already fail at successfully communicating levels of phantasia/aphantasia.
It’s not so much that there’s anything mysterious you can discover through intense introspection or meditation. There might be, but I haven’t found it.
It’s fundamentally that this capability exists at all.
Strip it all down to I think therefore I am. That is very bizarre because it doesn’t follow that such a thing would happen. It’s also not clear that this is even happening at all, and, as an outside observer, you would assess that it isn’t. However, from the inside, it is clear that it is.
I don’t have an explanation for anyone but I have basically given up and accepted that consciousness is epiphenomenal, like looking through a microscope.
The thing is that when you say “that capability”, I don’t quite know what you mean. The fact that we perceive inner processings of our mind isn’t any more surprising than that we perceive the outer world, or that a debugger is able to introspect its own program state. Continuous introspection has led me to realize that “qualia”, or “what it’s like to be X”, emotions and feelings, are just perceptions of inner phenomena, and that when you pay close attention, there is nothing more to that perception than its informational content.
All this talk about machine consciousness and I think I'm probably the only one that thinks it doesn't actually matter.
A conscious machine should treated be no different than livestock - heck, an even lower form of livestock - because if we start thinking we need to give thinking machines "rights" and to "treat them right" because they are conscious then it's already over.
My toaster does not get a 1st amendment because it's a toaster and can and never should be a person.
What do you mean? What is over? Do you mean the dominion of Homo Sapiens over the earth? If so, would that necessarily be bad?
The way you phrased it reminded me of some old Confederate writings I had read, saying that the question of whether black people are fully human, with souls and all, boils down to "if we do, our way of life is over, so they don't".
We do have forms of animal rights, including for livestock, and having them is not a controversial position.
I think this is actually a majority of everyone working on anything remotely related to artificial intelligence post-Searle.
> I think I'm probably the only one that thinks...
It's unlikely this is true for nearly every thought you may ever have, there's a lot of people
Should we have a thread about the actual paper (https://www.sciencedirect.com/science/article/pii/S136466132...) or is it enough to put the link in the toptext of this one?
When discussing consciousness what is often missed is that the notion of consciousness is tightly coupled with the notion of the perception of time flow. By any reasonable notion conscious entity must perceive the flow of time.
And then the time flow is something that physics or mathematics still cannot describe, see Wikipedia and other articles on the philosophical problem of time series A versus time series B that originated in a paper from 1908 by philosopher John McTaggart.
As such AI cannot be conscious since mathematics behind it is strictly about time series B which cannot describe the perception of time flow.
The stateless/timeless nature of LLMs comes from the rigid prompt-response structure. But I don't see why we cant in theory decouple the response from the prompt, and have them constantly produce a response stream from a prompt that can be adjusted asynchronously by the environment and by the LLMs themselves through the response tokens and actions therein. I think that would certainly simulate them experiencing time without the hairy questions about what time is.
Is consciousness coupled with "time flow" or specifically "cause and effect", i.e. prediction? LLMs learn to predict the next word, which teaches them more general cause and effect (required to predict next words in narratives).
As such humans cannot be conscious...
Consciousness implies self-awareness, in space and time. Consciousness implies progressive formation of the self. This is not acquired instantly by a type of design. This is acquired via a developmental process where some conditions have to be met. Keys to consciousness are closer to developmental neurobiology than the transformer architecture.
The substance / structure point is fascinating.
It gives us four quadrants.
Natural Substance, Natural Structure: Humans, dogs, ants, bacteria.
Natural Substance, Artificial Structure: enslaved living neurons (like the human brain cells that play pong 24/7), or perhaps a hypothetical GPT-5 made out of actual neurons instead of Nvidia chips.
Artificial Substance, Natural Structure: if you replace each of your neurons with a functional equivalent made out of titanium... would you cease to be conscious? At what point?
Artificial substance, Artificial structure: GPT etc., but also my refrigerator, which also has inputs (current temp), goals (maintain temp within range), and actions (turn cooling on/off).
The game SOMA by Frictional (of Amnesia fame!) goes into some depth on this subject.
The good news is we can just wait until the AI is superintelligent, then have it explain to us what consciousness really is, and then we can use that to decide if the AI is conscious. Easy peasy!
We can talk to bees, we know their language. How would you go to explain what it's like to be a human to a bee?
... and then listen to it debate whether or not mere humans are "truly conscious".
(Said with tongue firmly in cheek.)
Complexity of a single neuron is out of reach for all of the world's super computers. So we have to conclude that if the authors believe in a computational/functionalist instantiation of consciousness or self-awareness then they must also believe that the complexity of neurons is not necessary & is in fact some kind of accident that could be greatly simplified but still be capable of carrying out the functions in the relational/functionalist structure of conscious phenomenology. Hence, the digital neuron & unjustified belief that a properly designed boolean circuit & setting of inputs will instantiate conscious experience.
I have yet to see any coherent account of consciousness that manages to explain away the obvious obstructions & close the gap between lifeless boolean circuits & the resulting intentional subjectivity. There is something fundamentally irreducible about what is meant by conscious self-awareness that can not be explained in terms of any sequence of arithmetic/boolean operations which is what all functionalist specifications ultimately come down to, it's all just arithmetic & all one needs to do is figure out the right sequence of operations.
> irreducible
It seems like the opposite is true.
Only if you agree with the standard extensional & reductive logic of modern science but even then it is known that all current explanations of reality are incomplete, e.g. the quantum mechanical conception of reality consists of incessant activity that we can never be sure about.
It's not obvious at all why computer scientists & especially those doing work in artificial intelligence are convinced that they are going to eventually figure out how the mind works & then supply a sufficient explanation for conscious phenomenology in terms of their theories b/c there are lots of theorems in CS that should convince them of the contrary case, e.g. Rice's theorem. So even if we assume that consciousness has a functional/computable specification then it's not at all obvious why there would be a decidable test that could take the specification & tell you that the given specification was indeed capable of instantiating conscious experience.
It isn't surprising that "phenomenal consciousness" is the thing everyone gets hung about, after all we are all immersed in this water. The puzzle seems intractable but only because everyone is accepting the priors and not looking more carefully at it.
This is the endpoint of meditation, and the observation behind some religious traditions, which is look carefully and see that there was never phenomenal consciousness where we are a solid subject to begin with. If we can observe that behavior clearly, then we can remove the confusion in this search.
I see this comment nearly every time consciousness is brought up here and I’m pretty sure this is a misunderstanding of contemplative practices.
Are you a practitioner who has arrived at this understanding, or is it possible you are misremembering a common contemplative “breakthrough” that the self (as separate from consciousness) is illusory, and you’re mistakenly remembering this as saying consciousness itself is illusory?
Consciousness is the only thing we can be absolutely certain does actually exist.
As a very beginner practicer i've come to that conclusion myself, but how can the two be separate? If there is no self (or at least, there is a self but it exists in the same way that a nation or corporation "exists"), how can there be something to experience being? What separates the two?
My own experiential insight is not definitely not complete, so of course the guidance of a master or of course your own direct practice should be preferred.
But to the extent I have observed awareness, the idea of an entire "experiencer" is an extrapolation and fabrication. See how you generate that concept. And then, look closely at what's actually going on, there is "consciousness" of the components of the aggregate. (Maybe not dissimilar to some of the lower level mechanisms proposed in the article).
Phenomenal consciousness as being raised here, and probably in most people's minds, is probably taken to be the self or at least deeply intertwined with the concept of a separate self. The article tries to define it left and right, but I think most people will look at their own experience and then get stuck in this conversation.
"Consciousness" in the traditions is maybe closer to some of the lower abstraction proposals put out in the article.
I don't think the idea of illusory is necessarily the right view here. Maybe most clearly the thing to say is that there is "not" self and "not" consciousness. That these things are not separate entities and instead are dependently arisen. That consciousness is also dependently arisen is probably more contentious and different traditions make different claims on that point.
> Consciousness is the only thing we can be absolutely certain does actually exist.
A lot of philosophers would disagree with this.
Yeah sure, it's irrelevant to my actual question which is whether GP thinks consciousness doesn't exist or whether they're mistakenly replacing consciousness for self.
Let's say a genie hands you a magic wand.
The genie says "you can flick this wand at anything in the universe and - for 30 seconds - you will swap places with what you point it at."
"You mean that if I flick it at my partner then I will 'be' her for 30 seconds and experience exactly how she feels and what she thinks??"
"Yes", the genie responds.
"And when I go back to my own body I will remember what it felt like?"
"Absolutely."
"Awesome! I'm going to try it on my dog first. It won't hurt her, will it?"
"No, but I'd be careful if I were you", the genie replies solemnly.
"Why?"
"Because if you flick the magic wand at anything that isn't sentient, you will vanish."
"Vanish?! Where?" you reply incredulously.
"I'm not sure. Probably nowhere. Where do you vanish to when you die? You'll go wherever that is. So yeah. You probably die."
So: what - if anything - do you point the wand at?
A fly? Your best friend? A chair? Literally anyone? (If no, congratulations! You're a genuine solipsist.) Everything and anything? (Whoa... a genuine panpsychist!)
Probably your dog, though. Surely she IS a good girl and feels like one.
Whatever property you've decided that some things in the universe have and other things do not such that you "know" what you can flick your magic wand at and still live...
That's phenomenal consciousness. That's the hard problem.
Everything else? "Mere" engineering.
> congratulations! You're a genuine solipsist
Wrong, the genie is. The thought experiment is flawed/loaded.
I'm flipping it at the genie first, then removing the sentience requirement in 30 seconds.
Hey not fair!
While you're in there I have a few favors to ask...
Seems very bold to assume the genie is sentient
Eh, it's talking to me, it's the safest bet. It's either that or nothing.
Right. So do you flick it at ChatGPT? It's talking to you, after all.
(I honestly don't know. If there's any phenomenal consciousness there it would have to be during inference, but I doubt it.)
Well, if it gets to dogs, I'm not sure I wouldn't do ChatGPT first.
I think the illuminating part here is that only a magic wand could determine if something is sentient
How does the wand know what I'm flicking it at? What if I miss? Maybe the wand thinks I'm targeting some tiny organism that lives on the organism that I'm actually targeting. Can I target the wand with itself?
> How does the wand know what I'm flicking it at?
Magic! (i.e. not purely part of the thought experiment, unless I'm missing something interesting)
> What if I miss?
Panpsychism better be true :)
> Can I target the wand with itself?
John Malkovich? Is that you?!
It's magic. Chill out. It knows.
I'm waiting for someone to transcend the concept of I know it when I see it about consciousness.
What I love about this paper is that it is moving away from very fuzzily-defined and emotionally weighted terms like 'intelligence' and 'consciousness' and focusing on specific, measurable architectural features.
Some people behave as if there's something mysterious going on in LLMs, and that somehow, we must bracket our knowledge to create this artificial sense of mystery, like some kind of subconscious yearning for transcendence that's been perverted . "Ooo, what if this particular set of chess piece moves makes the board conscious??" That's what the "computational" view amounts to, and the best part of it is that it has all the depth of a high college student's ramblings about the multiverses that might occupy the atoms of his fingers. No real justification, no coherent or intelligible case made, just a big "what if" that also flies in the face of all that we know. And we're supposed to take it seriously, just like that.
"[S]uper-abysmal-double-low quality" indeed.
One objection I have to the initial framing of the problem concerns this characterization:
"Physical: whether or not a system is conscious depends on its substance or structure."
To begin with, by what right can we say that "physical" is synonymous with possessing "substance or structure"? For that, you would have to know:
1. what "physical" means and be able to distinguish it from the "non-physical" (this is where people either quickly realize they're relying on vague intuitions about what is physical or engaging in circular reasoning a la "physical is whatever physics tells us");
2. that there is nothing non-physical that has substance and structure.
In an Aristotelian-Thomistic metaphysics (which are much more defensible than materialism or panpsychism or any other Cartesian metaphysics and its derivatives), not only is the distinction between the material and immaterial understood, you can also have immaterial beings with substance and structure called "subsistent forms" or pure intellects (and these aren't God, who is self-subsisting being).
According to such a metaphysics, you can have material and immaterial consciousness. Compare this with Descartes and his denial of the consciousness of non-human animals. This Cartesian legacy is very much implicated in the quagmire of problems that these stances in the philosophy of mind can be bogged down in.
I look forward to other papers on spreadsheet consciousness and terminal emulator consciousness.
I think it's very unlikely any current LLMs are conscious, but these snarky comments are tiresome. I would be surprised if you read a significant amount of the post.
I believe the biggest issue creating a testable definition for conscientiousness. Unless we can prove we are sentient (and we really can't - I could just be faking it), this is not a discussion we can have in scientific terms.
It’s really trivial to prove but the issue is that sentience is not something you need to negate out of existence and then attempt reconstruct out of epistemological proofs. You’re not faking it, and if you were, then turn your sentience off and on again. If your idea comes from Dennett then he’s barking up completely the wrong tree.
You know at a deep level that a cat is sentient and a rock isn’t. You know that an octopus and a cat have different modes of sentience to that potentially in a plant, and same again for a machine running electrical computations on silicon. These are the kinds of certainties that all of your other experiences of the world hinge upon.
What, all two brief pages of it
The notion doesn't strike me as something that's much more ridiculous than consciousness of wet meat.
We'll know AGI has arrived when we finally see papers on Coca Cola Vending Machine Consciousness.
Following up on Anthropic's Project Vend [0] and given the rising popularity of Vending-Bench[1], it's actually quite likely that by the time an AI is deemed to possess consciousness, it will have already been tested in a vending machine.
[0] https://www.anthropic.com/research/project-vend-1
[1] https://andonlabs.com/evals/vending-bench
"Consciousness" is just what we call the thing we can't quite define that we believe separates us from other kinds of machine.
We’re not a machine
Where's the non-machine part of the human body that doesn't follow physical laws?
Are you aware of all possible laws of the universe? Furthermore, asking questions is not how one makes a positive & justifiable claim.
The underlying paper is from AE Studio people (https://arxiv.org/abs/2510.24797), who want to dress up their "AI" product with philosophical language, similar to the manner in which Alex Karp dresses up data base applications with language that originates in German philosophy.
Now I have to remember not to be mean to my Turing machine.
I abstain from making any conclusion about LLM consciousness. But the description in the article is fallacious to me.
Excluding LLMs from “something something feedback” but permitting mamba doesn’t make sense. The token predictions ARE fed back for additional processing. It might be a lossy feedback mechanism, instead of pure thought space recurrence, but recurrence is still there.
Especially given that it references the Anthropic paper on LLM introspection - which confirms that LLMs are somewhat capable of reflecting on their own internal states. Including their past internal states, attached to the past tokens and accessed through the attention mechanism. A weak and unreliable capability in today's LLMs, but a capability nonetheless.
https://transformer-circuits.pub/2025/introspection/index.ht...
I guess the earlier papers on the topic underestimated how much introspection the autoregressive transformer architecture permits in practice - and it'll take time for this newer research to set the record straight.
Has anyone read Hofstadter's I Am a Strange Loop?
> By ‘consciousness’ we mean phenomenal consciousness. One way of gesturing at this concept is to say that an entity has phenomenally conscious experiences if (and only if) there is ‘something it is like’ for the entity to be the subject of these experiences.
Stopped reading after this lol. Its just the turing test?
No.
https://en.wikipedia.org/wiki/What_Is_It_Like_to_Be_a_Bat%3F
One of the primary issues with Nagel's approach is that "what is it like" is - for reasons I have never been able to fathom - a phrase that imports the very ambiguity that Nagel is attempting to dispel.
The question of what it would feel like to awake one day to find that - instead of lying in your bed - you are hanging upside down as a bat is nearly the complete dual of the Turing test. And even then, the Turing test only asks whether your interlocutor is convincing you that it can perform the particulars of human behavior.
The "what it's like" is often bound up with the additional "what would it be like to wake up as", which is a different (and possibly nonsensical) question. Leaving aside consciousness transfer, there's an assumption baked into most consciousness philosophy that all (healthy, normal) humans have an interior point of view, which we refer to as consciousness, or in this paper and review as "phenomenal consciousness". Sometimes people discuss qualia in reference to this. One thing that I've noticed more very recently is the rise of people claiming that they, themselves, do not experience this internal point of view, and that there's nothing that it is like to be them, or, put another way, humans claiming that they are p-zombies, or that everyone is. Not sure what to make of that.
Ok so its like a deep comparator on the sensory and processing units in the "mind".
> Phenomenal consciousness is crazy. It doesn’t really seem possible in principle for matter to “wake up”.
> In 2004, neuroscientist Giulio Tononi proposed that consciousness depended on a certain computational property, the integrated information level, dubbed Φ. Computer scientist Scott Aaronson complained that thermostats could have very high levels of Φ, and therefore integrated information theory should dub them conscious. Tononi responded that yup, thermostats are conscious. It probably isn’t a very interesting consciousness. They have no language or metacognition, so they can’t think thoughts like “I am a thermostat”. They just sit there, dimly aware of the temperature. You can’t prove that they don’t.
For whatever reason HN does not like integrated information theory. Neither does Aaronson. His critique is pretty great, but beyond poking holes in IIT, that critique also admits that it's the rare theory that's actually quantified and testable. The holes as such don't show conclusively that the theory is beyond repair. IIT is also a moving target, not something that's frozen since 2004. (For example [1]). Quickly dismissing it without much analysis and then bemoaning the poor state of discussion seems unfortunate!
The answer to the thermostat riddle is basically just "why did you expect a binary value for consciousness and why shouldn't it be a continuum?" Common sense and philosophers will both be sympathetic to the intuition here if you invoke animals instead of thermostats. If you wanted a binary yes/no for whatever reason, just use an arbitrary cut-off I guess, which will lead to various unintuitive conclusions.. but play stupid games and win stupid prizes.
For the other standard objections, like a oldschool library card-catalogue or a hard drive that encodes a contrived Vandermonde matrix being paradoxically more conscious than people, variations on IIT are looking at normalizing phi-values to disentangle matters of redundancy of information "modes". I haven't read the paper behind TFA and definitely don't have in-depth knowledge of Recurrent Processing Theory or Global Workspace Theory at all. But speaking as mere bystander, IIT seems very generic in its reach and economical in assumptions. Even if it's broken in the details, it's hard to imagine that some minor variant on the basic ideas would not be able to express other theories.
Phi ultimately is about applied mereology moving from the world of philosophy towards math and engineering, i.e. "is the whole more than the sum of the parts, if so how much more". That's the closest I've ever heard to anything touching on the hard problem and phenomenology.
[1] https://pubs.aip.org/aip/cha/article/32/1/013115/2835635/Int...
I think this is one of the more interesting theories out there, because it makes "predictions" that come close to my intuitive understanding of consciousness.
I generally regard thinking about consciousness, unfortunately, a thing of madness.
"I think consciousness will remain a mystery. Yes, that's what I tend to believe... I tend to think that the workings of the conscious brain will be elucidated to a large extent. Biologists and perhaps physicists will understand much better how the brain works. But why something that we call consciousness goes with those workings, I think that will remain mysterious." - Ed Witten, probably the greatest living physicist
"The New AI Consciousness Paper – Reviewed By Scott Alexander" might be less confusing. He isn't an author of the paper in question, and "By Scott Alexander" is not part of the original title.
Scott Alexander, the prominent blogger and philosopher, has many opinions that I am interested in.
After encountering his participation in https://ai-2027.com/ I am not interested in hearing his opinions about AI.
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
Opinions vary, but I posted a link to a web page that he co-authored, which I would argue stands as a very significant and deep dismissal of his views on AI. If, after reading that essay, a person still feels that Scott Alexander has something interesting to say about AI, then I challenge them to defend that thesis.
Probably better for me to have remained silent out of politeness, but if anyone follows that link to the https://ai-2027.com/ page then I feel I have done my part to help inform that person of the lack of rigor in Scott Alexander's thinking around AI.
>After encountering his participation in https://ai-2027.com/ I am not interested in hearing his opinions about AI.
I'm not familiar with ai-2027 -- could you elaborate about why it would be distasteful to participate in this?
It is an attempt to predict a possible future in the context of AI. Basically a doomer fairy tale.
It is just phenomenally dumb.
Way worse than the worst bad scifi about the subject. It is presented as a cautionary tale and purports to be somewhat rationally thought out. But it is just so bad. It tries to delve into foreign policy and international politics but does so in such a naive way that it is painful to read.
It is not distasteful to participate in it -- it is embarrassing and, from my perspective, disqualifying for a commentator on AI.
Whole lot of "doomer", "fairy tale", "dumb", "bad scifi", "so bad", "naive", "embarassing".
Not any actual refutation. Maybe this opinion is a bit tougher to stomach for some reason than the rest you agree with...
I reject the premise that https://ai-2027.com/ needs "refutation". It is a story, nothing more. It does not purport to tell the future, but to enumerate a specific "plausible" future. The "refutation" in a sense will be easy -- none of its concrete predictions will come to pass. But that doesn't refute its value as a possible future or a cautionary tale.
That the story it tells is completely absurd is what makes it uninteresting and disqualifying for all participants in terms of their ability to comment on the future of AI.
Here is the prediction about "China Steals Agent-2".
> The changes come too late. CCP leadership recognizes the importance of Agent-2 and tells their spies and cyberforce to steal the weights. Early one morning, an Agent-1 traffic monitoring agent detects an anomalous transfer. It alerts company leaders, who tell the White House. The signs of a nation-state-level operation are unmistakable, and the theft heightens the sense of an ongoing arms race.
Ah, so CCP leadership tells their spies and cyberforce to steal the weights so they do. Makes sense. Totally reasonable thing to predict. This is predicting the actions of hypothetical people doing hypothetical things with hypothetical capabilities to engage in the theft of hypothetical weights.
Even the description of Agent-2 is stupid. Trying to make concrete predictions about what Agent-1 (an agent trained to make better agents) will do to produce Agent-2 is just absurd. Like Yudkowsky (who is far from clear-headed on this topic but at least has not made a complete fool of himself) has often pointed out, if we could predict what a recursively self-improving system could do then why do we need the system.
All of these chains of events are incredibly fragile and they all build on each other as linear consequences, which is just a naive and foolish way to look at how events occur in the real world -- things are overdetermined, things are multi-causal; narratives are ways for us to help understand things but they aren't reality.
An example
>The job market for junior software engineers is in turmoil: the AIs can do everything taught by a CS degree, but people who know how to manage and quality-control teams of AIs are making a killing.
AI doesn't look like a competition for a junior engineer and many of the people using not "managing" AI are going to be juniors in fact increasing what a junior can do and learn more quickly looks like one of the biggest potentials if they don't use it entirely as a crunch.
Meanwhile, it suggests leading-edge research into AI itself will proceed fully 50% faster than research not without AI but those using 6 months behind cutting edge. This appears hopelessly optimistic as does the idea that it will grow the US economy 30% in 2026 whereas a crash seems more likely.
Also it assumes that more compute will continue to be wildly more effective in short order assuming its possible to spend the money for magnitudes more compute. Either or both could easily fail to work out to plan.
I'm not sure why it's so distasteful, but they basically fear monger that AI will usurp control over all governments and kill us all in the next two years
Is there a reason why this text uses "-" as em-dashes "—"?
Since they are set open, I assume they are actually using them as if they were en-dashes and not em-dashes, which the more common style would be to set closed, but I’m guessing, in either case, the reason is “because you can type it on a normal keyboard without any special modification, Compose-key solution, or other processing, and the author doesn't care much about typography”.
EDIT: Though these the days it could also be an attempt at highly-visible “AI didn't write this” virtue signaling, too.
Yes; because - is on the keyboard and — isn't. (Don't tell me how to type —, I know how, but despite that it is the reason, which is what the parent comment asks about.)
Many people have for decades. Seems fine to me.
Is there a reason you phrased the question that way, instead of just asking whether it was written by AI?
It's just that I have the feeling that people avoid using the actual em-dash in fear of being accused that the text is ai generated. (Which isnt a valid indicator anyway) Maybe its just my perception that i notice this more since LLMs became popular.
my original word processor corrected “—-“ to an em-dash, which i would get rid of because it didnt render correctly somewhere in translation between plaintext- markdown- html (sort of how it butchered “- -“ just now on HN.)
but what youd see in your browser was “square blocks”
so i just ran output through some strings/awk /sed (server side) to clean up certain characters, that i now know specifying “ utf-8 “ encoding fixes altogether.
TLDR: the “problem” was “lets use wordpress as a CMS and composer, but spit it out in the same format as its predecessor software and keep generating static content that uses the design we already have”
em-dashes needed to be double dashes due to a longstanding oversight.
The Original Sin was Newsmaker, which had a proprietary format that didnt work in anything else and needed some perl magic to spit out plaintext.
I don’t work in that environment or even that industry anymore but took the hacky methodology my then-boss and I came up with together.
SO,
1) i still have a script that gets rid of them when publishing, even though its no longer necessary. and its been doing THAT longer than “LLMs” were mainstream.
and 2) now that people ask “did AI write this?” i still continue with a long standing habit of getting rid of them when manually composing something.
Funny story though after twenty years of just adding more and more post processing kludge. I finally screamed AAAAAAAAHAHHHH WHY DOES THIS PAGE STILL HAVE SQUARE BLOCKS ALL OVER IT at “Grok.”
All that kludge and post processing solved by adding utf-8 encoding in the <head>, which an “Ai” helpfully pointed out in about 0.0006s.
That was about two weeks ago. Not sure when I’ll finally just let my phone or computer insert one for me. Probably never. But thats it. I don’t hate the em-dash. I hate square blocks!
Absolutely nothing against AI. I had a good LONG recovery period where I could not sit there and read 40-100 page paper or a manual anymore, and i wasnt much better at composing my own thoughts. so I have a respect for its utility and I fully made use of that for a solid two years.
And it just fixed something that id overlooked because, well, im infrastructure. im not a good web designer.
Will we know AGI has been achieved when it stops using em-dashes?
Any AI smart enough not to use em-dashes will be smart enough to use them.
I don't see why it matters so much whether something is conscious or not. All that we care about is, whether something can be useful.
> All that we care about is, whether something can be useful
Anybody that thinks it's wrong to murder the terminally ill, disabled or elderly probably disagrees with you.
Anyone who knows that being conscious is not same as what you said, might disagree with you. Also, ever thought that chickens being killed all over America everyday, might have consciousness?
At the minimum it raises philosophical and ethical questions. If something is conscious, is it ethical to put it to work for you?
You mean it is not ethical to make them work for us without pay? Well, we had farm animals work for us. They were kind of conscious of the world around them. Ofcourse we fed them and took care of them. So why not treat these AI conscious things same as farm animals, except they work with their mind rather than muscle power.
I'll never ask if AI is conscious because I already know they are not. Consciousness must involve an interplay with the senses. It is naive to think we can achieve AGI by making Platonic machines ever more rational.
https://d1gesto.blogspot.com/2024/12/why-ai-models-cant-achi...
> Consciousness must involve an interplay with the senses.
an idea debated in philosophy for centuries, if not millenia, without consensus.
Maybe be a little more willing to be wrong about such matters?
Must be debated in physics/information theory. One cannot reach the Truth on Reason alone.
[refer to the Scientific Method]
AIs sense at least one thing: their inputs
Are our eyes, ears, nose (smell), touch, taste, and proprioception not just inputs to our brains?
Every time I try to think hard about this subject I can't help but notice that there are some key components making us different from LLMs:
- We have a greater number of inputs - We have the ability to synthesize and store new memories/skills in a way that is different from simply storing data (rote memorization) - Unlike LLMs our input/output loop is continuous - We have physiological drivers like hunger and feedback loops through hormonal interactions that create different "incentives" or "drivers"
The first 3 of those items seem solvable? Mostly through more compute. I think the memory/continuous learning point does still need some algorithmic breakthroughs though from what I'm able to understand.
It's that last piece that I think we will struggle with. We can "define" motivations for these systems but to what complexity? There's a big difference between "my motivation is to write code to accomplish XYZ" and "I really like the way I feel with financial wealth and status so I'm going to try my hardest to make millions of dollars" or whatever other myriad of ways humans are motivated.
Along those thoughts, we may not deem machines conscious until they operate with their own free will and agency. Seems like a scary outcome considering they may be exceptionally more intelligent and capable than your average wetware toting human.