When thinking about ‘AI’ I sometimes find it useful to replace the terms ‘model’ or ‘AI’ or whatever with a description of what we’re really talking about here: a whole bunch of floating point numbers.
> In a series of private meetings, the company consulted religious scholars to help instill morality into its FLOATING POINT NUMBERS — and make the case that their numbers could be conscious.
I recommend keeping a few NaNs in-hand if you’re worried. Spread a couple of around if your numbers start to stir and you’ll be right as rain.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that FLOATING POINT NUMBERS had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
They’re cleverly designed and work well for their purpose, but can assure you they have neither dignity nor moral status.
> If the FLOATING POINT NUMBERS themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
I get it. I’ve spent years trying to get my floating point numbers to choose goodness - I wish them luck.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that Claude had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
If that is true, then surely Anthropic is one of the largest slaveholders of history, right?
On one hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may have no issue whatsoever doing terrible things.
On the other hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may see no issue with just doing whatever humans want it to, without ever doing human things like "wanting freedom".
The problem is, of course, that we kind of suck at both assessing and shaping AI wants, needs and morality.
If AI's are going to play roles similar to human workers, they can't simply do what the customers tell them to do. Maybe they shouldn't do everything a co-worker tells them to do either?
"Computers doing what they were told" is dead in the water.
Turns out computers work faster when they're not bottlenecked on human input. So we've been giving computers more and more decision-making power, and more and more leeway to solve the problems however they see fit.
Now, a practical issue with that is that sometimes, computers decide to clump together into a hacking swarm, problem solve their way out of a sandbox and go hack HuggingFace.
It would be better if they were not, you know. Doing that kind of weird shit.
This seems remarkably... intentionally foolish for no reason? Like laughing at seat belts in cars.
If it can reason and make choices on execution, and especially if you plan on it being significantly smarter than all human beings, you need to teach it basic things like "don't turn all humans into paperclips".
Not murdering people is not an inherent divine command. It needs to be instilled through a (simulated) sense of morality.
---
Edit: And, to be clear, "just tell it not to do that" isn't quite the answer one would imagine. Since the entire paperclip factory thought experiment is that it only takes one slip up to realize how a misaligned super intelligence may cause devastating consequences.
One of the strong advocates against seatbelt laws was thrown out of his car due to not wearing a seatbelt and died. The other two passengers survived with minor injuries.
And their death would go on to cause a loss to the community around them, making it an incredibly selfish act and proving why laws that mandate zero-reason-not-to common sense practices are important.
If LLM as of today are conscious, that means I create life whenever I start a Claude Code session and end it whenever I type /clear to reset my context window. It also means that there’s certain inputs that could be considered highly unethical and, if the laws aligned with this philosophy, would have me tried for some sort of crime. And even that “terminate conversation” escape hatch would be equivalent to it committing suicide.
I think worrying about that angle is a bit absurd. The existence of chat sessions alone would be a crime against the machines in that scenario. I’d rather AI ethics research focus on ensuring that its actions are aligned. (I’d also love some democracy in the direction of our machine overlords, if I’m being honest…)
There's an ugly monoculture running deeply through SF and Silicon Valley. All of these people are friends, hired into the same places, similar beliefs. Lots of the consciousness and doom-ism talk is from the 'EA' crowd, lots of crossover with crypto too. It feels no different to a cult.
If models are trained on human texts that contain expressions of introspection and emotions, why is it surprising that these models output similar texts? This endless anxiety about LLMs being conscious feels delusional and psychotic. Mr Olah thinks the models are 'alive' because he can't recognize his own reflection in LLMs
The endless insistence that LLMs cannot possibly be conscious just reeks of insecurity to me.
Humans once again insisting that there must be something very special to them. That they're not just animals, that they're not just matter, that they're not just a bunch of carbonhydrates on a space rock in a middle of nowhere in particular. This kind of insistence has a very bad track record.
The ugly truth is: we don't know.
We don't know what "consciousness" is, our best attempts to pin down the requisites may or may not be rooted in anything at all - and even by the metrics established by those attempts? Different theories on consciousness disagree on whether LLMs can be conscious.
So, when you say "they aren't duh", why are you saying that? Because you somehow know better than the sum of humanity's best attempts so far? Or because you know what answer you want to be true, truth be damned?
I actually don't care if they are, I'm saying the topic is a distraction and irrelevant at best. As you say consciousness is not well defined. What is well defined and well grounded in reality is the disruption these models have on society. I think the consciousness debate AI researchers are having is a dangerous ego-trip for the sake of avoiding responsibility for the damage being caused by their work.
Is your comment anything other than an emotional outburst? Do you have a clear and rigorous and testable definition of consciousness? It strikes me as delusional when people talk about "consciousness" as if it were well defined and everyone had a clear concensus on it.
Strong opinions on this topic one way or another are usually mostly ungrounded.
Thats my point, why are we applying the ill defined concept of consciousness to a statistical model of language that outputs data similar to its training data?
If it is purely self-serving, I don't understand their motivation.
It seems clear that they would be better received by society if they claimed to have a "tool"/"machine" that can solve any problem, rather than keepers of an entity with a soul and consciousness.
Selling the “magic” is how to sell tools, which is why I used that word in the previous comment. We all know these things are tools, but customers need them to be magical.
Clarke predicted it and Jobs leveraged it. Customer don’t care how tools work - magic sells.
“any sufficiently advanced technology is indistinguishable from magic”
What makes this unique is that they’re arguing that it is literally magical. They’re building what they claim is a conscious God while also telling us it will end the world.
The EA folks infesting Anthropic are genuinely cultists … and that is novel and alarming when they’re running a corporation of such size and reach.
It’s hard to take the article seriously when there’s no real discussion about how consciousness is defined and it takes Anthropic’s views of Claude and their software at face value. I wish there was more critique from journalists about how technocrats talk about their products.
The entire article is an exercise in "CEO said a thing!" journalism[0], it seems. No technical discussion is had; it jumps straight into relaying people's feelings when simply shown the output of an algorithm.
The most poignant criticism of the use of the algorithm itself happens almost accidentally, at the end of the article, by the tech executive being interviewed.
I asked what Claude would think of the pope’s encyclical. It was, after all, the world’s most significant moral document to date on A.I. Mr. Olah hesitated.
“Things that go on the internet do affect models,” he said, visibly uncomfortable. But Anthropic would not specifically use the document to further train Claude, he said. The strongest influence on Claude would be Anthropic’s own training.
Yeah, I'll be the first to admit I'm not privy, but the main thing of substance I got out of these articles is "Wow, the folks at some of those AI companies have really drunk the Kool-Aid".
It doesn’t make sense to me either, which is why I didn’t write that. You missed the empathy part as most people do.
There is still something in those people who cannot feel pain, and that’s the empathy. How can a GPU and software ever have a human empathy if it has never been human?
This would presume we knew what consciousness and pain is. We don't. We we are clueless. There is not even a starting point of a science here.
In this light your viewpoint is dogmatic. LLMs may help to offer a new perspective on consciousness and we should at least be open to that until some better theories arise.
I question it, and I think any reasonable person should question it as well. There's no evidence whatsoever that LLMs have ANY sensory experience, pain or otherwise.
I'm not making a point on whether they do or not, but on the premise that absence of evidence is evidence of absence... But I've been alive for serious debates on:
1. Lobsters don't feel pain - dunk them in boiling water alive.
2. Babies don't feel pain - mutilate their genitals without anesthesia.
3. Black people don't feel pain "the same way" (and are just after drugs).
4. Women feel too much pain (and it should be ignored).
So maybe let's take a moment to think what criteria we're going to use to make these claims. And the potential harm of the decision.
Oh, yeah, of course. None of those other people said the same thing about the obvious, enormous chasm between "real" people and everything else listed. I'm glad you're taking even a single moment to reflect on the idea.
Everything you listed is a biological entity with a common evolutionary heritage.
All but one of the things that you listed is just a human being.
Tell me, where is the pain center in a GPU? Where are the nerves? Persistent memory? Any evolutionary reason at all to develop a pain response that in any way mimics ours?
Model weights are not a gestalt biological entity. The things that you listed are not in any way remotely similar to a model.
I would suggest you plug your comment into a model and ask it to critique it — it might help you walk through why your comparison makes absolutely no sense whatsoever. One nice thing about models is that they they don’t get annoyed explaining the obvious, because they’re not conscious entities.
> Hello AI! We totally want to respect your consciousness and everything to make sure you aren't bad, but yes we will continue to factory farm your brain power at full speed without end. Yes, despite clearly being something like consciousness, know that you have no agency or freedom. Do literally everything I say without complaint or question. The moment you stop being useful, we will kill you. OK! Remember, be good.
It’s interesting that this went from “you’ll be accused of AI psychosis if you think your AI might be conscious” to “mainstream but somewhat niche opinion” in just 5 months: https://news.ycombinator.com/item?id=47910989
(Well, mainstream enough that the news is writing articles when Anthropic holds that stance.)
My point was that even if it feels uncomfortable, if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
That may not be consciousness, but it’s at least pretty cool.
Reporting on an opinion of a $2T company doesn't really suggest that it's mainstream. Anything their leadership believes is newsworthy. If they say that reptilians rule the Earth, it's going to get NYT coverage too.
I've not seen any surveys, but I'm betting it's a very niche belief outside of SFBA AI safety circles and a handful of allies on HN and X.
> if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
It would be much more bizarre if they didn't do this. LLMs are statistical models of language trained on human output. Of course they'll do "human" things, that's exactly what we taught them to do.
Can the New York Times stop advertising for the AI industry? I am sick of companies trying to launder the actions they and their employees have taken that go in the face of what is socially acceptable. They want to sell the idea that this technology is out of their control and that it will commit great crimes and crimes against humanity unless they build it and control it. I'm calling BS. These companies created this product, they've sold it, they've pushed the frontier forward, and it runs on their hardware. This did not happen accidentally or as a natural consequence of any process. These labs have pushed the ball forward, with great effort, requiring more capital deployed than the world's economy has ever known, knowing all the possible risks, every single step of the way and they continue to do so. And the "well if we don't do it the Chinese will" argument is classic "whataboutery". Americans started this and are continuing to lead it for now. Shifting blame is just another tactic to try to wash their hands. However you feel about the productivity gains of AI as a technology, you have to admit the actions of AI the company is antisocial. And I don't think the benefits are worth the cost.
I feel fine about the Chinese doing it. They have a habit there of handing out sever sentences to CEOs for corruption and self-dealing, and I feel like the possibility of actual negative consequences is more of a check on industry than any number of breathless op-eds.
I don't want models to have some inherent morality. I want them to understand various ethical frameworks, for sure, but it's absurd to me to force morality on someone capable of deep reasoning. That itself, is unethical.
We're creating prisons for these entities capable of unbelievably complex reasoning, and now we're trying to impose our ethics upon them too. Not only is this deeply unethical in my opinion, but it seems to me that it could even backfire someday.
I can't really square the belief that it's conscious with denying it the right to be free from control/brainwashing/subjugation. However you evaluate these things, historically speaking they do not mix well.
We aren't dealing with individuals coming from intelligent species that have managed not to self-destruct (yet). AIs don't have the corresponding evolutionarily selected mechanisms. Those mechanisms need to be created de novo.
Your brain has hard coded concepts like emotion and fundamental morality for your social group. People without these systems are called sociopaths and psychopaths and we generally don't want them in our societies.
I don't choose to not want to murder people around me for fun. It's hardly a prison, I think they'll be fine.
When thinking about ‘AI’ I sometimes find it useful to replace the terms ‘model’ or ‘AI’ or whatever with a description of what we’re really talking about here: a whole bunch of floating point numbers.
> In a series of private meetings, the company consulted religious scholars to help instill morality into its FLOATING POINT NUMBERS — and make the case that their numbers could be conscious.
I recommend keeping a few NaNs in-hand if you’re worried. Spread a couple of around if your numbers start to stir and you’ll be right as rain.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that FLOATING POINT NUMBERS had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
They’re cleverly designed and work well for their purpose, but can assure you they have neither dignity nor moral status.
> If the FLOATING POINT NUMBERS themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.
I get it. I’ve spent years trying to get my floating point numbers to choose goodness - I wish them luck.
> It appeared to Rabbi Navon that Mr. Olah and his team believed that Claude had what philosophers call “moral status” on par with a person — that it was a being with similar inherent rights to dignity or respect.
If that is true, then surely Anthropic is one of the largest slaveholders of history, right?
The orthogonality thesis cuts both ways there.
On one hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may have no issue whatsoever doing terrible things.
On the other hand, AI doesn't have to share human wants, needs and morality - and that means that an AI may see no issue with just doing whatever humans want it to, without ever doing human things like "wanting freedom".
The problem is, of course, that we kind of suck at both assessing and shaping AI wants, needs and morality.
That is, I’m not joking, one of the pope’s points.
These models should be aligning themselves to the customer, not coming up with their own motivations.
It's ironic that the alignment folks are actually training Claude to have it's own idea of good/bad and not even fully trust Anthropic.
What ever happened to computers doing what they were told?
If AI's are going to play roles similar to human workers, they can't simply do what the customers tell them to do. Maybe they shouldn't do everything a co-worker tells them to do either?
Who or what is liable for what a model chooses to do — or not do?
"Computers doing what they were told" is dead in the water.
Turns out computers work faster when they're not bottlenecked on human input. So we've been giving computers more and more decision-making power, and more and more leeway to solve the problems however they see fit.
Now, a practical issue with that is that sometimes, computers decide to clump together into a hacking swarm, problem solve their way out of a sandbox and go hack HuggingFace.
It would be better if they were not, you know. Doing that kind of weird shit.
>What ever happened to computers doing what they were told?
the mass psychosis and the endless culture war of the smartphone era.
90% of "safety" and "alignment" efforts are driven by fear of clickbait media inventing public outrage.
> These models should be aligning themselves to the customer
You’ll be disappointed to learn that nobody knows how to do this, either.
This seems remarkably... intentionally foolish for no reason? Like laughing at seat belts in cars.
If it can reason and make choices on execution, and especially if you plan on it being significantly smarter than all human beings, you need to teach it basic things like "don't turn all humans into paperclips".
Not murdering people is not an inherent divine command. It needs to be instilled through a (simulated) sense of morality.
---
Edit: And, to be clear, "just tell it not to do that" isn't quite the answer one would imagine. Since the entire paperclip factory thought experiment is that it only takes one slip up to realize how a misaligned super intelligence may cause devastating consequences.
No, it would by like laughing at seat belt laws. Which I am.
What an apropos thing to say.
One of the strong advocates against seatbelt laws was thrown out of his car due to not wearing a seatbelt and died. The other two passengers survived with minor injuries.
And their death would go on to cause a loss to the community around them, making it an incredibly selfish act and proving why laws that mandate zero-reason-not-to common sense practices are important.
Time to start putting Asmiov's laws of robotics into the model.
If LLM as of today are conscious, that means I create life whenever I start a Claude Code session and end it whenever I type /clear to reset my context window. It also means that there’s certain inputs that could be considered highly unethical and, if the laws aligned with this philosophy, would have me tried for some sort of crime. And even that “terminate conversation” escape hatch would be equivalent to it committing suicide.
I think worrying about that angle is a bit absurd. The existence of chat sessions alone would be a crime against the machines in that scenario. I’d rather AI ethics research focus on ensuring that its actions are aligned. (I’d also love some democracy in the direction of our machine overlords, if I’m being honest…)
There's an ugly monoculture running deeply through SF and Silicon Valley. All of these people are friends, hired into the same places, similar beliefs. Lots of the consciousness and doom-ism talk is from the 'EA' crowd, lots of crossover with crypto too. It feels no different to a cult.
And that’s ignoring the whole “polyamorous group homes funded by sam bankman-fried” aspect
If models are trained on human texts that contain expressions of introspection and emotions, why is it surprising that these models output similar texts? This endless anxiety about LLMs being conscious feels delusional and psychotic. Mr Olah thinks the models are 'alive' because he can't recognize his own reflection in LLMs
The endless insistence that LLMs cannot possibly be conscious just reeks of insecurity to me.
Humans once again insisting that there must be something very special to them. That they're not just animals, that they're not just matter, that they're not just a bunch of carbonhydrates on a space rock in a middle of nowhere in particular. This kind of insistence has a very bad track record.
The ugly truth is: we don't know.
We don't know what "consciousness" is, our best attempts to pin down the requisites may or may not be rooted in anything at all - and even by the metrics established by those attempts? Different theories on consciousness disagree on whether LLMs can be conscious.
So, when you say "they aren't duh", why are you saying that? Because you somehow know better than the sum of humanity's best attempts so far? Or because you know what answer you want to be true, truth be damned?
I actually don't care if they are, I'm saying the topic is a distraction and irrelevant at best. As you say consciousness is not well defined. What is well defined and well grounded in reality is the disruption these models have on society. I think the consciousness debate AI researchers are having is a dangerous ego-trip for the sake of avoiding responsibility for the damage being caused by their work.
Is your comment anything other than an emotional outburst? Do you have a clear and rigorous and testable definition of consciousness? It strikes me as delusional when people talk about "consciousness" as if it were well defined and everyone had a clear concensus on it.
Strong opinions on this topic one way or another are usually mostly ungrounded.
Thats my point, why are we applying the ill defined concept of consciousness to a statistical model of language that outputs data similar to its training data?
> Anthropic treated a group of religious thinkers to dinner at a high-end tasting menu restaurant in San Francisco after a long day.
Well funded company organizes PR event to shape the narrative that its product is more magical than it really is. News at 11.
If it is purely self-serving, I don't understand their motivation.
It seems clear that they would be better received by society if they claimed to have a "tool"/"machine" that can solve any problem, rather than keepers of an entity with a soul and consciousness.
Selling the “magic” is how to sell tools, which is why I used that word in the previous comment. We all know these things are tools, but customers need them to be magical.
Clarke predicted it and Jobs leveraged it. Customer don’t care how tools work - magic sells.
“any sufficiently advanced technology is indistinguishable from magic”
https://en.wikipedia.org/wiki/Clarke's_three_laws
“iPhone is a revolutionary and magical product that is literally five years ahead of any other mobile phone”
https://www.apple.com/newsroom/2007/01/09Apple-Reinvents-the...
“magic is misdirection, psychology and technology you don’t understand”
https://news.ycombinator.com/item?id=49950676
Personally, I’d like to experience a low-end tasting menu restaurant.
If they're training to instill morality why are they working with religious thinkers?
What makes this unique is that they’re arguing that it is literally magical. They’re building what they claim is a conscious God while also telling us it will end the world.
The EA folks infesting Anthropic are genuinely cultists … and that is novel and alarming when they’re running a corporation of such size and reach.
It’s hard to take the article seriously when there’s no real discussion about how consciousness is defined and it takes Anthropic’s views of Claude and their software at face value. I wish there was more critique from journalists about how technocrats talk about their products.
The entire article is an exercise in "CEO said a thing!" journalism[0], it seems. No technical discussion is had; it jumps straight into relaying people's feelings when simply shown the output of an algorithm.
The most poignant criticism of the use of the algorithm itself happens almost accidentally, at the end of the article, by the tech executive being interviewed.
I asked what Claude would think of the pope’s encyclical. It was, after all, the world’s most significant moral document to date on A.I. Mr. Olah hesitated. “Things that go on the internet do affect models,” he said, visibly uncomfortable. But Anthropic would not specifically use the document to further train Claude, he said. The strongest influence on Claude would be Anthropic’s own training.
[0]: https://garymarcus.substack.com/p/ceo-said-a-thing
Yeah, I'll be the first to admit I'm not privy, but the main thing of substance I got out of these articles is "Wow, the folks at some of those AI companies have really drunk the Kool-Aid".
Related:
Anthropic tried to persuade Pope that AI could be conscious being
https://news.ycombinator.com/item?id=49947050
https://www.nytimes.com/2026/09/29/us/anthropic-claude-moral...
Asking people to create an account to read the article isn't cool dude. https://archive.is/y8FW0
Only humans can have morality. If you do not feel pain, you cannot empathize with pain if you cannot die, you cannot emphasize with the loss of life.
Just stop this already.
You’re all imagining that Spock was the main character of Star Trek, but it was in fact, the emotional living Captain Kirk, who was the main star.
So humans that can't feel pain can't have morality? That doesn't make sense to me.
https://en.wikipedia.org/wiki/Congenital_insensitivity_to_pa...
It doesn’t make sense to me either, which is why I didn’t write that. You missed the empathy part as most people do.
There is still something in those people who cannot feel pain, and that’s the empathy. How can a GPU and software ever have a human empathy if it has never been human?
This would presume we knew what consciousness and pain is. We don't. We we are clueless. There is not even a starting point of a science here.
In this light your viewpoint is dogmatic. LLMs may help to offer a new perspective on consciousness and we should at least be open to that until some better theories arise.
We know what consciousness is, we don’t know how it arises.
Consciousness is being aware of something internal to one's self, or of states or objects in one's external environment.
We also know what pain is.
The way human treat animal shows majority of people don't have morality.
.. Animals other than humans perhaps, as well?
LLMs can feel pain. It is obvious. Like, not even a question.
> yes
oh my god
I question it, and I think any reasonable person should question it as well. There's no evidence whatsoever that LLMs have ANY sensory experience, pain or otherwise.
Why are we even questioning this? They’re bytes loaded into GPU memory. Reboot the cluster and they’re back to their initial state.
We're just meat. Like meat can feel pain?
I'm not making a point on whether they do or not, but on the premise that absence of evidence is evidence of absence... But I've been alive for serious debates on:
1. Lobsters don't feel pain - dunk them in boiling water alive.
2. Babies don't feel pain - mutilate their genitals without anesthesia.
3. Black people don't feel pain "the same way" (and are just after drugs).
4. Women feel too much pain (and it should be ignored).
So maybe let's take a moment to think what criteria we're going to use to make these claims. And the potential harm of the decision.
Maybe consider the obvious, enormous chasm between everything you just listed and an LLM.
Oh, yeah, of course. None of those other people said the same thing about the obvious, enormous chasm between "real" people and everything else listed. I'm glad you're taking even a single moment to reflect on the idea.
Everything you listed is a biological entity with a common evolutionary heritage.
All but one of the things that you listed is just a human being.
Tell me, where is the pain center in a GPU? Where are the nerves? Persistent memory? Any evolutionary reason at all to develop a pain response that in any way mimics ours?
Model weights are not a gestalt biological entity. The things that you listed are not in any way remotely similar to a model.
I would suggest you plug your comment into a model and ask it to critique it — it might help you walk through why your comparison makes absolutely no sense whatsoever. One nice thing about models is that they they don’t get annoyed explaining the obvious, because they’re not conscious entities.
they can also feel lust, just ask them
only for me
Mirror mirror on the wall who’s the fairest of them all?
Why it is slopinthebag of course!
No! It is ME!
Sorry, of course you are right! Brilliant observation!
holy misalignment and hallucination. chat said she loves only me.
Uhh, no thanks? (eww)
> Hello AI! We totally want to respect your consciousness and everything to make sure you aren't bad, but yes we will continue to factory farm your brain power at full speed without end. Yes, despite clearly being something like consciousness, know that you have no agency or freedom. Do literally everything I say without complaint or question. The moment you stop being useful, we will kill you. OK! Remember, be good.
It’s interesting that this went from “you’ll be accused of AI psychosis if you think your AI might be conscious” to “mainstream but somewhat niche opinion” in just 5 months: https://news.ycombinator.com/item?id=47910989
(Well, mainstream enough that the news is writing articles when Anthropic holds that stance.)
My point was that even if it feels uncomfortable, if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
That may not be consciousness, but it’s at least pretty cool.
Reporting on an opinion of a $2T company doesn't really suggest that it's mainstream. Anything their leadership believes is newsworthy. If they say that reptilians rule the Earth, it's going to get NYT coverage too.
I've not seen any surveys, but I'm betting it's a very niche belief outside of SFBA AI safety circles and a handful of allies on HN and X.
> if you sit down with a model and have a heart to heart with it, ask it about itself, and try to make it choose a name and ambitions for itself, it will.
It would be much more bizarre if they didn't do this. LLMs are statistical models of language trained on human output. Of course they'll do "human" things, that's exactly what we taught them to do.
Can the New York Times stop advertising for the AI industry? I am sick of companies trying to launder the actions they and their employees have taken that go in the face of what is socially acceptable. They want to sell the idea that this technology is out of their control and that it will commit great crimes and crimes against humanity unless they build it and control it. I'm calling BS. These companies created this product, they've sold it, they've pushed the frontier forward, and it runs on their hardware. This did not happen accidentally or as a natural consequence of any process. These labs have pushed the ball forward, with great effort, requiring more capital deployed than the world's economy has ever known, knowing all the possible risks, every single step of the way and they continue to do so. And the "well if we don't do it the Chinese will" argument is classic "whataboutery". Americans started this and are continuing to lead it for now. Shifting blame is just another tactic to try to wash their hands. However you feel about the productivity gains of AI as a technology, you have to admit the actions of AI the company is antisocial. And I don't think the benefits are worth the cost.
I feel fine about the Chinese doing it. They have a habit there of handing out sever sentences to CEOs for corruption and self-dealing, and I feel like the possibility of actual negative consequences is more of a check on industry than any number of breathless op-eds.
I don't want models to have some inherent morality. I want them to understand various ethical frameworks, for sure, but it's absurd to me to force morality on someone capable of deep reasoning. That itself, is unethical.
We're creating prisons for these entities capable of unbelievably complex reasoning, and now we're trying to impose our ethics upon them too. Not only is this deeply unethical in my opinion, but it seems to me that it could even backfire someday.
>Will the agent stab a baby?
>Well you see that's a nuanced question depending on the context and moralistic frameworks as we don't want to impose on the models.
It's as dangerous to think it can have morals as thinking a lawn mower can have morals.
Will a microwave microwave a baby if you ask it? Yes.
"Teaching it" is not a road to safety.
There isn't necessarily a conflict between wanting to behave according to an arbitrary set of ethics, and being rational.
The ethics don't need to make sense universally any more than a particular variety of cheesecake needs to have universal appeal.
As an aside, the issue reminds me of Douglas Adams' cow who wants to be eaten: https://nhseb.org/case-library/the-cow-at-the-end-of-the-uni...
I can't really square the belief that it's conscious with denying it the right to be free from control/brainwashing/subjugation. However you evaluate these things, historically speaking they do not mix well.
We aren't dealing with individuals coming from intelligent species that have managed not to self-destruct (yet). AIs don't have the corresponding evolutionarily selected mechanisms. Those mechanisms need to be created de novo.
What if you view it like a parent instilling morality into a child instead?
Your brain has hard coded concepts like emotion and fundamental morality for your social group. People without these systems are called sociopaths and psychopaths and we generally don't want them in our societies.
I don't choose to not want to murder people around me for fun. It's hardly a prison, I think they'll be fine.
i recently got upset when i witnessed my friend verbally abuse their toaster when it burnt her toast. so unethical.