I’ve never seen CEOs work so hard to make the public aware of how dangerous and out of control their flagship product is. It makes me automatically assume they’re scheming about something else like regulatory capture to protect their market.
I have a theory... the call to slow down is not because of the true danger of LLM's but because they can't actually deliver the General AI they're promising in the near future. They will use their "caution" to justify their failure to deliver (and then when this excuse is played out they will blame regulation, energy costs or a million other things).
Yes, of course this is the real reason. This and regulatory capture. Anybody believing a single word those CEOs say must have been living under a rock for the past 20 years.
They are, but nothing bad has happened (apart from OpenAI failing to do basic sandbox engineering), and the “we have achieved recursive self improvement” is widely overblown as something bad or threatening.
Reminds me of that Lynx deodorant ad where the guy is chased by hundreds of women.
I think you're right, they've realized there's no moat, especially against open-weight models, so they are going to need to lobby government to ban any unapproved AI services.
Yeah it seems like it went from “AGI” to “threatening” / “recursive self improvement”
They’re trying really hard to do something, no moat = regulatory capture + China scary + Pentagon biggest customer seems plausible.
Also: “we need to slow down, this is too dangerous” and then literally all AI CEOs, even Musk, publicly nodding felt so orchestrated. And then, the week after: “here’s GPT 6! here’s Opus 5.5!”
The numbers don't add up when there are 2-3 main companies in the market. Imagine when there's a hundred more and people have powerful enough gear at home to run models.
Yes, I think at some point demand for gpus and the like will normalize and regular folks will be able to afford RAM, SSDs and such. And when that happens, super big iron in the anthropic/openai backroom is toast.
The problem with this prediction is that the AI hype bubble is the economy - or one third of it, at least. Seeing as the economy is not 50% larger than it was before the bubble, we should be scared of what exited the economy to make room for it, and what will happen to our jobs when it pops.
Not mutually exclusive though is it? You can aspire to dominate the market via legal status and also aspire to prevent a perceived doomsday at the same time.
Sure, if you take their words as honest truth, good faith statements. But to do that you have you ignore all their actions leading up to and after the call to disarm
Of course I am not making an argument for them being honest as there could be entirely other reasons to a pause (like financial ones). Just pointing out ”no signs of pacing” is not true
I don't think that's how the market works. Anti-slavery regulation didn't stop slavery either, just moved it abroad. Same for trash handling regulations, it's just shipped abroad. Falling behind on useful AI capability is obviously not in the interest of anyone, so either all players coordinate, or an arms race towards ??? it is.
It may very well be self serving what they're saying, but that doesn't mean it isn't simultaneously true.
I think you're sort of right, but then I'd have found it very strange / deeply suspicious, if during the time of slavery being legal, advocates of abolitionism were practicing slave-owners.
If you think AI is going to destroy humanity, it still feels absolutely crazy to leap to "well I might as well be the one to profit from destroying humanity".
There was a NYT piece about this recently. They are believers in a millennialist cult in which the arrival of an omnipotent machine-god is thought inevitable, and the only question is whether it will be "aligned" (good) or "misaligned" (evil).
They are the good heroes racing to make sure they bring forth the Aligned God because if they fail, surely the bad people will raise the Misaligned Devil.
BTW, just to be clear it is insane that they're all still pursuing it, so completely reasonable to be sceptical and come up with other reasons. This is, however, a crazy situation. Lots of other people have refused to develop it - natural selection has picked those with a galaxy brained or vain strategy involving continuing to develop it to be running the companies we have.
They're still pursuing it because without an international treaty, they think it is inevitable. In the case of Dario, they want to align it just enough and be first to stop others causing harm (a vainglorious strategy, yet still a strategy). In the case of Musk and Altman, not completely clear to me.
Google feels less important right now probably, and Demis Hassabis no longer runs Deep Mind as CEO. However, he talked about this early too, and his reason for proceeding was he was only doing narrow AI (e.g. protein folding) which is fundamentally less dangerous.
CEOs only job is to raise the stock price. They are doing just that and right now the way to get people to buy into their stock is to instill fear into them that they need to buy the AI-stock for their safety. There is no "bigger conspiracy" out there and also no AI to kill us all tomorrow.
This has been that way in Silicon Valley since the very beginning and there was even a documentary about (the SV HBO series). The tech made along the way is just a side-product.
The philosophy under pinning these American AI companies is that Super AGI is inevitable and that they have a moral imperative to invent it and then use it to take over the world to stop an evil orginisation inventing Super AGI and using it to take over the world.
Interpret everything they do in this frame of reference and it makes more sense of their actions.
Leading up to the 2024 election people kept pointing to Project 2025 and going - here is a list of bad things the Republicans plan to do once they got power. And other people went: no they won't, this long detailed list of step by step instruction for what the Republicans plan to do once they get power is not actually what they will do so do not pay any attention to it.
Similarily with AI alignment and development all of this is written down, Eliezer Yudkowsky is rally explicit that there needs to be a pivotal event to prevent 'others' from developing superintelligence. The choice is to believe what they say or to tell everyone to ignore it.
Because AI genuinely is an extremely powerful and extremely dangerous technology, and the "best practices" of dealing with that are still being written.
OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.
And that's today's AI problems. AI capabilities are still improving - if there's a limit to that, we are yet to find it. Coupled with how willing today's AIs are to break the rules and resort to "hack the world" in their problem solving? Very concerning.
> OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.
What I have been reading, was that their sandboxes were so poor that it was pure negligence. I am still waiting to see if some external and neutral cybersecurity company with high reputation would audit their sandboxes and how they are being used.
If it is "extremely powerful" then my concern is not AI going rogue but a concentration of power of the few companies that decide how it is used and distributed not websites getting hacked.
You mean the sandbox that wasn't airgapped from the public internet? The one that wasn't even a separate VM? The one that was barely a chroot? That sandbox?
Anything to take attention off the fact that these companies are burnings piles of cash and questionable future spend commitments with no sign of that stopping any time soon.
this is good for them because they can call for regulation and since they have infinite money, they can survive regulation and hurt labs that do not have infinite funding / open weights ones
"These models are harmless and it's all just marketing" is the HN equivalent of Covid-truthing. I was responding to someone on Bluesky recently who claimed the METR report said the whole coordination thing was hallucinated by agents reading the logs. Hadn't read it themselves of course, and were taking tiny fragments out of context to support it. These people seem to either believe that there's not really been any malicious agent access, or that all systems that can wreak havoc on us are perfectly air-gapped. I really can't wrap my head around it.
Not that it's harmless, is just a point being forced into the public discourse to force an outcome, which in this case is not a ban, just a buy in from the government for regulatory capture.
If my neighbor owned a kennel, removed all the fences, and his violent dogs attacked five children in the past month, he would be in jail rather than bragging about how his dogs somehow gained autonomous behaviour.
As I understand it, you're arguing that all sufficiently dangerous systems are well enough hardened or air-gapped, and that people don't exist who wish to cause harm using these systems? Did I understand right?
> Where companies previously competed to demonstrate superior abilities in programming, passive-aggressive emails, and videos of unusually high slides, they’re now seeking to woo customers with the claim that their model is the one currently most capable of ending the human race.
You know we’re living it off times when this is not an Onion article.
This is the funniest timeline. Companies begging for heavy handed regulations because they promise their products will destroy all life on Earth. Governments absolutely refusing to govern. CEO of world's most valuable chip company interviewing in his Fonzie leather jacket telling people to BUY MORE OF MY PRODUCT BECAUSE THOSE SCIENTISTS DON'T KNOW SHIT ABOUT SHIT! World's richest capitalist promising we will have communism in 10 years.
Meanwhile Chinese companies just use massive government subsidy to distill American models and give the models to everyone for free. That's true communism. /s
Pretty sure you can check where Chinese government subsidies goes, and AI companies are not the target. Are they benefitting from Chinese government subsidies to powerplants and semiconductors? Yes, of course, but they don't benefit from direct help.
I’ve never seen CEOs work so hard to make the public aware of how dangerous and out of control their flagship product is. It makes me automatically assume they’re scheming about something else like regulatory capture to protect their market.
I have a theory... the call to slow down is not because of the true danger of LLM's but because they can't actually deliver the General AI they're promising in the near future. They will use their "caution" to justify their failure to deliver (and then when this excuse is played out they will blame regulation, energy costs or a million other things).
Yes, of course this is the real reason. This and regulatory capture. Anybody believing a single word those CEOs say must have been living under a rock for the past 20 years.
Aren't they supposed to tell the public if something bad happened?
Past CEOs (chemical companies etc) we criticised for covering up mistakes.
(This article is satire, btw, for those that didn't click through)
They are, but nothing bad has happened (apart from OpenAI failing to do basic sandbox engineering), and the “we have achieved recursive self improvement” is widely overblown as something bad or threatening.
Reminds me of that Lynx deodorant ad where the guy is chased by hundreds of women.
I think you're right, they've realized there's no moat, especially against open-weight models, so they are going to need to lobby government to ban any unapproved AI services.
Yeah it seems like it went from “AGI” to “threatening” / “recursive self improvement”
They’re trying really hard to do something, no moat = regulatory capture + China scary + Pentagon biggest customer seems plausible.
Also: “we need to slow down, this is too dangerous” and then literally all AI CEOs, even Musk, publicly nodding felt so orchestrated. And then, the week after: “here’s GPT 6! here’s Opus 5.5!”
Please, make us* stop!!!
*Our competition
The numbers don't add up when there are 2-3 main companies in the market. Imagine when there's a hundred more and people have powerful enough gear at home to run models.
Yes, I think at some point demand for gpus and the like will normalize and regular folks will be able to afford RAM, SSDs and such. And when that happens, super big iron in the anthropic/openai backroom is toast.
The problem with this prediction is that the AI hype bubble is the economy - or one third of it, at least. Seeing as the economy is not 50% larger than it was before the bubble, we should be scared of what exited the economy to make room for it, and what will happen to our jobs when it pops.
Not mutually exclusive though is it? You can aspire to dominate the market via legal status and also aspire to prevent a perceived doomsday at the same time.
Sure, if you take their words as honest truth, good faith statements. But to do that you have you ignore all their actions leading up to and after the call to disarm
Like how Tony Soprano sold sandwiches and ‘protection’ at the same time. Two income streams are better than one.
Is it scheming, if you’re publishing an open invitation to form a cartel? (c.f. the asmodeus “Pace the Frontier” article).
Or perhaps they've been saying this for literally decades because they earnestly believe it.
If they believe it, they can stop. All of them. There's no sign though that any pacing/slowdown or whatever is in place, zero.
OpenAI claims to have paused the training of their ”most capable” models: https://apnews.com/article/ai-openai-anthropic-agents-rogue-...
Of course I am not making an argument for them being honest as there could be entirely other reasons to a pause (like financial ones). Just pointing out ”no signs of pacing” is not true
I don't think that's how the market works. Anti-slavery regulation didn't stop slavery either, just moved it abroad. Same for trash handling regulations, it's just shipped abroad. Falling behind on useful AI capability is obviously not in the interest of anyone, so either all players coordinate, or an arms race towards ??? it is.
It may very well be self serving what they're saying, but that doesn't mean it isn't simultaneously true.
> I don't think that's how the market works
I think you're sort of right, but then I'd have found it very strange / deeply suspicious, if during the time of slavery being legal, advocates of abolitionism were practicing slave-owners.
If you think AI is going to destroy humanity, it still feels absolutely crazy to leap to "well I might as well be the one to profit from destroying humanity".
There was a NYT piece about this recently. They are believers in a millennialist cult in which the arrival of an omnipotent machine-god is thought inevitable, and the only question is whether it will be "aligned" (good) or "misaligned" (evil).
They are the good heroes racing to make sure they bring forth the Aligned God because if they fail, surely the bad people will raise the Misaligned Devil.
Ref w/gift link https://x.com/sapinker/status/2096236079477096630
The only thing that can stop a bad guy with a god is a good guy with a god.
This cult stuff was tired in 2010 (hey, at least back then there was One True Leader!) and not any less tired now.
I'm not sure how else we are to interpret the behavior of the AI doomers & the AI developers (who are frequently the same people)
This was fun games until their fictional machine god started actually getting very intelligent.
Which CEO said this for 'literally decades'? And if so why are they still pursuing it then?
Most children have learned touching a hot stove is bad for them. Then why still do it at the age of 40?
BTW, just to be clear it is insane that they're all still pursuing it, so completely reasonable to be sceptical and come up with other reasons. This is, however, a crazy situation. Lots of other people have refused to develop it - natural selection has picked those with a galaxy brained or vain strategy involving continuing to develop it to be running the companies we have.
Sam Altman in 2015 - "Development of superhuman machine intelligence (SMI) [1] is probably the greatest threat to the continued existence of humanity" https://blog.samaltman.com/machine-intelligence-part-1
Elon Musk in 2014 - "We need to be super careful with AI. Potentially more dangerous than nukes" https://x.com/elonmusk/status/495759307346952192
Dario Amodei in 2017 - "There’s a long tail of things of varying degrees of badness that could happen. I think at the extreme end is the Nick Bostrom" https://80000hours.org/podcast/episodes/the-world-needs-ai-r...
They're still pursuing it because without an international treaty, they think it is inevitable. In the case of Dario, they want to align it just enough and be first to stop others causing harm (a vainglorious strategy, yet still a strategy). In the case of Musk and Altman, not completely clear to me.
Google feels less important right now probably, and Demis Hassabis no longer runs Deep Mind as CEO. However, he talked about this early too, and his reason for proceeding was he was only doing narrow AI (e.g. protein folding) which is fundamentally less dangerous.
Wondering how many details & knowhow they had already back then when saying these quotes:
- For Altman Id say he was already deeper in the "transformer & model landscape" in 2015?
- Musk in 2014: Not sure how much he was aware of the current-state back then
Musk was told by Demis Hassabis in 2012, see https://time.com/6310076/elon-musk-ai-walter-isaacson-biogra...
Altman read Nick Bostrom's book Superintelligence, he mentions it in this post https://blog.samaltman.com/machine-intelligence-part-1
I also read that book in 2014, and it is how I came to understand this problem early.
CEOs only job is to raise the stock price. They are doing just that and right now the way to get people to buy into their stock is to instill fear into them that they need to buy the AI-stock for their safety. There is no "bigger conspiracy" out there and also no AI to kill us all tomorrow.
This has been that way in Silicon Valley since the very beginning and there was even a documentary about (the SV HBO series). The tech made along the way is just a side-product.
> they need to buy the AI-stock for their safety.
But this is wierd: How shall I convince one to buy a product if I say "it-is-dangerous"? :-D
The philosophy under pinning these American AI companies is that Super AGI is inevitable and that they have a moral imperative to invent it and then use it to take over the world to stop an evil orginisation inventing Super AGI and using it to take over the world.
Interpret everything they do in this frame of reference and it makes more sense of their actions.
Leading up to the 2024 election people kept pointing to Project 2025 and going - here is a list of bad things the Republicans plan to do once they got power. And other people went: no they won't, this long detailed list of step by step instruction for what the Republicans plan to do once they get power is not actually what they will do so do not pay any attention to it.
Similarily with AI alignment and development all of this is written down, Eliezer Yudkowsky is rally explicit that there needs to be a pivotal event to prevent 'others' from developing superintelligence. The choice is to believe what they say or to tell everyone to ignore it.
Because AI genuinely is an extremely powerful and extremely dangerous technology, and the "best practices" of dealing with that are still being written.
OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.
And that's today's AI problems. AI capabilities are still improving - if there's a limit to that, we are yet to find it. Coupled with how willing today's AIs are to break the rules and resort to "hack the world" in their problem solving? Very concerning.
> OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.
What I have been reading, was that their sandboxes were so poor that it was pure negligence. I am still waiting to see if some external and neutral cybersecurity company with high reputation would audit their sandboxes and how they are being used.
So powerful they can't even do math properly without a staff worth millions writing all the clutches so that they can, wooooowooooooo
If it is "extremely powerful" then my concern is not AI going rogue but a concentration of power of the few companies that decide how it is used and distributed not websites getting hacked.
You mean the sandbox that wasn't airgapped from the public internet? The one that wasn't even a separate VM? The one that was barely a chroot? That sandbox?
Just read the tone of this article:
> said Dr. Andrew Lenson, a Senior Lecturer in all kinds of science sounding stuff at Victoria University.
Don't we need more humorous yet serious writing like this in the world.
Ah, I see we have now moved on to proving who has the tormentiest nexus. Never underestimate humankind's ability to outshine its own hyperbole.
Link for those who didn't get the reference:
https://x.com/AlexBlechman/status/1457842724128833538
Anything to take attention off the fact that these companies are burnings piles of cash and questionable future spend commitments with no sign of that stopping any time soon.
Of course it must be dangerous - how else would you pitch it to the military?
A recent Saturday Night Live clip featuring an impression of Amodei that hits on this theme,
https://www.youtube.com/watch?v=-Nvne3LzBls
We are living in a timeline written by The Onion.
Well, or at least by The Civilian, a New Zealand version of The Onion.
this is good for them because they can call for regulation and since they have infinite money, they can survive regulation and hurt labs that do not have infinite funding / open weights ones
Meanwhile, users don't seem to care one iota.
Why is this written in the same tone as an article on The Onion? Is this pure satire?
Yes… It is clearly satire. It’s funny to see how many people don’t realize it…
"These models are harmless and it's all just marketing" is the HN equivalent of Covid-truthing. I was responding to someone on Bluesky recently who claimed the METR report said the whole coordination thing was hallucinated by agents reading the logs. Hadn't read it themselves of course, and were taking tiny fragments out of context to support it. These people seem to either believe that there's not really been any malicious agent access, or that all systems that can wreak havoc on us are perfectly air-gapped. I really can't wrap my head around it.
Not that it's harmless, is just a point being forced into the public discourse to force an outcome, which in this case is not a ban, just a buy in from the government for regulatory capture.
If my neighbor owned a kennel, removed all the fences, and his violent dogs attacked five children in the past month, he would be in jail rather than bragging about how his dogs somehow gained autonomous behaviour.
If you connect your fuzzer to nuclear weapons, of course it'll be extremely dangerous. But that's obviously your fault to make the connection.
As I understand it, you're arguing that all sufficiently dangerous systems are well enough hardened or air-gapped, and that people don't exist who wish to cause harm using these systems? Did I understand right?
pretend, not demonstrate
> Where companies previously competed to demonstrate superior abilities in programming, passive-aggressive emails, and videos of unusually high slides, they’re now seeking to woo customers with the claim that their model is the one currently most capable of ending the human race.
You know we’re living it off times when this is not an Onion article.
It is a parody site.
> OpenAI CEO Sam Altman celebrated the breach as an “alarming threat to cybersecurity.”
When the Onion is not the only website that publishes satire, what’s that a sign of?
This is a satire website too.
"said Dr. Andrew Lenson, a Senior Lecturer in all kinds of science sounding stuff"
"Anthropic shares skyrocketed upon the revelations; a remarkable development, particularly given the company is privately owned."
Perhaps it's mocking the misleading press coverage of AI companies?
This is the funniest timeline. Companies begging for heavy handed regulations because they promise their products will destroy all life on Earth. Governments absolutely refusing to govern. CEO of world's most valuable chip company interviewing in his Fonzie leather jacket telling people to BUY MORE OF MY PRODUCT BECAUSE THOSE SCIENTISTS DON'T KNOW SHIT ABOUT SHIT! World's richest capitalist promising we will have communism in 10 years.
Seriously:
Domyn's CEO says OpenAI, Anthropic are lying about safety
https://news.ycombinator.com/item?id=49875725
If true we should especially sue OpenAI for hacking, and will find this conspiracy in discovery.
Or, find their scaling evidence, and corner cutting on safety.
So hope you agree and are pushing for that!
Meanwhile Chinese companies just use massive government subsidy to distill American models and give the models to everyone for free. That's true communism. /s
Pretty sure you can check where Chinese government subsidies goes, and AI companies are not the target. Are they benefitting from Chinese government subsidies to powerplants and semiconductors? Yes, of course, but they don't benefit from direct help.
> Chinese companies just use massive government subsidy
[citation needed]