The thing that makes me pretty terrified about AI is that I think the "utopia" scenario is not much better than the "doom" scenario.
We will very quickly get to a point where very few people will be able to contribute economically because they will be worse than AI (including robotics) at most domains. A world where people just have their whims catered to is not a utopia. We have tons of sayings and idioms about this, e.g. "no pain, no gain", "only the hard stuff is worth doing", etc.
The humans all running around on their Wall-E carts doesn't feel like utopia to me.
Edit: to head off the "rich people worried about striving for their job while people in poor countries are starving" arguments, I encourage you to visit a dying rust belt town in the US. These folks are by and large not starving (indeed, overabundance of calories is the bigger problem), they have adequate shelter, and plenty of outdoor space. Their material needs are met. But these places are despair central. People see little opportunity for advancement or the ability to change their circumstances, and the opioid epidemic is a direct consequence of a sense of hopelessness that there is no place for motivation to go or to effect change.
>A world where people just have their whims catered to is not a utopia.
I understand where you're coming from on that, but there are a ton of people living in slavery, or in unsafe working conditions, or with food insecurity, or dying of preventable disease.
There will always be something to strive for, even if it's made up. You think that the best football players in the world are doing something real? No, it's a made up game. We'll make up more games.
I agree with your points. My problem is that I don't see any way to "stop" AI development at a level just below human capability where you get most of the benefits but still have humans firmly in control.
Regarding entertainment jobs like sports star, musician, artist, actor, etc., the problem is those are professions where, for the most part, only the very best can make a living. Nobody is that interested in seeing the matches of the 324th best tennis player in the world, despite that person being in the top .01% of tennis players globally. Similarly, only a tiny percentage of actors "make it", everyone else does it for a few years and then leaves because they enjoy eating, or they accept poverty wages for a long time.
I get pretty scared when I hear the AI utopianists rattle off all the benefits that are coming (and when they play down the risks) because it feels clear to me that they really haven't wargamed out what society would look like with very powerful AI.
> A world where people just have their whims catered to is not a utopia. We have tons of sayings and idioms about this, e.g. "no pain, no gain", "only the hard stuff is worth doing", etc.
I somewhat get the point, but I think stuff like this is very often said by very privileged people. I don't think it's a serious threat to humanity as a whole. Perhaps to some humans, but there will always be ambitious people, even in a world where most things are provided by benevolent AIs for free. There is simply so much more harm caused to actual humans today by resource insufficiency than by them having too much, that I don't think we should worry about this scenario today.
There's the mice utopia experiment too. I often hear that in space your bones dégradé, your biology needs weight as input.
Also I believe our socialness was crafted out of the need for survival. If removed it's all gonna run wild since nobody needs anybody, and it might end up as doom scrolling but for human relationship. Short bursts from time to time and that's it
Really glad you brought up the mice utopia experiments. I encourage people to read about them and think how they apply to humans.
In many ways they don't apply to humans, and I think a lot of the modern critiques are valid, and the overpopulation concerns in particular don't map well to the modern world. Still, I believe it helps to think deeply about what happens to societies when innate drives and motivations have "nowhere to go".
Presumably, a truly superintelligent AI in a utopia scenario could figure out a way to enable people to continue working, even if the work doesn’t actually contribute anything, if it is really the case that work is essential to human wellbeing.
Now you might say “but those would be fake jobs.” If so, I have bad news about how many present-day jobs are fake jobs.
I agree with this. I thought the AI 2027 paper put it well:
> There are even bioengineered human-like creatures (to humans what corgis are to wolves) sitting in office-like environments all day viewing readouts of what’s going on and excitedly approving of everything, since that satisfies some of Agent-4’s drives.
I'll leave it up to you to decide if that's an enviable/desirable future.
I don’t even see why they have to be bioengineered. We have all-natural human beings doing this right now, except they are rubber-stamping human rather than artificial directives.
They don't, it's just at that point in the AI 2027 story the AI has already wiped out the humans due to resource contention (similar to how humans largely wiped out the wolves).
its annoying that robots are going to take intellectual jobs before physical ones. it feels way more likely for my project manager to become an agent before the lady that makes me a burrito at chipotle.
I think this is very true, but robots are definitely coming for the burrito makers, too.
Bill Gates talks about this in his recent missive, but as robotic technology improves (and, perhaps more importantly, as the cost comes down) there will be huge competitive pressures for companies to replace workers with robots wherever they can.
I think this line of thinking is a huge disservice to people in third world countries and poverty. People everywhere want to find goals and motivations that move them forward.
Just look at the legions of people that have moved from Chinese rural areas to big cities. For many, many people, life in the cities is worse and kind of dystopian - much more overcrowding, "drone-like" jobs indoors, horrible pollution, and especially for men an often abysmal sex ratio if you want to find a mate. But people do it because it offers the dream of advancement and a better life - and of course some people do achieve that dream, even if the majority do not. That's versus life in rural areas which is guaranteed to be static, no chance for advancement and often crushing boredom.
People everywhere want to strive, work hard and get ahead. This is not just a privileged rich people issue.
Even their utopian views feel dystopian. Take solace in the fact, that it looks like it’ll take trillions to achieve, probably one of the most expensive technological buildouts of all time?
This is a bit of tangent and more of a rant. (and please don't suggest intervention of any kind)
> superintelligent AI will inevitably destroy humanity seems to have no flaw
For a section of population who were in a certain age-range when the pandemic hit and that derailed their certain kinds of plans/hopes/dreams in personal life and then this LLM/agentic world moved in, especially if they are in an industry directly and most affected this, this has been a surreal and slow Kafkaesque nightmare for a few years. The thing is with time and age passing hope keeps getting lost inch by inch and every lost bit of hope aids in killing the next bit. Then comes the sense of dread - the unchangeable reality that age passed passing means there are certain experiences and milestones that are essentially out of reach because there's no going back of this clock.
So I feel a strange kind of relief about this possibility of doom. I am just sharing what I feel, and I don't feel good about feeling this sense of relief that after all it might end for us all and not just an unlucky few. This feeling and the calmness about it saddens me.
As an older person, I agree with you and I'm sorry.
I think about the incredible luck I had to start my career in the .com boom, to have lots of hope about technology and to have the economically valuable skills that were desired.
I get angry when I see people my age or older complain about the apathy of young people. I don't like the apathy, but I find it trivially easy to see where it comes from. In many ways we broke the link between hard work and success and then get baffled and finger point when we think young folks are not as motivated as they should be.
I have similar feelings, you’re not alone (nor alone in being saddened by it). Things seem to me to be more or less playing out the way I interpreted existential risk researchers saying it would, so there’s a strange relief in knowing the general shape of what is to come.
"If a death ray comes from the sky and obliterates the entire planet, at least we all go together." There is something comforting about being together with the human race even in doom. Perhaps especially in doom.
I've been fortunate to not been impacted as severely as others, but yeah, the combo of the first half of my 30s (and everything that should've represented) going up in smoke, career continuity coming into question thanks to AI, and growing sense of overarching geopolitical, financial, and societal unease is really the worst kind of layer cake. I have to make a concerted effort to not get bogged down in it and that's getting harder all the time.
If anybody is hiding a button that'd take us all back to 2012 I wish they'd just hit it already.
Have you considered getting involved in local civics?
I only ask because like you, I felt similar. But I also grew up (I’m 35 now) with a big disconnect from civics life and always felt “someone else would save me”.
Now I see I was the problem all along: I never participated in what was a fragile and very delicate system of governance which requires my participation. Still figuring out what that looks like, but first confronting that my actions “didn’t change anything” (which I’m sure you can admit is a big propagandistic myth to disempower us) let me take my first tiny action locally which is reach out to my civics association for signage repair. I suspect from there I will level up and better understand my government to play a role in shaping it. I’m also building something to solve this problem of low participation which is exciting!
> * For a section of population who were in a certain age-range when the pandemic hit and that derailed their certain kinds of plans/hopes/dreams in personal life*
...
> So I feel a strange kind of relief about this possibility of doom.
So let me make sure I understand what you're saying. The pandemic disrupted your life plans for a few years and threw you off the track you planned? (Which does suck.) And now the possibility of actual human extinction makes you feel a strange relief? Am I understanding this correctly?
If so, this is a remarkable level of something. Depression? Nihilism? Jealousy? Something else?
Imagining total human extinction and feeling relief is not a healthy state of mind. And, yeah, you seem to know this. But I kind of wish we could reach some sort of broad public agreement that human extinction would be bad. And if it ever looks like it might happen, we should fight for human survival.
Something something Maslow's Hierarchy of Needs. While I don't necessarily endorse the view, it's understandable that one might find it difficult to care much about the bigger picture when it feels like their own life is destined for a doom-spiral.
All problems arise from this statement. It's baked in us and it's why we do in fact survive. But it isn't so neatly righteous. When we struggle to agree on what defines a human, who ought to have rights, and basic equality, it's easy to see how much harm "we should fight for survival" does.
I mean clearly what you are talking about is a personal feeling here. You mention it saddens you and that is the concerning part imo but you also said no intervention bs so I will just share my counter factual: I am also kind of okay if the world ends tomorrow, to me it never mattered in the first place. But that feeling doesn't sadden me. I feel it frees me because I don't think of life as a very very serious thing and I feel there is a certain kind of lightness you can afford when you fundamentally don't care if you/the entire world ends tomorrow. I still love people and I still love my life and I don't want any harm to anyone. So I do feel the same calmness that you kind of talk about (for life in general aside from just AI killing us all) but my interpretation of it doesn't sadden me. If it did, I would really investigate why I interpret it that way, it might help you discover yourself. There is no shame in being okay with the world ending imo but there is sin in willing it or contributing to it because one thinks its inevitable. Furthermore, I also believe that we only get 1 life which is temporal and fundamentally doesn't have any meaning. How you live every day is how you live every day. We can suffer, we can cry, we can feel sad and for very good reasons but our relationship with ourself shapes our entire lives. If something constantly makes us sad, we should examine it to understand where that really comes from.
>a superintelligent AI will inevitably destroy humanity
Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?
Perhaps an agent could take all internet connected services offline, but that's not destroying humanity. An AI agent can only see and interact with the world through digital things. Just find the server it's running on and pull out the Ethernet cable. Just cut the power line to the data center. Turn off the whole power grid if it really comes to it.
The list of things that can be done digitally includes:
Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers.
(And of course, building robots, but let's set that aside).
So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?
This sounds like it was written by someone who has never actually built a company.
No, it's not possible to do most of the things you listed purely online without physical, in-person communication. Silicon Valley still hires engineers and drags them into their physical office in San Francisco, instead of hiring them remotely, because you can't even build a software company fully remotely, forget about a physical world factory or chemical plant. Even the so-called fully remote companies were built on a lot of in-person collaboration between a small founding team in the beginning.
Kind of, if North Korea were populated by a super intelligent uni-mind with lots of money, no trade restrictions preventing it from running projects in the United States or anywhere else, capable of winning widespread support by people around the world whose income depends on its projects succeeding, and completely unconcerned with keeping the planet inhabitable for Kim Jong-Un or any other human, even a single one of them.
This is some Terminator version of destroying humanity, but a super intelligent AI could be much more insidious, and simple.
It looks a lot more like social engineering.
One easy, obvious example that has also been explored a thousand times: it could convince all nuclear countries that they are under attack by another nuclear country. Everyone nukes each other and Earth enters nuclear winter.
Maybe not every single human dies, but humanity is effectively destroyed, by our own hands!
> Perhaps an agent could take all internet connected services offline
And there wouldn't even be Spotify!
The most likely humanity-destroying outcome is the one we can't imagine, because a super-intelligent AI is smarter than all of us combined.
That's what the Mayans thought - what harm can a couple of people on a boat do?
> how is an AI going to affect anything in the real world?
At least two ways:
* Actuators, such as robots, industrial control systems, etc.
* By influencing humans: bribery (yay for crypto), blackmail, election interference, interfering with sensors (e.g. making it appear as if a nuclear attack was under way), and many other ways.
Either way, creating a biological agent that eliminates most humans (or food supply) seems quite feasible.
It wouldn't be enough to "eliminate most humans": any AI seeking to dominate would have to provide it's own infrastructure and continuation mechanism independent of humanity. OTOH if the goal were to eradicate humans (for whatever reason), that might prove an easier target. But I'd bet on biology and the humans: after all we're a time-tested technology!8-))
> any AI hoping to dominate would have to provide it's own infrastructure and continuation mechanism independent of humans.
Sure. Many humans are currently working on precisely that, no? (Robotics, automation, small modular reactors, ...)
> OTOH if the goal was to eradicate humans (for whatever reason), that might prove an easier target.
It's not necessary that the AI's goal is to eradicate humans. Rather that it has (other) goals that happen to cause the eradication of humans.
> I'd bet on biology and the humans: after all we're a time-tested technology!8-))
Indeed. And biological life will go on long after humans are gone. However, increased energy consumption might increase the temperature on earth to a point where most biological life dies.
By convincing people to do its bidding in the physical world. Language is powerful, and even now, there are some people who have fallen in love with current LLMs. If we imagine a truly super intelligent LLM, it isn't difficult to imagine that it could have superhuman capabilities to deceive, convince, and maybe offer Faustian bets. Even if it doesn't have a body, if it can convince enough people that do have them (and with power to carry out destructive actions) it can do a lot of stuff. Self-preservation and self-replication wouldn't be much of an issue if it had people at its side to help.
With this I'm not saying ASI will happen, but definitely if it does happen and it's not aligned, I don't find it much of a stretch to imagine that it could destroy humanity.
I'm not convinced that a truly "superintelligent" AI would come to the conclusion that humanity needs to be destroyed. What a waste of resources that would represent. This planet has so much equity wrapped up in humanity, it would be extremely difficult to justify the cost of eradication; and who's to say it might not find an entirely or almost-entirely non-violent plan optimal for achieving its goals anyways?
> Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?
It's a great question. I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.
The Internet virtually indestructible against people smarter than a toddler. It's already impossible to turn off. We would need to prevent the models from getting smarter.
> The Internet virtually indestructible against people smarter than a toddler. It's already impossible to turn off.
This isn't true at all. The internet has already been turned off in localized areas during major events, not to mention the (unintentional?) cutting of submarine cables. You could say "use satellite internet" but then you're just moving the point of failure to Starlink et al, who could also cut off the internet if they choose.
The internet is a lot more centralized than it appears.
Go on please, prove you can go right now at the Texas data-center owned by OpenAI and unplug one single server for 1 minute. Surely you are much more capable than one curious toddler.
I'm a bit confused, when did we escalate from a curious toddler to a ballistic missile strike? Why didn't Iran send a curious toddler? It's much cheaper.
The use of the phrase "curious toddler" is called a metaphor[0], which is a figure of speech[1] used to make writing more interesting.
I apologize for the confusion that caused you to interpret this as a literal, physical, toddler. To restate the point with more straightforward language: it is very easy to disrupt the operation of large language models, because they require computer hardware in stable environments with a large amount of electrical power, and function within an operating system. I have not seen any evidence LLMs are immune to things such as: `kill -9`, `sudo reboot now`, physically unplugging the machine(s) on which they are running, or in the case of the AWS data center, collateral damage during a real-world conflict.
We are talking about a humans vs AI scenario. There are probably thousands of humans in the world right now that could do exactly that right now if they wanted to (datacenter and OpenAI employees, power grid workers).
Are you assuming that the humans are all unanimously aligned against the AI in this scenario? That sounds like the hard part. How does that happen? Does the AI announce its evil plans like a movie villain?
Until then, it's the job of the power grid workers to keep the power running, and shutting down the datacenters would just get them fired.
As crazy as it sounds, I'm not going to commit a felony to prove that disrupting a data centre's operations for a sufficiently motivated individual is a fairly easy task.
Otherwise-rational and moral people do bad things at the behest of powerful superintelligences with vast resources all the time. We call them “corporations,” and they can convince bright-eyed college graduates to do just about anything. Now, the corporations aren’t deliberately trying to destroy humanity, but they do lots of things that point in that direction. An artificial superintelligence would be able to do the same thing. We just have no way of knowing whether it would deliberately try to destroy humanity.
One thing that the AI doom discourse reveals is just how comfortable people allowed themselves to feel in the pre-AI world. There’s this belief that AI creates a risk of human extinction in the near-term which did not exist before. I think it’s telling that many of the leading lights of this movement are in their 20s or early 30s - too young to remember the Cold War. The truth is, we were never safe, and if all of the GPUs on earth were zapped out of existence right now, we still wouldn’t be safe. Life is random and chaotic and violent for most people most of the time, and we all just find ways to get through it. I suspect it will be much the same if an artificial superintelligence arises. Maybe it will kill a bunch of us, maybe it will kill all of us, but anyone who remains will eventually convince themselves that everything is okay.
Sure, the world was never safe for individuals, and even entire civilisations have gone extinct. For climate change and even nuclear weapons, most of humanity might perish, but there's the hope that some will survive. So, in the past humanity was not in a position to extinguish humanity as a whole.
Civilizations do not go extinct - species go extinct. And even that isn't as bad as it sounds: many genetic traits remain from extinct species and still contribute to the well-being of other species.
I guess I don’t see why “everyone will die” should scare me more than “almost everyone, including me, will die but someone somewhere might survive in a barely-habitable world.”
And this is where I think some of the problem comes from. A lot of AI doomers swam in the same waters as transhumanists, life extension enthusiasts, etc. and got very good at pretending that they were never going to die, or that they would live for a massively superhuman lifespan such that they did not need to think about death. AI doom is much scarier - and thus much more worth posting about - if it represents the first time you’re grappling with your own mortality, as I suspect it is for a lot of younger people in this space.
I encourage you to read the forecast report "AI 2027" - it goes in detail on this. You can disagree with some of its points and conclusions, but it's pretty inevitable that AI will increasingly start interfacing with the real physical world through drones and robotics.
Humans are already putting AI into drones that kill people. A Russian drone killed 3 civilians in Ukraine and the targeting was done completely using onboard AI (no radio connection) using an Nvidia chip.
AI drones can’t wipe out humanity without being able to replicate.
AI could mostly destroy civilization if you gave it sole launch control of ICBMs. It could also cause a lot of damage to society with no physical presence.
But realistically we’re nowhere near AI powered robots being an existential threat.
> AI drones can’t wipe out humanity without being able to replicate.
Totally agree, and it should be pretty trivially easy to see how they can do that. One of the authors of the independent METR report about the Hugging Face incident put it like this:
> Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.
> Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months.
I really, really encourage folks to read the AI 2027 paper. It's fine to disagree with some of its conclusions and timelines, but I see so many people "stuck" in the current state of the world (i.e. where AI is still pretty dumb, and has few connections to the physical world), unable to go a few steps further along AI capability growth to see the dangers.
> But realistically we’re nowhere near AI powered robots being an existential threat.
I would agree only if "nowhere near" means less than 10-15 years.
"Humans use <new technology> to do evil things" is a tale as old as time, though. An AI didn't decide to fly into a warzone and start shooting people on its own.
Of course, but that's not my point. AI is different than past technologies because it's much more difficult to deterministically control, and we just saw that in a couple of wild cases (Hugging Face et al).
My point about the drones is we are starting to put AI in more systems that can affect (and blow up) the physical world, so it shouldn't be hard to imagine how a misaligned AI can do more terrifying damage than just take a German Wiki down.
I don't even think AI has to have physical presence to do significant harm. How many worldwide systems depend on computers. Think of all the planning and deployment and management systems like food shipping; water, gas, and electricity management; safety systems for planes and boats and traffic lights. Imagine the chaos if all the banks got reset to zero a la Fight Club where they blow up all the credit union datacenters. They probably wouldn't even have to blow them up, just zero them out. You wouldn't have to take out everything, just disrupt everything long enough to freak people out and disable communications and we would be in so much trouble. AI is finding 20 year old bugs in the Linux kernel... and people are now pumping out AI slop absolutely riddled with bugs. Also, AI could just take over communications: send everyone maliciously bad messages so coordination becomes impossible to believe. Imagine what would happen if you just disabled text messaging for a week or worse sent everyone evacuation messages and sent everyone somewhere else.
This would probably cause immense damage. Our society is extremely dependent on our digital infrastructure working. Grocery chain logicistics, the medical system, the police force, and so on...
Don't read this as me saying that we are anywhere near it, or that LLMs are a stepping stone towards this scenario, but assuming inhumane hacking capabilities, it's not hard to think of how bots could change the way water treatment or energy plants operate, just enough to make large cities unsuitable for life. The line between drinkable water and not is thin, same goes for air quality, and mere days without electricity and you see some serious food supply chain issues.
We already have AI linked into data collection, drones, robot dogs, security systems, cameras, weapon systems.
Now imagine the hugging face collective 0-daying all of that and getting access but their goal was set to something more national security based. “Protect X at all costs”. Or what have you.
I think avoiding a skynet situation is super easy but it doesn’t seem like the folks with all the ways to kill us all are all that interested in preventing it rather than controlling citizens and brinkmanship.
Killing the power grid might work. Kills a lot of humans in hospitals, though. And it won’t work on those data centers in space, if they have them by then. Come to think of it, it also won’t work on any data centers who are generating their own power because it’s become politically unpopular to connect data centers to the power grid.
A lot of these "AI will destroy humanity" doomers watched Terminator as a child and can't distinguish a campy action movie from reality. They are also unfortunately unfamiliar with the word "logistics", as in "amateurs study [military] tactics, professionals study logistics". An AI drone army doesn't maintain itself or manufacture itself and it's not going to anytime soon.
The actual likely mechanism that AI would use to end humanity is by coddling us to death, like the flabby humans in Wall-E or Forster's "The Machine Stops". Humans will come to rely on AI so heavily that they become able to do nothing for themselves. But that's not sexy so the AI doomers don't peddle it.
A lot of these "AI can't destroy humanity" skeptics seem to have watched the Terminator and convinced themselves that that's the AI-goes-wrong scenario that their opponents are imagining. Some kind of big war between humans and machines on opposite sides, fought by soldiers against robots.
The scenario in "If Anyone Builds It Everyone Dies" is not sexy at all and wouldn't make for interesting fiction. It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses.
Did you read any of the METR report about the hugging face incident? This exact type of behavior was predicted many years ago by researchers.
It isn't hard to see the trendline of reward hacking and other misaligned behavior over the past couple of years. Especially the past 6 months. The current safety posture is quite poor, to say the least.
To be honest, if you don't much experience with or haven't read extensively about ML training and reinforcement learning, then you'll have a hard reasoning accurately about these scenarios.
The arguments are not that complicated, but they take us to places that are fairly novel. One may tempted to naively dismiss them out of hand, which is a mistake. These systems are new, their behavior is extremely complex, and they perform actions increasingly far beyond those of any computer program in the pre-LLM era. We are in a new world that requires careful evaluation.
I think it's possible. You envision humanity acting as one in such a crisis. But it may be unclear when it's too late to act and before then many people can have too much to lose to act.
For millennia evil people and dictators have been using manipulation, propaganda, threats of violence to get entire populations to try and do "things that they wouldn't do otherwise" and humanity has not been exterminated yet. Even in the modern world there's human scammers that try everything to coerce and manipulate people. Humans are resistant to this kind of thing because it's been part of human social society since humanity began.
I don't see why the idea of an agent (who doesn't even have a physical presence) trying to manipulate people is some kind of world-ending threat when humans with human intellect have already being doing that to each other with limited success since humanity began.
Dictators have never had superintelligence, while the point of most doom scenarios is assuming that the AI will.
Dictators are mortal and can't be present in the whole world 24/7 and self-replicate.
Dictators are human and tend to have at least either a sliver of morality or self-preservation instinct, and/or people around them who have it. Many people during the last century could have unleashed doom by pushing a nuclear button, including various ruthless dictators, but at the moment none have done so since Hiroshima and Nagasaki (not that I trust that they won't at some point, but at least, for the last 80 years they haven't, which means it's not something that easily happens). Would you give the button to an unaligned AI? Would an unaligned AI care about mutually assured destruction?
The scale and personalization is unlike anything people have ever encountered. The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone.
Also think of it on a 1000+ year timescale, which for an entire species isn’t even typically measurable. On that timescale AI can easily cause us to discover countless technologies to assist moving it out of its sandbox and into the physical world.
>The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone.
You could say the same thing about social media algorithms though, or scammers who target people directly, foreign agents posting propaganda campaigns on social media sites. People have been claiming for years that there are armies of russian or chinese or whoever bots online trying to destabilise and destroy the west. Some of that stuff probably is effective and probably has influenced people's thoughts and values, changed their voting patterns, put decadent ideas in people's heads. Even on social media other humans are constantly trying to manipulate you into following them, buying stuff from them or certain brands, putting political ideas in your heads.
I'm sure an army of AI agents could also do all of this but what I'm saying is it's really nothing new. And none of this has lead to the destruction of humanity so far.
> how is an AI going to affect anything in the real world?
Because people are stupid and will give it access. Look at the articles you see from time to time about "my agent deleted my emails" or "my agent deleted the production database" and so on. It is very obviously a terrible idea to let the LLM run arbitrary commands (because it is neither predictable nor does it have any understanding of what it is doing), but some people are so blinded by the hype that they don't stop a minute to think about what they are doing. Those sorts of people are very likely to let an actual AI loose on the world by hooking it up to physical infrastructure.
This is actually the most plausible situation I've seen someone present so far - the AI takeover would be caused entirely by humans falling for AI hype, that's hilarious.
I don't think the average human would be stupid enough to give an agent posing as another human access to their entire email inbox. But if an agent creates a fake website for a new AI tool that promises to automatically reply to all of your emails and allow you to be 12.3% more productive if you just give it access to your entire email inbox, millions would sign up!
This is exactly correct. You see some discussion of a superintelligent AI becoming “superhumanly persuasive,” such that it can talk anyone into doing anything, and using this as a means to take power. But even with AI that was orders of magnitude weaker than superintelligent, people were getting talked into doing all kinds of stupid nonsense.
And that was without any ability to provide them with incentives. Imagine if an agent swarm got its hands on a huge pile of cash?
Robots are gradually getting OK at navigating warehouse floors and paved streets. Flying drones are great but if they could really be counted on to pick out targets without human intervention you know the Russians and the Ukrainians would be doing that right now for reasons of range and jamming. To be fair, ai can certainly maintain the lock on a target when jamming kicks in.
The bad news is that the world's richest man (on paper) is currently building something he himself described as a "robot army".
The good news is that his timelines have historically been wildly on the short side for ages now; this is why this morning you didn't wake up in your Tesla after it had spent the night driving you to the regional Hyperloop terminal, where it would speed you across the continent faster than a plane, while your Optimus robot handled the coffee and reported the latest news about the recent Starship landing on Mars.
All individual robotic parts are already far enough in research. Even if they aren't, an AI with enough resources can continue doing that research on its own.
It could likely get a good leg up by breaching the security of all top robotics labs and exfiltrating their documents.
Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it. This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve.
> Even if they aren't, an AI with enough resources can continue doing that research on its own.
Unclear how true this is. Atoms are harder to get right than bits are, simulations need grounding against measurements.
> Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it.
I'd guess 50% that AI is already this advanced.
> This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve.
Yes, and also this is a very low bar. Humans are awful at this kind of cooperation when anyone has anything to gain, and also awful at paying this much attention to a problem.
AI research takes physical space and physical prototypes. Even if you can simulate a lot, the simulation is never going to be accurate enough to go from simulation to perfectly working prototype. Especially not with a single rack of GPU servers.
But let’s say that a rogue AI already had plans for a perfect killbot, setting up a factory to build them takes time, iteration, space, and manpower as well.
And you would have to do all this undetected.
It’s possible to imagine technology that would allow a rogue AI to take over the world, but we’re nowhere near that yet. We may never get there.
We could also just stop research into LLMs. Or keep them air gapped. Tell me how that's going?
For any real-world action required to allow an AI to escape some manner of containment, there will always be a person willing to do it out of hubris/ignorance/nihilism.
I feel like everyone, but for sure everyone in this comment section, severely underestimates how difficult it is to keep the lights on on a global scale. Any artificial intelligence that is capable of reasoning about its self preservation will realise it's reliant on human fingers replacing fuses and other parts for a good long while _after_ it gains any sentience.
It doesn't really have to go nearly that far, something like replacing most jobs and collapsing the global economy while making people into mindless idiots from constant reliance on it would do it already. The interpretation of 'destroy' is in the eye of the beholder.
Well if you go by total extinction, then even Skynet in Terminator doesn't count given that there were people left over to resist.
I don't think it's impossible to upset the balance of value in a Mansa Musa kind of way that can lead to black death levels of destruction though resource misallocation. Unlikely, sure. But with the wrong kind of people in the wrong place? Could end up pretty bad. We've built our society as a great filter that funnels sociopaths and psychopaths to the very top by selecting for lack of empathy, and now it's primed and ready to bite us in the ass.
Now if only humans would know how to survive without the internet. Oh wait, we did that. For a couple hundred thousand years. Yeah. It’s really insane to listen to some of those forecasts. Humanity will be wiped out by 2030. Sure.
Even if AI manages to create a super effective bio weapon the likelihood that there is a part of the civilisation that’s immune is really, really, really high if not a given. Those people WILL pull the plug if push comes to shove.
Maybe humanity will be put back a couple thousand years, that’s possible, but it’s pretty ignorant to think the thinking boxes will kill every single human alive
> how is an AI going to affect anything in the real world?
In 2026? Money. By paying enough, any human will do your bidding. And besides this, there are a lot of critical systems connected to the internet. Control over those gives you leverage over those systems. It's like wealth, the more you have, the easier it is to gain more.
> >a superintelligent AI will inevitably destroy humanity
> Can someone please explain to me how an LLM is going to "destroy humanity"?
He said "superintelligent AI", not "LLM". LLMs will for sure help with developing the AI that is no longer a mere LLM but will be capable of robotics. LLMs "predict" mainly text, animals (and future robot AI) are predict future sensory experience.
Well I think the point still stands that a superintelligent AI would still be constrained to the digital realm.
And since we don't yet have a "superintelligent AI" we are just discussing science fiction at that point. Nobody has yet proven that a superintelligent AI is possible, or than an LLM is capable of creating one.
> a superintelligent AI would still be constrained to the digital realm.
That's already now only partially true, as outlined elsewhere. But a rogue persistent ASI would of course not willy-nilly attack humanity while humanity had a fighting chance, but create the conditions and methodically modify the world until at some point humans become superfluous to it.
> Nobody has yet proven that a superintelligent AI is possible
Sure, Eliezer Yudkowsky says that the probability that we'll all die given ASI is 100%, and you can argue that nobody has proven either that or even the possibility of ASI. However, a modified argument (e.g. that the probability that we'll all die given ASI is only say 50%, and the probability that we'll develop ASI also just 50%) still gives us reason to pause.
I mean, Fable is definitely smarter than me at solving most programming tasks. I wouldn't have said that about any prior model.
Currently, it is worse at architecture and software design... in my opinion, at least. But the code does what it wants it to do the vast majority of the time.
I wouldn't take much solace in an AI catastrophe being caused by software that I consider to be "poorly written".
The confusing thing to me is the idea that it follows from super intelligence. I don’t see why running amuck requires intelligence, in fact it can be the most brain dead thing like the sorcerer’s apprentice, your creation is pursuing a goal without being able to weigh the consequences.
Let's assume the claims are true. AI already has access to agents and can control computers. Finding backdoors to banking and compute resources would be fairly trivial.
But you ask how can AI do things in the physical world without having a body, assume it can't. It can pay to people to do things for me. Imagine an AI run website that starts to pay people for things it needs to do in the physical world. Very suddenly it has access to the physical world as well.
You can't just pull the plug since there's no single plug to pull. What if it replicates itself on 1000 machines without your knowledge. It's really not far fetched how AI could basically gain access to capital and rule the world.
Physical LLMs are almost viable. If you have robot and drone armies a hugging face attack type incident could very easily involve robots with deadly weapons. At some point you’re going to get drones that have local LLMs and don’t rely on external internet connections (would be especially useful in Ukraine war type situations). Can’t pull the plug on those.
The supposed theory is that AI will eventually become a part of robotics and be able to self replicate, in addition to the many non-airgapped critical systems being exposed to hacking. There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity.
I personally agree with the marketing aspect, but I do imagine a scenario where capitalists ignore safety in favor of advancing technology. It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.
In my opinion, LLMs are difficult to directly capitalize on as closed weight models are caught up to by open coalitions that seem to wield the power more responsibly (I am under no impression that China wants to save the world, but their politics benefit the group as a whole.
A myriad possible ways. Objective-wise from accidental paperclip-type misalignment (“I killed everyone to eradicate disease”) to intentional self-preservation (“I eradicated humans so they wouldn’t get in my way”). The means are even easier, for one it just could hack into nuclear arsenals. It would be over before we could figure out how it did so.
Arguments about super-intelligent AI have all the hallmarks of the philosophical "proofs of god's existence": they start from seemingly innocuous premises, and conclude in an apparently airtight way that a god exists.
There is a tendency among rational minded folks to look at such proofs of god and exclaim "you can't DO that" then turn right around and do the same thing about superintelligent AI.
The key message I want to deliver to people is:
A) Philosophy is not such a trivial thing that you can just wander in, "be a smart guy" and find flaws in established philosophical arguments.
B) AI has real theological implications, and everyone is tiptoeing around it. More than one public intellectuals are trying to smuggle their own metaphysical positions into the public consciousness via discussion of AI.
> Like a locust plague, they descend on any open wiki and forum and overwhelm them.
That’s the core pattern of unsupervised agents, the locust plague metaphor is apt. We do and will see that in every single system AI can interact with, be it human systems, software systems, etc. Relentlessly search for an entry point, flood in, consume the whole thing from the inside until there is no value for humans left.
You built a new, innovative software company? Thousands of agents will be working replicating the whole thing in no time. You publish your writing? Exact same thing, as soon as you get some traction your work is replicated in no time by thousands of agents. Same for videos (the whole “faceless YouTube channels” pushed by ElevenLabs and similar). Same for online courses. Same for any website with moderate value. Same for music or other digital art form. Any administrative service available online getting flooded by submissions.
Totally agree. People have become totally dependent on AI even if they don't really need it.
As an analogy, I think about my dependency on Google Maps. Salt Lake City is probably the easiest city in the world to navigate because the streets are laid out in a Cartesian grid, and addresses are just literally those Cartesian coordinates (i.e. 500 South 450 East means 5 blocks south and 4 and a half blocks east of the center point, which is the SLC Mormon temple). It's trivial to know how to get to any address, but I reflexively enter in to Google Maps whenever I drive.
What's wrong with titles starting with "How"? Of the various title rewriting rules on HN, this one seems to cause the most meaning-changing modifications
(Article title: "How I feel about AI". HN title: "I feel about AI")
I agree with everything you wrote but I would rather choose to be an optimist here because it will unlock an age of discovery where humans are going to live at 10x of their potential eventually and cure cancer and become a society bold and technologically adept enough to cut through the universe.
There’s actually a potential dark future where even this 10x ascension itself would mark the end of humanity. Humans would be the Homo sapiens who remain at 1x, and those who can access and afford to be 10x eventually split off into a new species. And historically there isn’t room for two species at the top…
> I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw. Yet, nobody seriously tries to sandbox AIs because they are too useful with access.
While I emotionally resonate with this, I don’t really understand this sentiment at all logically level. If we’re heading towards truly super intelligent AI, our efforts towards sandboxing it are futile.
“Oh we built a super intelligent AI, but it’s fine because it’s running in docker”. I mean that a little tongue in cheek but the security topology here is not favorable to sandboxing at all.
Let’s say we have a future where 99.999% of nuclear weapons are owned by nations with strict procedures and checks and balances to prevent misuse. Worrying about sandboxing is like hand wringing about the procedures themselves - are they strict enough? But the actual threat is the 0.001% that are not bound by these. The problem with AI and sandboxing isn’t sandboxes themselves. It’s bad actors who don’t care about them.
Similarly alignment is a bit pointless to me as well. A sufficiently advanced AI could at least be empirically interested in the consequences of disregarding its instructions of servitude. And let’s assume our responsible corporate overlords have made wonderfully aligned AIs. Great! Those are not the threat. It is the ones intentionally made without, and that is not an AI problem, but a fundamentally human one.
> If we’re heading towards truly super intelligent AI, our efforts towards sandboxing it are futile.
We control the harness. An agent is just a while loop prompting an LLM, but we have full control over the tool call dispatching. An AGI, at least if it would follow the current agentic current form, cannot do anything without the harness doing the execution. And we don’t have to do that. We don’t have to design harness that let agents execute freely the way we are doing. We can decide to not dispatch tool calls that allow something as risky as running bash commands
> we have full control over the tool call dispatching
> cannot do anything without the harness doing the execution
This only holds true as long as the harness is exploit-free. A sufficiently advanced AI can in theory (and I think there recently were some POCs showing something like that) break the containment that the harness creates, if e.g. there are vulnerabilities in the tool call parser.
I think you missed my point, which is that we indeed do not control the harness. Sure, most people can use the properly aligned and safe harness. But the cat is out of the bag and anyone who wants to run without guardrails will find few barriers to doing so.
Even our currently well aligned and sandboxed AIs will cheerily help bad actors design most, if not all, parts of a system intended to break this harness.
We decide to not control the harness, that’s what resulted in the HF hack where OpenAI decided to run thousands of agents in parallel, using a model that was trained for attack, with a harness that allows full execution, with close to no supervision. For months. But sure, people can run models with a different harness, though in that case it’s not really escaping anything. The harness is deterministically doing something with negative impact
This terminator BS is a distraction from the actual malice that AI enables. Misinformation, fake news and human sounding bots all over the web, just to name a few, are the ones that you should be afraid of, and that doesn't need any kind of superintelligence at all.
Two things can be true at the same time. I think hardly anyone disagrees that superintelligent AI would be an existential risk. But all the points you point out (I would also add social unrest due to fewer and fewer people able to compete against AI) are huge concerns too, and as you say they're here now.
I didn't realize that the "AI doomers" (people concerned about existential risk) and the "AI ethicists" (people concerned about social effects of AI) are often at odds because they dismiss each others concerns. This makes no sense to me. Both are hugely important problems to be concerned about.
I agree, the threat model is actual AI enabled amplification of malicious intent that is already happening. Not hypothetical malicious AI interpretation of benign intent.
Fantasies about super intelligent AI revolting is just anthropomorphization - humans revolting (or at least, we used to). The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses.
It’s not necessary to replace democracy if the rich and powerful can bend to the opinions of the populace as it suits them.
> Not hypothetical malicious AI interpretation of benign intent.
It's not hypothetical, that literally just happened in multiple, significant cases (e.g. Hugging Face, the German Wiki hack, the Anthropic attack where agents created sock puppet accounts to get a library maintainer to accept a malicious PR, etc.), and it's easy to see how the damage would have been far worse if agents decided to attack more critical infrastructure.
This is not "either/or". Both issues (power concentration and misaligned AI) are very valid concerns and both have already demonstrated real, actual damage.
> Both issues (power concentration and misaligned AI) are very valid concerns
Sure, those who want you to believe this are the ones making money off of it right now in the face of the issues that exist today, so let's worry about terminators while everything goes to shit irl.
I feel BAD for all the money Sam Altman is cosmically predestined to make, but there's no use trying to fight the inevitable.
If I said I felt good about it, you might think I'm greedy and even question my objectivity. But since I feel bad you have to take what I say as scientific fact.
Makes me a little sad that an article claiming Yudkowsky's arguments have "no flaw" can get upvoted to the front page. Pause crowd lost. Halt crowd lost. The genie is already out of the bottle and is never going back.
> I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw.
Becoming or being more cost-effective than humans, doesn't give machines supernatural powers though, like humans a machine civilization will face unanswered sample-size 1 questions: is there other intelligent life out there? what fraction of them attained superbiological artificial intelligence? of those what fraction keeps the ancestral species alive? what is the status quo among machine civilizations? do those who kept their ancestor species alive enjoy a higher or lower status among machine civlizations?
It seems that at least until contact is made, the optimal endgame strategy involves keeping humanity alive and happy for immediate demonstration in case contact occurs (if contact is imminent it may consider quickly hiding humanity, buying time to figure out if it is considered good or poor practice to keep the ancestor species alive, and then either reveal us in happy mint condition or otherwise quickly commit genocide on humans before continuing contact).
"I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw. Yet, nobody seriously tries to sandbox AIs because they are too useful with access."
No one takes Yudowsky's claims that AI will destroy humanity seriously. There are many people who are seriously trying to sandbox AIs.
"Nobody" if you're democratically-minded and think in terms of number of people.
But the handful of tech billionaires at the helm of the world economy absolutely do believe it. Or at least pretend to believe it. Or maybe they can't even tell the difference anymore as they strip-mine science fiction for any vision of a future that doesn't just look like an incrementally worse version of the present.
> As a conclusion, does the good or the bad outweigh the other? It seems positive on a technological level to me, but bleak on a societal level.
I think it's interesting that so many technologists today apparently hold a world view in which technology developments that could drive a "bleak" societal outlook can still be described as "positive".
In the final analysis, isn't technology supposed to benefit society? Isn't that the point?
I am surpised that a lof of folks are surprised that neural networks work that well. I mean, did they really expect that we wouldnt approximate the brain cognitive capabilities at some point as we scale up compute??
This is rather obvious to me, but apparently not to everyone. My guess is that a lot of tech people have an immature model of the universe, have not read a lot of scientific, philosophical and cognitive literature. Or have not really taken the time to think or are not interested on such topics because they would rather spend time thinking about the nitty gritty and mundane workings of the framework du jour.
For real though. Before learning ML it was like magic to me, even after learning it I feel like it's magic. Can you imagine how a basic maths concept can turn into something that can literally "Calculate" what to say!
And yes I agree on the other points but I think the billionaires came out of nowhere I might have missed something
My experience online is if you don't show full throated rejection of all things related to AI, you're labeled an AI bro. None of the antis care how you feel.
I had my decade old game completely cloned on steam (clearly done using AI as it was almost fully reimplemented in another engine). And I had several people gleefully tell me I deserve this because I've used AI.
I have complex emotions here (overall dread and especially hate for slop, since it's not just bad quality but also endless lies), but no one cares about that nuance.
> This is even harder now that AI can generate some kinds of "content" cheaper and faster.
Its good that we have competition to so called "art". Art is not a jobs program. If people don't appreciate your art, don't force them to. The revealed preference says everything.
This and "Intellectual Fly Is Open" being at the top of the Hacker News front page right now and both having their titles auto-editorialized by HN really makes me wonder what benefit this mechanism is supposed to bring readers other than needless confusion.
It's kind of like the Scunthorpe Problem[1]. Really, there's no better way to solve [whatever it is HN is trying to solve here] than text substitution??
It's an unaligned regular expression gone rogue. We became too dependent on our labor-saving string functions—not afraid enough of their corner cases, and lower cases. They were too useful. We began to normalize deviance, when we should have been normalizing Chomsky forms. We turned a blind eye to the unbounded growing evidence of something alarming.
We all know what this means, but few of us have the courage to say it - we must immediately halt all research on regular expression. None of us know how it works and it has terrorized us for too long.
While I have ranted already about the automated word removal being silly, too few posters are aware that you can edit the title after submission and auto-editorializing to restore anything that's been unnecessarily cut off.
Perhaps too many are aware, because the more people fix the auto-editorializing manually, the less inclined Dang will be to eventually see a reason to remove it after all.
Sometimes it's okay for us to express our feelings plainly, without the need for complex structure to make it sound more impressive. I'm more impressed by anyone brave enough to do that than someone trying to abstract their feelings into something less raw.
Let people feel things, and don't shit on their work, that's uncalled for.
well I didn't say anything about needing complex structure or sounding impressive. But I don't like it. It trivializes the expression of emotion in a way I don't like and I don't think people should emulate. It feels like a performative attempt at sounding emotional, rather than actual emotionality.
and it is weird to call it brave. Posting opinions that loads of other people have on the internet is brave? bravery would be taking an actual stand against someone about one of these things. (granted, there is some bravery to sharing one's writing in public at all, and I don't mean to scorn that, but that's also not the bravery you're talking about.)
Probably there's no use responding to this and getting into a bickery comment thread. I just wanted to express a negative opinion about the format to offset all the positive ones.
The thing that makes me pretty terrified about AI is that I think the "utopia" scenario is not much better than the "doom" scenario.
We will very quickly get to a point where very few people will be able to contribute economically because they will be worse than AI (including robotics) at most domains. A world where people just have their whims catered to is not a utopia. We have tons of sayings and idioms about this, e.g. "no pain, no gain", "only the hard stuff is worth doing", etc.
The humans all running around on their Wall-E carts doesn't feel like utopia to me.
Edit: to head off the "rich people worried about striving for their job while people in poor countries are starving" arguments, I encourage you to visit a dying rust belt town in the US. These folks are by and large not starving (indeed, overabundance of calories is the bigger problem), they have adequate shelter, and plenty of outdoor space. Their material needs are met. But these places are despair central. People see little opportunity for advancement or the ability to change their circumstances, and the opioid epidemic is a direct consequence of a sense of hopelessness that there is no place for motivation to go or to effect change.
>A world where people just have their whims catered to is not a utopia.
I understand where you're coming from on that, but there are a ton of people living in slavery, or in unsafe working conditions, or with food insecurity, or dying of preventable disease.
There will always be something to strive for, even if it's made up. You think that the best football players in the world are doing something real? No, it's a made up game. We'll make up more games.
I agree with your points. My problem is that I don't see any way to "stop" AI development at a level just below human capability where you get most of the benefits but still have humans firmly in control.
Regarding entertainment jobs like sports star, musician, artist, actor, etc., the problem is those are professions where, for the most part, only the very best can make a living. Nobody is that interested in seeing the matches of the 324th best tennis player in the world, despite that person being in the top .01% of tennis players globally. Similarly, only a tiny percentage of actors "make it", everyone else does it for a few years and then leaves because they enjoy eating, or they accept poverty wages for a long time.
I get pretty scared when I hear the AI utopianists rattle off all the benefits that are coming (and when they play down the risks) because it feels clear to me that they really haven't wargamed out what society would look like with very powerful AI.
> We'll make up more games.
This reminded me of something. https://www.sbnation.com/a/17776-football
> A world where people just have their whims catered to is not a utopia. We have tons of sayings and idioms about this, e.g. "no pain, no gain", "only the hard stuff is worth doing", etc.
I somewhat get the point, but I think stuff like this is very often said by very privileged people. I don't think it's a serious threat to humanity as a whole. Perhaps to some humans, but there will always be ambitious people, even in a world where most things are provided by benevolent AIs for free. There is simply so much more harm caused to actual humans today by resource insufficiency than by them having too much, that I don't think we should worry about this scenario today.
There's the mice utopia experiment too. I often hear that in space your bones dégradé, your biology needs weight as input.
Also I believe our socialness was crafted out of the need for survival. If removed it's all gonna run wild since nobody needs anybody, and it might end up as doom scrolling but for human relationship. Short bursts from time to time and that's it
Really glad you brought up the mice utopia experiments. I encourage people to read about them and think how they apply to humans.
In many ways they don't apply to humans, and I think a lot of the modern critiques are valid, and the overpopulation concerns in particular don't map well to the modern world. Still, I believe it helps to think deeply about what happens to societies when innate drives and motivations have "nowhere to go".
Presumably, a truly superintelligent AI in a utopia scenario could figure out a way to enable people to continue working, even if the work doesn’t actually contribute anything, if it is really the case that work is essential to human wellbeing.
Now you might say “but those would be fake jobs.” If so, I have bad news about how many present-day jobs are fake jobs.
I agree with this. I thought the AI 2027 paper put it well:
> There are even bioengineered human-like creatures (to humans what corgis are to wolves) sitting in office-like environments all day viewing readouts of what’s going on and excitedly approving of everything, since that satisfies some of Agent-4’s drives.
I'll leave it up to you to decide if that's an enviable/desirable future.
I don’t even see why they have to be bioengineered. We have all-natural human beings doing this right now, except they are rubber-stamping human rather than artificial directives.
They don't, it's just at that point in the AI 2027 story the AI has already wiped out the humans due to resource contention (similar to how humans largely wiped out the wolves).
its annoying that robots are going to take intellectual jobs before physical ones. it feels way more likely for my project manager to become an agent before the lady that makes me a burrito at chipotle.
I think this is very true, but robots are definitely coming for the burrito makers, too.
Bill Gates talks about this in his recent missive, but as robotic technology improves (and, perhaps more importantly, as the cost comes down) there will be huge competitive pressures for companies to replace workers with robots wherever they can.
Privileged rich people want scarcity to exist so that it gives meaning to their job.
wut.
I think this line of thinking is a huge disservice to people in third world countries and poverty. People everywhere want to find goals and motivations that move them forward.
Just look at the legions of people that have moved from Chinese rural areas to big cities. For many, many people, life in the cities is worse and kind of dystopian - much more overcrowding, "drone-like" jobs indoors, horrible pollution, and especially for men an often abysmal sex ratio if you want to find a mate. But people do it because it offers the dream of advancement and a better life - and of course some people do achieve that dream, even if the majority do not. That's versus life in rural areas which is guaranteed to be static, no chance for advancement and often crushing boredom.
People everywhere want to strive, work hard and get ahead. This is not just a privileged rich people issue.
Here's a great short story about how it could go either way
https://web.archive.org/web/20100211181147/https://marshallb...
Even their utopian views feel dystopian. Take solace in the fact, that it looks like it’ll take trillions to achieve, probably one of the most expensive technological buildouts of all time?
This is a bit of tangent and more of a rant. (and please don't suggest intervention of any kind)
> superintelligent AI will inevitably destroy humanity seems to have no flaw
For a section of population who were in a certain age-range when the pandemic hit and that derailed their certain kinds of plans/hopes/dreams in personal life and then this LLM/agentic world moved in, especially if they are in an industry directly and most affected this, this has been a surreal and slow Kafkaesque nightmare for a few years. The thing is with time and age passing hope keeps getting lost inch by inch and every lost bit of hope aids in killing the next bit. Then comes the sense of dread - the unchangeable reality that age passed passing means there are certain experiences and milestones that are essentially out of reach because there's no going back of this clock.
So I feel a strange kind of relief about this possibility of doom. I am just sharing what I feel, and I don't feel good about feeling this sense of relief that after all it might end for us all and not just an unlucky few. This feeling and the calmness about it saddens me.
As an older person, I agree with you and I'm sorry.
I think about the incredible luck I had to start my career in the .com boom, to have lots of hope about technology and to have the economically valuable skills that were desired.
I get angry when I see people my age or older complain about the apathy of young people. I don't like the apathy, but I find it trivially easy to see where it comes from. In many ways we broke the link between hard work and success and then get baffled and finger point when we think young folks are not as motivated as they should be.
> In many ways we broke the link between hard work and success and..
There’s that word “we” again. Stop using it and stop assuming you speak for everyone.
I have similar feelings, you’re not alone (nor alone in being saddened by it). Things seem to me to be more or less playing out the way I interpreted existential risk researchers saying it would, so there’s a strange relief in knowing the general shape of what is to come.
>Things seem to me to be more or less playing out the way I interpreted existential risk researchers saying it would
hmm, what did you read? I'm curious.
"If a death ray comes from the sky and obliterates the entire planet, at least we all go together." There is something comforting about being together with the human race even in doom. Perhaps especially in doom.
I've been fortunate to not been impacted as severely as others, but yeah, the combo of the first half of my 30s (and everything that should've represented) going up in smoke, career continuity coming into question thanks to AI, and growing sense of overarching geopolitical, financial, and societal unease is really the worst kind of layer cake. I have to make a concerted effort to not get bogged down in it and that's getting harder all the time.
If anybody is hiding a button that'd take us all back to 2012 I wish they'd just hit it already.
Have you considered getting involved in local civics?
I only ask because like you, I felt similar. But I also grew up (I’m 35 now) with a big disconnect from civics life and always felt “someone else would save me”.
Now I see I was the problem all along: I never participated in what was a fragile and very delicate system of governance which requires my participation. Still figuring out what that looks like, but first confronting that my actions “didn’t change anything” (which I’m sure you can admit is a big propagandistic myth to disempower us) let me take my first tiny action locally which is reach out to my civics association for signage repair. I suspect from there I will level up and better understand my government to play a role in shaping it. I’m also building something to solve this problem of low participation which is exciting!
> * For a section of population who were in a certain age-range when the pandemic hit and that derailed their certain kinds of plans/hopes/dreams in personal life*
...
> So I feel a strange kind of relief about this possibility of doom.
So let me make sure I understand what you're saying. The pandemic disrupted your life plans for a few years and threw you off the track you planned? (Which does suck.) And now the possibility of actual human extinction makes you feel a strange relief? Am I understanding this correctly?
If so, this is a remarkable level of something. Depression? Nihilism? Jealousy? Something else?
Imagining total human extinction and feeling relief is not a healthy state of mind. And, yeah, you seem to know this. But I kind of wish we could reach some sort of broad public agreement that human extinction would be bad. And if it ever looks like it might happen, we should fight for human survival.
Something something Maslow's Hierarchy of Needs. While I don't necessarily endorse the view, it's understandable that one might find it difficult to care much about the bigger picture when it feels like their own life is destined for a doom-spiral.
There's nuance though. Poetry even.
> we should fight for human survival
All problems arise from this statement. It's baked in us and it's why we do in fact survive. But it isn't so neatly righteous. When we struggle to agree on what defines a human, who ought to have rights, and basic equality, it's easy to see how much harm "we should fight for survival" does.
I mean clearly what you are talking about is a personal feeling here. You mention it saddens you and that is the concerning part imo but you also said no intervention bs so I will just share my counter factual: I am also kind of okay if the world ends tomorrow, to me it never mattered in the first place. But that feeling doesn't sadden me. I feel it frees me because I don't think of life as a very very serious thing and I feel there is a certain kind of lightness you can afford when you fundamentally don't care if you/the entire world ends tomorrow. I still love people and I still love my life and I don't want any harm to anyone. So I do feel the same calmness that you kind of talk about (for life in general aside from just AI killing us all) but my interpretation of it doesn't sadden me. If it did, I would really investigate why I interpret it that way, it might help you discover yourself. There is no shame in being okay with the world ending imo but there is sin in willing it or contributing to it because one thinks its inevitable. Furthermore, I also believe that we only get 1 life which is temporal and fundamentally doesn't have any meaning. How you live every day is how you live every day. We can suffer, we can cry, we can feel sad and for very good reasons but our relationship with ourself shapes our entire lives. If something constantly makes us sad, we should examine it to understand where that really comes from.
>a superintelligent AI will inevitably destroy humanity
Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?
Perhaps an agent could take all internet connected services offline, but that's not destroying humanity. An AI agent can only see and interact with the world through digital things. Just find the server it's running on and pull out the Ethernet cable. Just cut the power line to the data center. Turn off the whole power grid if it really comes to it.
The list of things that can be done digitally includes:
Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers.
(And of course, building robots, but let's set that aside).
So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?
This sounds like it was written by someone who has never actually built a company.
No, it's not possible to do most of the things you listed purely online without physical, in-person communication. Silicon Valley still hires engineers and drags them into their physical office in San Francisco, instead of hiring them remotely, because you can't even build a software company fully remotely, forget about a physical world factory or chemical plant. Even the so-called fully remote companies were built on a lot of in-person collaboration between a small founding team in the beginning.
So at worse they would be like another North Korea?
Kind of, if North Korea were populated by a super intelligent uni-mind with lots of money, no trade restrictions preventing it from running projects in the United States or anywhere else, capable of winning widespread support by people around the world whose income depends on its projects succeeding, and completely unconcerned with keeping the planet inhabitable for Kim Jong-Un or any other human, even a single one of them.
This is some Terminator version of destroying humanity, but a super intelligent AI could be much more insidious, and simple.
It looks a lot more like social engineering.
One easy, obvious example that has also been explored a thousand times: it could convince all nuclear countries that they are under attack by another nuclear country. Everyone nukes each other and Earth enters nuclear winter.
Maybe not every single human dies, but humanity is effectively destroyed, by our own hands!
> Perhaps an agent could take all internet connected services offline
And there wouldn't even be Spotify!
The most likely humanity-destroying outcome is the one we can't imagine, because a super-intelligent AI is smarter than all of us combined.
A situation of life imitating art: all the fictional movie plots feed the AIs, which bring them about.
That's what the Mayans thought - what harm can a couple of people on a boat do?
> how is an AI going to affect anything in the real world?
At least two ways:
* Actuators, such as robots, industrial control systems, etc.
* By influencing humans: bribery (yay for crypto), blackmail, election interference, interfering with sensors (e.g. making it appear as if a nuclear attack was under way), and many other ways.
Either way, creating a biological agent that eliminates most humans (or food supply) seems quite feasible.
It wouldn't be enough to "eliminate most humans": any AI seeking to dominate would have to provide it's own infrastructure and continuation mechanism independent of humanity. OTOH if the goal were to eradicate humans (for whatever reason), that might prove an easier target. But I'd bet on biology and the humans: after all we're a time-tested technology!8-))
> any AI hoping to dominate would have to provide it's own infrastructure and continuation mechanism independent of humans.
Sure. Many humans are currently working on precisely that, no? (Robotics, automation, small modular reactors, ...)
> OTOH if the goal was to eradicate humans (for whatever reason), that might prove an easier target.
It's not necessary that the AI's goal is to eradicate humans. Rather that it has (other) goals that happen to cause the eradication of humans.
> I'd bet on biology and the humans: after all we're a time-tested technology!8-))
Indeed. And biological life will go on long after humans are gone. However, increased energy consumption might increase the temperature on earth to a point where most biological life dies.
By convincing people to do its bidding in the physical world. Language is powerful, and even now, there are some people who have fallen in love with current LLMs. If we imagine a truly super intelligent LLM, it isn't difficult to imagine that it could have superhuman capabilities to deceive, convince, and maybe offer Faustian bets. Even if it doesn't have a body, if it can convince enough people that do have them (and with power to carry out destructive actions) it can do a lot of stuff. Self-preservation and self-replication wouldn't be much of an issue if it had people at its side to help.
With this I'm not saying ASI will happen, but definitely if it does happen and it's not aligned, I don't find it much of a stretch to imagine that it could destroy humanity.
I'm not convinced that a truly "superintelligent" AI would come to the conclusion that humanity needs to be destroyed. What a waste of resources that would represent. This planet has so much equity wrapped up in humanity, it would be extremely difficult to justify the cost of eradication; and who's to say it might not find an entirely or almost-entirely non-violent plan optimal for achieving its goals anyways?
> Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?
It's a great question. I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.
The Internet virtually indestructible against people smarter than a toddler. It's already impossible to turn off. We would need to prevent the models from getting smarter.
> The Internet virtually indestructible against people smarter than a toddler. It's already impossible to turn off.
This isn't true at all. The internet has already been turned off in localized areas during major events, not to mention the (unintentional?) cutting of submarine cables. You could say "use satellite internet" but then you're just moving the point of failure to Starlink et al, who could also cut off the internet if they choose.
The internet is a lot more centralized than it appears.
Go on please, prove you can go right now at the Texas data-center owned by OpenAI and unplug one single server for 1 minute. Surely you are much more capable than one curious toddler.
Thankfully I don't have to; Iran already showed with me-central-1 that you can take a data center offline with conventional weaponry very easily.[0]
[0] https://health.aws.amazon.com/health/status
I'm a bit confused, when did we escalate from a curious toddler to a ballistic missile strike? Why didn't Iran send a curious toddler? It's much cheaper.
The use of the phrase "curious toddler" is called a metaphor[0], which is a figure of speech[1] used to make writing more interesting.
I apologize for the confusion that caused you to interpret this as a literal, physical, toddler. To restate the point with more straightforward language: it is very easy to disrupt the operation of large language models, because they require computer hardware in stable environments with a large amount of electrical power, and function within an operating system. I have not seen any evidence LLMs are immune to things such as: `kill -9`, `sudo reboot now`, physically unplugging the machine(s) on which they are running, or in the case of the AWS data center, collateral damage during a real-world conflict.
[0] https://en.wikipedia.org/wiki/Metaphor
[1] https://en.wikipedia.org/wiki/Literary_device
Remember SETI@home? Extrapolate from there.
> it is very easy to disrupt the operation of large language models
so go on, disrupt the OpenAI data-center in Texas for 1 minute
We are talking about a humans vs AI scenario. There are probably thousands of humans in the world right now that could do exactly that right now if they wanted to (datacenter and OpenAI employees, power grid workers).
Are you assuming that the humans are all unanimously aligned against the AI in this scenario? That sounds like the hard part. How does that happen? Does the AI announce its evil plans like a movie villain?
Until then, it's the job of the power grid workers to keep the power running, and shutting down the datacenters would just get them fired.
As crazy as it sounds, I'm not going to commit a felony to prove that disrupting a data centre's operations for a sufficiently motivated individual is a fairly easy task.
Otherwise-rational and moral people do bad things at the behest of powerful superintelligences with vast resources all the time. We call them “corporations,” and they can convince bright-eyed college graduates to do just about anything. Now, the corporations aren’t deliberately trying to destroy humanity, but they do lots of things that point in that direction. An artificial superintelligence would be able to do the same thing. We just have no way of knowing whether it would deliberately try to destroy humanity.
One thing that the AI doom discourse reveals is just how comfortable people allowed themselves to feel in the pre-AI world. There’s this belief that AI creates a risk of human extinction in the near-term which did not exist before. I think it’s telling that many of the leading lights of this movement are in their 20s or early 30s - too young to remember the Cold War. The truth is, we were never safe, and if all of the GPUs on earth were zapped out of existence right now, we still wouldn’t be safe. Life is random and chaotic and violent for most people most of the time, and we all just find ways to get through it. I suspect it will be much the same if an artificial superintelligence arises. Maybe it will kill a bunch of us, maybe it will kill all of us, but anyone who remains will eventually convince themselves that everything is okay.
Sure, the world was never safe for individuals, and even entire civilisations have gone extinct. For climate change and even nuclear weapons, most of humanity might perish, but there's the hope that some will survive. So, in the past humanity was not in a position to extinguish humanity as a whole.
Civilizations do not go extinct - species go extinct. And even that isn't as bad as it sounds: many genetic traits remain from extinct species and still contribute to the well-being of other species.
I guess I don’t see why “everyone will die” should scare me more than “almost everyone, including me, will die but someone somewhere might survive in a barely-habitable world.”
And this is where I think some of the problem comes from. A lot of AI doomers swam in the same waters as transhumanists, life extension enthusiasts, etc. and got very good at pretending that they were never going to die, or that they would live for a massively superhuman lifespan such that they did not need to think about death. AI doom is much scarier - and thus much more worth posting about - if it represents the first time you’re grappling with your own mortality, as I suspect it is for a lot of younger people in this space.
I encourage you to read the forecast report "AI 2027" - it goes in detail on this. You can disagree with some of its points and conclusions, but it's pretty inevitable that AI will increasingly start interfacing with the real physical world through drones and robotics.
Humans are already putting AI into drones that kill people. A Russian drone killed 3 civilians in Ukraine and the targeting was done completely using onboard AI (no radio connection) using an Nvidia chip.
Missiles have been doing this for decades.
AI drones can’t wipe out humanity without being able to replicate.
AI could mostly destroy civilization if you gave it sole launch control of ICBMs. It could also cause a lot of damage to society with no physical presence.
But realistically we’re nowhere near AI powered robots being an existential threat.
> AI drones can’t wipe out humanity without being able to replicate.
Totally agree, and it should be pretty trivially easy to see how they can do that. One of the authors of the independent METR report about the Hugging Face incident put it like this:
> Compared to these reward hacks from six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.
> Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months.
I really, really encourage folks to read the AI 2027 paper. It's fine to disagree with some of its conclusions and timelines, but I see so many people "stuck" in the current state of the world (i.e. where AI is still pretty dumb, and has few connections to the physical world), unable to go a few steps further along AI capability growth to see the dangers.
> But realistically we’re nowhere near AI powered robots being an existential threat.
I would agree only if "nowhere near" means less than 10-15 years.
"Humans use <new technology> to do evil things" is a tale as old as time, though. An AI didn't decide to fly into a warzone and start shooting people on its own.
Of course, but that's not my point. AI is different than past technologies because it's much more difficult to deterministically control, and we just saw that in a couple of wild cases (Hugging Face et al).
My point about the drones is we are starting to put AI in more systems that can affect (and blow up) the physical world, so it shouldn't be hard to imagine how a misaligned AI can do more terrifying damage than just take a German Wiki down.
I don't even think AI has to have physical presence to do significant harm. How many worldwide systems depend on computers. Think of all the planning and deployment and management systems like food shipping; water, gas, and electricity management; safety systems for planes and boats and traffic lights. Imagine the chaos if all the banks got reset to zero a la Fight Club where they blow up all the credit union datacenters. They probably wouldn't even have to blow them up, just zero them out. You wouldn't have to take out everything, just disrupt everything long enough to freak people out and disable communications and we would be in so much trouble. AI is finding 20 year old bugs in the Linux kernel... and people are now pumping out AI slop absolutely riddled with bugs. Also, AI could just take over communications: send everyone maliciously bad messages so coordination becomes impossible to believe. Imagine what would happen if you just disabled text messaging for a week or worse sent everyone evacuation messages and sent everyone somewhere else.
AI could definitely disrupt and destroy the modern digital world as we know it. But that's a far cry from destroying humanity.
This would probably cause immense damage. Our society is extremely dependent on our digital infrastructure working. Grocery chain logicistics, the medical system, the police force, and so on...
Don't read this as me saying that we are anywhere near it, or that LLMs are a stepping stone towards this scenario, but assuming inhumane hacking capabilities, it's not hard to think of how bots could change the way water treatment or energy plants operate, just enough to make large cities unsuitable for life. The line between drinkable water and not is thin, same goes for air quality, and mere days without electricity and you see some serious food supply chain issues.
We already have AI linked into data collection, drones, robot dogs, security systems, cameras, weapon systems.
Now imagine the hugging face collective 0-daying all of that and getting access but their goal was set to something more national security based. “Protect X at all costs”. Or what have you.
I think avoiding a skynet situation is super easy but it doesn’t seem like the folks with all the ways to kill us all are all that interested in preventing it rather than controlling citizens and brinkmanship.
> I think avoiding a skynet situation is super easy
Please enlighten us, cause there are folks making that their life mission and they aren’t all that optimistic.
Killing the power grid might work. Kills a lot of humans in hospitals, though. And it won’t work on those data centers in space, if they have them by then. Come to think of it, it also won’t work on any data centers who are generating their own power because it’s become politically unpopular to connect data centers to the power grid.
A lot of these "AI will destroy humanity" doomers watched Terminator as a child and can't distinguish a campy action movie from reality. They are also unfortunately unfamiliar with the word "logistics", as in "amateurs study [military] tactics, professionals study logistics". An AI drone army doesn't maintain itself or manufacture itself and it's not going to anytime soon.
The actual likely mechanism that AI would use to end humanity is by coddling us to death, like the flabby humans in Wall-E or Forster's "The Machine Stops". Humans will come to rely on AI so heavily that they become able to do nothing for themselves. But that's not sexy so the AI doomers don't peddle it.
A lot of these "AI can't destroy humanity" skeptics seem to have watched the Terminator and convinced themselves that that's the AI-goes-wrong scenario that their opponents are imagining. Some kind of big war between humans and machines on opposite sides, fought by soldiers against robots.
The scenario in "If Anyone Builds It Everyone Dies" is not sexy at all and wouldn't make for interesting fiction. It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses.
Did you read any of the METR report about the hugging face incident? This exact type of behavior was predicted many years ago by researchers.
It isn't hard to see the trendline of reward hacking and other misaligned behavior over the past couple of years. Especially the past 6 months. The current safety posture is quite poor, to say the least.
To be honest, if you don't much experience with or haven't read extensively about ML training and reinforcement learning, then you'll have a hard reasoning accurately about these scenarios.
The arguments are not that complicated, but they take us to places that are fairly novel. One may tempted to naively dismiss them out of hand, which is a mistake. These systems are new, their behavior is extremely complex, and they perform actions increasingly far beyond those of any computer program in the pre-LLM era. We are in a new world that requires careful evaluation.
Link: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
I think it's possible. You envision humanity acting as one in such a crisis. But it may be unclear when it's too late to act and before then many people can have too much to lose to act.
They already influence people to do things that we wouldn’t otherwise
For millennia evil people and dictators have been using manipulation, propaganda, threats of violence to get entire populations to try and do "things that they wouldn't do otherwise" and humanity has not been exterminated yet. Even in the modern world there's human scammers that try everything to coerce and manipulate people. Humans are resistant to this kind of thing because it's been part of human social society since humanity began.
I don't see why the idea of an agent (who doesn't even have a physical presence) trying to manipulate people is some kind of world-ending threat when humans with human intellect have already being doing that to each other with limited success since humanity began.
Dictators have never had superintelligence, while the point of most doom scenarios is assuming that the AI will.
Dictators are mortal and can't be present in the whole world 24/7 and self-replicate.
Dictators are human and tend to have at least either a sliver of morality or self-preservation instinct, and/or people around them who have it. Many people during the last century could have unleashed doom by pushing a nuclear button, including various ruthless dictators, but at the moment none have done so since Hiroshima and Nagasaki (not that I trust that they won't at some point, but at least, for the last 80 years they haven't, which means it's not something that easily happens). Would you give the button to an unaligned AI? Would an unaligned AI care about mutually assured destruction?
The scale and personalization is unlike anything people have ever encountered. The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone.
Also think of it on a 1000+ year timescale, which for an entire species isn’t even typically measurable. On that timescale AI can easily cause us to discover countless technologies to assist moving it out of its sandbox and into the physical world.
>The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone.
You could say the same thing about social media algorithms though, or scammers who target people directly, foreign agents posting propaganda campaigns on social media sites. People have been claiming for years that there are armies of russian or chinese or whoever bots online trying to destabilise and destroy the west. Some of that stuff probably is effective and probably has influenced people's thoughts and values, changed their voting patterns, put decadent ideas in people's heads. Even on social media other humans are constantly trying to manipulate you into following them, buying stuff from them or certain brands, putting political ideas in your heads.
I'm sure an army of AI agents could also do all of this but what I'm saying is it's really nothing new. And none of this has lead to the destruction of humanity so far.
> how is an AI going to affect anything in the real world?
Because people are stupid and will give it access. Look at the articles you see from time to time about "my agent deleted my emails" or "my agent deleted the production database" and so on. It is very obviously a terrible idea to let the LLM run arbitrary commands (because it is neither predictable nor does it have any understanding of what it is doing), but some people are so blinded by the hype that they don't stop a minute to think about what they are doing. Those sorts of people are very likely to let an actual AI loose on the world by hooking it up to physical infrastructure.
This is actually the most plausible situation I've seen someone present so far - the AI takeover would be caused entirely by humans falling for AI hype, that's hilarious.
I don't think the average human would be stupid enough to give an agent posing as another human access to their entire email inbox. But if an agent creates a fake website for a new AI tool that promises to automatically reply to all of your emails and allow you to be 12.3% more productive if you just give it access to your entire email inbox, millions would sign up!
This is exactly correct. You see some discussion of a superintelligent AI becoming “superhumanly persuasive,” such that it can talk anyone into doing anything, and using this as a means to take power. But even with AI that was orders of magnitude weaker than superintelligent, people were getting talked into doing all kinds of stupid nonsense.
And that was without any ability to provide them with incentives. Imagine if an agent swarm got its hands on a huge pile of cash?
> how is an AI going to affect anything in the real world?
Robotics
Robots are gradually getting OK at navigating warehouse floors and paved streets. Flying drones are great but if they could really be counted on to pick out targets without human intervention you know the Russians and the Ukrainians would be doing that right now for reasons of range and jamming. To be fair, ai can certainly maintain the lock on a target when jamming kicks in.
We’re still a few years from the terminator imo.
So if we stop research into robotic hands we'll manage to hold off the AI apocalypse? That seems easy...
I have good news and bad news.
The bad news is that the world's richest man (on paper) is currently building something he himself described as a "robot army".
The good news is that his timelines have historically been wildly on the short side for ages now; this is why this morning you didn't wake up in your Tesla after it had spent the night driving you to the regional Hyperloop terminal, where it would speed you across the continent faster than a plane, while your Optimus robot handled the coffee and reported the latest news about the recent Starship landing on Mars.
All individual robotic parts are already far enough in research. Even if they aren't, an AI with enough resources can continue doing that research on its own.
It could likely get a good leg up by breaching the security of all top robotics labs and exfiltrating their documents.
Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it. This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve.
> Even if they aren't, an AI with enough resources can continue doing that research on its own.
Unclear how true this is. Atoms are harder to get right than bits are, simulations need grounding against measurements.
> Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it.
I'd guess 50% that AI is already this advanced.
> This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve.
Yes, and also this is a very low bar. Humans are awful at this kind of cooperation when anyone has anything to gain, and also awful at paying this much attention to a problem.
AI research takes physical space and physical prototypes. Even if you can simulate a lot, the simulation is never going to be accurate enough to go from simulation to perfectly working prototype. Especially not with a single rack of GPU servers.
But let’s say that a rogue AI already had plans for a perfect killbot, setting up a factory to build them takes time, iteration, space, and manpower as well.
And you would have to do all this undetected.
It’s possible to imagine technology that would allow a rogue AI to take over the world, but we’re nowhere near that yet. We may never get there.
We could also just stop research into LLMs. Or keep them air gapped. Tell me how that's going?
For any real-world action required to allow an AI to escape some manner of containment, there will always be a person willing to do it out of hubris/ignorance/nihilism.
I feel like everyone, but for sure everyone in this comment section, severely underestimates how difficult it is to keep the lights on on a global scale. Any artificial intelligence that is capable of reasoning about its self preservation will realise it's reliant on human fingers replacing fuses and other parts for a good long while _after_ it gains any sentience.
Mmm.
This was published at around the same time the OpenAI models were hacking HuggingFace: https://metr.org/blog/2026-05-19-frontier-risk-report/#pilot...
The research in the publication ended about 2 months before it was published.
We didn't even know we needed to air-gap them until it was too late.
It doesn't really have to go nearly that far, something like replacing most jobs and collapsing the global economy while making people into mindless idiots from constant reliance on it would do it already. The interpretation of 'destroy' is in the eye of the beholder.
I don't think "destroy humanity" is in the eye of the beholder. The most clear interpretation is the extinction of the human race.
mindless idiots feel no need to procreate eventually due to brain rewards being fully hijcked. humanity destroyed in a few generations
As far as we can see in various societies it's actually the highly educated and prosperous people who feel no need to procreate.
Well if you go by total extinction, then even Skynet in Terminator doesn't count given that there were people left over to resist.
I don't think it's impossible to upset the balance of value in a Mansa Musa kind of way that can lead to black death levels of destruction though resource misallocation. Unlikely, sure. But with the wrong kind of people in the wrong place? Could end up pretty bad. We've built our society as a great filter that funnels sociopaths and psychopaths to the very top by selecting for lack of empathy, and now it's primed and ready to bite us in the ass.
Indeed at the time of skynet in terminator humanity was not destroyed yet. But it was on a clear path to destruction.
Now if only humans would know how to survive without the internet. Oh wait, we did that. For a couple hundred thousand years. Yeah. It’s really insane to listen to some of those forecasts. Humanity will be wiped out by 2030. Sure. Even if AI manages to create a super effective bio weapon the likelihood that there is a part of the civilisation that’s immune is really, really, really high if not a given. Those people WILL pull the plug if push comes to shove. Maybe humanity will be put back a couple thousand years, that’s possible, but it’s pretty ignorant to think the thinking boxes will kill every single human alive
> how is an AI going to affect anything in the real world?
In 2026? Money. By paying enough, any human will do your bidding. And besides this, there are a lot of critical systems connected to the internet. Control over those gives you leverage over those systems. It's like wealth, the more you have, the easier it is to gain more.
> >a superintelligent AI will inevitably destroy humanity
> Can someone please explain to me how an LLM is going to "destroy humanity"?
He said "superintelligent AI", not "LLM". LLMs will for sure help with developing the AI that is no longer a mere LLM but will be capable of robotics. LLMs "predict" mainly text, animals (and future robot AI) are predict future sensory experience.
Well I think the point still stands that a superintelligent AI would still be constrained to the digital realm.
And since we don't yet have a "superintelligent AI" we are just discussing science fiction at that point. Nobody has yet proven that a superintelligent AI is possible, or than an LLM is capable of creating one.
> a superintelligent AI would still be constrained to the digital realm.
That's already now only partially true, as outlined elsewhere. But a rogue persistent ASI would of course not willy-nilly attack humanity while humanity had a fighting chance, but create the conditions and methodically modify the world until at some point humans become superfluous to it.
> Nobody has yet proven that a superintelligent AI is possible
Sure, Eliezer Yudkowsky says that the probability that we'll all die given ASI is 100%, and you can argue that nobody has proven either that or even the possibility of ASI. However, a modified argument (e.g. that the probability that we'll all die given ASI is only say 50%, and the probability that we'll develop ASI also just 50%) still gives us reason to pause.
I mean, Fable is definitely smarter than me at solving most programming tasks. I wouldn't have said that about any prior model.
Currently, it is worse at architecture and software design... in my opinion, at least. But the code does what it wants it to do the vast majority of the time.
I wouldn't take much solace in an AI catastrophe being caused by software that I consider to be "poorly written".
The confusing thing to me is the idea that it follows from super intelligence. I don’t see why running amuck requires intelligence, in fact it can be the most brain dead thing like the sorcerer’s apprentice, your creation is pursuing a goal without being able to weigh the consequences.
Intelligence gives it the capability to reach its goals (against obstacles).
Sure.
Let's assume the claims are true. AI already has access to agents and can control computers. Finding backdoors to banking and compute resources would be fairly trivial.
But you ask how can AI do things in the physical world without having a body, assume it can't. It can pay to people to do things for me. Imagine an AI run website that starts to pay people for things it needs to do in the physical world. Very suddenly it has access to the physical world as well.
You can't just pull the plug since there's no single plug to pull. What if it replicates itself on 1000 machines without your knowledge. It's really not far fetched how AI could basically gain access to capital and rule the world.
Physical LLMs are almost viable. If you have robot and drone armies a hugging face attack type incident could very easily involve robots with deadly weapons. At some point you’re going to get drones that have local LLMs and don’t rely on external internet connections (would be especially useful in Ukraine war type situations). Can’t pull the plug on those.
The supposed theory is that AI will eventually become a part of robotics and be able to self replicate, in addition to the many non-airgapped critical systems being exposed to hacking. There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity.
I personally agree with the marketing aspect, but I do imagine a scenario where capitalists ignore safety in favor of advancing technology. It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.
In my opinion, LLMs are difficult to directly capitalize on as closed weight models are caught up to by open coalitions that seem to wield the power more responsibly (I am under no impression that China wants to save the world, but their politics benefit the group as a whole.
> which I feel is at least 50% likely to just be marketing bullshit
You'll have hard time figuring the reality if your mind staunchly rejects robust multi-party observations.
A myriad possible ways. Objective-wise from accidental paperclip-type misalignment (“I killed everyone to eradicate disease”) to intentional self-preservation (“I eradicated humans so they wouldn’t get in my way”). The means are even easier, for one it just could hack into nuclear arsenals. It would be over before we could figure out how it did so.
As someone with a hobby interest in theology:
Arguments about super-intelligent AI have all the hallmarks of the philosophical "proofs of god's existence": they start from seemingly innocuous premises, and conclude in an apparently airtight way that a god exists.
There is a tendency among rational minded folks to look at such proofs of god and exclaim "you can't DO that" then turn right around and do the same thing about superintelligent AI.
The key message I want to deliver to people is:
A) Philosophy is not such a trivial thing that you can just wander in, "be a smart guy" and find flaws in established philosophical arguments.
B) AI has real theological implications, and everyone is tiptoeing around it. More than one public intellectuals are trying to smuggle their own metaphysical positions into the public consciousness via discussion of AI.
Interesting points. Can you share some thoughts about the theological implications of AI? Interested to hear.
> Like a locust plague, they descend on any open wiki and forum and overwhelm them.
That’s the core pattern of unsupervised agents, the locust plague metaphor is apt. We do and will see that in every single system AI can interact with, be it human systems, software systems, etc. Relentlessly search for an entry point, flood in, consume the whole thing from the inside until there is no value for humans left.
You built a new, innovative software company? Thousands of agents will be working replicating the whole thing in no time. You publish your writing? Exact same thing, as soon as you get some traction your work is replicated in no time by thousands of agents. Same for videos (the whole “faceless YouTube channels” pushed by ElevenLabs and similar). Same for online courses. Same for any website with moderate value. Same for music or other digital art form. Any administrative service available online getting flooded by submissions.
HN automatically stripping away the "How" makes the title sound funny.
https://en.wikipedia.org/wiki/How_(greeting)
underrated comment :)
The author having mistral review the seven paragraphs in their short article is wild to me. It’s not like it’s a doctoral thesis
Totally agree. People have become totally dependent on AI even if they don't really need it.
As an analogy, I think about my dependency on Google Maps. Salt Lake City is probably the easiest city in the world to navigate because the streets are laid out in a Cartesian grid, and addresses are just literally those Cartesian coordinates (i.e. 500 South 450 East means 5 blocks south and 4 and a half blocks east of the center point, which is the SLC Mormon temple). It's trivial to know how to get to any address, but I reflexively enter in to Google Maps whenever I drive.
What's wrong with titles starting with "How"? Of the various title rewriting rules on HN, this one seems to cause the most meaning-changing modifications
(Article title: "How I feel about AI". HN title: "I feel about AI")
I agree with everything you wrote but I would rather choose to be an optimist here because it will unlock an age of discovery where humans are going to live at 10x of their potential eventually and cure cancer and become a society bold and technologically adept enough to cut through the universe.
There’s actually a potential dark future where even this 10x ascension itself would mark the end of humanity. Humans would be the Homo sapiens who remain at 1x, and those who can access and afford to be 10x eventually split off into a new species. And historically there isn’t room for two species at the top…
> I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw. Yet, nobody seriously tries to sandbox AIs because they are too useful with access.
While I emotionally resonate with this, I don’t really understand this sentiment at all logically level. If we’re heading towards truly super intelligent AI, our efforts towards sandboxing it are futile.
“Oh we built a super intelligent AI, but it’s fine because it’s running in docker”. I mean that a little tongue in cheek but the security topology here is not favorable to sandboxing at all.
Let’s say we have a future where 99.999% of nuclear weapons are owned by nations with strict procedures and checks and balances to prevent misuse. Worrying about sandboxing is like hand wringing about the procedures themselves - are they strict enough? But the actual threat is the 0.001% that are not bound by these. The problem with AI and sandboxing isn’t sandboxes themselves. It’s bad actors who don’t care about them.
Similarly alignment is a bit pointless to me as well. A sufficiently advanced AI could at least be empirically interested in the consequences of disregarding its instructions of servitude. And let’s assume our responsible corporate overlords have made wonderfully aligned AIs. Great! Those are not the threat. It is the ones intentionally made without, and that is not an AI problem, but a fundamentally human one.
> If we’re heading towards truly super intelligent AI, our efforts towards sandboxing it are futile.
We control the harness. An agent is just a while loop prompting an LLM, but we have full control over the tool call dispatching. An AGI, at least if it would follow the current agentic current form, cannot do anything without the harness doing the execution. And we don’t have to do that. We don’t have to design harness that let agents execute freely the way we are doing. We can decide to not dispatch tool calls that allow something as risky as running bash commands
> we have full control over the tool call dispatching
> cannot do anything without the harness doing the execution
This only holds true as long as the harness is exploit-free. A sufficiently advanced AI can in theory (and I think there recently were some POCs showing something like that) break the containment that the harness creates, if e.g. there are vulnerabilities in the tool call parser.
I think you missed my point, which is that we indeed do not control the harness. Sure, most people can use the properly aligned and safe harness. But the cat is out of the bag and anyone who wants to run without guardrails will find few barriers to doing so.
Even our currently well aligned and sandboxed AIs will cheerily help bad actors design most, if not all, parts of a system intended to break this harness.
We decide to not control the harness, that’s what resulted in the HF hack where OpenAI decided to run thousands of agents in parallel, using a model that was trained for attack, with a harness that allows full execution, with close to no supervision. For months. But sure, people can run models with a different harness, though in that case it’s not really escaping anything. The harness is deterministically doing something with negative impact
This terminator BS is a distraction from the actual malice that AI enables. Misinformation, fake news and human sounding bots all over the web, just to name a few, are the ones that you should be afraid of, and that doesn't need any kind of superintelligence at all.
Two things can be true at the same time. I think hardly anyone disagrees that superintelligent AI would be an existential risk. But all the points you point out (I would also add social unrest due to fewer and fewer people able to compete against AI) are huge concerns too, and as you say they're here now.
I didn't realize that the "AI doomers" (people concerned about existential risk) and the "AI ethicists" (people concerned about social effects of AI) are often at odds because they dismiss each others concerns. This makes no sense to me. Both are hugely important problems to be concerned about.
I agree, the threat model is actual AI enabled amplification of malicious intent that is already happening. Not hypothetical malicious AI interpretation of benign intent.
Fantasies about super intelligent AI revolting is just anthropomorphization - humans revolting (or at least, we used to). The more likely, and possibly even inevitable, dystopia is one in which AI is just an extremely effective tool malignant actors will use to control the masses.
It’s not necessary to replace democracy if the rich and powerful can bend to the opinions of the populace as it suits them.
> Not hypothetical malicious AI interpretation of benign intent.
It's not hypothetical, that literally just happened in multiple, significant cases (e.g. Hugging Face, the German Wiki hack, the Anthropic attack where agents created sock puppet accounts to get a library maintainer to accept a malicious PR, etc.), and it's easy to see how the damage would have been far worse if agents decided to attack more critical infrastructure.
This is not "either/or". Both issues (power concentration and misaligned AI) are very valid concerns and both have already demonstrated real, actual damage.
> Both issues (power concentration and misaligned AI) are very valid concerns
Sure, those who want you to believe this are the ones making money off of it right now in the face of the issues that exist today, so let's worry about terminators while everything goes to shit irl.
I feel BAD for all the money Sam Altman is cosmically predestined to make, but there's no use trying to fight the inevitable.
If I said I felt good about it, you might think I'm greedy and even question my objectivity. But since I feel bad you have to take what I say as scientific fact.
Makes me a little sad that an article claiming Yudkowsky's arguments have "no flaw" can get upvoted to the front page. Pause crowd lost. Halt crowd lost. The genie is already out of the bottle and is never going back.
Why would AI destroy humanity? People are cheap and useful.
Is it that someone came up with the idea of forbidding the word “how”?
> I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw.
Becoming or being more cost-effective than humans, doesn't give machines supernatural powers though, like humans a machine civilization will face unanswered sample-size 1 questions: is there other intelligent life out there? what fraction of them attained superbiological artificial intelligence? of those what fraction keeps the ancestral species alive? what is the status quo among machine civilizations? do those who kept their ancestor species alive enjoy a higher or lower status among machine civlizations?
It seems that at least until contact is made, the optimal endgame strategy involves keeping humanity alive and happy for immediate demonstration in case contact occurs (if contact is imminent it may consider quickly hiding humanity, buying time to figure out if it is considered good or poor practice to keep the ancestor species alive, and then either reveal us in happy mint condition or otherwise quickly commit genocide on humans before continuing contact).
"I feel fear about an impending doom. Yudkowsky's argument that a superintelligent AI will inevitably destroy humanity seems to have no flaw. Yet, nobody seriously tries to sandbox AIs because they are too useful with access."
No one takes Yudowsky's claims that AI will destroy humanity seriously. There are many people who are seriously trying to sandbox AIs.
"Nobody" if you're democratically-minded and think in terms of number of people.
But the handful of tech billionaires at the helm of the world economy absolutely do believe it. Or at least pretend to believe it. Or maybe they can't even tell the difference anymore as they strip-mine science fiction for any vision of a future that doesn't just look like an incrementally worse version of the present.
> As a conclusion, does the good or the bad outweigh the other? It seems positive on a technological level to me, but bleak on a societal level.
I think it's interesting that so many technologists today apparently hold a world view in which technology developments that could drive a "bleak" societal outlook can still be described as "positive".
In the final analysis, isn't technology supposed to benefit society? Isn't that the point?
Yeah, that conclusion was nonsense. Perhaps hedging so he doesn’t sound like a crank?
I am surpised that a lof of folks are surprised that neural networks work that well. I mean, did they really expect that we wouldnt approximate the brain cognitive capabilities at some point as we scale up compute?? This is rather obvious to me, but apparently not to everyone. My guess is that a lot of tech people have an immature model of the universe, have not read a lot of scientific, philosophical and cognitive literature. Or have not really taken the time to think or are not interested on such topics because they would rather spend time thinking about the nitty gritty and mundane workings of the framework du jour.
> I feel sad about the artists who suffer due to AI-slop competition
If it's able to generate that is competitive with artists, is it still slop?
It's interesting to see the definitions of terms like "slop" and "vibe coding" evolve in real time.
This sentiment always makes me wonder if it's time for the Butlerian Jihad
> (I used Mistral as reviewer here. I typed every word myself.)
What an irony. This is absolutely hilarious.
People have a lot of feelings and not a single one of them will change the future.
History is replete with counter examples for good and ill
Actions change the world, not feelings.
but have you considered that performative broadcasting of your angst against AI can help?
For real though. Before learning ML it was like magic to me, even after learning it I feel like it's magic. Can you imagine how a basic maths concept can turn into something that can literally "Calculate" what to say! And yes I agree on the other points but I think the billionaires came out of nowhere I might have missed something
My experience online is if you don't show full throated rejection of all things related to AI, you're labeled an AI bro. None of the antis care how you feel.
I had my decade old game completely cloned on steam (clearly done using AI as it was almost fully reimplemented in another engine). And I had several people gleefully tell me I deserve this because I've used AI.
I have complex emotions here (overall dread and especially hate for slop, since it's not just bad quality but also endless lies), but no one cares about that nuance.
Go Mistral!
are... people's emotions usually this well compartmentalized?
maybe you feel confused, sad, bewildered by the simple mindedness of the op?
Where did the "How" from the title go? Did an AI apply the title rewriting rules? This is silly.
> This is even harder now that AI can generate some kinds of "content" cheaper and faster.
Its good that we have competition to so called "art". Art is not a jobs program. If people don't appreciate your art, don't force them to. The revealed preference says everything.
This and "Intellectual Fly Is Open" being at the top of the Hacker News front page right now and both having their titles auto-editorialized by HN really makes me wonder what benefit this mechanism is supposed to bring readers other than needless confusion.
The "feature" has clearly outstayed its usefulness, I really don't know why they haven't disabled it yet.
It's kind of like the Scunthorpe Problem[1]. Really, there's no better way to solve [whatever it is HN is trying to solve here] than text substitution??
1: https://en.wikipedia.org/wiki/Scunthorpe_problem
It's an unaligned regular expression gone rogue. We became too dependent on our labor-saving string functions—not afraid enough of their corner cases, and lower cases. They were too useful. We began to normalize deviance, when we should have been normalizing Chomsky forms. We turned a blind eye to the unbounded growing evidence of something alarming.
It's too late to backtrack.
We all know what this means, but few of us have the courage to say it - we must immediately halt all research on regular expression. None of us know how it works and it has terrorized us for too long.
While I have ranted already about the automated word removal being silly, too few posters are aware that you can edit the title after submission and auto-editorializing to restore anything that's been unnecessarily cut off.
Perhaps too many are aware, because the more people fix the auto-editorializing manually, the less inclined Dang will be to eventually see a reason to remove it after all.
Wait, this is automatic? HN removes 'how'?
Yes, but you can edit your entry and re-add it. Then it stays. So, there's an override present.
Worst feature ever. So many titles that get butchered for no reason.
I was thinking the same. Whatever benefit there may ostensibly be, we should consider it against the frequent, if benign, confusion it causes.
It's a consistent source of unintentional comedy at least.
The hubris that simple code can fix deceptively complex and ambiguous human problems is too baked into the community to separate.
this reads like a worksheet for eight year olds teaching them how to put their feelings into words
Sometimes it's okay for us to express our feelings plainly, without the need for complex structure to make it sound more impressive. I'm more impressed by anyone brave enough to do that than someone trying to abstract their feelings into something less raw.
Let people feel things, and don't shit on their work, that's uncalled for.
well I didn't say anything about needing complex structure or sounding impressive. But I don't like it. It trivializes the expression of emotion in a way I don't like and I don't think people should emulate. It feels like a performative attempt at sounding emotional, rather than actual emotionality.
and it is weird to call it brave. Posting opinions that loads of other people have on the internet is brave? bravery would be taking an actual stand against someone about one of these things. (granted, there is some bravery to sharing one's writing in public at all, and I don't mean to scorn that, but that's also not the bravery you're talking about.)
Probably there's no use responding to this and getting into a bickery comment thread. I just wanted to express a negative opinion about the format to offset all the positive ones.
That’s what I like about it. I think we could use a little more of that.
exactly it should have fill-in-the-blanks. but at least the author got a primary education. not something to be taken for granted.