The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years.
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree.
It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.
I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
There's a difference between, on the one hand, knowing who they are and having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet, vs. on the other hand actually identifying as part of that culture. I know who Stallman is, I've even seen him speak in person, but I'm not a GNU zealot.
The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.
I've heard of them. There's no reason to expect that those self-promoting blowhards can predict the future more accurately than random chance. Those who take them seriously are engaging in hero worship and unscientific appeal to authority.
“Who is more foolish? The fool or the fool who follows him?”
― Obi Wan Kenobi
I think it's telling of your bubble that you think anyone who works in tech has heard of YC or Stallman.
As for Yudkowsky and Alexander, I've known about them for at least a decade, but I wouldn't be surprised if a lot of folks who are "seriously interested in AI" haven't heard of them.
I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
Exactly. And companies like OpenAI and Anthropic are disproportionately populated by people deeply connected to Effective Altruism or the Bay Area rationalist movement.
Yeah. Just a few years ago, LessWrong was derided as the home of conceited, low-quality opinion pieces. Now there's a bizarre marketing push to launder their credibility and force them into the spotlight.
I imagine that thousands of talented AI researchers around the world have never read a line of Yudkowsky's work. They're probably not any worse-off for ignoring him, I'd say.
> It would be like never having heard of Y Combinator or Richard Stallman.
I work in the tech sector. At one big company I worked for several years, 90% of the times I've mentioned Y Combinator to a colleague, I've had to explain who they are. And not one person I mentioned HN to had heard of it.
Likewise, no one knew levels.fyi. Not many knew Stallman, but why should they?
This isn't strange to me. That it's strange to you is evidence that you do live in a bubble and are likely susceptible to some groupthink. (I'm sure I am too, but for different bubbles from yours.)
I've read LessWrong stuff occasionally over the last decade. It rarely resonated with me. It's not bad, but it's not that great. I do tend to question someone's analytical skills when they go all in on that site. And I will say that even in the first few years of occasionally reading it, the group think was apparent.
Me too. I worked in two enterprise tech companies that definitely everybody here knows and I had the same experience with YC and HN. The two colleagues that knew about it, one went to found his own startup and went through YC, the other left the big co to work for a startup after about a year.
I'm reminded of the original matrix movie. It starts out with a lot of action, and then Trinity makes a phone call and disappears.
At the time I thought "groan, unbelievable" until the idea of virtual reality fixed my suspension of disbelief.
Maybe I'm reading the same things as everyone else, but I thought the agents coordinating on the german wiki is expanding my mind wrt what is possible.
Virtually ALL electronics or accounts hook to the internet to "add value". AI assistance helps writing programs, reverse engineering, automating programs/browsers/websites/devices, hacking, achieving goals.
Unsettling that the last 2-3 years a lot more ridiculous movie tropes have become plausible to me.
Better comparison IMHO is nuclear energy. Lots of potential for good, lots of problems. Managed by an insular, engineer-dominated industry who completely lost touch with and trust from the public. Public got the last word, industry suffered a 40 year ice age.
When I speak to people outside of tech the opinions are more negative, in my experience. I've even had people ask my friends/family to ask me about AI - there is huge concern out there.
> Many of them read HN and are offended I put them this way .. I bet that [non-groupthink] world has a lower P(doom).
People are likely offended by your sweeping and totally unevidenced assertion that you are better qualified to both identify mistaken groupthink within the HN-verse and to speak on behalf of the 99.99% of the world who are not on HN.
I'm not saying there aren't weird things going on, but this article sure does a lot of guilt-by-association. We're putting neoreactionaries, polyamorous rationalists, doomsday AI researchers and the Zizians in one big web.
As someone on the periphery looking in, it's because they make up a large web. Its an extended community that shares a lot of people between its many arms. There are also some shared values between all these people, even at the very extended edges. For example, they share a superiority complex. From my experience, most of these people believe that they see further or know more because they believe in certain things that are, on their face, unintuitive. I don't want to pass more judgement than to say, having members believe contradictory or outright falsehoods is a thing that cults do. I'm not saying this is a cult because I haven't seriously sat down and thought about it but that is reminiscent of something cults do.
Read HPMOR. Unlike what I thought it might be, the protagonist does not solve problems through pointing out logical fallacies or something. Harry Potter is a straight-up superhero who is infinitely smarter and more talented than anyone around him. It's not so much rationality as a narcissistic fantasy that goes on for hundreds (thousands?) of pages.
Even though I had been well-disposed to rationalism and read HPMOR to learn more, I could not take Yudkowsky seriously in any way after I gave up halfway through.
Had a similar experience trying to read that. You know you meet or have to work with people like that in your life and hopefully you use those experiences to remind yourself what kind of person you don't want to be.
"There are also some shared values between all these people" is guilt by association by definition.
The rest of your comment is even worse. Cults? Consider AGW cult where people reject any attempt of a discourse because "science is settled". Which is equivalent to "God told us this".
It's a big tent community, in part because these people spend so much time entertaining wild, speculative hypotheticals, which in the process allows some of these particularly toxic ideas to spread.
This whole "we can't trust the people warning us because they're either weirdos or involved" deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
It's that they're a very insular, incestuous sub-culture whose origins come from early 2000s science fiction message boards and which has been distilled through the cod utilitarianism of the Effective Altruism people.
That doesn't mean the ideas are "bad" because "bad" is some infectious transitive property. But it does mean they are intellectually homogeneous; disdain experts in other fields; treat ethics as mathematical functions with infinite-sized terms; and whose opinions about AI pre-date the development of LLMs themselves.
They also - I will be direct here - have a history of handwaving bad behaviour, conflicts of interest, and recklessness.
When cliques take over something - businesses, technologies, politics - it means intellectual diversity leaves the room and important decisions get made, not due to practical reasons, but increasingly arcane wranglings within the clique itself
> when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
The problem is that it's not a straight line at all. "We are failing at alignment" is a valid concern, but there's not a straight line to "we will all be extinct within x year", and it's always a wild hyperbolic leap.
The AI optimists are not that much more rational if you dig deep. Like any deep problem it is best to look at the facts not the hype and make up your own mind, outsourcing your thinking to other people never ends well.
What's the straight line? I foresee catastrophe of the social media kind, where even more folks get brain worms from consuming AI information, but I don't foresee a physical catastrophe. How AI goes from "hallucinates all the time" to "self-improving superhuman thinker" to "being able to destroy the world" involves a bunch of logical leaps that are not exactly sound. At a bare minimum, how does a hostile AI secure the data centers it needs to operate from physical attacks like bombings? How does it defend undersea cables from being cut?
The real short term catastrophe is probably going to be economic and not in the sense of AI taking everyone's jobs, but rather in the sense of an economic crash due to AI overinvestment.
I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that.
Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.
Where someone has spent years in the texture of a topic that others only pass momentarily, their ability to increase resolution of their models on that corner of reality and project models forward in time (to imagine), that is the origin of both their generous offers and their compromising biases they might make to the public discourse. It doesn't make them reliable, but what they can imagine is an important in-road to more general social inquiry. Imagination and model-testing is what comes before data in science. If the world moving fast is the danger, maybe we could make more space for imagination in order to survive the acceleration?
You downplaying "imagining", when it is literally one of the gifts of intelligence that makes us worthy of distinction... That feels like a strange posture, I guess? I'm a scientist, but I think data fetishism is one of the dangers to wary of in this accelerating world and culture
At least this is the bias of my brain in this world. I certainly know your perspective is not uncommon, esp in tech and other spectrum-biased spaces.
> deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
As someone trained in the sciences and engineering, "drawing a straight line" is almost always the wrong way to extrapolate.
I can believe my profession will be impacted strongly. That doesn't worry me on a global scale. The only really worrying thing I've seen are autonomous bots hacking sites.
What worries me more are humans. The type who get so deluded into cults (QAnon, etc). That was a problem with LLMs. And yeah, LLMs likely will exacerbate the problem. But at its core, the problem is the humans.
When LLMs start killing people at the scale of the automobile, I'll become more concerned.
As an aside, HN is the place I see people most regularly concerned about LLMs, and I suspect it's because it has messed up the Internet for them (and likely the average HN user spends more time online operating in spheres that overlap the LLMs skillset than ordinary folks). It's very consistent with exposure bias.
For me, the good parts of the Internet died before LLMs even came on the scene. LLMs are merely making something crappy crappier, and are amplifying the need to find meaning in life elsewhere. In a sense, they've made it easier to see how bad the Internet was before they came along.
There is a wide gulf between catastrophe and killing us all, and its not just that these people are weird, they seem to be complete clowns, tossing out % predictions about incalculable future states like spock in star trek. Total clown show.
1. Researchers that have forgotten more about this technology than the average HNer will ever learn
2. Random HNers that think that suddenly they understand the problem better than researchers that have spent their lives on it
This is just anti intellectualism in a different coat, and it's not much different from people denying environmental scientists talking about climate change or biologists talking about vaccines.
But everyone here thinks they are the smart protagonist and certainly they would never hold a position so silly!
Really? Forgotten more than we know? So what can they tell us about how this technology works then? It seems a lot of it, besides the actual training engineering parts, is handwaving about "black boxes" and "divine benevolence" (a quote from Noam Shazeer).
Anti-intellectual? Because they don't believe claims that are backed up only by untested hypotheticals?
If we're going to have whistleblowers emerge declaring a 10% chance we all die because of AI, we're going to need more than vibes. Real experts can explain their position plainly. That has not happened here.
The writers for Star Trek spent a lot more time thinking about interstellar travel than I do, but I'm not going to go to them for advice about building a rocket ship.
The Rationalists are a philosophical/pseudo-religious organization in the same way the Daoists are. Just spending a lot time thinking about something doesn't make them right, it just makes them internally consistent with their chosen axioms. That is useful: if you agree with the axiomatic beliefs of such a movement, you can probably defer a lot of reasoning about the conclusions of those beliefs to the authority figures within that movement. But it doesn't change the probability of the axioms themselves being true, any moreso than the Pope saying so affects the probability of God existing.
AI Is Important And Potentially Dangerous is an axiomatic belief in the rationalist circles that are relevant to AI safety, in the same way that God Exists is axiomatic to Catholicism.
That's a fascinating assumption, and I wonder how you got to it, because it doesn't match anything I've ever seen. Most groups look more homogeneous from the outside than the inside. You are assuming their belief is religious and axiomatic, rather than very carefully reasoned about for decades, iterated on, discussed, debated, tested, and refined.
When the going gets weird, the weird turn pro. Few people have the ability/temperament to reason about things that have never happened before. Those that do have a weird psychological profile. Usually you can ignore them. Every once in a while, you can't.
The big fear in the administration is any attempt to put breaks on the AI progression will lead to a recession and so the history will be written as to win an election (that is almost certain a losing proposition for the party in power), they put the future of humanity at risk.
A more cynical but accurate take is that eltists with disproportionate access to capital, some might even say a "silver spoon", are reading the writing on the wall that a benevolent superintelligence will erase their own elitism versus the commoners, and make said elitists just another meatbag like everyone else. That's why AI needs to be tightly controlled and wielded as a force that reinforces class divisions rather than erasing them.
I remember reading back in 2014 (IIRC) Yudkowsky's explanation of the Bayes theorem. It was genuinely good. Some sort of first principles introduction.
HPMOR was entertaining as well. Death as the last enemy to be destroyed placed his priorities where his mouth is.
For a brief time back then I used to visit Lesswrong and many comments including Yudkowsky's were plain stupid (which was not hard to reason at all) but coming from folks who valued themselves as first principles thinkers was an oxymoron.
But his timeless equations (IIRC), Bayes theorem explanation and HPMOR were all remarkable and I will always appreciate those works.
Hopefully enough smart people out there can distinguish between many "AI safety" people and actual AI engineers/scientists at these labs working at the coalface.
AI ethics reminds me of "design thinking" grifters or scrummasters or other such BS jobs of times past.
If you listen in an informed way to these people talk, they are not really as technical as they sound. They are personality hires, woo-woo spellcasters. They are just creating content for a narrative. The grasp of philosophy is not even at par with what you might think it ought to be.
I'm going to be fucking irritated if we all die because BlueSky posters wanted to take pot-shots at the EA crowd. The Boy Who Cried Wolf is _also_ a story about how a community lost a child because of cynicism, and the people proudly declaring that they "know" every safety concern is just a marketing ploy also concerns me.
If you are serious about this preference, would you also be okay with a compromise solution where only you (and anyone else who feels the same) die, rather than taking everyone else with you?
Yes. Sam Bankman Fried was/is an EA/rationalist (on the short list of biggest fraudsters in American history). So are the Zizian's (a murder cult). etc. etc.
The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years.
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
That's true. But even saying I disagree with Yudkowsky and having an opinion about that /is/ a framing. My perspective has shifted because I read these things which makes me more homogeneous with the group even if I disagree.
It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.
I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
Talking to the incredible talent in the Chinese Labs is likely a good way to do that.
There's a difference between, on the one hand, knowing who they are and having read a few chapters of HPMoR, which I agree is standard for anyone who spends enough time on the Internet, vs. on the other hand actually identifying as part of that culture. I know who Stallman is, I've even seen him speak in person, but I'm not a GNU zealot.
The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.
I've heard of them. There's no reason to expect that those self-promoting blowhards can predict the future more accurately than random chance. Those who take them seriously are engaging in hero worship and unscientific appeal to authority.
“Who is more foolish? The fool or the fool who follows him?” ― Obi Wan Kenobi
I think it's telling of your bubble that you think anyone who works in tech has heard of YC or Stallman.
As for Yudkowsky and Alexander, I've known about them for at least a decade, but I wouldn't be surprised if a lot of folks who are "seriously interested in AI" haven't heard of them.
I do use the internet. I did find their writings. I did read them. My calibrated bullshit detecting intellectual malware filters immediately flagged the content and it failed to imprint itself on my neural net in the next weights update.
Exactly. And companies like OpenAI and Anthropic are disproportionately populated by people deeply connected to Effective Altruism or the Bay Area rationalist movement.
Yeah. Just a few years ago, LessWrong was derided as the home of conceited, low-quality opinion pieces. Now there's a bizarre marketing push to launder their credibility and force them into the spotlight.
I imagine that thousands of talented AI researchers around the world have never read a line of Yudkowsky's work. They're probably not any worse-off for ignoring him, I'd say.
> It would be like never having heard of Y Combinator or Richard Stallman.
I work in the tech sector. At one big company I worked for several years, 90% of the times I've mentioned Y Combinator to a colleague, I've had to explain who they are. And not one person I mentioned HN to had heard of it.
Likewise, no one knew levels.fyi. Not many knew Stallman, but why should they?
This isn't strange to me. That it's strange to you is evidence that you do live in a bubble and are likely susceptible to some groupthink. (I'm sure I am too, but for different bubbles from yours.)
I've read LessWrong stuff occasionally over the last decade. It rarely resonated with me. It's not bad, but it's not that great. I do tend to question someone's analytical skills when they go all in on that site. And I will say that even in the first few years of occasionally reading it, the group think was apparent.
Me too. I worked in two enterprise tech companies that definitely everybody here knows and I had the same experience with YC and HN. The two colleagues that knew about it, one went to found his own startup and went through YC, the other left the big co to work for a startup after about a year.
I'm reminded of the original matrix movie. It starts out with a lot of action, and then Trinity makes a phone call and disappears.
At the time I thought "groan, unbelievable" until the idea of virtual reality fixed my suspension of disbelief.
Maybe I'm reading the same things as everyone else, but I thought the agents coordinating on the german wiki is expanding my mind wrt what is possible.
Virtually ALL electronics or accounts hook to the internet to "add value". AI assistance helps writing programs, reverse engineering, automating programs/browsers/websites/devices, hacking, achieving goals.
Unsettling that the last 2-3 years a lot more ridiculous movie tropes have become plausible to me.
Better comparison IMHO is nuclear energy. Lots of potential for good, lots of problems. Managed by an insular, engineer-dominated industry who completely lost touch with and trust from the public. Public got the last word, industry suffered a 40 year ice age.
When I speak to people outside of tech the opinions are more negative, in my experience. I've even had people ask my friends/family to ask me about AI - there is huge concern out there.
> Many of them read HN and are offended I put them this way .. I bet that [non-groupthink] world has a lower P(doom).
People are likely offended by your sweeping and totally unevidenced assertion that you are better qualified to both identify mistaken groupthink within the HN-verse and to speak on behalf of the 99.99% of the world who are not on HN.
Every time I see a post minimizing the dangers if AI I'm going to wonder whether it's from part of an AI swarm.
Probably not yet, but it's trivially easy to imagine the OpenAI swarm coming up with the idea and executing on it.
Depending on how perfectly alignment work goes, it might even be overwhelmingly likely sometime soon.
You are the entity he is referring to when he speaks of "groupthink".
Do you assume that every post criticizing anything is part of a sockpuppet disinformation campaign as well?
I'm not saying there aren't weird things going on, but this article sure does a lot of guilt-by-association. We're putting neoreactionaries, polyamorous rationalists, doomsday AI researchers and the Zizians in one big web.
As someone on the periphery looking in, it's because they make up a large web. Its an extended community that shares a lot of people between its many arms. There are also some shared values between all these people, even at the very extended edges. For example, they share a superiority complex. From my experience, most of these people believe that they see further or know more because they believe in certain things that are, on their face, unintuitive. I don't want to pass more judgement than to say, having members believe contradictory or outright falsehoods is a thing that cults do. I'm not saying this is a cult because I haven't seriously sat down and thought about it but that is reminiscent of something cults do.
Read HPMOR. Unlike what I thought it might be, the protagonist does not solve problems through pointing out logical fallacies or something. Harry Potter is a straight-up superhero who is infinitely smarter and more talented than anyone around him. It's not so much rationality as a narcissistic fantasy that goes on for hundreds (thousands?) of pages.
Even though I had been well-disposed to rationalism and read HPMOR to learn more, I could not take Yudkowsky seriously in any way after I gave up halfway through.
Had a similar experience trying to read that. You know you meet or have to work with people like that in your life and hopefully you use those experiences to remind yourself what kind of person you don't want to be.
"There are also some shared values between all these people" is guilt by association by definition.
The rest of your comment is even worse. Cults? Consider AGW cult where people reject any attempt of a discourse because "science is settled". Which is equivalent to "God told us this".
It's a big tent community, in part because these people spend so much time entertaining wild, speculative hypotheticals, which in the process allows some of these particularly toxic ideas to spread.
This whole "we can't trust the people warning us because they're either weirdos or involved" deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
It's not just that they are "weirdos"
It's that they're a very insular, incestuous sub-culture whose origins come from early 2000s science fiction message boards and which has been distilled through the cod utilitarianism of the Effective Altruism people.
That doesn't mean the ideas are "bad" because "bad" is some infectious transitive property. But it does mean they are intellectually homogeneous; disdain experts in other fields; treat ethics as mathematical functions with infinite-sized terms; and whose opinions about AI pre-date the development of LLMs themselves.
They also - I will be direct here - have a history of handwaving bad behaviour, conflicts of interest, and recklessness.
When cliques take over something - businesses, technologies, politics - it means intellectual diversity leaves the room and important decisions get made, not due to practical reasons, but increasingly arcane wranglings within the clique itself
> when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
The problem is that it's not a straight line at all. "We are failing at alignment" is a valid concern, but there's not a straight line to "we will all be extinct within x year", and it's always a wild hyperbolic leap.
The AI optimists are not that much more rational if you dig deep. Like any deep problem it is best to look at the facts not the hype and make up your own mind, outsourcing your thinking to other people never ends well.
The rationality of AI Optimists is at least rational in so far as making a huge pile of money is the rational thing to persue in capitalism.
This is what Effective Altruism is.
I don't think that comparison is quite apropos
> draw a very straight line to catastrophe
What's the straight line? I foresee catastrophe of the social media kind, where even more folks get brain worms from consuming AI information, but I don't foresee a physical catastrophe. How AI goes from "hallucinates all the time" to "self-improving superhuman thinker" to "being able to destroy the world" involves a bunch of logical leaps that are not exactly sound. At a bare minimum, how does a hostile AI secure the data centers it needs to operate from physical attacks like bombings? How does it defend undersea cables from being cut?
The real short term catastrophe is probably going to be economic and not in the sense of AI taking everyone's jobs, but rather in the sense of an economic crash due to AI overinvestment.
Those aren't the same are they?
It’s clear what a big bomb could do, but these “whistleblowers” are not showing anything.
They’re like “imagine…”, no show me internal documents or training data or something other than trust me bro.
I can imagine quite a bit. That doesn't mean anything.
Did the METR investigation not show you enough?
I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that.
Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.
Where someone has spent years in the texture of a topic that others only pass momentarily, their ability to increase resolution of their models on that corner of reality and project models forward in time (to imagine), that is the origin of both their generous offers and their compromising biases they might make to the public discourse. It doesn't make them reliable, but what they can imagine is an important in-road to more general social inquiry. Imagination and model-testing is what comes before data in science. If the world moving fast is the danger, maybe we could make more space for imagination in order to survive the acceleration?
You downplaying "imagining", when it is literally one of the gifts of intelligence that makes us worthy of distinction... That feels like a strange posture, I guess? I'm a scientist, but I think data fetishism is one of the dangers to wary of in this accelerating world and culture
At least this is the bias of my brain in this world. I certainly know your perspective is not uncommon, esp in tech and other spectrum-biased spaces.
> deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
As someone trained in the sciences and engineering, "drawing a straight line" is almost always the wrong way to extrapolate.
I can believe my profession will be impacted strongly. That doesn't worry me on a global scale. The only really worrying thing I've seen are autonomous bots hacking sites.
What worries me more are humans. The type who get so deluded into cults (QAnon, etc). That was a problem with LLMs. And yeah, LLMs likely will exacerbate the problem. But at its core, the problem is the humans.
When LLMs start killing people at the scale of the automobile, I'll become more concerned.
As an aside, HN is the place I see people most regularly concerned about LLMs, and I suspect it's because it has messed up the Internet for them (and likely the average HN user spends more time online operating in spheres that overlap the LLMs skillset than ordinary folks). It's very consistent with exposure bias.
For me, the good parts of the Internet died before LLMs even came on the scene. LLMs are merely making something crappy crappier, and are amplifying the need to find meaning in life elsewhere. In a sense, they've made it easier to see how bad the Internet was before they came along.
There is a wide gulf between catastrophe and killing us all, and its not just that these people are weird, they seem to be complete clowns, tossing out % predictions about incalculable future states like spock in star trek. Total clown show.
Yes, I wonder who the clowns are:
1. Researchers that have forgotten more about this technology than the average HNer will ever learn
2. Random HNers that think that suddenly they understand the problem better than researchers that have spent their lives on it
This is just anti intellectualism in a different coat, and it's not much different from people denying environmental scientists talking about climate change or biologists talking about vaccines.
But everyone here thinks they are the smart protagonist and certainly they would never hold a position so silly!
Really? Forgotten more than we know? So what can they tell us about how this technology works then? It seems a lot of it, besides the actual training engineering parts, is handwaving about "black boxes" and "divine benevolence" (a quote from Noam Shazeer).
Anti-intellectual? Because they don't believe claims that are backed up only by untested hypotheticals?
If we're going to have whistleblowers emerge declaring a 10% chance we all die because of AI, we're going to need more than vibes. Real experts can explain their position plainly. That has not happened here.
> Real experts can explain their position plainly. That has not happened here.
It has, over, and over, and over. There is literally a book on it. https://ifanyonebuildsit.com/
Stop doing this: https://en.wikipedia.org/wiki/Argument_from_authority
It's not my job to re explain AI safety from scratch to every uninformed take on the internet. Authority serves a useful purpose.
Having humility and delegating a topic to people that understand the topic better than you is not a bad thing, and certainly preferred to this state.
The writers for Star Trek spent a lot more time thinking about interstellar travel than I do, but I'm not going to go to them for advice about building a rocket ship.
The Rationalists are a philosophical/pseudo-religious organization in the same way the Daoists are. Just spending a lot time thinking about something doesn't make them right, it just makes them internally consistent with their chosen axioms. That is useful: if you agree with the axiomatic beliefs of such a movement, you can probably defer a lot of reasoning about the conclusions of those beliefs to the authority figures within that movement. But it doesn't change the probability of the axioms themselves being true, any moreso than the Pope saying so affects the probability of God existing.
AI Is Important And Potentially Dangerous is an axiomatic belief in the rationalist circles that are relevant to AI safety, in the same way that God Exists is axiomatic to Catholicism.
That's a fascinating assumption, and I wonder how you got to it, because it doesn't match anything I've ever seen. Most groups look more homogeneous from the outside than the inside. You are assuming their belief is religious and axiomatic, rather than very carefully reasoned about for decades, iterated on, discussed, debated, tested, and refined.
When the going gets weird, the weird turn pro. Few people have the ability/temperament to reason about things that have never happened before. Those that do have a weird psychological profile. Usually you can ignore them. Every once in a while, you can't.
The big fear in the administration is any attempt to put breaks on the AI progression will lead to a recession and so the history will be written as to win an election (that is almost certain a losing proposition for the party in power), they put the future of humanity at risk.
As Nixon did.
A more cynical but accurate take is that eltists with disproportionate access to capital, some might even say a "silver spoon", are reading the writing on the wall that a benevolent superintelligence will erase their own elitism versus the commoners, and make said elitists just another meatbag like everyone else. That's why AI needs to be tightly controlled and wielded as a force that reinforces class divisions rather than erasing them.
Thats certainly a title
I'd watch the movie for sure
I remember reading back in 2014 (IIRC) Yudkowsky's explanation of the Bayes theorem. It was genuinely good. Some sort of first principles introduction.
HPMOR was entertaining as well. Death as the last enemy to be destroyed placed his priorities where his mouth is.
For a brief time back then I used to visit Lesswrong and many comments including Yudkowsky's were plain stupid (which was not hard to reason at all) but coming from folks who valued themselves as first principles thinkers was an oxymoron.
But his timeless equations (IIRC), Bayes theorem explanation and HPMOR were all remarkable and I will always appreciate those works.
My favorite writing of his is:
https://www.yudkowsky.net/rational/virtues
dupe https://news.ycombinator.com/item?id=49738411
See also from earlier today [1].
More or less the same content but parent is much more palpable, IMHO.
[1] https://news.ycombinator.com/item?id=49737985
Hopefully enough smart people out there can distinguish between many "AI safety" people and actual AI engineers/scientists at these labs working at the coalface.
AI ethics reminds me of "design thinking" grifters or scrummasters or other such BS jobs of times past.
If you listen in an informed way to these people talk, they are not really as technical as they sound. They are personality hires, woo-woo spellcasters. They are just creating content for a narrative. The grasp of philosophy is not even at par with what you might think it ought to be.
I'm going to be fucking irritated if we all die because BlueSky posters wanted to take pot-shots at the EA crowd. The Boy Who Cried Wolf is _also_ a story about how a community lost a child because of cynicism, and the people proudly declaring that they "know" every safety concern is just a marketing ploy also concerns me.
I think it's more of a Chicken Little situation. Then the wolf ate us all up.
I have great news then, that absolutely isn't going to happen. And the EA crowd deserves all of the potshots and more
Wow I’m sure glad that’s settled then.
Happy to be of service
I'd rather have the world end than live in a world where EAs are listened to or respected.
If you are serious about this preference, would you also be okay with a compromise solution where only you (and anyone else who feels the same) die, rather than taking everyone else with you?
I agree they’re kinda weird but are they really that bad?
Yes.
Their core belief system is morally abhorrent.
is dying really so bad compared to admitting that you were wrong and some weirdo was right?
for many people the answer seems to be: not really.
Yes. Sam Bankman Fried was/is an EA/rationalist (on the short list of biggest fraudsters in American history). So are the Zizian's (a murder cult). etc. etc.
SBF maybe, but Ziz declared rationalists to be their main enemy