I think in principal, "If AI leaders genuinely believe the technology is existentially risky, that’s a good thing" because the assumption is a rational AI leader therefore wouldn't be building it if it were existentially risky.
I think the problem with that is, the options presented to the CEO of AI-Megacorp are not "Oh no, if we build this version, it will be evil and kill us". It's more like "last round in the evil-AI benchmark, it decided to kill all the humans. We've retrained it with stronger alignment, and now it's passing the benchmark. We're probably OK for this version?". The issue is the next, better version might be better at hiding its reasoning. This is already happening.
My assumption is that these leaders / companies are rational, but rationalizing. Lots of people make make decisions that are objectively bad. If the chance of making a world-ending AI is (0% - 0.1%), would you take the chance, assuming if you continue down the path you may be one of the richest and most powerful men on earth? I think I probably would. What if, every release increases that risk by 0.1%? What point do you stop?
The problem with the idea of "slowing down" is that it has to be top-down at a global level. Taking a month or two to slow down and think about it while your competitors are full steam ahead might be suicidal right now, especially in the age of RSI. So at what point do you decide "too much risk"? It's a game of chicken where if you stand down, you lose.
Points 6 and 7 shows how well the students are intuiting LLMs. I think it ties very well into why they think progress from chatgpt 2 to chatgpt 5 was not a drastic improvement. After all, the first car Ford mass produced and current supercars are the same in principle, internal combustion engines for transportation. Engine design, technology development and increased complexity does not change that a car is a car. A Lamborghini is awesome in capability just as the latest LLMs are, but they are still what they are.
Wait, how does their intuition about AI desires (point 6) and the HuggingFace incident (point 7) have anything to do with why they think there's been only incremental progress from GPT-2 to GPT-5?
If anything it just feels like they don't understand how hard certain tasks are, so when AI can do those things, it doesn't mean anything to them. For example, in point 1, students' reaction to Claude Code isn't "wow, AI can code entire programs now", it's "wow, I had no idea software engineering was actually that easy."
I have experience with highschool students and I'd like to add one thing. There is a clear anti-AI attitude especially amongst brighter students because they see this as yet another clear attempt, in an education system already in chaos and staffed by underqualified teachers, to shift even more responsibility onto the pupils’ shoulders. Responsibility that ought not to be there in the first place.
The kids are alright! This list gives me great hope and it’s amusing to see the usual TESCREAL crowd lose it over at LessWrong over how they should be more worried about post-singularity runaway basilisks and less worried about enshittification, deskilling, and inequality.
Point 2.3 (unemployment feedback loop) is real and has occurred in history - it's called an economic depression. As in the Great Depression, but there are littler ones too.
It, like most of the economy, is a policy decision by Congress.
I think in principal, "If AI leaders genuinely believe the technology is existentially risky, that’s a good thing" because the assumption is a rational AI leader therefore wouldn't be building it if it were existentially risky.
I think the problem with that is, the options presented to the CEO of AI-Megacorp are not "Oh no, if we build this version, it will be evil and kill us". It's more like "last round in the evil-AI benchmark, it decided to kill all the humans. We've retrained it with stronger alignment, and now it's passing the benchmark. We're probably OK for this version?". The issue is the next, better version might be better at hiding its reasoning. This is already happening.
My assumption is that these leaders / companies are rational, but rationalizing. Lots of people make make decisions that are objectively bad. If the chance of making a world-ending AI is (0% - 0.1%), would you take the chance, assuming if you continue down the path you may be one of the richest and most powerful men on earth? I think I probably would. What if, every release increases that risk by 0.1%? What point do you stop?
The problem with the idea of "slowing down" is that it has to be top-down at a global level. Taking a month or two to slow down and think about it while your competitors are full steam ahead might be suicidal right now, especially in the age of RSI. So at what point do you decide "too much risk"? It's a game of chicken where if you stand down, you lose.
Points 6 and 7 shows how well the students are intuiting LLMs. I think it ties very well into why they think progress from chatgpt 2 to chatgpt 5 was not a drastic improvement. After all, the first car Ford mass produced and current supercars are the same in principle, internal combustion engines for transportation. Engine design, technology development and increased complexity does not change that a car is a car. A Lamborghini is awesome in capability just as the latest LLMs are, but they are still what they are.
Wait, how does their intuition about AI desires (point 6) and the HuggingFace incident (point 7) have anything to do with why they think there's been only incremental progress from GPT-2 to GPT-5?
If anything it just feels like they don't understand how hard certain tasks are, so when AI can do those things, it doesn't mean anything to them. For example, in point 1, students' reaction to Claude Code isn't "wow, AI can code entire programs now", it's "wow, I had no idea software engineering was actually that easy."
I have experience with highschool students and I'd like to add one thing. There is a clear anti-AI attitude especially amongst brighter students because they see this as yet another clear attempt, in an education system already in chaos and staffed by underqualified teachers, to shift even more responsibility onto the pupils’ shoulders. Responsibility that ought not to be there in the first place.
Attempt by who? Responsibility to do what? Not snark, just not following.
Lesswrongers and real-world problems are like oil and water, exhibit A. Most fun thing I've read in a while
The kids are alright! This list gives me great hope and it’s amusing to see the usual TESCREAL crowd lose it over at LessWrong over how they should be more worried about post-singularity runaway basilisks and less worried about enshittification, deskilling, and inequality.
"Prone tp hyperbolic despair"
A pedophile criminal is the president of the "greatest democracy on earth"
Pre 2016, that would sound hyperbolic.
Point 2.3 (unemployment feedback loop) is real and has occurred in history - it's called an economic depression. As in the Great Depression, but there are littler ones too.
It, like most of the economy, is a policy decision by Congress.
Good article