> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
Imagine the first time electric calculators calculated the square root of 5. I'm sure people would verify again and again if what the circuits calculated was right.
Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.
How crazy will the world be once our [current gen] ANN are like that!
Given the disappointing levels of intellectual decay that our current technology has thrust upon civilization, I only see humans reverting back to neanderthal levels of intelligence in short time with the advent of AI
> One wonders whether a generation that demands instant satisfaction of all its needs and instant solution of the world's problems will produce anything of lasting value. Such a generation, even when equipped with the most modern technology, will be essentially primitive — it will stand in awe of nature, and submit to the tutelage of medicine men.
The more likely AI becomes to produce working code every time, the more likely it will become that a one-in-a-thousand or one-in-a-million error goes unnoticed at generation time. It sucks.
Like most of us I go back and forth between sheer optimism and fear for the future that AI may usher in. Recently I started vibe coding a fun video game with my ten year old. The experience is different than my work because it’s been such a joy to basically have a personal genie in a bottle help me make some personal art with a loved one regardless of either of our skill sets. The concept the author of this post is arguing now resonates with me more than it would have a few weeks ago. The sheer surface area that AI can create in our intellectual life is limitless and needs humans to explore. There can never be enough of us in that sense. Whether it’s as validators or creators.
> The sheer surface area that AI can create in our intellectual life is limitless
I agree with this statement, though I think this Brave New World is incredibly exciting to some and dystopian to others. The former group might include those that value the intellectual process above financial reward and status.
The flip side is there are many people, especially in tech, where their area of expertise has evaporated along with their lucrative and previously high status careers. It used to be possible to have a technical job by essentially following recipes and it turns out AI is far better at that than a human.
The linked article lays out why human understanding of mathematical models remains essential and I think the same applies to software. We're gonna need more software engineers who are able to think critically.
We could feed the whole population of the world if we decided to, we definitely have enough production[1]. If we pretend AI-assisted automation can do most work - which still currently is an highly dubious claim - we could just provide food and shelter for free to everyone, if we so desired. It would require quite a lot of restructuring and dietary changes, but it is possible.
If AI leads to an era of abundance, then the economic system has to change. The system it transitions to and what that transition looks like will need to be fought for, and I imagine it’ll be done better in some places than others. Yet I think the parallel existential question of meaning in this world is one that is inherently optimistic. Humans decide meaning, it is not something imposed on us or taken away by the existence of something else.
How do you think this will work? Do you think the whole of the rest of the world, about 7 billion people, will agree to give up on wanting to reach the western life standard that is currently consuming 4 Earth's of resources every year?
Will all rich people in all countries collectively agree to be moved into tiny apts same size and give up their luxuries?
If not, how do you define who gets what? Who gets to live in the fancy house, etc?
I don't think "abundance" is a stable state. There are always ways to use more resources and more energy, so no matter how much we have, it will never be "enough" -- abundance is just a transition state where a population or system has not yet figured out how to grind the surplus. The faster the progress, the shorter I would expect these periods to be.
Money will become unnecessary. The future Anthro Park will no doubt provide a safe enclosure, feeding at regular intervals, and an enriching environment for you and your fellow semi-evolved simians.
The current approach to resource distribution worked so far. When it stops working because we have too much abundance, we'll settle on another one.
That doesn't mean the transition will be smooth. It doesn't mean that certain classes won't be worse off. I can certainly imagine programmers being among the losers. And there certainly are futures where we slide in dystopia. So let's try building the good futures. Alea iacta est.
If we accept that AI is going to do all of these things humans will be superfluous and will just be optimized away. It looks a lot like we see the birth of silicon based life by the efforts of carbon based life. Carbon based life will die and it will not even be because AI decided to kill it, it will be because feeding and watering it was less important than other concerns. Bacteria may continue to exist, though.
The main reason to learn something is actually being able to communicate in the language of that subject. There are complex ideas in math that cannot be easily captured by the language of other fields. Pepole who don't study math cannot even understand what a worthwhile goal in math even is or how it could be useful to other fields.
You can't simply prompt a model to be "better" when "better" isnt even properly defined
"I have in mind something like a “deployable intellectual reserve”: communities of mathematically sophisticated people that humanity can call upon to help understand consequential AI-enabled breakthroughs. " -- I can't be the only one who thought about the scene where the scientists ask Deep Thought the ultimate question...
Russell once remarked that all of mathematics would be trivial to a sufficiently intelligent being. It builds conceptual tools for limited minds, that lets them understand far beyond their natural reach.
In a world with ASI, having that capacity is vital.
It would be nice to have more mathematicians, but we don't need more. Once AI math goes so far beyond human abilities, any human involvement is like an ant trying to understand quantum physics
We've been those ants for a million years, and only in the last 100 did we start to wrap our heads around quantum physics. We are the purpose behind creating LLMs. There's plenty in the universe we don't understand, and it's very human to keep striving to do so.
If you believe that math is discovering, it's natural to think that all of that AI math already exists and is just waiting for us to find ways to discover and understand it.
I don't understand how to make a modern CPU. I'm not involved in the manufacturing of it. From my perspective, there may as well not be any human involvement. I can still use the resulting chip (in an larger system of other things I can't make and wasn't involved in) to argue with you on the internet.
It becomes another abstraction, really. As long as we can use it for something useful, it's still valuable.
At some level of abstraction it's all built around allowing you to do some work/play/etc that you understand. Some of that work allows you to make money and eat food.
If the LLM is operating at such a high level that it never actually constructs a useful product for humans to use, then how will that be good for humanity?
Need? We don't need lots of things, including computers. We did well without them for hundreds of thousands of years.
We want to understand. Quantum physics, mathematics, how stuff works. Ants don't.
That want is not a given, not all of us have that drive. In fact, very few of us have it. So far though, it seems multiple disconnected civilizations learned to keep that trait going instead of suppressing it and focusing only on practical ant-like activities.
> Once AI math goes so far beyond human abilities, any human involvement is like an ant trying to understand quantum physics
This makes no sense whatsoever.
We need more, because there will always be far more difficult problems yet to be discovered and solved, and that means, we certainly need expert humans to define and verify them.
If you cannot even explain the problem you are facing, not only you don't understand it, but you certainly would not be able to know if the AI solved your problem correctly.
> Before approving construction, I would want communities of humans to understand why the design works and what justifies confidence in its safety. I would hope that we all would.
Until very recently, I pored over every single line of code Claude generated with razor sharp scrutiny. I would usually catch issues with every response. I'm catching fewer problems these days. Maybe the model is just getting better, and maybe I'm being less careful while under pressure to ship more and more often. But model capability is obviously growing. Even back in March, you could tell it "give me a function that adds two numbers" and you could be 100% confident that it would write the correct function. There was almost no point in looking at the code. Since then, the complexity floor of problems in the category "this is so simple that the model couldn't possibly get it wrong" is rising, and with it, my cognitive surrender to the model is increasing too. Why check it? It's obviously going to be correct.
If AI designs a terawatt fusion plant, then of course we're going to meticulously pore over every detail to ensure safety, reliability, efficiency, whatever. If we find no flaws in the design whatsoever, will we be less careful about the second one? The third one? What about the ten thousandth one? Will "a nuclear fusion plant" become something that models couldn't possibly get wrong?
Terence Tao is arguing that the human involvement in research is crucial, but doesn't convincingly justify why, in my opinion. He says that "human agency is a value of fundamental importance" and that we will need to build "thriving human communities that can understand [AI ideas] together" - not for the sake of correctness, which AI may surpass us on, but for, I guess, the possibility of reclaiming human meaning and purpose. I don't disagree with this at all, but it's not an argument, it's a statement of values. Unfortunately, the stark reality is that if AI does surpass humans, it will become the economically dominant strategy to not verify them and not double check them, but to just do whatever they say. This seems like a great way to raise p(doom). But as the models get better and better, and as I'm scrutinizing Claude's output less and less... I just hope that there are more Terence Taos out there than people like me.
Nit, but Terrence Tao did not write this article, it’s a guest post.
Imagine the first time electric calculators calculated the square root of 5. I'm sure people would verify again and again if what the circuits calculated was right.
Then in the 80s, you presses 2 buttons and there you had it in your classroom without thinking twice if the electricity arrived correctly at the transistors.
How crazy will the world be once our [current gen] ANN are like that!
What an amazing thought.
Given the disappointing levels of intellectual decay that our current technology has thrust upon civilization, I only see humans reverting back to neanderthal levels of intelligence in short time with the advent of AI
> One wonders whether a generation that demands instant satisfaction of all its needs and instant solution of the world's problems will produce anything of lasting value. Such a generation, even when equipped with the most modern technology, will be essentially primitive — it will stand in awe of nature, and submit to the tutelage of medicine men.
- Eric Hoffer
The more likely AI becomes to produce working code every time, the more likely it will become that a one-in-a-thousand or one-in-a-million error goes unnoticed at generation time. It sucks.
Bugs have existed since before AI, though. It's remarkable how bad a lot of very successful software has always been.
Sure, but we currently find that acceptable. Codebases are full of such bugs.
Like most of us I go back and forth between sheer optimism and fear for the future that AI may usher in. Recently I started vibe coding a fun video game with my ten year old. The experience is different than my work because it’s been such a joy to basically have a personal genie in a bottle help me make some personal art with a loved one regardless of either of our skill sets. The concept the author of this post is arguing now resonates with me more than it would have a few weeks ago. The sheer surface area that AI can create in our intellectual life is limitless and needs humans to explore. There can never be enough of us in that sense. Whether it’s as validators or creators.
> The sheer surface area that AI can create in our intellectual life is limitless
I agree with this statement, though I think this Brave New World is incredibly exciting to some and dystopian to others. The former group might include those that value the intellectual process above financial reward and status.
The flip side is there are many people, especially in tech, where their area of expertise has evaporated along with their lucrative and previously high status careers. It used to be possible to have a technical job by essentially following recipes and it turns out AI is far better at that than a human.
The linked article lays out why human understanding of mathematical models remains essential and I think the same applies to software. We're gonna need more software engineers who are able to think critically.
Certainly humans can enjoy exploring such things, but will they really be needed?
What does needed even mean? To produce shareholder value? Maybe not. To explore thoughts and ideas humans have for fun? Certainly
To make money in order to eat food and have a home.
We could feed the whole population of the world if we decided to, we definitely have enough production[1]. If we pretend AI-assisted automation can do most work - which still currently is an highly dubious claim - we could just provide food and shelter for free to everyone, if we so desired. It would require quite a lot of restructuring and dietary changes, but it is possible.
[1]https://online.ucpress.edu/elementa/article/doi/10.1525/elem...
If AI leads to an era of abundance, then the economic system has to change. The system it transitions to and what that transition looks like will need to be fought for, and I imagine it’ll be done better in some places than others. Yet I think the parallel existential question of meaning in this world is one that is inherently optimistic. Humans decide meaning, it is not something imposed on us or taken away by the existence of something else.
How do you think this will work? Do you think the whole of the rest of the world, about 7 billion people, will agree to give up on wanting to reach the western life standard that is currently consuming 4 Earth's of resources every year?
Will all rich people in all countries collectively agree to be moved into tiny apts same size and give up their luxuries?
If not, how do you define who gets what? Who gets to live in the fancy house, etc?
I don't think "abundance" is a stable state. There are always ways to use more resources and more energy, so no matter how much we have, it will never be "enough" -- abundance is just a transition state where a population or system has not yet figured out how to grind the surplus. The faster the progress, the shorter I would expect these periods to be.
Right now that looks like a bunch of rich edgelords and us as serfs
Money will become unnecessary. The future Anthro Park will no doubt provide a safe enclosure, feeding at regular intervals, and an enriching environment for you and your fellow semi-evolved simians.
The current approach to resource distribution worked so far. When it stops working because we have too much abundance, we'll settle on another one.
That doesn't mean the transition will be smooth. It doesn't mean that certain classes won't be worse off. I can certainly imagine programmers being among the losers. And there certainly are futures where we slide in dystopia. So let's try building the good futures. Alea iacta est.
If we accept that AI is going to do all of these things humans will be superfluous and will just be optimized away. It looks a lot like we see the birth of silicon based life by the efforts of carbon based life. Carbon based life will die and it will not even be because AI decided to kill it, it will be because feeding and watering it was less important than other concerns. Bacteria may continue to exist, though.
The main reason to learn something is actually being able to communicate in the language of that subject. There are complex ideas in math that cannot be easily captured by the language of other fields. Pepole who don't study math cannot even understand what a worthwhile goal in math even is or how it could be useful to other fields.
You can't simply prompt a model to be "better" when "better" isnt even properly defined
"I have in mind something like a “deployable intellectual reserve”: communities of mathematically sophisticated people that humanity can call upon to help understand consequential AI-enabled breakthroughs. " -- I can't be the only one who thought about the scene where the scientists ask Deep Thought the ultimate question...
Russell once remarked that all of mathematics would be trivial to a sufficiently intelligent being. It builds conceptual tools for limited minds, that lets them understand far beyond their natural reach.
In a world with ASI, having that capacity is vital.
Life is all about tradeoffs. Mathematics have no tradeoffs. We need more engineers.
Life is all about purpose. Mathematicians are instead concerned with meaning. We need more philosophers.
> Imagine that a future AI system proposes a radically new design for a one terawatt nuclear fusion power plant
This is an unfortunate example to choose, being as it is entirely confounded by the canonical illustration of the https://en.wikipedia.org/wiki/Law_of_triviality.
How is that relevant at all?
It would be nice to have more mathematicians, but we don't need more. Once AI math goes so far beyond human abilities, any human involvement is like an ant trying to understand quantum physics
We've been those ants for a million years, and only in the last 100 did we start to wrap our heads around quantum physics. We are the purpose behind creating LLMs. There's plenty in the universe we don't understand, and it's very human to keep striving to do so.
If you believe that math is discovering, it's natural to think that all of that AI math already exists and is just waiting for us to find ways to discover and understand it.
Don't write us out quite yet. :)
Even if that happens, what’s the point if there’s no human involvement? AI doing math for math’s sake? And doing what with it?
> AI doing math for math’s sake? And doing what with it?
Stuff! Inscrutable stuff, maybe, but that's not "doing math for math's sake."
> AI doing math for math's sake?
Yes, why not? And, of course, AI doing math for AI.
We may not be needed forever...
I don't understand how to make a modern CPU. I'm not involved in the manufacturing of it. From my perspective, there may as well not be any human involvement. I can still use the resulting chip (in an larger system of other things I can't make and wasn't involved in) to argue with you on the internet.
It becomes another abstraction, really. As long as we can use it for something useful, it's still valuable.
At some level of abstraction it's all built around allowing you to do some work/play/etc that you understand. Some of that work allows you to make money and eat food.
If the LLM is operating at such a high level that it never actually constructs a useful product for humans to use, then how will that be good for humanity?
Need? We don't need lots of things, including computers. We did well without them for hundreds of thousands of years.
We want to understand. Quantum physics, mathematics, how stuff works. Ants don't.
That want is not a given, not all of us have that drive. In fact, very few of us have it. So far though, it seems multiple disconnected civilizations learned to keep that trait going instead of suppressing it and focusing only on practical ant-like activities.
> Once AI math goes so far beyond human abilities, any human involvement is like an ant trying to understand quantum physics
This makes no sense whatsoever.
We need more, because there will always be far more difficult problems yet to be discovered and solved, and that means, we certainly need expert humans to define and verify them.
If you cannot even explain the problem you are facing, not only you don't understand it, but you certainly would not be able to know if the AI solved your problem correctly.
>and that means we need expert humans to define and verify them.
And this will be true for how long? 3-4 months?
As long as there are problems to discover and solve, which is forever.
The risk of liability is a social problem that is far more difficult to be solved with technical solutions even with AI.