This is something that really needs to be formally studied.
My inclination is to say that the assertion is consistent with my experience, but I can't say it's not confirmation bias.
As a meticulous person generally looking for a very specific code outcome, I prompt in a way intended to get exactly the thing I have in mind, and my results reflect that. But on the other hand, I have coworkers who type ten-word prompts with very limited specificity, and they seem to get results that way as well, and that makes me wonder.
It would certainly be beneficial for my career and financial well-being for the assertion to be true, because it means I don't have to worry about being pushed out of my job by an army of $15/hr vibe coders. But the convenience of that assumption is exactly why I think it's important to be skeptical.
> In the 2010s, if you had technical gaps (say, you couldn’t write CSS), you had to either rely on a skilled colleague or just hope that the answer to your exact problem was out there on the internet.
You could read some general reference/guide/tutorial documentation on CSS, and then probably solve your problem (without searching for "how to center a div", or whatever your exact problem was, and copy&pasting the answer and moving on), also becoming more knowledgeable in the process.
The rest of the short blog post has some good points, but the first sentence sounds like it's targeted at the percentage of developers who did StackOverflow copy&paste to close Jira tickets, never becoming experts.
Delegating to LLM-ish AI is just a natural evolution of that. The question is whether they can still add value if kept in the loop.
The article author suggests that the answer is to be expert, and is addressing people who... "either rely on a skilled colleague or just hope that the answer to your exact problem was out there on the internet."
> You could read some general reference/guide/tutorial documentation on CSS, and then probably solve your problem
Hours + Hours of reading and a lot of trial-and-error. The loop was so long and sooo slow. Now it's instant. As if your very first Google search just solved the problem for you immediately.
Yes, but sometimes I don't need and want to learn. One example from my recent experience in research -- building custom dashboard pages for results of scientific analyses. Each analysis is bespoke, and building interactive webpages is simply not the skill many researchers have (and it's boring IMO). But here with LLM you could easily explore the results visually/share them with collaborators etc. There are plenty examples like that.
But certainly there are cases where learning is required.
You seem to be hung up on the part about "learning". Do you know how x86 registers work? Or atomic instructions available on ARM, SPARC, or POWER architectures?
No, of course not. Because all of that got abstracted to higher-level instructions decades ago.
I don't think AI use is supposed to replace foundational learning such as reading a C++ book or Python book or CSS tutorial when you're a beginner. You still have to do those things if you want to be a professional or a strong amateur. But many people just want to get the thing done. They don't want to become a mechanic, they just want to drive from A to B.
> They don't want to become a mechanic, they just want to drive from A to B.
I'm fairly certain the article is directed at professionals, or at least the AI companies are basing their valuations off of directly taking a slice of that professional "productivity".
I've seen junior engineers be productive on their first day in the industry because of AI, so I don't think the article is the whole truth.
The example math is boundary-pushing and definitely not a solved problem. But most of us work on CRUD backends with a React frontend. Those are more or less solved problems that have well-documented solutions. For those kinds of tasks, LLMs just reward usage.
I can count on one hand the number of times in my career I've needed to solve a problem that's not described on Stack Overflow.
Except LLMs will only tend to share the most common or average of what it knows as the standard and deviating from it (including new ways) it can be resistant to.
An expert can lay a different kind of frame to prevent the llm to fell out of its way of being generally too verbose, and that can transfer as well to code generation and complication.
Not sure I agree with this. The math guy at anthropic's prompts are essentially:
"suppose you’ve gotta resolve the $CONJECTURE, like absolutely have to, everything depends on it. think really hard, and try to come up with a bunch of ideas to try. but remember to trust yourself and not necessarily in conventional wisdom!!"
https://claude.ai/share/25740bd5-aa97-4bd7-bf58-c4df3793fda7
https://xcancel.com/__alpoge__/status/2083855298239078748
Tao's chat was for him to gain intuition, not to solve the problem from the outset.
What's funny is that every other person gets a different conclusion about who these models reward/empower. I've seen people say that the generalist stands to gain the most and others say that it's the experts. Like all of life, maybe the "winner" is the person who just does stuff.
It depends on the levels. People with differing fitness levels and ages run at very different paces. Now, do cars make them more equal or less? On the bottom end, the tide lifts all boats. Most healthy people can learn to drive and will drive "fine", they get from A to B. Out there in the city streets the car flattens the differences, everyone roughly takes the same time to get from A to B in a car.
But at the top of top, the gap probably widens. A professional F1 driver will drive laps around some random guy. It amplifies reflexes etc, because at that speed little differences in timing make a big difference.
Now, AI coding isn't exactly analogous, but I think it also has these two regimes. It flattens things for simple tasks. If your task is to shovel data, do some trivial compiler wrangling staring at badly designed error messages, looking through GitHub issues hunting for the comment with many tadaa emojis to fix an issue etc, those things can now be done by anyone. Just as grandpa can also drive to the grocery store. But if you're pushing at things on a higher level, now only your above-AI ability matters. If all the things that AI can do well are subtracted out, how much other expertise do you have left? This will be proportionally a bigger and bigger difference between different people.
So you're saying that it depends whether you're measuring "success at task X" (where in average everyone improves) vs comparative success (where people with knowledge can do far and away better at specific tasks).
This works better for math because math is self-verifiable. Once you have a proof it needs no outside evidence.
Expertise is needed to evaluate model outputs where it can't verify itself, or at the very least one's expertise can help steer the model in the right direction.
However this is irrelevant if models themselves are better at evaluating/leveraging expertise/information.
Corollary to this is an important part of LLM usage is what I call pinning it to reality. That is, designing verification steps that interact with the real world in some way not easy to hallucinate or work around. This means things like having code that interacts with the physical world, round trip tests, arriving at the same result using different paths, interoperability / replication with external libraries / competing products, performance improvement projects that start with robust performance test suites, and similar sorts of things that reduce to "how do I provide evidence that's difficult to fool myself about".
This includes things like "before you start fixing this bug, write two tests that fail proving it exists".
Expertise is good, but a wise expert will set up methods for the machine to prove to itself that a desired result is achieved removing the expert from the tight development loop.
There was one math proof that was AI generated going around Twitter and the chat transcript to generate it was basically the human prompting "keep going" until it solved it.
Tao's chat was fascinating because the questions he was asking belied expert knowledge of the subject that only a handful of people could have asked.
Yup I linked that in my other comment but copy-pasted here for everyone's convenience:
The counterexample of the Dinitz-Garg-Goemans conjecture was basically just "keep going" and finally "enough of partial results. now finish with a complete unconditional counterexample"
Yes, we're in the low hanging fruit stage where folks can just say "here's the problem" and "keep going" until a result is found and it will work sometimes.
The low hanging fruit will run short. Ultimately mathematics is a field of subjective selections of problems and proofs as beautiful and interesting. Machines absolutely will struggle with what to study, what theorems are desirable, and when do be done with a proof.
> Machines absolutely will struggle with what to study, what theorems are desirable, and when do be done with a proof
And why do you think this would be the case? I'm not talking about today but in 1-2 years. For reference o1 was released less than 2 years ago, and we've had reasonable coding agents for 9 months or so.
Mathematics is ultimately an aesthetic pursuit. Outside of a well defined goal ML models don't have any sense of taste and regardless of the scaling that's been enabled in the last year or so of capability if they haven't memorized the process of doing something they have the same limitations of inability to make choices about unknowns not trained into them.
Real synthetic intelligence seems to me to be still very far away and not a matter of making models bigger or more efficient.
LLMs are a collection of biases. Humans are also a collection of biases. So we project our biases as input through the biases of an LLM and get an output. Hence why I think getting optimal output requires being an optimal person. And in that sentence there are many points of expression.
Finally, we train our LLMs on who we are. Another reinforcement of biases.
I agree. When I talk to LLMs about fields I am familiar with, I can push back on bad suggestions and ignore faulty/incorrect advice and assumptions, which is much harder for unfamiliar subjects.
Of course, simple common sense and extremely basic Googling on unfamiliar subjects can produce similar results, but it's much faster if you are truly understanding what the AI is suggesting.
I do find that "signalling expertise" is important. "I have a significant background in biblical scholarship. You can assume I've read the most important works in NT studies in particular. Do not translate Greek, Latin, Hebrew, or Syriac. Now, I would like to know . . ." That changes things significantly. So does telling it you have 20+ years of experience with C programming, that you have a robust understanding of machine organization, memory layouts, embedded systems, etc.
For sure. On a personal coding project I said "I'm a professional software engineer, and while this is a hobby project I'm not just vibe-coding and want to build reliable software" and the agent suddenly started suggesting all kinds of things to make its code more robust.
The amplifying mirror analogy works best here. LLMs are ultimately a reflection of your own interactions with its weights, the tone you use, the structure with which you construct your prompt, aspects of an issue you tend to focus on, your breadth of vocabulary and world knowledge and whatnot.
People who (carefully) use it as an extension of their own mind and senses will very likely thrive, and those who use it as a replacement for their minds and their senses will struggle.
One of the Claude skills I made Claude itself generate was the 'learning a concept across tiers' skill -- from ELI5 level to a PhD level, and it triggers whenever I ask it a very general question on a complex topic that isn't my bread-and-butter. The fact that I'm able to choose explanation level from a super smart LLM (that's available 24x7) that can explain any topic under the sun would've been mind-bogglingly sci-fi-ish just 4 years ago in 2022.
The short version I give to non-technical people who ask me about whether "AI will replace coding" is this: it accelerates you. You can get much further much more quickly.
If you don't know where you're going or how to get there, or even if you're just not paying enough attention, it will get you very far in the wrong direction before you've realised.
I totally see this. I just did 3 hours of bot sitting to put together some thrash loops that thrash our provisioning working flow for a BLE gadget we make. It was pretty straightforward and productive. But then, I have a lot of experience with BLE, and a quite a bit of experience with python and shell scripting. So I was able to guide the process through stages, do some intermediate testing, make some adjustments, and proceed. Domain experience made this really easy and straightforward. Me two junior engineers who have only superficial/high level knowledge of BLE and some of the other pieces, couldn't have done this as effectively.
Where my angst comes, is worrying that no one will ever get that experience anymore. They might have had some eventual success, who knows what monstrosity a much less guided LLM would have done, but experential learning may be mostly a thing of the past. And it creates a real tension between the person with experience and the person without.
The LLM is like the death star. If you don't know exactly where to point it, you will likely miss your target and have no/negative effect. The further away the target, the more accurate your firing solution needs to be. If all you need to do is add something like a dark mode theme to an existing product, this is probably a point blank shot in this metaphor. Building an entire codebase from zero, or even refactoring a legacy codebase into a new codebase, are lightyears away by comparison. You can still land the shot, but you need to deeply understand the metrology and astrodynamics.
The information system required to encode the aesthetic preferences needed to make a technology experience not suck is likely in excess of what any near-term solution will offer. Knowing when to say "no" is perhaps the most important skill here. You can't just say it arbitrarily either. You really have to mean it and be willing to fight other humans for it.
There's a growing and fascinating divide between people who see LLMs as more of a "bicycle for the mind" in the vein of Jobs vs those who see them as whollly supplanting the role of human intelligence. I can't help but wonder if these aren't primarily two human archetypes more than anything - the LLMs can be both and they erect a mirror of the human using them. Some humans really don't want deep individual expertise and intelligence to be the deciding factor because they don't identify with that. Others are completely the opposite. We really can't tell which will be more effective yet, because LLMs are very good in both modes. But most of the predictions currently are people executing on wishful thinking about what they hope will be the outcome.
Well, nice post. Actually, there may be some truth behind it, but basically, it captures what I—as a programmer—want to read: expertise will remain valuable.
But how I am observing is different, though. Since LLMs the gap between experts and non-experts has been shrinking. And yes, there is still a gap, but vanishing.
I am feeling this a lot lately. Getting the most out of agents seems to require being able to ask the right question. And how can you ask the right questions without deep domain expertise?
Yes sometimes it’s a matter of just using the right word. You can talk to an agent about a general concept for hours and hours and it may never mention $Concept_X, but you mention $Keyword_Y and all of a sudden the AI is going on about how $Concept_X is foundational to understanding the whole thing.
The counterexample of the Dinitz-Garg-Goemans conjecture was basically just "keep going" and finally "enough of partial results. now finish with a complete unconditional counterexample" lol
Might be a very noob question but in this era of LLMs, let me ask the reverse, how do you gain expertise? It seems this rewards people who had expertise pre LLMs, but what about people who don’t have that in a specific domain? What approaches are viable now in this current system?
The same as it's been! Make things without using LLMs. Don't debug with them, don't use them to research things, just do it yourself. It'll be painful and that pain is learning.
This matches my experience. Just Talk To It is the best method for working with LLMs if you're an expert.
I've seen this at work (as eng manager/lead/principal/whoevenknowsanymore) – all the big APIs give you stats. We see how much people burn in tokens and we know how much output they produce. There is a pretty strong inverse correlation between token burn and output.
The more tokens people burn, the less likely they are to produce a good outcome.
Yes. I agree with most, if not all of this. For instance, I am seeing folks either relying in the LLM as an _assumed_ expert or, assuming someone - who knows the structure of skill definitions - also has some expertise (in the area of the skill).
It's a difficult situation; there is not much point in explaining _why_ the LLM output or skill in use (on a domain problem) isn't what the person actually _needs_ to address the domain problem, because the person isn't a domain expert or indeed, adjacent to domain expertise.
But, it is an interesting experiemnt to arm folk with little domain expertise with the _skill_ necessary to be able to extract the right solution from the model.
This is also why people's experience with LLMs/AI varies so much, because some people can see a use for AI for their needs and go about using the tool, while others do not as it relates to whatever they're working on and so they may say "LLMs/AI are useless" (it doesn't mean they're not experts though, although some people who have totally no expertise might also see no use for AI for themselves).
In my experience (scientific programming) AI is a giant multiplier for people with specialized knowledge.
But it’s also a giant devaluer for that same knowledge as people with no idea what they’re doing can clog the field with plausible bullshit.
It’s now the case that if someone tells me they’ve done something, and I look into it and find out it’s completely AI slop, then I will have spent more time on the project than the person who “made” it. The situation is completely untenable and only serves to drain time and resources from people with better things to do.
this will have educational consequences (that I'm trying to solve). I don't think that we can adjust without rapid education and making extreme specialists of us all.
This requires coordination, certification, licensing, and other tiers of authenticity. False experts can ruin sample gathering, can ruin training. False expertise is exemplified by the current American Administration. Look at Robert F. Kennedy Jr.; he's a false expert. He is responsible for the measles outbreak. He is responsible for ivermectin abuse by humans. False expertise is overtaking real expertise. And the results are continuously disastrous and large-scale.
I often do my best to represent a genuine interest in the subject at hand and learning in general to models. Imagine the model's response prose and mannerisms being on the other polar end of answering questions simply to get the correct answers as they're often scoped for on quantitative benchmarks. Not sure I explained this well, sorry. An LLM could help
I have lengthy conversations with my LLM, almost like an interview. I agree on the expertise part, because I wouldn't be able to go in depth on a subject with it if I lacked the expertise.
Some work is a result of design and negotiations in those designs.
I don't think Tao's style works with everyone/thing, especially if we don't know what style he's tuned his LLM on.
Overall, I agree, when I ask things I'm an expert in and do professionally every day. I get very good useful answers. When, for example, our marketing people, ask about the science, they often get confusing and wrong answers.
Many, including myself, report having a lot of success with braindumping and not structuring anything. Just talking into speech recognition for 2-10 minutes as a stream of consciousness about what my context is, what I want, what I know already, what I have a vague hunch about, how it fits into a bigger picture, what aspects are most important to me, any footguns I already know about, really like having a chat with a person on the phone, with someone you have to guide remotely because they have to implement the thing right now but you have to be out of office and so your only interface is speech. Except you can be more structureless because the AI won't be offended. Just keep on rambling, and press enter, don't even correct mistranscriptions. It will understand it anyway.
Now, the key is, that while rambling without structure, you do have to drop the key facts into your speech, and you have to know what you're talking about in at least a good portion of it.
I think people are afraid of doing it, because it seems "not the right way" or "not scientific" or whatnot. They want to believe there is some magic to writing the right prompt. So let me tell you, it works.
I don't completely disagree with the concept of giving a free association thought process ramble into context. But I also bet that when you start getting it to actually generate code and link modules of things together, subroutines, functions, code structure and filenames, you still pay attention to what it does and you guide it into the architecture that makes logical sense to you.
For real work yes. For personal projects, less and less since Fable came out (probably the same if true of the other frontier models). You can get a lot done if it's just some one off, or a personal tool, even without looking at the code, just trying the application. Frontier models now automatically test it before handing the thing to you, they take screenshots, they fix the superficial issues themselves. To get something up and running, it's enough to send chat messages.
Unfortunately the software industry is saying things like "don't look at the code", "LLMs have made developers 10-100x faster", etc. The only way they can make such claims is by saying what you said above: "Claude, make me Microsoft Flight Simulator, make no mistakes".
Additionally engineers are facing pressures via deadlines to work in the paradigm of "Claude, make me Microsoft Flight Simulator, make no mistakes"...
The question i wonder about is, when will an event come along that persuades everyone that human understanding is still required? Or will it never come?
The event could be when fair pricing comes from the model providers. We're still at the cash burning stage. When the economy crashes a little and departments start monitoring their spending, and the prices for inference are 10x what they are, there will be less tolerance for employees to substitute constant AI usage for understanding.
I wonder the same thing. I think we've already seen some of this happening, however the consequences haven't been large enough to the organization, for example:
Such an event would have to be pretty catastrophic at this point to slow down the inertia. Perhaps the tech debt will just pile up until someone's product implodes, or there's a massive safety issue that causes loss of life, or some big hedge fund goes bust.
This is something that really needs to be formally studied.
My inclination is to say that the assertion is consistent with my experience, but I can't say it's not confirmation bias.
As a meticulous person generally looking for a very specific code outcome, I prompt in a way intended to get exactly the thing I have in mind, and my results reflect that. But on the other hand, I have coworkers who type ten-word prompts with very limited specificity, and they seem to get results that way as well, and that makes me wonder.
It would certainly be beneficial for my career and financial well-being for the assertion to be true, because it means I don't have to worry about being pushed out of my job by an army of $15/hr vibe coders. But the convenience of that assumption is exactly why I think it's important to be skeptical.
> In the 2010s, if you had technical gaps (say, you couldn’t write CSS), you had to either rely on a skilled colleague or just hope that the answer to your exact problem was out there on the internet.
You could read some general reference/guide/tutorial documentation on CSS, and then probably solve your problem (without searching for "how to center a div", or whatever your exact problem was, and copy&pasting the answer and moving on), also becoming more knowledgeable in the process.
The rest of the short blog post has some good points, but the first sentence sounds like it's targeted at the percentage of developers who did StackOverflow copy&paste to close Jira tickets, never becoming experts.
Delegating to LLM-ish AI is just a natural evolution of that. The question is whether they can still add value if kept in the loop.
The article author suggests that the answer is to be expert, and is addressing people who... "either rely on a skilled colleague or just hope that the answer to your exact problem was out there on the internet."
> You could read some general reference/guide/tutorial documentation on CSS, and then probably solve your problem
Hours + Hours of reading and a lot of trial-and-error. The loop was so long and sooo slow. Now it's instant. As if your very first Google search just solved the problem for you immediately.
>Hours + Hours of reading and a lot of trial-and-error. The loop was so long and sooo slow.
But that's how you learn...
We don't need to learn CSS anymore. Just like we don't need to learn ASM since GCC does a great job generating it from higher-level code.
I want to believe this is sarcasm, but it's so hard to tell these days.
It's not sarcasm.
We've all accepted that code-generation has been required and accepted for decades.
It is not the only way to learn.
Yes, but sometimes I don't need and want to learn. One example from my recent experience in research -- building custom dashboard pages for results of scientific analyses. Each analysis is bespoke, and building interactive webpages is simply not the skill many researchers have (and it's boring IMO). But here with LLM you could easily explore the results visually/share them with collaborators etc. There are plenty examples like that. But certainly there are cases where learning is required.
This "instant" loop is so fast because it doesn't involve the part where you learn
You seem to be hung up on the part about "learning". Do you know how x86 registers work? Or atomic instructions available on ARM, SPARC, or POWER architectures?
No, of course not. Because all of that got abstracted to higher-level instructions decades ago.
Yes, I do. And it wouldn't take you that long to learn, either.
I don't think AI use is supposed to replace foundational learning such as reading a C++ book or Python book or CSS tutorial when you're a beginner. You still have to do those things if you want to be a professional or a strong amateur. But many people just want to get the thing done. They don't want to become a mechanic, they just want to drive from A to B.
> They don't want to become a mechanic, they just want to drive from A to B.
I'm fairly certain the article is directed at professionals, or at least the AI companies are basing their valuations off of directly taking a slice of that professional "productivity".
I've seen junior engineers be productive on their first day in the industry because of AI, so I don't think the article is the whole truth.
The example math is boundary-pushing and definitely not a solved problem. But most of us work on CRUD backends with a React frontend. Those are more or less solved problems that have well-documented solutions. For those kinds of tasks, LLMs just reward usage.
I can count on one hand the number of times in my career I've needed to solve a problem that's not described on Stack Overflow.
As they said in the 80s or maybe earlier RTFM. I think if you got a good enough duster TFM was still readable in 2010.
the author forgot you could also do a secret third thing: learn!
Except LLMs will only tend to share the most common or average of what it knows as the standard and deviating from it (including new ways) it can be resistant to.
An expert can lay a different kind of frame to prevent the llm to fell out of its way of being generally too verbose, and that can transfer as well to code generation and complication.
Not sure I agree with this. The math guy at anthropic's prompts are essentially:
Tao's chat was for him to gain intuition, not to solve the problem from the outset.What's funny is that every other person gets a different conclusion about who these models reward/empower. I've seen people say that the generalist stands to gain the most and others say that it's the experts. Like all of life, maybe the "winner" is the person who just does stuff.
It depends on the levels. People with differing fitness levels and ages run at very different paces. Now, do cars make them more equal or less? On the bottom end, the tide lifts all boats. Most healthy people can learn to drive and will drive "fine", they get from A to B. Out there in the city streets the car flattens the differences, everyone roughly takes the same time to get from A to B in a car.
But at the top of top, the gap probably widens. A professional F1 driver will drive laps around some random guy. It amplifies reflexes etc, because at that speed little differences in timing make a big difference.
Now, AI coding isn't exactly analogous, but I think it also has these two regimes. It flattens things for simple tasks. If your task is to shovel data, do some trivial compiler wrangling staring at badly designed error messages, looking through GitHub issues hunting for the comment with many tadaa emojis to fix an issue etc, those things can now be done by anyone. Just as grandpa can also drive to the grocery store. But if you're pushing at things on a higher level, now only your above-AI ability matters. If all the things that AI can do well are subtracted out, how much other expertise do you have left? This will be proportionally a bigger and bigger difference between different people.
So you're saying that it depends whether you're measuring "success at task X" (where in average everyone improves) vs comparative success (where people with knowledge can do far and away better at specific tasks).
This works better for math because math is self-verifiable. Once you have a proof it needs no outside evidence.
Expertise is needed to evaluate model outputs where it can't verify itself, or at the very least one's expertise can help steer the model in the right direction.
However this is irrelevant if models themselves are better at evaluating/leveraging expertise/information.
Corollary to this is an important part of LLM usage is what I call pinning it to reality. That is, designing verification steps that interact with the real world in some way not easy to hallucinate or work around. This means things like having code that interacts with the physical world, round trip tests, arriving at the same result using different paths, interoperability / replication with external libraries / competing products, performance improvement projects that start with robust performance test suites, and similar sorts of things that reduce to "how do I provide evidence that's difficult to fool myself about".
This includes things like "before you start fixing this bug, write two tests that fail proving it exists".
Expertise is good, but a wise expert will set up methods for the machine to prove to itself that a desired result is achieved removing the expert from the tight development loop.
And what about problems that cannot be one-shotted but helped along?
There was one math proof that was AI generated going around Twitter and the chat transcript to generate it was basically the human prompting "keep going" until it solved it.
Tao's chat was fascinating because the questions he was asking belied expert knowledge of the subject that only a handful of people could have asked.
Yup I linked that in my other comment but copy-pasted here for everyone's convenience:
The counterexample of the Dinitz-Garg-Goemans conjecture was basically just "keep going" and finally "enough of partial results. now finish with a complete unconditional counterexample"
https://x.com/DmitryRybin1/status/2079904005652893709
https://chatgpt.com/share/6a60b2eb-0b64-83ee-9c76-7931ca1de0...
Yes, we're in the low hanging fruit stage where folks can just say "here's the problem" and "keep going" until a result is found and it will work sometimes.
The low hanging fruit will run short. Ultimately mathematics is a field of subjective selections of problems and proofs as beautiful and interesting. Machines absolutely will struggle with what to study, what theorems are desirable, and when do be done with a proof.
> Machines absolutely will struggle with what to study, what theorems are desirable, and when do be done with a proof
And why do you think this would be the case? I'm not talking about today but in 1-2 years. For reference o1 was released less than 2 years ago, and we've had reasonable coding agents for 9 months or so.
> why do you think this would be the case?
Mathematics is ultimately an aesthetic pursuit. Outside of a well defined goal ML models don't have any sense of taste and regardless of the scaling that's been enabled in the last year or so of capability if they haven't memorized the process of doing something they have the same limitations of inability to make choices about unknowns not trained into them.
Real synthetic intelligence seems to me to be still very far away and not a matter of making models bigger or more efficient.
> who these models reward/empower
The easy, straightforward answer is "the people who own the models". Who else benefits feels like a more complex question and we'll have to see...
> Like all of life, maybe the "winner" is the person who just does stuff.
Someone who just does stuff still has to be able to deal with errors and failures. That’s where an expert or a generalist may have an advantage.
LLMs are a collection of biases. Humans are also a collection of biases. So we project our biases as input through the biases of an LLM and get an output. Hence why I think getting optimal output requires being an optimal person. And in that sentence there are many points of expression.
Finally, we train our LLMs on who we are. Another reinforcement of biases.
I agree. When I talk to LLMs about fields I am familiar with, I can push back on bad suggestions and ignore faulty/incorrect advice and assumptions, which is much harder for unfamiliar subjects.
Of course, simple common sense and extremely basic Googling on unfamiliar subjects can produce similar results, but it's much faster if you are truly understanding what the AI is suggesting.
I do find that "signalling expertise" is important. "I have a significant background in biblical scholarship. You can assume I've read the most important works in NT studies in particular. Do not translate Greek, Latin, Hebrew, or Syriac. Now, I would like to know . . ." That changes things significantly. So does telling it you have 20+ years of experience with C programming, that you have a robust understanding of machine organization, memory layouts, embedded systems, etc.
For sure. On a personal coding project I said "I'm a professional software engineer, and while this is a hobby project I'm not just vibe-coding and want to build reliable software" and the agent suddenly started suggesting all kinds of things to make its code more robust.
The amplifying mirror analogy works best here. LLMs are ultimately a reflection of your own interactions with its weights, the tone you use, the structure with which you construct your prompt, aspects of an issue you tend to focus on, your breadth of vocabulary and world knowledge and whatnot.
People who (carefully) use it as an extension of their own mind and senses will very likely thrive, and those who use it as a replacement for their minds and their senses will struggle.
One of the Claude skills I made Claude itself generate was the 'learning a concept across tiers' skill -- from ELI5 level to a PhD level, and it triggers whenever I ask it a very general question on a complex topic that isn't my bread-and-butter. The fact that I'm able to choose explanation level from a super smart LLM (that's available 24x7) that can explain any topic under the sun would've been mind-bogglingly sci-fi-ish just 4 years ago in 2022.
The short version I give to non-technical people who ask me about whether "AI will replace coding" is this: it accelerates you. You can get much further much more quickly.
If you don't know where you're going or how to get there, or even if you're just not paying enough attention, it will get you very far in the wrong direction before you've realised.
I totally see this. I just did 3 hours of bot sitting to put together some thrash loops that thrash our provisioning working flow for a BLE gadget we make. It was pretty straightforward and productive. But then, I have a lot of experience with BLE, and a quite a bit of experience with python and shell scripting. So I was able to guide the process through stages, do some intermediate testing, make some adjustments, and proceed. Domain experience made this really easy and straightforward. Me two junior engineers who have only superficial/high level knowledge of BLE and some of the other pieces, couldn't have done this as effectively.
Where my angst comes, is worrying that no one will ever get that experience anymore. They might have had some eventual success, who knows what monstrosity a much less guided LLM would have done, but experential learning may be mostly a thing of the past. And it creates a real tension between the person with experience and the person without.
The LLM is like the death star. If you don't know exactly where to point it, you will likely miss your target and have no/negative effect. The further away the target, the more accurate your firing solution needs to be. If all you need to do is add something like a dark mode theme to an existing product, this is probably a point blank shot in this metaphor. Building an entire codebase from zero, or even refactoring a legacy codebase into a new codebase, are lightyears away by comparison. You can still land the shot, but you need to deeply understand the metrology and astrodynamics.
The information system required to encode the aesthetic preferences needed to make a technology experience not suck is likely in excess of what any near-term solution will offer. Knowing when to say "no" is perhaps the most important skill here. You can't just say it arbitrarily either. You really have to mean it and be willing to fight other humans for it.
There's a growing and fascinating divide between people who see LLMs as more of a "bicycle for the mind" in the vein of Jobs vs those who see them as whollly supplanting the role of human intelligence. I can't help but wonder if these aren't primarily two human archetypes more than anything - the LLMs can be both and they erect a mirror of the human using them. Some humans really don't want deep individual expertise and intelligence to be the deciding factor because they don't identify with that. Others are completely the opposite. We really can't tell which will be more effective yet, because LLMs are very good in both modes. But most of the predictions currently are people executing on wishful thinking about what they hope will be the outcome.
Well, nice post. Actually, there may be some truth behind it, but basically, it captures what I—as a programmer—want to read: expertise will remain valuable.
But how I am observing is different, though. Since LLMs the gap between experts and non-experts has been shrinking. And yes, there is still a gap, but vanishing.
I am feeling this a lot lately. Getting the most out of agents seems to require being able to ask the right question. And how can you ask the right questions without deep domain expertise?
Yes sometimes it’s a matter of just using the right word. You can talk to an agent about a general concept for hours and hours and it may never mention $Concept_X, but you mention $Keyword_Y and all of a sudden the AI is going on about how $Concept_X is foundational to understanding the whole thing.
Agree with this. LLMs multiply the human user's ability. More ability, more impact!
And unfortunately, more ineptitude, more chaos.
The counterexample of the Dinitz-Garg-Goemans conjecture was basically just "keep going" and finally "enough of partial results. now finish with a complete unconditional counterexample" lol
https://x.com/DmitryRybin1/status/2079904005652893709
https://chatgpt.com/share/6a60b2eb-0b64-83ee-9c76-7931ca1de0...
Might be a very noob question but in this era of LLMs, let me ask the reverse, how do you gain expertise? It seems this rewards people who had expertise pre LLMs, but what about people who don’t have that in a specific domain? What approaches are viable now in this current system?
Same way you get strong in an age of heavy machinery: Lift heavy weights yourself.
Skills will have to be built through artificial constraints. Pen & paper, reading books, not using AI, etc.
The same as it's been! Make things without using LLMs. Don't debug with them, don't use them to research things, just do it yourself. It'll be painful and that pain is learning.
I'm not sure I understand the question. What would prevent you from doing what these people did now that LLMs are here?
Those people had no choice. In my opinion, it’s harder to grind through problems knowing very well an answer is a prompt away.
I think if we have a large population of generalists, then none of them are generalists after all
I believe they would still be called generalists.
Prompt an image or video generator without knowledge in photography or art skills and your results will look sloppy.
This matches my experience. Just Talk To It is the best method for working with LLMs if you're an expert.
I've seen this at work (as eng manager/lead/principal/whoevenknowsanymore) – all the big APIs give you stats. We see how much people burn in tokens and we know how much output they produce. There is a pretty strong inverse correlation between token burn and output.
The more tokens people burn, the less likely they are to produce a good outcome.
Yes. I agree with most, if not all of this. For instance, I am seeing folks either relying in the LLM as an _assumed_ expert or, assuming someone - who knows the structure of skill definitions - also has some expertise (in the area of the skill). It's a difficult situation; there is not much point in explaining _why_ the LLM output or skill in use (on a domain problem) isn't what the person actually _needs_ to address the domain problem, because the person isn't a domain expert or indeed, adjacent to domain expertise. But, it is an interesting experiemnt to arm folk with little domain expertise with the _skill_ necessary to be able to extract the right solution from the model.
This feels like a moment in time, not the end state of AI.
Like I read there was a time when teams of people + AI could beat pure AI at chess. But that these days, pure AI wins.
For all the things people say about "how AI works" you have to add the missing piece "how current AI works".
This is also why people's experience with LLMs/AI varies so much, because some people can see a use for AI for their needs and go about using the tool, while others do not as it relates to whatever they're working on and so they may say "LLMs/AI are useless" (it doesn't mean they're not experts though, although some people who have totally no expertise might also see no use for AI for themselves).
I came up with a catch phrase on this very topic a few days ago:
Using AI squares your abilities. The issue is that some abilities were below 1 to begin with.
You're basically playing the role of team lead to the LLM's junior dev.
I love larping as a vacant scrum master
This is true but also false.
In my experience (scientific programming) AI is a giant multiplier for people with specialized knowledge.
But it’s also a giant devaluer for that same knowledge as people with no idea what they’re doing can clog the field with plausible bullshit.
It’s now the case that if someone tells me they’ve done something, and I look into it and find out it’s completely AI slop, then I will have spent more time on the project than the person who “made” it. The situation is completely untenable and only serves to drain time and resources from people with better things to do.
we are slowly punishing reading comprehension
this will have educational consequences (that I'm trying to solve). I don't think that we can adjust without rapid education and making extreme specialists of us all.
This requires coordination, certification, licensing, and other tiers of authenticity. False experts can ruin sample gathering, can ruin training. False expertise is exemplified by the current American Administration. Look at Robert F. Kennedy Jr.; he's a false expert. He is responsible for the measles outbreak. He is responsible for ivermectin abuse by humans. False expertise is overtaking real expertise. And the results are continuously disastrous and large-scale.
I often do my best to represent a genuine interest in the subject at hand and learning in general to models. Imagine the model's response prose and mannerisms being on the other polar end of answering questions simply to get the correct answers as they're often scoped for on quantitative benchmarks. Not sure I explained this well, sorry. An LLM could help
I have lengthy conversations with my LLM, almost like an interview. I agree on the expertise part, because I wouldn't be able to go in depth on a subject with it if I lacked the expertise. Some work is a result of design and negotiations in those designs.
I don't think Tao's style works with everyone/thing, especially if we don't know what style he's tuned his LLM on.
Overall, I agree, when I ask things I'm an expert in and do professionally every day. I get very good useful answers. When, for example, our marketing people, ask about the science, they often get confusing and wrong answers.
I cant have an overall opinion but in my personal experience i have analysed that LLMs do reward concreteness
Skilled use may or may not matter, depending on the task. Do you need to do what Terence Tao is doing?
Yes. This is called the Matthew Principle:
> For to every one who has will more be given, and he will have abundance; but from him who has not, even what he has will be taken away.
LLMs reward architecture knowledge of how to structure things and how to not just say "Claude, make me Microsoft Flight Simulator, make no mistakes".
Many, including myself, report having a lot of success with braindumping and not structuring anything. Just talking into speech recognition for 2-10 minutes as a stream of consciousness about what my context is, what I want, what I know already, what I have a vague hunch about, how it fits into a bigger picture, what aspects are most important to me, any footguns I already know about, really like having a chat with a person on the phone, with someone you have to guide remotely because they have to implement the thing right now but you have to be out of office and so your only interface is speech. Except you can be more structureless because the AI won't be offended. Just keep on rambling, and press enter, don't even correct mistranscriptions. It will understand it anyway.
Now, the key is, that while rambling without structure, you do have to drop the key facts into your speech, and you have to know what you're talking about in at least a good portion of it.
I think people are afraid of doing it, because it seems "not the right way" or "not scientific" or whatnot. They want to believe there is some magic to writing the right prompt. So let me tell you, it works.
I don't completely disagree with the concept of giving a free association thought process ramble into context. But I also bet that when you start getting it to actually generate code and link modules of things together, subroutines, functions, code structure and filenames, you still pay attention to what it does and you guide it into the architecture that makes logical sense to you.
For real work yes. For personal projects, less and less since Fable came out (probably the same if true of the other frontier models). You can get a lot done if it's just some one off, or a personal tool, even without looking at the code, just trying the application. Frontier models now automatically test it before handing the thing to you, they take screenshots, they fix the superficial issues themselves. To get something up and running, it's enough to send chat messages.
Unfortunately the software industry is saying things like "don't look at the code", "LLMs have made developers 10-100x faster", etc. The only way they can make such claims is by saying what you said above: "Claude, make me Microsoft Flight Simulator, make no mistakes". Additionally engineers are facing pressures via deadlines to work in the paradigm of "Claude, make me Microsoft Flight Simulator, make no mistakes"...
The question i wonder about is, when will an event come along that persuades everyone that human understanding is still required? Or will it never come?
The event could be when fair pricing comes from the model providers. We're still at the cash burning stage. When the economy crashes a little and departments start monitoring their spending, and the prices for inference are 10x what they are, there will be less tolerance for employees to substitute constant AI usage for understanding.
I wonder the same thing. I think we've already seen some of this happening, however the consequences haven't been large enough to the organization, for example:
- https://www.theguardian.com/technology/2026/mar/20/meta-ai-a...
- https://tech.yahoo.com/articles/ai-code-wreaked-havoc-amazon...
- https://alexeyondata.substack.com/p/how-i-dropped-our-produc...
We can only hope that engineers working in safety critical systems haven't fallen to these working conditions.
Such an event would have to be pretty catastrophic at this point to slow down the inertia. Perhaps the tech debt will just pile up until someone's product implodes, or there's a massive safety issue that causes loss of life, or some big hedge fund goes bust.
That question makes me think about Boeing. Or NASA. Or Enron. Reality always wins, no matter what management and Investor Relations says.
It makes me think about The Terminator.
Yes. If you use the right technical terms together it’s lights up more specific feature spaces to your task.
Specificity matters to LLMs a lot.