"In the METR study, developers predicted AI would make them 24% faster before starting. After finishing 19% slower, they still believed they'd been 20% faster."
I hadn't heard of this study before. Seems like it's been mentioned on HN before but not got much traction.
I see it brought up almost every week! It's a firm favorite of the "LLMs don't actually help write code" contingent, probably because there are very few other credible studies they can point to in support of their position.
Most people who cite it clearly didn't read as far as the table where METR themselves say:
> We do not provide evidence that:
> 1) AI systems do not currently speed up many or most software developers. Clarification: We do not claim that our developers or repositories represent a majority or plurality of software development work
> 2) AI systems do not speed up individuals or groups in domains other than software development. Clarification: We only study software development
> 3) AI systems in the near future will not speed up developers in our exact setting. Clarification: Progress is difficult to predict, and there has been substantial AI progress over the past five years [3]
> 4) There are not ways of using existing AI systems more effectively to achieve positive speedup in our exact setting. Clarification: Cursor does not sample many tokens from LLMs, it may not use optimal prompting/scaffolding, and domain/repository-specific training/finetuning/few-shot learning could yield positive speedup
Weird, you shouldn't really need to list the things your study doesn't prove! I guess they anticipated that the study might be misrepresented and wanted to get ahead of that.
Their study still shows something interesting, and quite surprising. But if you choose to extrapolate from this specific setting and say coding assistants don't work in general then that's not scientific and you need to be careful.
I think the studyshould probably decrease your prior that AI assistants actually speed up development, even if developers using AI tell you otherwise. The fact it feels faster when it is slower is super interesting.
The lesson I took from the study is that developers are terrible at estimating their own productivity based on a new tool.
Being armed with that knowledge is useful when thinking about my own productivity, as I know that there's a risk of me over-estimating the impact of this stuff.
But then I look at https://github.com/simonw which currently lists 530 commits over 46 repositories for the month of December, which is the month I started using Opus 4.5 in Claude Code. That looks pretty credible to me!
That's certainly an impressive month! However, it's conceivable that you are an outlier (in the best possible way!)
I liked the way they did that study and I would be interested to see an updated version with new tools.
I'm not particularly sceptical myself and my guess is that using Opus 4.5 would probably have produced a different result to the one in the original study.
The lesson I learned is that agentic coding uses intermittent reinforcement to mimic a slot machine.
It (along with the hundreds of billions in investments hinging on it), explains the legions of people online who passionately defend their "system". Every gambler has a "system" and they usually earnestly believe it is helping them.
Some people even write popular (and profitable!) blogs about playing slots machines where they share their tips and tricks.
Plenty of people have been (too) quick to dismiss that study as not generally applicable because it was about highly experienced OSS devs rather than your average corporation programmer drone.
The issue I have with the paper is that it seems (based on my skimming) that they did not pick developers who were already versed with AI tooling. So they're comparing (experienced dev working in the way they're comfortable) vs (experienced dev working with new tool for the first time and not having passed the productivity slump from onboarding).
The thing I find interesting is that there is trillions of dollars in valuations hinging upon this question and yet the appetite to spend a little bit of money to repeat this study and then release the results publicly is apparently very low.
It reminds me of global warming where on one side of the debate there some scientists with very little money running experiments and on the other side there were some ridiculously wealthy corporations poking holes in those experiments.
Longitudinal studies are definitely needed, but of course at the time the research for this paper was done there weren't any programmers experienced with AI assist out there yet.
That's interesting context for sure, but the fact these were experienced developers makes it all the more surprising that they didn't realise the LLM slowed them down.
Measuring programming productivity in general is notoriously difficult, subjectively measuring your own programming productivity is even worse. A magic LoC machine saying brrrrrt gives an overoptimistic sense of getting things done.
If we take out most of frontend work, and the easy backend/Ops tasks where writing the code/config is 99% of the work, i think my overall productivity with the latest gen (basically Opus 4.5) improve by 15-20%. I also am _very_ sure that with the previous generation (Sonnet 4, sonnet 4.5, Codex 5.1), my team overall velocity decreased, even taking into account the frontend and the "easy" tasks. The amount of production bug we had to deal with this year is crazy. To much code is generated, and me and the other senior on my team just can't carefully review everything, we have to trust sometime (especially data structures).
The worse part is reading a PR, and catching a reintroduced bug that was fixed a few commit ago. The first time i almost lost my cool at work and said a negative thing to a coworker.
This would be my advice to juniors (and i mean basically: devs who don't yet understand the underlying business/architecture): use the AI to explain how stuff work, generate basic functions maybe, but write code logic/algorithm yourself until you are sure you understand what you're doing and why. Work and reflect on the data structures by yourself, even if generated by the AI, and ask for alternatives. Always ask for alternatives, it helps understanding.
You might not see huge productivity gains from AI, but you will improve first, and then productivity will improve very fast, from your brain first, then from AI.
>> Kernighan's Law - Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.
Now question is..
is AI providing solutions smarter than the developer using it might have produced?
And perhaps more importantly, How much time it takes AI to write code and human to debug it, even if both are producing equally smart solutions.
Just to add to your advice to juniors working with AI:
* Force the AI to write tests for everything. Ensure those tests function. Writing boring unit tests used to be arduous. Now the machine can do it for you. There's no excuse for a code regression making it's way into a PR because you actually ran the tests before you did the commit, right? Right? RIGHT?
* Force the AI to write documentation and properly comment code, then (this is the tricky part) you actually read what it said it was doing and ensure that this is what you wanted it to do before you commit.
Just doing these two things will vastly improve the quality and prevent most of the dumb regressions that are common with AI generated code. Even if you're too busy/lazy to read every line of code the AI outputs just ensuring that it passes the tests and that the comments/docs describe the behavior you asked for will get you 90% of the way there.
Sometimes the AI is all too good at writing tests.
I agree with the idea, I do it too, but you need to make sure the test don't just validate the incorrect behavior or that the code is not updated to pass the test in a way that actually "misses the point".
I've had this happen to me on one or two tests every time
For some reason Gemini seems to be worse at it than Claude lately. Since mostly moving to 3 I've had it go back and change the tests rather than fixing the bug on what seems to be a regular basis. It's like it's gotten smart enough to "cheat" more. You really do still have to pay attention that the tests are valid.
Even more important, those tests need to be useful. Often unit tests are simply testing the code works as written which is generally doing more harm than good.
To give some further advice to juniors: if somebody is telling you writing unit tests is boring, they haven’t learned how to write good tests. There appears to be a large intersection between devs who think testing is a dull task and devs who see a self proclaimed speed up from AI. I don’t think this is a coincidence.
Writing useful tests is just as important as writing app code, and should be reviewed with equal scrutiny.
AI almost always reduces the time from "I need to implement this feature" to "there is some code that implements this feature".
However in my experience, the issue with AI is the potential hidden cost down the road. We either have to:
1. Code review the AI generated code line by line to ensure it's exactly what you'd have produced yourself when it is generated or
2. Pay an unknown amount of tech tebt down the road when it inevitably wasn't what you'd have done yourself and it isn't extensible, scalable, well written code.
Exactly. Optimizations in one area will simply move the bottleneck so in order to truly recognize gains you have to optimize the entire software pipeline.
Exactly right. It turns out that writing code is hardly ever the real bottleneck. People should spend some time learning the basics of queueing theory.
RE 2: It's not that far down the road either. Laziliy reviewed or unreviewed LLM code rapidly turns your codebase into an absolute mess that LLMs can't maintain either. Very quickly you find yourself with lots of redundant code and duplicated logic, random unused code that's called by other unused code that gets called inside a branch that only tests will trigger, stuff like that. Eventually LLMs start fixing the code that isn't used and then confidently report that they solved the problem, filling up a context window with redundant nonsense every prompt, so they can't get anywhere. Yolo AI coding is like the payday loan of tech debt.
This can happen sooner than you think too. I asked for what I thought was a simple feature and the AI wrote and rewrote a number of times trying to get it right, and eventually (not making this up) it told me the file was corrupt and could I please restore it from backup. This happened within about 20-30 minutes of asking for the change.
Have you considered having AI code review the AI code before giving them off to a human? I've been experimenting with having claude work on some code and commit it, and then having codex review the changes in the most recent git commit, then eyeballing the recommendations and either having codex work the changes, or giving them back to claude. That has seemed to be quite effective so far.
A lot of the time, AI allows you to exercise basic competence at tasks for which you'd otherwise be incompetent. I think this is why it feels so powerful. You can jump into more or less any task below a certain level of complexity. (eg: you're not going to write an operating system with an LLM but you can set up and configure Wordpress if you'd never done it before.)
I think for users this _feels_ incredibly powerful, however this also has its own pitfalls: Any topic which you're incompetent at is one which you're also unequipped to successfully review.
I think there are some other productivity pitfalls for LLMs:
- Employees use it to give their boss emails / summaries / etc in the language and style their boss wants. This makes their boss happy, but doesn't actually modify productivity whatsoever since the exercise was a waste of time in the first place.
- Employees send more emails, and summarize more emails. They look busier, but they're not actually writing the emails or really reading them. The email volume has increased, however the emails themselves were probably a waste of time in the first place.
- There is more work to review all around and much of it is of poor quality.
I think these issues play a smaller part than some of the general issues raised (eg: poor quality code / lack of code reviews / etc.) but are still worth noting.
It's like Excel: It's really powerful to enable someone who actually knows what needs done to build a little tool that does that thing. It often doesn't have to be professional-quality, let alone perfect. It just has to be better than doing the same thing manually. There are massive productivity gains to be had there... for people with that kind of problem.
This is completely orthogonal to productivity gains for full time professional developers.
If anyone ever wonder why they don't see productivity improvement, they really need to read Mythical Man-Month.
Garage Duo can out-compete corporate because there is less overhead. But Garage Duo can't possibly output the sheer amount of work matching with corporate.
In my view the reasons why LLMs may be less effective in a corporate environment is quite different from the human factors in mythical man month.
I think that the reason LLMs don't work as well in a corporate environment with large codebases and complex business logic, but do work well in greenfield projects, is linked to the amount of context the agents can maintain.
Many types of corporate overhead can be reduced using an LLM. Especially following "well meant but inefficient" process around JIRA tickets, testing evidence, code review, documentation etc.
I've found that something very similar to those "inefficient" processes works incredibly well when applied to LLMs. All of those processes are designed to allow for seamless handoff to different people who may not be familiar with the project or code which is exactly what an LLM behaves like when you clear its context.
There have been methods to reduce overhead available over the history of our industry. Unfortunately almost all the times it involves using productive tools that would in some way reduce the head counts required to do large projects.
The way this works is you eventually have to work with languages like Lisp, Perl, Prolog, and then some one comes up with a theory that programming must be optimised for the mostly beginners and power tooling must be avoided. Now you are forced to use verbose languages, writing, maintaining and troubleshooting take a lot of people.
The thing is this time around, we have a way to make code by asking an AI tool questions. So you get the same effect but now with languages like JS and Python.
Sounds like AI slopish article. A whole section about "Why most enterprises don't" with many words but no actual data or analysis. Just assumptions based on orthogonal report.
AI won't give you much productivity if the problem you're challenged with is the human problem. That could happen both to startups and enterprises.
I think coding agents require fundamentally different development practices in order to produce efficiency improvements. And just like any new tool, they benefit from wisdom in how they are applied, which we are just starting to develop as an industry. I expect that over time we will grow to understand and also expand the circumstances in which they are a net benefit, while also appreciating where they are a hindrance, leading to an overall efficiency increase as we avoid the productivity hit resulting from their misapplication.
This article simply reinforces existing (and outdated) biases.
Complex legacy refactoring + Systems with poor documentation or unusual patterns + Architectural decisions requiring deep context: These go hand in hand. LLMs are really good at pulling these older systems apart, documenting, then refactoring them, tests and all. Exacerbated by poor documentation of domain expectations. Get your experts in a room weekly and record their rambling ideas and history of the system. Synthesize with an LLM against existing codebase. You'll get to 80% system comprehension in a matter of months.
Novel problem-solving with high stakes: This is the true bottleneck, and where engineers can shine. Risk assessment and recombination of ideas, with rapid prototyping.
Coz I have always done coding this way with humans I started out using LLMs to do simple bits of refactoring where tests could be used to validate that the output still worked.
I did not get the impression from this that LLMs were great coders. They would frequently miss stuff, make mistakes and often just ignore the instructions i gave them.
Sometimes they would get it right but not enough. The agentic coding loop still slowed me down overall. Perhaps if i were more junior it would have been a net boost.
In my experience, it’s basically impossible to accurately measure productivity of knowledge work. Whenever I see a stat associated to productivity gain/loss I get skeptical.
If you go the pure subjective route, I’ve found that people conflate “speed” or “productivity” with “ease.”
A key point missing from a lot of the AI debate is how much work is useless. From as simple as a feature that’s never turned on to a more extreme version of a job that doesn’t need to exist.
We have a lot of useless work being done, and AI is absolutely going to be a 10x speed up for this kind of work.
Actually though. We had one device that was over 10 years older without any MDM etc. and it outperformed a new laptop building the same product because of the corporate anti virus crap.
In programming we've often embraced spending time to learn new tools. The AI tools are just another set of tools, and they're rapidly changing as well.
I've been experimenting seriously with the tools for ~3 years now, and I'm still learning a lot about their use. Just this past weekend I started using a whole new workflow, and it one-shotted building a PWA that implements a fully-featured calorie tracking app (with social features, pre-populating foods from online databases, weight tracking and graphing, avatars, it's on par with many I've used in the past that cost $30+/year).
Someone just starting out at chat.openai.com isn't going to get close to this. You absolutely have to spend time learning the tooling for it to be at all effective.
I've worked at a number of non-tech companies the past few years. They bought every SaaS product, Palantir, Databricks, multi-cloud, their dev teams adopted every pattern popularized by big tech and the results were always mixed. Any gains were wiped out by being buried under technical debt. They had all the data catalogs & 'ontologies' with none of the governance to go make it work. Turns out that benefiting from all this tech requires you to re-organize and change your culture. For a lot of companies, they're just not going to see big gains from AI or tech in general at this point.
I think AI would have better general acceptance if we stopped mythologizing it's utility. It's so wildly over exaggerated it can't ever live up to the hype. If AI can't adapt to a reality-based universe, the bubble is going to burst all the sooner.
The thing you're likely missing is that you've forgotten what programming is at a high level.
A program is a series of instructions that tell a computer how to perform a task. The specifics of the language aren't as important as the ability to use them to get the machine to perform the tasks instructed.
We can now use English as that language, which allows more people than ever to program. English isn't as expressive as Python wielded by an expert, yet. It will be. This is bad for people who used to leverage the difficulty of the task to their own advantage, but good for everyone else.
Also, keep in mind that todays LLM's are the worst they'll ever be. They will continue to improve, and you will stagnate if you don't learn to use the new tools effectively.
> you will stagnate if you don't learn to use the new tools effectively
I've been going the other way, learning the old tools, the old algorithms. Specifically teaching myself graphics and mastering the C language. Tons of new grads know how to use Unity, how many know how to throw triangles directly onto the GPU at the theoretical limit of performance? Not many!
I did some of that when I was younger. I started with assembly and C, even though everyone told me to skip it and start with at least C++ or something further up the abstraction ladder. Ignoring them and gaining that knowledge has proven invaluable over the years.
Understanding a "deeper" abstraction layer is almost always to your advantage, even if you seldom use it in your career. It just gives you a glimpse behind the curtain.
That said, you have to also learn the new tools unless you tend to be a one man band. You'll find that employers don't want esoteric knowledge or all-knowing wizards who can see the matrix. Mostly, they just want a team member who can cooperate with other folks to get things done in whatever tool they can find enough skilled folks to use.
> you will stagnate if you don't learn to use the new tools effectively.
This is the first technology in my career where the promoters feel the need to threaten everyone who expresses any sort of criticism, skepticism, or experience to the contrary.
How old is your career then? I've been hearing some variation on "evolve or die" for about 30 years now, and it's been true every time... Except for COBOL. Some of those guys are still doing the same thing they were back then. Literally everything else has changed and the people that didn't keep up are gone.
I've seen interns (with academic background) build advanced UIs for projects while not having a background in coding. This would not have been possible without LLMs.
It doesn't really matter how "skilled" the designer is. Figma's MCP already provides HTML and CSS that's basically ready, and all AI needs to do is translate that into React or whatever. Or if you mean that AI wouldn't be able to make a proper interface without the human, that's also not true. The only reason I use Figma MCP is that my company uses Figma and has a dedicated Figma person. My opinion is that is just a bottleneck, and it would be easier to prompt AI to make whatever interface.
> The only reason I use Figma MCP is that my company uses Figma and has a dedicated Figma person. My opinion is that is just a bottleneck, and it would be easier to prompt AI to make whatever interface.
Here's where our opinions differ - I think replacing that Figma person with AI prompts will negatively affect product in a way that is noticeable to the end-user and effects their experience.
It does of course depend what kind of product you're making, but I'd say most of the time this holds.
I'm not even arguing that you should replace the Figma person with AI. I am arguing that even without AI, having Figma persons is a bottleneck. It is much faster to just use some kind of component library, like shadcn, and let the developer figure it out. And with AI that would be even faster, as the developer wouldn't have to write any code, just check the AI output and prompt to make changes if needed. Unless of course you need one of those really fancy landing pages, but even then, you would likely need a specialized developer, and not a Figma person.
If you work in B2B SaaS, sure, I guess. That's a lot of HN by virtue of being a lot of SF VC, but only a tiny part of all tech. Elsewhere shadcn isn't a realistic option.
I think LLMs have aggressively facilitated the rise of illiteracy in people attending software development university programs.
I think graduates of these programs are far, far worse software developers than they were in the recent past.
edit: i think you mean "irrelevant", not "irreverent". that being said, my response is an expansion of the point made in my comment that you replied to.
> I think LLMs have aggressively facilitated the rise of illiteracy in people attending software development university programs.
But this subthread is about interns who did not study CS, and are able to create advanced UIs using LLMs in the short time they had left to finish their project.
I'll start by saying that this seems irreverent to my previous comment.
That being said, I half agree but I think we see things differently. Based on what I've seen, the "illiterate" are those who would have otherwise dropped out or done a poor job previously. Now instead of exiting the field, or slowly shipping code they didn't understand (because that has always been a thing) they are shovelling more slop.
That's a problem, but it's at most gotten worse rather than come out of thin air.
But, there are still competent software engineers and I have seen with my own eyes how AI usage makes them more productive.
Similarly, some of those "illiterate" are those who now have the ability to make small apps for themselves to solve a problem they would not be able to before, and I argue that's a good thing.
Ultimately, people care about the solution to their problems, not the code. If (following the original anecdote) someone with an LLM can build a UI for their project I frankly don't think it matters whether they understood the code. The UI is there, it works, and they can get one with the thing that is actually important: using the UI for their bigger goal.
So what about Cursor's tab autocomplete? Seems like there is a spectrum of tooling between raw assembly all the way to vibe coding and I'm trying to see where you draw the line. Is it "if it uses AI, its bad" or are you more against the "hey build me something and I'm not even gonna check the results."
Can you describe anything about the difference between using ChatGPT with GPT-4o to write code in January 2025 and using Opus 4.5 with Claude Code in December 2025?
This isn't the smart, snappy reply that you believe it to be.
As a comment reader this exchange with Simon translates directly to "no, but you have forced me to try and misdirect because I can't reply in good faith to an expert who has forgotten more about LLMs than I'll ever know".
well you don't have to "think" anything lol, if you've never tried it yourself that's step one. step two is to not assume everyone out there is a shill or lying because that's awful convenient. also its not black or white.
developers can exist in a small team, solo, large enterprise all with their mandates and cultures so just saying LLMs increase/decrease is reductive.
You're alienating a lot of people just by calling them "LLM addicts," and I suspect you're not arguing in good faith given your language.
There are a lot of sad people who have developed parasocial relationships with ChatGPT, but that's entirely a separate issue from whether agents are a good tool for software engineering.
"Real" woodworkers often have similar reactions when they see people incorporating CNC machines into their creative process as you appear to have when it comes to your engagement style on this topic.
The METR study cited here is very interesting.
"In the METR study, developers predicted AI would make them 24% faster before starting. After finishing 19% slower, they still believed they'd been 20% faster."
I hadn't heard of this study before. Seems like it's been mentioned on HN before but not got much traction.
I see it brought up almost every week! It's a firm favorite of the "LLMs don't actually help write code" contingent, probably because there are very few other credible studies they can point to in support of their position.
Most people who cite it clearly didn't read as far as the table where METR themselves say:
> We do not provide evidence that:
> 1) AI systems do not currently speed up many or most software developers. Clarification: We do not claim that our developers or repositories represent a majority or plurality of software development work
> 2) AI systems do not speed up individuals or groups in domains other than software development. Clarification: We only study software development
> 3) AI systems in the near future will not speed up developers in our exact setting. Clarification: Progress is difficult to predict, and there has been substantial AI progress over the past five years [3]
> 4) There are not ways of using existing AI systems more effectively to achieve positive speedup in our exact setting. Clarification: Cursor does not sample many tokens from LLMs, it may not use optimal prompting/scaffolding, and domain/repository-specific training/finetuning/few-shot learning could yield positive speedup
https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...
Weird, you shouldn't really need to list the things your study doesn't prove! I guess they anticipated that the study might be misrepresented and wanted to get ahead of that.
Their study still shows something interesting, and quite surprising. But if you choose to extrapolate from this specific setting and say coding assistants don't work in general then that's not scientific and you need to be careful.
I think the studyshould probably decrease your prior that AI assistants actually speed up development, even if developers using AI tell you otherwise. The fact it feels faster when it is slower is super interesting.
The lesson I took from the study is that developers are terrible at estimating their own productivity based on a new tool.
Being armed with that knowledge is useful when thinking about my own productivity, as I know that there's a risk of me over-estimating the impact of this stuff.
But then I look at https://github.com/simonw which currently lists 530 commits over 46 repositories for the month of December, which is the month I started using Opus 4.5 in Claude Code. That looks pretty credible to me!
That's certainly an impressive month! However, it's conceivable that you are an outlier (in the best possible way!)
I liked the way they did that study and I would be interested to see an updated version with new tools.
I'm not particularly sceptical myself and my guess is that using Opus 4.5 would probably have produced a different result to the one in the original study.
The lesson I learned is that agentic coding uses intermittent reinforcement to mimic a slot machine.
It (along with the hundreds of billions in investments hinging on it), explains the legions of people online who passionately defend their "system". Every gambler has a "system" and they usually earnestly believe it is helping them.
Some people even write popular (and profitable!) blogs about playing slots machines where they share their tips and tricks.
Plenty of people have been (too) quick to dismiss that study as not generally applicable because it was about highly experienced OSS devs rather than your average corporation programmer drone.
The issue I have with the paper is that it seems (based on my skimming) that they did not pick developers who were already versed with AI tooling. So they're comparing (experienced dev working in the way they're comfortable) vs (experienced dev working with new tool for the first time and not having passed the productivity slump from onboarding).
The thing I find interesting is that there is trillions of dollars in valuations hinging upon this question and yet the appetite to spend a little bit of money to repeat this study and then release the results publicly is apparently very low.
It reminds me of global warming where on one side of the debate there some scientists with very little money running experiments and on the other side there were some ridiculously wealthy corporations poking holes in those experiments.
Longitudinal studies are definitely needed, but of course at the time the research for this paper was done there weren't any programmers experienced with AI assist out there yet.
That's interesting context for sure, but the fact these were experienced developers makes it all the more surprising that they didn't realise the LLM slowed them down.
Measuring programming productivity in general is notoriously difficult, subjectively measuring your own programming productivity is even worse. A magic LoC machine saying brrrrrt gives an overoptimistic sense of getting things done.
If we take out most of frontend work, and the easy backend/Ops tasks where writing the code/config is 99% of the work, i think my overall productivity with the latest gen (basically Opus 4.5) improve by 15-20%. I also am _very_ sure that with the previous generation (Sonnet 4, sonnet 4.5, Codex 5.1), my team overall velocity decreased, even taking into account the frontend and the "easy" tasks. The amount of production bug we had to deal with this year is crazy. To much code is generated, and me and the other senior on my team just can't carefully review everything, we have to trust sometime (especially data structures).
The worse part is reading a PR, and catching a reintroduced bug that was fixed a few commit ago. The first time i almost lost my cool at work and said a negative thing to a coworker.
This would be my advice to juniors (and i mean basically: devs who don't yet understand the underlying business/architecture): use the AI to explain how stuff work, generate basic functions maybe, but write code logic/algorithm yourself until you are sure you understand what you're doing and why. Work and reflect on the data structures by yourself, even if generated by the AI, and ask for alternatives. Always ask for alternatives, it helps understanding. You might not see huge productivity gains from AI, but you will improve first, and then productivity will improve very fast, from your brain first, then from AI.
>> Kernighan's Law - Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.
Now question is..
is AI providing solutions smarter than the developer using it might have produced?
And perhaps more importantly, How much time it takes AI to write code and human to debug it, even if both are producing equally smart solutions.
Just to add to your advice to juniors working with AI:
* Force the AI to write tests for everything. Ensure those tests function. Writing boring unit tests used to be arduous. Now the machine can do it for you. There's no excuse for a code regression making it's way into a PR because you actually ran the tests before you did the commit, right? Right? RIGHT?
* Force the AI to write documentation and properly comment code, then (this is the tricky part) you actually read what it said it was doing and ensure that this is what you wanted it to do before you commit.
Just doing these two things will vastly improve the quality and prevent most of the dumb regressions that are common with AI generated code. Even if you're too busy/lazy to read every line of code the AI outputs just ensuring that it passes the tests and that the comments/docs describe the behavior you asked for will get you 90% of the way there.
Sometimes the AI is all too good at writing tests.
I agree with the idea, I do it too, but you need to make sure the test don't just validate the incorrect behavior or that the code is not updated to pass the test in a way that actually "misses the point".
I've had this happen to me on one or two tests every time
I agree 100%.
For some reason Gemini seems to be worse at it than Claude lately. Since mostly moving to 3 I've had it go back and change the tests rather than fixing the bug on what seems to be a regular basis. It's like it's gotten smart enough to "cheat" more. You really do still have to pay attention that the tests are valid.
Even more important, those tests need to be useful. Often unit tests are simply testing the code works as written which is generally doing more harm than good.
To give some further advice to juniors: if somebody is telling you writing unit tests is boring, they haven’t learned how to write good tests. There appears to be a large intersection between devs who think testing is a dull task and devs who see a self proclaimed speed up from AI. I don’t think this is a coincidence.
Writing useful tests is just as important as writing app code, and should be reviewed with equal scrutiny.
And, you actually wrote the regression test when you fixed the bug, right? Right?
AI almost always reduces the time from "I need to implement this feature" to "there is some code that implements this feature".
However in my experience, the issue with AI is the potential hidden cost down the road. We either have to:
1. Code review the AI generated code line by line to ensure it's exactly what you'd have produced yourself when it is generated or
2. Pay an unknown amount of tech tebt down the road when it inevitably wasn't what you'd have done yourself and it isn't extensible, scalable, well written code.
Exactly. Optimizations in one area will simply move the bottleneck so in order to truly recognize gains you have to optimize the entire software pipeline.
Exactly right. It turns out that writing code is hardly ever the real bottleneck. People should spend some time learning the basics of queueing theory.
http://lpd2.com/
RE 2: It's not that far down the road either. Laziliy reviewed or unreviewed LLM code rapidly turns your codebase into an absolute mess that LLMs can't maintain either. Very quickly you find yourself with lots of redundant code and duplicated logic, random unused code that's called by other unused code that gets called inside a branch that only tests will trigger, stuff like that. Eventually LLMs start fixing the code that isn't used and then confidently report that they solved the problem, filling up a context window with redundant nonsense every prompt, so they can't get anywhere. Yolo AI coding is like the payday loan of tech debt.
This can happen sooner than you think too. I asked for what I thought was a simple feature and the AI wrote and rewrote a number of times trying to get it right, and eventually (not making this up) it told me the file was corrupt and could I please restore it from backup. This happened within about 20-30 minutes of asking for the change.
This is why I say LLMs are for idiots
>Code review the AI generated code line by line
Have you considered having AI code review the AI code before giving them off to a human? I've been experimenting with having claude work on some code and commit it, and then having codex review the changes in the most recent git commit, then eyeballing the recommendations and either having codex work the changes, or giving them back to claude. That has seemed to be quite effective so far.
Maybe it's turtles all the way down?
A lot of the time, AI allows you to exercise basic competence at tasks for which you'd otherwise be incompetent. I think this is why it feels so powerful. You can jump into more or less any task below a certain level of complexity. (eg: you're not going to write an operating system with an LLM but you can set up and configure Wordpress if you'd never done it before.)
I think for users this _feels_ incredibly powerful, however this also has its own pitfalls: Any topic which you're incompetent at is one which you're also unequipped to successfully review.
I think there are some other productivity pitfalls for LLMs:
- Employees use it to give their boss emails / summaries / etc in the language and style their boss wants. This makes their boss happy, but doesn't actually modify productivity whatsoever since the exercise was a waste of time in the first place.
- Employees send more emails, and summarize more emails. They look busier, but they're not actually writing the emails or really reading them. The email volume has increased, however the emails themselves were probably a waste of time in the first place.
- There is more work to review all around and much of it is of poor quality.
I think these issues play a smaller part than some of the general issues raised (eg: poor quality code / lack of code reviews / etc.) but are still worth noting.
"There is more work to review all around and much of it is of poor quality."
This is the average software developer's experience of LLMs
It's like Excel: It's really powerful to enable someone who actually knows what needs done to build a little tool that does that thing. It often doesn't have to be professional-quality, let alone perfect. It just has to be better than doing the same thing manually. There are massive productivity gains to be had there... for people with that kind of problem.
This is completely orthogonal to productivity gains for full time professional developers.
If anyone ever wonder why they don't see productivity improvement, they really need to read Mythical Man-Month.
Garage Duo can out-compete corporate because there is less overhead. But Garage Duo can't possibly output the sheer amount of work matching with corporate.
In my view the reasons why LLMs may be less effective in a corporate environment is quite different from the human factors in mythical man month.
I think that the reason LLMs don't work as well in a corporate environment with large codebases and complex business logic, but do work well in greenfield projects, is linked to the amount of context the agents can maintain.
Many types of corporate overhead can be reduced using an LLM. Especially following "well meant but inefficient" process around JIRA tickets, testing evidence, code review, documentation etc.
I've found that something very similar to those "inefficient" processes works incredibly well when applied to LLMs. All of those processes are designed to allow for seamless handoff to different people who may not be familiar with the project or code which is exactly what an LLM behaves like when you clear its context.
The limited LLM context windows could be an argument in favor of a microservices architecture with each service or library in its own repository.
the productivity improvement is the Big Lie
>>there is less overhead.
There have been methods to reduce overhead available over the history of our industry. Unfortunately almost all the times it involves using productive tools that would in some way reduce the head counts required to do large projects.
The way this works is you eventually have to work with languages like Lisp, Perl, Prolog, and then some one comes up with a theory that programming must be optimised for the mostly beginners and power tooling must be avoided. Now you are forced to use verbose languages, writing, maintaining and troubleshooting take a lot of people.
The thing is this time around, we have a way to make code by asking an AI tool questions. So you get the same effect but now with languages like JS and Python.
Sounds like AI slopish article. A whole section about "Why most enterprises don't" with many words but no actual data or analysis. Just assumptions based on orthogonal report.
AI won't give you much productivity if the problem you're challenged with is the human problem. That could happen both to startups and enterprises.
I think coding agents require fundamentally different development practices in order to produce efficiency improvements. And just like any new tool, they benefit from wisdom in how they are applied, which we are just starting to develop as an industry. I expect that over time we will grow to understand and also expand the circumstances in which they are a net benefit, while also appreciating where they are a hindrance, leading to an overall efficiency increase as we avoid the productivity hit resulting from their misapplication.
This article simply reinforces existing (and outdated) biases.
Complex legacy refactoring + Systems with poor documentation or unusual patterns + Architectural decisions requiring deep context: These go hand in hand. LLMs are really good at pulling these older systems apart, documenting, then refactoring them, tests and all. Exacerbated by poor documentation of domain expectations. Get your experts in a room weekly and record their rambling ideas and history of the system. Synthesize with an LLM against existing codebase. You'll get to 80% system comprehension in a matter of months.
Novel problem-solving with high stakes: This is the true bottleneck, and where engineers can shine. Risk assessment and recombination of ideas, with rapid prototyping.
When producing code is cheap, you can spend more time on verification testing.
Force the LLM to follow a workflow, have it do TDD, use task lists, have it write implementation plans.
LLMs are great coders, but subpar developers, help them be a good developer and you will see massive returns.
Coz I have always done coding this way with humans I started out using LLMs to do simple bits of refactoring where tests could be used to validate that the output still worked.
I did not get the impression from this that LLMs were great coders. They would frequently miss stuff, make mistakes and often just ignore the instructions i gave them.
Sometimes they would get it right but not enough. The agentic coding loop still slowed me down overall. Perhaps if i were more junior it would have been a net boost.
In my experience, it’s basically impossible to accurately measure productivity of knowledge work. Whenever I see a stat associated to productivity gain/loss I get skeptical.
If you go the pure subjective route, I’ve found that people conflate “speed” or “productivity” with “ease.”
A key point missing from a lot of the AI debate is how much work is useless. From as simple as a feature that’s never turned on to a more extreme version of a job that doesn’t need to exist.
We have a lot of useless work being done, and AI is absolutely going to be a 10x speed up for this kind of work.
[dead]
Lets give 99% of the company devices with 16gb of ram or less and force them to use 85% of it for security scans
- corporate
WHY CANT OUR DEVICES RUN TECHNOLOGIES ??????
- also corporate
Actually though. We had one device that was over 10 years older without any MDM etc. and it outperformed a new laptop building the same product because of the corporate anti virus crap.
If you don't exclude your build folders from the scan it will slow everything down tremendously.
>The AI fluency tax. This isn't free to learn.
In programming we've often embraced spending time to learn new tools. The AI tools are just another set of tools, and they're rapidly changing as well.
I've been experimenting seriously with the tools for ~3 years now, and I'm still learning a lot about their use. Just this past weekend I started using a whole new workflow, and it one-shotted building a PWA that implements a fully-featured calorie tracking app (with social features, pre-populating foods from online databases, weight tracking and graphing, avatars, it's on par with many I've used in the past that cost $30+/year).
Someone just starting out at chat.openai.com isn't going to get close to this. You absolutely have to spend time learning the tooling for it to be at all effective.
Another day, another evidently AI-written article about AI on the front page of HN...
Yup, closed as soon as I saw the classic "it's not x, it's y" pattern.
I've worked at a number of non-tech companies the past few years. They bought every SaaS product, Palantir, Databricks, multi-cloud, their dev teams adopted every pattern popularized by big tech and the results were always mixed. Any gains were wiped out by being buried under technical debt. They had all the data catalogs & 'ontologies' with none of the governance to go make it work. Turns out that benefiting from all this tech requires you to re-organize and change your culture. For a lot of companies, they're just not going to see big gains from AI or tech in general at this point.
I think AI would have better general acceptance if we stopped mythologizing it's utility. It's so wildly over exaggerated it can't ever live up to the hype. If AI can't adapt to a reality-based universe, the bubble is going to burst all the sooner.
[flagged]
The thing you're likely missing is that you've forgotten what programming is at a high level.
A program is a series of instructions that tell a computer how to perform a task. The specifics of the language aren't as important as the ability to use them to get the machine to perform the tasks instructed.
We can now use English as that language, which allows more people than ever to program. English isn't as expressive as Python wielded by an expert, yet. It will be. This is bad for people who used to leverage the difficulty of the task to their own advantage, but good for everyone else.
Also, keep in mind that todays LLM's are the worst they'll ever be. They will continue to improve, and you will stagnate if you don't learn to use the new tools effectively.
> you will stagnate if you don't learn to use the new tools effectively
I've been going the other way, learning the old tools, the old algorithms. Specifically teaching myself graphics and mastering the C language. Tons of new grads know how to use Unity, how many know how to throw triangles directly onto the GPU at the theoretical limit of performance? Not many!
I did some of that when I was younger. I started with assembly and C, even though everyone told me to skip it and start with at least C++ or something further up the abstraction ladder. Ignoring them and gaining that knowledge has proven invaluable over the years.
Understanding a "deeper" abstraction layer is almost always to your advantage, even if you seldom use it in your career. It just gives you a glimpse behind the curtain.
That said, you have to also learn the new tools unless you tend to be a one man band. You'll find that employers don't want esoteric knowledge or all-knowing wizards who can see the matrix. Mostly, they just want a team member who can cooperate with other folks to get things done in whatever tool they can find enough skilled folks to use.
I think this guy is smarter than every LLM user in the thread
> you will stagnate if you don't learn to use the new tools effectively.
This is the first technology in my career where the promoters feel the need to threaten everyone who expresses any sort of criticism, skepticism, or experience to the contrary.
It is very odd. I do not care for it.
"you will stagnate if you don't learn to use the new tools effectively."
this hostile marketing scheme is the reason for my hostile opposition to LLMs and LLM idiots.
LLMs do not make you smarter or a more effective developer.
You are a sucker if you buy into the hype.
Are you arguing that you can work in technology without learning new things?
Have you considered a career in plumbing? Their technology moves at a much slower rate and does not require you to learn new things.
How old is your career then? I've been hearing some variation on "evolve or die" for about 30 years now, and it's been true every time... Except for COBOL. Some of those guys are still doing the same thing they were back then. Literally everything else has changed and the people that didn't keep up are gone.
I've seen interns (with academic background) build advanced UIs for projects while not having a background in coding. This would not have been possible without LLMs.
Can they do it without LLMs?
If they can't, did they really do it in the first place?
Are they actually literate in the programming languages they're using?
I don't write any front-end code at work anymore. I use Figma MCP and Cursor and it can implement the design near perfectly on first try.
To be fair, this is presumably because a skilled human spends time properly making the design for you in Figma.
It doesn't really matter how "skilled" the designer is. Figma's MCP already provides HTML and CSS that's basically ready, and all AI needs to do is translate that into React or whatever. Or if you mean that AI wouldn't be able to make a proper interface without the human, that's also not true. The only reason I use Figma MCP is that my company uses Figma and has a dedicated Figma person. My opinion is that is just a bottleneck, and it would be easier to prompt AI to make whatever interface.
> The only reason I use Figma MCP is that my company uses Figma and has a dedicated Figma person. My opinion is that is just a bottleneck, and it would be easier to prompt AI to make whatever interface.
Here's where our opinions differ - I think replacing that Figma person with AI prompts will negatively affect product in a way that is noticeable to the end-user and effects their experience.
It does of course depend what kind of product you're making, but I'd say most of the time this holds.
I'm not even arguing that you should replace the Figma person with AI. I am arguing that even without AI, having Figma persons is a bottleneck. It is much faster to just use some kind of component library, like shadcn, and let the developer figure it out. And with AI that would be even faster, as the developer wouldn't have to write any code, just check the AI output and prompt to make changes if needed. Unless of course you need one of those really fancy landing pages, but even then, you would likely need a specialized developer, and not a Figma person.
If you work in B2B SaaS, sure, I guess. That's a lot of HN by virtue of being a lot of SF VC, but only a tiny part of all tech. Elsewhere shadcn isn't a realistic option.
I'm curios, where is "elsewhere"?
This is such a tired argument.
Could people write scientific code without python? If they can't, did they really do it in the first place?
Could people write code without use after free bugs without using a GC'd language? If they can't, did they really do it in the first place?
Could people make a website without WYSIWYG editor? If they can't, did they really make a website?
I think LLMs have aggressively facilitated the rise of illiteracy in people attending software development university programs.
I think graduates of these programs are far, far worse software developers than they were in the recent past.
edit: i think you mean "irrelevant", not "irreverent". that being said, my response is an expansion of the point made in my comment that you replied to.
> I think LLMs have aggressively facilitated the rise of illiteracy in people attending software development university programs.
But this subthread is about interns who did not study CS, and are able to create advanced UIs using LLMs in the short time they had left to finish their project.
I'll start by saying that this seems irreverent to my previous comment.
That being said, I half agree but I think we see things differently. Based on what I've seen, the "illiterate" are those who would have otherwise dropped out or done a poor job previously. Now instead of exiting the field, or slowly shipping code they didn't understand (because that has always been a thing) they are shovelling more slop.
That's a problem, but it's at most gotten worse rather than come out of thin air.
But, there are still competent software engineers and I have seen with my own eyes how AI usage makes them more productive.
Similarly, some of those "illiterate" are those who now have the ability to make small apps for themselves to solve a problem they would not be able to before, and I argue that's a good thing.
Ultimately, people care about the solution to their problems, not the code. If (following the original anecdote) someone with an LLM can build a UI for their project I frankly don't think it matters whether they understood the code. The UI is there, it works, and they can get one with the thing that is actually important: using the UI for their bigger goal.
Do you think IDE's, type checking, refactoring tools and autocomplete make developers stupider too? Serious question.
not at all, I think these are valuable tools
would you agree that LLMs make developer stupider?
edit: answer my question
So what about Cursor's tab autocomplete? Seems like there is a spectrum of tooling between raw assembly all the way to vibe coding and I'm trying to see where you draw the line. Is it "if it uses AI, its bad" or are you more against the "hey build me something and I'm not even gonna check the results."
... they are not going to give you a satisfying answer to your totally reasonable line of inquiry.
Looking at the brief history of their account, I don't think anything they are saying or asking is in remotely good faith.
Can you describe anything about the difference between using ChatGPT with GPT-4o to write code in January 2025 and using Opus 4.5 with Claude Code in December 2025?
[flagged]
This isn't the smart, snappy reply that you believe it to be.
As a comment reader this exchange with Simon translates directly to "no, but you have forced me to try and misdirect because I can't reply in good faith to an expert who has forgotten more about LLMs than I'll ever know".
what reads to you as "an expert who has forgotten more about LLMs than I'll ever know" reads to me as a crack cocaine smoker.
just write the code
The only person coming off as unhinged and out of touch with reality here is you.
Yeah, I don't think you've used any of the technology you are criticizing here.
well you don't have to "think" anything lol, if you've never tried it yourself that's step one. step two is to not assume everyone out there is a shill or lying because that's awful convenient. also its not black or white.
developers can exist in a small team, solo, large enterprise all with their mandates and cultures so just saying LLMs increase/decrease is reductive.
have a feeling i'm being trolled tho.
I'm not trolling, this is my sincere opinion
I think LLM addicts are particularly susceptible to flattery.
You're alienating a lot of people just by calling them "LLM addicts," and I suspect you're not arguing in good faith given your language.
There are a lot of sad people who have developed parasocial relationships with ChatGPT, but that's entirely a separate issue from whether agents are a good tool for software engineering.
"Real" woodworkers often have similar reactions when they see people incorporating CNC machines into their creative process as you appear to have when it comes to your engagement style on this topic.
They don't emerge looking credible, either.
equating a CNC router with Claude Code or Chat GPT is an egregious false equivalency.