This article feels like the result of having unlimited tokens at your disposal. It’s got some very expensive practices, where near-free practices would produce the same or better results.
There’s a long section in the middle where the article describes per-change verification via skill files and “ask Claude to use the chrome-for-Claude tool” when checking if changes have introduced regressions.
But there’s already tools which do this without spending tokens tokens. Playwright will achieve equal quality but deterministic results across the same verification attempts. It is write-once-use-often, so has a marginal cost near zero.
> It’s got some very expensive practices, where near-free practices would produce the same or better results.
I think tokens are near-free already, even for those of us outside Anthropic.
More importantly, your philosophy comes from a time where thinking had to be front-loaded: you try to predict everything that can go wrong, then design deterministic tests that can catch those things and a decision tree to figure out what to do in each case. This approach was the only viable way until now, but maybe that will change. Having the equivalent of a senior engineer thoroughly review each error with full context of what the application and feature is trying to achieve could perhaps uncover different insights, or cover edge cases more smoothly.
> you try to predict everything that can go wrong, then design deterministic tests that can catch those things and a decision tree to figure out what to do in each case
No real software engineer think that way, as things that can go wrong are near infinite. Instead you design the right thing and write tests that verify that it’s indeed the right thing and add some more that are the wrong thing, especially at the edge of definition. Then you rely on the dev to not make very obvious errors like coding something that is not in the requirements or is completely opposite.
A key pillar was introducing things like static analysis to provide cheap deterministic checks as the first goal and only then moving to other forms of verification.
You are not agreeing with the OP. You are just saying that there are many levels to testing and each give their own swiss cheese level of verification.
Sure you can have linting, compilation, unit tests, e2e tests. Do you really think Anthropic doesn't do this?
I was agreeing that this page as presented felt like it was from the POV of people will unlimited tokens to burn.
In particular constructing ( and often deconstructing on change ) differential tests is very expensive relative to cheaper things to set up.
I felt this article's argument was poorly presented because of this seeming contradiction. The video I linked was much clearer about what works and what doesn't when scaling to inhuman levels of PR merges.
I don't agree with "very expensive" claim. My friend works in a f500. They do releases like 2-3 times a week. Assuming each release uses $500 of tokens (higher estimate) just for verification, that's like $6000 a month and that's just a small percent over the tokens used for coding. What's the big deal? Their team brings in way more money than that - its a drop in the ocean.
I also know of companies that run very expensive test suites using spot instances in AWS that cost around $100 each time. No one complains about it.
I think all the posts complaining about expenses forget to compare the orders of magnitudes.
I had a developer friend who wanted to automate gym reservations at night, his first idea was to make use of ChatGPT Computer use, but he encountered refusal of the AI to enter his password, and it was difficult to convince it to do it without approval as well.
I just told him to make the AI write a small selenium python script to do the clicking and it worked amazingly.
What makes you think they are not using Playwright ?
For me this article is more aimed at devs that are still stuck treating code generation and verification as two separate tasks. Giving agents the right tools (including playwright) to do both at once makes things more token efficient, not less.
Quite possibly. But, a thought I had some years back:
In the future when AI is able to do everything, there will not be one single super-AI that does it all, but a whole ecosystem of different AI. This is the same as how in the biological world, while there are various degrees of generalists, there's no single cell, plant, or animal which does absolutely everything. If our body needs more blood, it makes more blood cells, it doesn't repurpose intestine or skin for the task.
Similar argument in economics, comparative advantage: an AI which can do everything is best used for the things which only it and no other AI can do, and leaving to other (smaller, cheaper) AI relatively simple tasks like face recognition.
Even where AI can do this, that doesn't mean the best solution is AI rather than a traditional API.
First, last I checked AlphaZero is a much better chess player than any LLM, and by a large margin.
Second, this isn't even about finding the best model for small tasks, it's about best use of finite resources, compute or energy, which means that it still holds even if a future AI (call it Alice) did everything better than AlphaZero does chess, it would still make sense for Alice to offload chess to a merely-good model like AlphaZero so that Alice could spend more time doing the things that nothing else besides Alice could do:
So the future is about replacing deterministic procedures with stochastic procedures?
On some level you are right, LLMs and the agents built with the various models and architectures will get cheaper. But there are some significant drawbacks by introducing this risk, regardless of cost. The future will need the capacity to understand this too.
Its almost become a meme at this point: well actually you can achieve the same using X so why waste tokens?!
The reason you should use Claude and not playwright always is because the the invariant you wanna capture is at the natural language level.
Like imagine I wanted to verify that Claude Code works and doesn't break. I can describe the invariant I want in natural language. Like: make sure that subagents don't block the original thread.
Doing this in playwright means that you have already bought into some previously existing architecture which means your tests are now correlated to your code. Having the invariants written in natural language allows you to decouple code vs the invariant.
I expect a lot of greybeard conservative programmers to keep opposing using AI here until eventually, not using AI will be seen as more risky.
The better thing is do use both, use claude to convert the invariant to deterministic code, and review said code to be sure it matches your natural language idea of it.
The issue with only having the natural language specification is that it is non-deterministic.
Wait, are you saying a company that makes money from selling tokens generated a content piece recommending people solve their problems by using a lot of tokens?
I do understand that it is extreme, but I think the principled stand makes my code and my skills better in the long run. I'm banking on building less, but building better. Otherwise where do you draw the line in losing quality and understanding? And for what I'm working on, typing is rarely the bottleneck anyways.
I still use LLMs (cautiously) for searching and troubleshooting API things.
Letting go of the typing isn’t a light decision. And it is harder to get good output when it’s also simply a lot of output.
I’m banking on that being an evolving, but necessary skillset. For a while at least. And I’m still gaining experience in the process.
Plenty of exposure, plenty of prior experience, it’s a weird mix of management-ish context handling and sme mentorship while also providing the research benefits you mentioned.
The overall process that I am using does use a lot more time and AI spend than just asking AI to do something. I still do the latter for simple or noncritical changes. I had to switch to using DeepSeek as my workhorse to keep down my spend.
Why aren't you spending $12k on tokens? Or better, $200k? Coding is solved, you can now infinite money glitch; just give your current money, which will be useless in the Agentic Future anyway, to Anthropic now.
You should fire yourself and have your boss go all in on anthropic duh. If you are the boss then you either shouldn't have gone down this path, accept your code won't be verified, or shut your doors.
Exactly this. I build large systems like erp, ipaas just from a Claude team with two premium licences. everything builds, verifies, deploys itself. I can track everything and implement features in a days. security and big fixing is background work. I would love to know what 2000$ worth of tokens do in your business.
> Nobody can read every change, so we use failures as one way to decide what needs a person.
Whether a human can or should review every diff is a conscious project level decision. I have yet to see a project where it is not feasible. The trick is to break things into small enough pieces that you can review them. Just like one would do if you were doing such changes manually. Nobody in their right mind performs non-mechanical changes of 100s of files.
The "right mind" of companies is a bit different from that of individuals. Generally they care about three things: Revenue, costs and risks.
To companies, some risks are worth taking, if it can increase revenue or lower costs. Many companies happily do not-quite-legal things, taking into account the potential of being caught and the size of the fine.
In the case of Anthropic, from that perspective, it almost seems like a no-brainer that they would want to push an "our AI can do it all, no humans needed" narrative in everything they do. They need hype like we need oxygen.
I would agree that for many others, the risk is too high, and the reward too low. So they'd indeed have to be out of their mind to deploy code nobody understands. But depends on the situation of course.
Just trying to say, in a really lengthy and possibly patronising fashion, that "good engineering" is unfortunately not something many (most?) companies prioritise.
In my opinion, the output of a stochastic process can only be truly verified through a deterministic process. Tests already do this for functional accuracy. Linters can do this for line-level code, with code treated as text. But there is currently no such method of deterministic verification for the structure of the entire program, or more importantly the viability of the architecture when you throw in connectivity and IOPS.
Perhaps we need a level above linters: a tool that constructs the abstract syntax tree and verifies its various branches against deterministic rules.
The more I read articles like this, the more I realize Claude is good for process prototyping, but you need to "bake" the process with flat code/script/organization to decrease costs.
The future of IA would actually be to large scale prototype easily, even more like Python was a prototype language.
> Flaky tests get the same treatment. Claude can't skip them, add sleeps or retry until they pass. If it can't find the root cause, it deletes the test and writes down why.
If the tests are hermetic, you should always be able to find the root cause (by bisecting to determine when the test became flaky).
If the tests are not hermetic, it should still be possible to identify the dependency that is making them flaky, and use that as a justification if it still makes sense to delete the test.
Issues are automatically closed after being inactive for too long. It's also true that many issues that are filed there have nothing to do with Claude Code proper. To conclude, these numbers (either many open, or many closed, or any relationship between them) a-priori mean nothing either way.
HN comments love to use the curmudgeonly tone but did you not stop to think that they are going as far as possible to let model do what it can and do everything else?
Did you forget that they literally publish benchmarks that show how much of the task can be done independently?
They poached a lot of people from companies like Google that have solid engineering practices. I imagine it's a culture issue if anything, not a knowledge/background issue.
Googles engineers are good in those types of environments. Most environments aren't Google like at all. Ive seen so many small and medium companies hire googlers and burn money because of it. I actually watched a start up die because of a few key middle management/staff engineer hires from Google. They just could not operate for anyone other then themselves and were oblivious to the scale they worked on.
Apparently, the remaining 10% too, but not attributed correctly, so they are careful of claiming it as Claude-written. (Can't remember where I've read that.)
This article feels like the result of having unlimited tokens at your disposal. It’s got some very expensive practices, where near-free practices would produce the same or better results.
There’s a long section in the middle where the article describes per-change verification via skill files and “ask Claude to use the chrome-for-Claude tool” when checking if changes have introduced regressions.
But there’s already tools which do this without spending tokens tokens. Playwright will achieve equal quality but deterministic results across the same verification attempts. It is write-once-use-often, so has a marginal cost near zero.
> It’s got some very expensive practices, where near-free practices would produce the same or better results.
I think tokens are near-free already, even for those of us outside Anthropic.
More importantly, your philosophy comes from a time where thinking had to be front-loaded: you try to predict everything that can go wrong, then design deterministic tests that can catch those things and a decision tree to figure out what to do in each case. This approach was the only viable way until now, but maybe that will change. Having the equivalent of a senior engineer thoroughly review each error with full context of what the application and feature is trying to achieve could perhaps uncover different insights, or cover edge cases more smoothly.
> you try to predict everything that can go wrong, then design deterministic tests that can catch those things and a decision tree to figure out what to do in each case
No real software engineer think that way, as things that can go wrong are near infinite. Instead you design the right thing and write tests that verify that it’s indeed the right thing and add some more that are the wrong thing, especially at the edge of definition. Then you rely on the dev to not make very obvious errors like coding something that is not in the requirements or is completely opposite.
I agree completely, there was a much better talk on this by Lauren Tan: https://www.youtube.com/watch?v=O_zcc6Xl12g
A key pillar was introducing things like static analysis to provide cheap deterministic checks as the first goal and only then moving to other forms of verification.
You are not agreeing with the OP. You are just saying that there are many levels to testing and each give their own swiss cheese level of verification.
Sure you can have linting, compilation, unit tests, e2e tests. Do you really think Anthropic doesn't do this?
I was agreeing that this page as presented felt like it was from the POV of people will unlimited tokens to burn.
In particular constructing ( and often deconstructing on change ) differential tests is very expensive relative to cheaper things to set up.
I felt this article's argument was poorly presented because of this seeming contradiction. The video I linked was much clearer about what works and what doesn't when scaling to inhuman levels of PR merges.
I don't agree with "very expensive" claim. My friend works in a f500. They do releases like 2-3 times a week. Assuming each release uses $500 of tokens (higher estimate) just for verification, that's like $6000 a month and that's just a small percent over the tokens used for coding. What's the big deal? Their team brings in way more money than that - its a drop in the ocean.
I also know of companies that run very expensive test suites using spot instances in AWS that cost around $100 each time. No one complains about it.
I think all the posts complaining about expenses forget to compare the orders of magnitudes.
If you take the post at face value they are saying some of those tests aren't valuable
That developer brags about slopping out 2000+ PRs a month, how is anything they have to say useful?
I had a developer friend who wanted to automate gym reservations at night, his first idea was to make use of ChatGPT Computer use, but he encountered refusal of the AI to enter his password, and it was difficult to convince it to do it without approval as well.
I just told him to make the AI write a small selenium python script to do the clicking and it worked amazingly.
It seems like a new stack where you try to push things down as low as possible for efficiency.
What makes you think they are not using Playwright ?
For me this article is more aimed at devs that are still stuck treating code generation and verification as two separate tasks. Giving agents the right tools (including playwright) to do both at once makes things more token efficient, not less.
They are living in the future.
I think future AI cost will be as cheap as doing an API call in classic sense.
Many API calls can be answered by something as underpowered as an Raspi. But AI needs very expensive GPUs to run.
So no way it will be as cheap. That is just wishfull thinking.
Quite possibly. But, a thought I had some years back:
In the future when AI is able to do everything, there will not be one single super-AI that does it all, but a whole ecosystem of different AI. This is the same as how in the biological world, while there are various degrees of generalists, there's no single cell, plant, or animal which does absolutely everything. If our body needs more blood, it makes more blood cells, it doesn't repurpose intestine or skin for the task.
Similar argument in economics, comparative advantage: an AI which can do everything is best used for the things which only it and no other AI can do, and leaving to other (smaller, cheaper) AI relatively simple tasks like face recognition.
Even where AI can do this, that doesn't mean the best solution is AI rather than a traditional API.
You are dangerously close to dismissing the Bitter Lesson.
https://en.wikipedia.org/wiki/Bitter_lesson
Hardly.
First, last I checked AlphaZero is a much better chess player than any LLM, and by a large margin.
Second, this isn't even about finding the best model for small tasks, it's about best use of finite resources, compute or energy, which means that it still holds even if a future AI (call it Alice) did everything better than AlphaZero does chess, it would still make sense for Alice to offload chess to a merely-good model like AlphaZero so that Alice could spend more time doing the things that nothing else besides Alice could do:
https://en.wikipedia.org/wiki/Comparative_advantage
So the future is about replacing deterministic procedures with stochastic procedures?
On some level you are right, LLMs and the agents built with the various models and architectures will get cheaper. But there are some significant drawbacks by introducing this risk, regardless of cost. The future will need the capacity to understand this too.
Its almost become a meme at this point: well actually you can achieve the same using X so why waste tokens?!
The reason you should use Claude and not playwright always is because the the invariant you wanna capture is at the natural language level.
Like imagine I wanted to verify that Claude Code works and doesn't break. I can describe the invariant I want in natural language. Like: make sure that subagents don't block the original thread.
Doing this in playwright means that you have already bought into some previously existing architecture which means your tests are now correlated to your code. Having the invariants written in natural language allows you to decouple code vs the invariant.
I expect a lot of greybeard conservative programmers to keep opposing using AI here until eventually, not using AI will be seen as more risky.
The better thing is do use both, use claude to convert the invariant to deterministic code, and review said code to be sure it matches your natural language idea of it.
The issue with only having the natural language specification is that it is non-deterministic.
Wait, are you saying a company that makes money from selling tokens generated a content piece recommending people solve their problems by using a lot of tokens?
Scandalous.
Here's how to properly use XYZ, yes it'll cost you more but we manufacture it so to us it's no issue.
Do you not see the propaganda through the pseudo helpfulness?
It's the same reason they give you a recipe for Chex Mix on the back of every box of Chex.
> Do you not see the propaganda through the pseudo helpfulness?
I find it amusing that people phrase bog-standard "what can our product do" documents as "propaganda", even when it's a dogfooding exercise.
Like, I guess? In the sense that all marketing can be seen as capitalist propaganda?
Marketing is for products, propaganda is for agendas.
Most businesses have an agenda of "sell more product".
Tokens are their product.
It behooves the token merchant to espouse the benefits of having more tokens.
Not that it is an explicit, conscious direction, but it very much is one.
You’re not wrong, in that free flowing tokens are enough to convince such a group it’s simply good advice.
The more you use AI to write your codebase, the more you need AI to manage your codebase.
The more you use product X the more product X you need to buy. Nice to be selling product X.
I'm not working for a manager, so I don't need to use it at all.
This feels a bit like the other extreme.
It can meaningfully “type” faster than you.
Take the handicap if you really wish to, I suppose.
I do understand that it is extreme, but I think the principled stand makes my code and my skills better in the long run. I'm banking on building less, but building better. Otherwise where do you draw the line in losing quality and understanding? And for what I'm working on, typing is rarely the bottleneck anyways.
I still use LLMs (cautiously) for searching and troubleshooting API things.
Letting go of the typing isn’t a light decision. And it is harder to get good output when it’s also simply a lot of output.
I’m banking on that being an evolving, but necessary skillset. For a while at least. And I’m still gaining experience in the process.
Plenty of exposure, plenty of prior experience, it’s a weird mix of management-ish context handling and sme mentorship while also providing the research benefits you mentioned.
Chinese gonna beat anthropic because they gluttonous use of AI.
You don't get a nice juicy IPO valuation by trying to save money or be efficient with your token usage!
This is the main thing that people are missing with their agentic coding.
After I created this skill for verification, it completely changed my level of trust with AI coding.
https://github.com/gregwebs/skills-sdlc/blob/main/skills/ver...
The overall process that I am using does use a lot more time and AI spend than just asking AI to do something. I still do the latter for simple or noncritical changes. I had to switch to using DeepSeek as my workhorse to keep down my spend.
https://github.com/gregwebs/skills-sdlc/blob/main/skills/imp...
> With them, you can run dozens or hundreds in parallel.
Im already spending my $2k usd per month in tokens with just one or two agents. How do they expect me to do that?
I don't know why you are comparing your personal spend with how a profitable enterprise company works.
I have worked in companies that spend millions a month on some random infra. I'm obviously not going to complain that its not sustainable?!
Edit: can't reply but Token maxxing is not over. We are just getting started. And please keep in mind two things are happening
1. AI costs are reducing by 2x every few months
2. This causes more usage because of Jevons Paradox
3. Also, running verification tests on releases is not that frequent and it happens only once for a whole team of 5-6 people
Why aren't you spending $12k on tokens? Or better, $200k? Coding is solved, you can now infinite money glitch; just give your current money, which will be useless in the Agentic Future anyway, to Anthropic now.
This is probably aimed at gullible governments to burn tokens using tax payer money.
You should fire yourself and have your boss go all in on anthropic duh. If you are the boss then you either shouldn't have gone down this path, accept your code won't be verified, or shut your doors.
Exactly this. I build large systems like erp, ipaas just from a Claude team with two premium licences. everything builds, verifies, deploys itself. I can track everything and implement features in a days. security and big fixing is background work. I would love to know what 2000$ worth of tokens do in your business.
> Nobody can read every change, so we use failures as one way to decide what needs a person.
Whether a human can or should review every diff is a conscious project level decision. I have yet to see a project where it is not feasible. The trick is to break things into small enough pieces that you can review them. Just like one would do if you were doing such changes manually. Nobody in their right mind performs non-mechanical changes of 100s of files.
The "right mind" of companies is a bit different from that of individuals. Generally they care about three things: Revenue, costs and risks.
To companies, some risks are worth taking, if it can increase revenue or lower costs. Many companies happily do not-quite-legal things, taking into account the potential of being caught and the size of the fine.
In the case of Anthropic, from that perspective, it almost seems like a no-brainer that they would want to push an "our AI can do it all, no humans needed" narrative in everything they do. They need hype like we need oxygen.
I would agree that for many others, the risk is too high, and the reward too low. So they'd indeed have to be out of their mind to deploy code nobody understands. But depends on the situation of course.
Just trying to say, in a really lengthy and possibly patronising fashion, that "good engineering" is unfortunately not something many (most?) companies prioritise.
> Nobody in their right mind performs non-mechanical changes of 100s of files.
We're in a brave new world, many many big name companies are doing this.
Their status pages reflect that.
> Nobody in their right mind performs non-mechanical changes of 100s of files.
Not 6 months ago, sure. Today? It's not totally crazy. In a year it will be routine.
In my opinion, the output of a stochastic process can only be truly verified through a deterministic process. Tests already do this for functional accuracy. Linters can do this for line-level code, with code treated as text. But there is currently no such method of deterministic verification for the structure of the entire program, or more importantly the viability of the architecture when you throw in connectivity and IOPS.
Perhaps we need a level above linters: a tool that constructs the abstract syntax tree and verifies its various branches against deterministic rules.
How to do this for architecture: unsure.
The more I read articles like this, the more I realize Claude is good for process prototyping, but you need to "bake" the process with flat code/script/organization to decrease costs.
The future of IA would actually be to large scale prototype easily, even more like Python was a prototype language.
> Flaky tests get the same treatment. Claude can't skip them, add sleeps or retry until they pass. If it can't find the root cause, it deletes the test and writes down why.
If the tests are hermetic, you should always be able to find the root cause (by bisecting to determine when the test became flaky).
If the tests are not hermetic, it should still be possible to identify the dependency that is making them flaky, and use that as a justification if it still makes sense to delete the test.
excellent content from the team. very concise and clear
> https://github.com/anthropics/claude-code/issues has over 16k open issues
this u?
And 90k closed. Looks like it's doing a pretty good job.
Issues are automatically closed after being inactive for too long. It's also true that many issues that are filed there have nothing to do with Claude Code proper. To conclude, these numbers (either many open, or many closed, or any relationship between them) a-priori mean nothing either way.
Opening and closing GitHub issues does not necessarily mean doing some job.
A better signal would be average issue age.
Why is the team thinking? Their product should be doing that for them
HN comments love to use the curmudgeonly tone but did you not stop to think that they are going as far as possible to let model do what it can and do everything else?
Did you forget that they literally publish benchmarks that show how much of the task can be done independently?
Why is there a team? Their product should be the team. /s
Can they not afford someone with an engineering background?
They poached a lot of people from companies like Google that have solid engineering practices. I imagine it's a culture issue if anything, not a knowledge/background issue.
Googles engineers are good in those types of environments. Most environments aren't Google like at all. Ive seen so many small and medium companies hire googlers and burn money because of it. I actually watched a start up die because of a few key middle management/staff engineer hires from Google. They just could not operate for anyone other then themselves and were oblivious to the scale they worked on.
Formal vibefication
Ironic that this page loads partially styled with broken CSS:
"The resource at “https://claude.dev/_next/static/chunks/241gnzjfqbp9f.css” preloaded with link preload was not used within a few seconds. Make sure all attributes of the preload tag are set correctly."
> ...On the Claude Code team, the vast majority of our code (~90%) is written and reviewed by Claude...
I am more interested in the remaining 10%.
Apparently, the remaining 10% too, but not attributed correctly, so they are careful of claiming it as Claude-written. (Can't remember where I've read that.)
Boris has said it when asked, such as here;
“Also Claude, but we're fixing up the attribution to be super confident before claiming it”
https://x.com/bcherny/status/2111207278653399460?a=CQxYPj8Va...
So a lot of slop...