Garbage. You can't include training by the companies that develop an llm in the comparison against companies that merely use the same llm. Apples and potatoes.
It'll get paid from revenue, not by redirecting employee salaries. All that AI+compute is literally what customers pay Anthropic for.
Big AI labs are not software companies where payroll dominates expenses. They're capex-heavy industrial entities; it just so happens that the "machines" (whose output they sell) are nominally the same category as the devices that their knowledge worker employees use on their desks.
xAI is discounting the first few months of their rental to Anthropic, which will help them seem profitable for a bit longer. We'll need to see if that lasts.
I wonder if they ever will be. If the chinese open source models are only 3-6 months behind every major frontier model release, I can't see the business model. GLM-5.2 is supposedly on par to Opus depending on the case. And everybody and their mother can run that model in their datacenter and charge Dollars for tokens.
There is distillation going on where chinese providers give the model lots of outputs. We don't live in a world where chinese providers are not doing this so we can't compare the advantage of this distillation, but there is some advantage to it otherwise they wouldn't do it.
If Anthropic can block distillations somehow (which are fair game imo given that Anthropic et al did the same with the written works of mankind), then they might stop or slow down the chinese from catching up.
Chinese also have like 40% of the AI researchers of the world, plus they have access to a lot of cheap labour for writing training data. I'm sure an hour of training data creation from one of China's 162 million university educated people is much cheaper than an hour of work from one of US's 97 million. Probably still cheaper than someone from the grand area.
China is behind in AI chips/GPUs but they are catching up. One thing where they have a hard dependence on outside is their energy imports: they have to import a lot of stuff from third party countries. The US on the other hand is energy self sufficient.
I think the panic around distillation misses the fact that US labs also benefit heavily from Chinese breakthroughs like Deepseek's work on sparsity, MoE and training architecture
It may be that US labs use Chinese models for distillation but we'd ofc never know because they can host the models themselves
Stopping distilation is a condradiction to growth: The more active users Anthropic gets, the easier it will be for chinese companies to distile the model. Heck, I can see a paid browser extension being issued that does nothing but send copies of your AI chatbox prompts+results to China - with a "hidden" feature that creates distillation prompts every now and then. Give each 5$ a month for installing the browser extension, and you got an unstoppable distillation botnet.
Luckily we're not, because then your tip to a waiter/waitress would be dependent on their student loans remaining, especially considering how many expensive liberal arts majors struggle to find a sufficient career.
Maybe an ESL thing? "Potatoes" are literally called "earth apples" in some languages (e.g. pommes de terre in French; Erdäpfel in some German dialects.)
OpenAI and Anthropic aren't charities, so whatever cost they inccur for training will be passed down to the companies using the models. So you absolute should include it.
OpenAI and Anthropic aren't charities, so whatever cost they inccur for training will be passed down to the companies using the models
You should, but with two important caveats. First, you don't know what their amortization schedule is like so you don't know what the impact on the pricing will be (are they going to pass the cost on over 5 years or over 20 years?), and second they may go bust before paying the cost down so they may not get a chance to pass it all on. If someone buys the company then they'll get a discount on the value, which means the training costs are just eaten by the investors.
Anthropic spends [...] about $2m of compute per employee per year against a likely all-in comp of $500k+.
The rest of the software market trails. The top 1% of companies spend $89k per engineer per year on AI
This framing makes no sense. The reason Anthropic spends so much on compute per employee is that they are building models. Anthropic employees aren't opening Claude Code and spending $2m in inference every year, so comparing it to other software companies, where AI expense is mostly inference, is completely incoherent.
Yes, the cost has to be passed down eventually, but it's not passed down to one company; it's passed down to all of Anthropic's customers, so the actual share of that money will be distributed among Anthropic's clients.
Look, I 100% agree with the idea that OpenAI and Anthropic are both unsustainable companies that have dug themselves so far into a debt hole that, most likely, the only way they'll be rescued is with government intervention, but this is still a terrible article.
Because the money that OpenAI and Anthropic have to pay those costs with comes from the users who pay for the service. There is no "passing the cost" of anything. The consumer always pays all costs.
Apples and potatoes are both something people will need to eat if we want to see it from the human utility perspective, and they both require some land space to be allocated for their culture (though one can of course conjugate both culture).
If you want to take the DDG LLM summary at fate value, apples are lower in calories and sugar but higher in fiber compared to potatoes, which are richer in vitamins and minerals like potassium and vitamin B6. Overall, apples provide more dietary fiber, while potatoes offer more protein and essential nutrients.
Comparison rarely lead to one obvious all superior option that discard every other considerations.
the saying "comparing x and y" implies that you compare something that one of them can't compete ; if people praise the softness of the skin first and foremost, comparing apples and potatoes won't lead interesting results
Yes, that’s certainly what people mean generally. Now if we consider perspectives like the one elaborated by Marcel Detienne in Comparer l'incomparable[1], we can go a bit further.
The comparison no longer starts with the goal to assess distinct objects in the frame of a given more or less established framework, and instead our attention is framed toward challenging ourself. That is, anchored toward finding what frameworks would allow to assess anything meaningful. And latter on, what does frameworks and framework creation reveals about ourself.
I don't know, compute is compute. Arguably making complex software with LLMs isn't all that different from training a model to do a thing. You're throwing a lot of compute at the problem and hoping for a stochastic solution. The distinction will become even blurrier with time.
Though I agree it might be informative to split it by industry sector.
Which is probably a backwards anti-pattern companies have built.
Your most expensive engineer's time is most valuable, so if you give them standard issue which is half the speed, you are throttling the value you can get from your engineer. Not to mention the mental drain of your cursor barely being able to move due to all the bloated virtual networking systemization.
It would seem to make sense to give more valuable employees faster equipment, so that their time isn't spent toiling with the slow machine, but rather actually producing value.
> more expensive engineers don't use more expensive computers
They don't? If you give your best engineers substandard hardware to work on, you're going to get worse output from them compared to if you give them more expensive computers to work with.
... and that actually shows - senior engineers have spent actual paid time to train juniors. Plus they used to spent time contributing to open source projects or Stack Overflow, all the stuff which every company benefits from.
why stop there? Count how long and how much energy it took for evolution to produce that 3 chimp brain that is then educated, and add how long it took culture to produce the knowledge in text books for said education to be possible.
Evian use 1.25 million litres of water per employee per year. When can we expect other non-bottled-water corporations to rise to this level of water usage?
A.I suffers from the last-mile problem. It can do 90% of the work in 20 minutes but then the remaining 10% ends up taking 20 million hours to actually finish. It frustrating to the point that I sometimes want to throw the whole thing out and start from scratch.
Some people would argue that this is the best way to use AI as it exists today. Generate a POC and then if that POC makes sense, then rebuild it from scratch by hand. Maybe you can still use AI for a bit of boilerplate generation, but you should write all of the business logic and verify it by hand.
Personally, I'm starting to lean more and more towards this approach.
Though, I have to admit, for a well defined bug ticket, AI can be super useful to knock those out.
I find AI useful for boilerplate stuff, very generic code like mappers etc.
For more complex stuff, I find that the best workflow is usually treating AI like a kind of stupid, but very motivated intern you're pair programming with. Nothing unsupervised and you might have to touch up/do manually the really critical parts, but it can help with a lot of the bitchwork.
That has been my experience at the beginning, but not anymore.
I have developed some intuition of how large tasks I can give it so that it will complete them well, probably erring on the conservative side.
I am using it daily for all my code writing and honestly don't remember the last time I had the feeling that I had to spend a lot of work to get the last few % done.
Open-weight models are going to completely shatter these forecasts. It takes a little more effort – right now, probably won’t be true in three months – but you can achieve the same at 1/10th of the cost.
> but you can achieve the same at 1/10th of the cost.
For some tasks, sure. But not for all tasks. And for some tasks, cost per token is irrelevant if it provides real benefits that are oom compared to what you had.
Local models are indeed becoming "good enough" for some tasks, but there are still tasks that they can't touch. There's a recent benchmark for kernel writing. Fable wrote a kernel that provides ~30% more throughput per unit of compute compared to the latest Opus max / gpt max. Does it matter how much that session cost in terms of one session if you can take that kernel, deploy it on your inference fleet and "magically" get 30% more tokens served to your clients? There are companies that would pay millions for such a "leap". Because they can make more millions down the line.
You're looking at the status quo and ignoring the trajectory. The best current open models are about as good as closed models from ~1.5 generations ago. The rate of improvement of all models is converging to zero. It follows that in a few generations, open models inferencing will be about as good as closed model inferencing.
The problem is going to become that there's no incentive for anyone to run the stupidly-expensive training phase. May God have mercy on the stock market.
> The rate of improvement of all models is converging to zero.
That's so obviously not true that I don't even think it's worth the energy to even debate it. It's been said for years, yet here we are, constantly improving. People really don't get RL / the bitter lesson, do they?
> It follows that in a few generations, open models inferencing will be about as good as closed model inferencing.
Not a chance. There's hundreds of billions of dollars on one side, and oom less on the other. There's also scaling laws and information theory. No matter how good, a 30B model will not be able to be better than a 3T+ model, all things being equal.
You are mistaking models becoming "good enough" for an increasingly number of tasks, which I agree is happening, with SotA models stagnating, hitting walls etc. That will not happen for many many years to come.
On that aspect, I agree. Smaller / open models are becoming "good enough" at an increasing number of tasks. And that's great for us, consumers. But there will always be tasks that are "worth" pursuing with better models, and cost is irrelevant for those tasks. That was the point I was trying to make.
That’s true, there will always be demand for ultra-intelligent assistants, especially if they surpass what humans can achieve at similar cost. For the other 90%, the average frontier model will be good enough.
I don't disagree with you, but it's important to pay attention to where the money is. Cheap non frontier models is something that Anthropic and open AI could do too, but who's willing to pay a premium for using them? It will be like competing to sell rice, lots of demand at Rock bottom margins.
And 10x the headache. Money can be exchanged for goods and services, and people pay money to not have to deal with things. If you don't have the money for it, you pay for it in dealing-with-bullshit credits.
Working regularly with AI is like managing a small team of unbelievably knowledgeable, very smart, and occasionally crashingly naïve junior developers. Because they're so knowledgeable and smart, they can get a lot done very quickly. Because they make a proportion of howling errors, you have to keep a close eye on them -- or carefully train another agent to do it for you, in which case you now have to keep a close eye on that agent as well.
So, overall, you get more done that without AI, at the cost of spending almost all of your time writing specs and doing code review and almost none of it writing code.
Do you get 3.3x the work done? Probably not. Do you get 2x the work done? I think maybe, if you can hack the dynamics of the new job as a manager of eager robots. For me the jury's still out on the second point.
I get the feeling that either I'm using LLMs wrong, or everyone else is.
Outside of enthusiastic use of Tab and some one-off scripts, I don't really tell it to write code. Instead I ask vague questions about the codebase and its inner workings.
Reading other people's code has always been my Achilles' heel - particularly if it's a huge project and has a lot of undocumented conventions. LLMs are brilliant at explaining this sort of stuff.
Shit, I know people that don't code that are asking LLMs for advice about personal matters. As far as AI companies are concerned, as long as you're using their product and are paying them for it, who cares what you're actually asking it for.
The layoffs are irrelevant to the discourse. It's typically considered by management to be good, for mature companies, to periodically fire as many employees as they can sustain without visibly impacting operations, and then re-hire cheaper workers only where strictly necessary. This allows them to keep costs down, reduce risks of excessive worker entrenchment, and overcome the drawbacks of contingent hiring-sprees.
Excuses for these exercises will vary, AI is just the latest; but it's fundamentally just a labor-containment/efficiency-seeking strategy.
The thing AI has been the most useful at is showing without a doubt that the emperor does in fact not have any clothes.
It's exposed the incompetence, hubris and sheer out-of-touchness of the tech leadership caste, open for everyone to see.
They're not smarter than you, they don't have any great strategic insights. They're just rich kids that happened to be at the right place at the right time and now have a cadre of sycophants blowing smoke up their ass.
I think its a fallacy to believe people like Zuckerberg or any other stupidly rich person aren't extremely calculative about this. I am very sure they have surrounded themselves by top tier engineers making very informed decisions while their top tier marketing teams make very calculated decisions on how its expressed to the public. The public generally is NOT in favor of AI outside of tech circles so it makes sense to communicate critique of AI to the public.
Executives are salivating at the prospect of AI being able to execute their plans instead of real humans. It is not even about human payroll cost, you can just tell that many higher up executives just complain about how hard it is to steer the ship and get people to work on the right things.
Unfortunately they also don't realize just how much decision-making real people do lower down the org-chart. Critical decisions are often done by the leaf nodes, often without even discussing it internally with the leaf-node team. AI will likely not be very good at this kind of decision making or realize any decision needs to be made at all.
At that level it becomes hard not to surround yourself with obsequious yes men who will instinctively agree with all your harebrained ideas about virtual reality or AI.
I'm currently interning and solely spending ~10-20x on AI than my monthly spend as a part of my job (we're getting training rigs too) probably not the best comparison but it's very real haha
I've not seen anyone yet implement a true cost to productivity assessment or guardrails for AI usage yet. Sure this is hard to do with people, but performance management is a well understood field with a hundred years of practice for knowledge workers.
We don't get unlimited hiring budget, so we also won't get unlimited token budgets, and we as the operators will be responsible for the productivity of our agents.
What does performance management for engineers look like when dollar token cost is included in reviews? I think it's going to change a lot of assumptions and a lot of strategy around AI use.
I think that token usage by engineers continues to increase, probably at a very high rate for many years (we are in the middle of the S curve of adoption and it isn’t yet clear where this will plateaux) but an increasing percentage of those tokens are cheap, because we use expensive models for goals and design and cheap models for implementations and workflows.
A missing thread by the author for how Anthropic's training expenses becomes expenses for employee workplace expenses.
And this is before we start adding Anthropic engineer's ability to use it's tools/models for far less than market price.
Is not one or the other. AI is a tool for the Engineer. Costs more? Depends on how you use it. You can reduce AI costs in multiple ways, accepting the tradeoffs.
Even if the current generation of frontier models becomes 10x cheaper, companies will still end up spending much more per employee than they do today.
Lower prices will not reduce AI spend. They will simply increase usage.
There is no real ceiling on how much companies can delegate to AI. The only limit is the floor where spend too little, and you simply stop being competitive.
My company recently got a ton of AI credits on Linear and they are testing out this feature where whenever an issue is created in linear, it triggers an AI agent to automatically fix the issue. The idea is that when the dev gets to it, he will just check the preview-URL or run unit tests and rubber-stamp the PR.
It is unlikely this kind of agentic workflow will ever get cheaper. Agents get stuck in doom-loops quite often, just burning tokens without any value. Especially by prompts created by people unfamiliar with the codebase.
And it is becoming increasingly obvious that better models just use more tokens (and take longer to execute on prompts). So this kind of human-out-of-the-loop workflows will be forced to use cheaper models and be time-gated in order to not waste tokens. And then they will also produce worse results than a manual change or a more powerful model...
If tokens get cheaper you just put a better model for this kind of problem and let it run for longer.
But what is more insane is that we are using a ton of cloud VM time on top of a ton of tokens just to save a few minutes from a developer doing the same on his machine...
I don't think my company will keep this system once the free credits run out once they realize how much it actually costs.
Ignoring the bizarre inclusion of training compute for the AI company estimates, the other comparisons are still valid.
> The rest of the software market trails. The top 1% of companies spend $89k per engineer per year on AI, 40% of a fully-loaded $224k senior engineer salary. The median spends $137. That is the gap : ... 0.4x at the top of the market, near zero at the median.
So it's not more expensive than an engineer it's 40% as expensive, and for many companies use-cases the cost is virtually negligible.
Even here in Europe where developers are much cheaper than in the US, it still makes sense to pay for the LLM Enterprise subscriptions.
My company has a Claude Code and Codex one and I use Claude Code because I am more familiar with it. That said, I just use Opus for planning and Sonnet for implementation and it's pretty cheap. Codex seems decent too so I should try it out some more.
But you can get an awful lot done even with just like $200 a month at API pricing if you are careful not to waste a powerful model on an easy task, or carry around a bloated context window etc.
I think a lot of the 'tokenmaxxing' people spending thousands every month are simply using the tools ineffectively (like having loads of Opus agents doing tasks that Sonnet or even Haiku could do). I suspect this will only get worse now with the release of Fable, but Anthropic must love it.
When you say the cheaper models do you mean like Deepseek or GLM? I haven't tried those but they look interesting. It'd be nice to shift to open weights and not be tied to one company.
With cheaper models I really meant cheaper subscriptions but used the wrong vocabulary. We still use Claude Opus (if thats what 4.6 is?). We just have the 20 bucks subscription and I barely use up my token limits in my day to day work.
I often wonder what kinda features other devs implement compared to me, if they need that many tokens?
It kind of feels impractical to bloat up an app with features one barely understands? I've just been reading about these devs using x-amount of tokens, having that y-amount of steps perfected AI workflow, but none of them ever talk about what they actually implement all day...
Using Anthropic in the comparison is obviously bullshit. That's like mentioning that a local construction company spends more on concrete and timber than on workers, but framing it as if they were spending more on power tools than on their employees
But even ignoring that: if AI was making Engineers 10x more productive (bear with me), wouldn't spending 2x the engineers salary on AI be the rational thing to do. In effect, what we are seeing here is a crude proxy for the benefit each company sees in AI. Whether that benefit is real or only in manager's heads is a different thing these numbers can't tell us
I'm not a VC guru but in my opinion you can't include the time and money it takes to grow a tree and mine the iron to compare the time it takes to hammer in a nail with a hammer versus using your fist.
That’s how policy makers and concentrated decision power class get completely disconnected from actual resources at stake and what actual constraints need to be weighted. If a job require to put a blindfold and a sound blocker headset preventing to hear the things people scream, people in the role will happily accelerate against the wall the are induced to ignore.
This is not even specific to capitalism or VC mind you. Look how PRC led to the Great Chinese Famine. That’s why actual democracies (not the inter-elected aristocraties ), despite all their downsides, are so damn interesting. Corruption, negligence, or mere error with catastrophic follows, is easily spread in a situation where small core of individuals monopolize greatest part of decision weight, but is logistically impossible to achieve in a system optimized for widespread and highly redundant power responsibilities.
> Anthropic spends 2.3x its payroll on compute.1 With ~5,000 employees & roughly $10b in inference & training spend in 2026, that works out to about $2m of compute per employee per year against a likely all-in comp of $500k+.2
> The rest of the software market trails.
This shows how VC firms see things and why we have such a lopsided market where grift rises to top easily.
Yes the rest of the software market trails in comparision to the compute costs at Anthropic if you including training the actual models. Like is this the insight? Biggest AI company spends a lot of money to make AI models?
Sure you can find anthropic's business model risky/not feseable but using this as your starting point shows a lack of basic understanding at best and malicious intent to make a stupid point at worst
Excellent, with stunning insight like this, you can see why this VC is earning the big bucks.
This is almost economics level of line projection.
It would be good to understand _why_ anthropics "AI" bill is so high. First, They are going to be renting a lot of inference compute just to service customers (Meta's Capex bill is about 2x its wage bill) It then also needs a huge amount of infra to both run training and experimentation. THats probably a third of the cost. (storage and physical infra to get the most out of storage and compute is hard. Then getting it reliable, so that shit state doesn't propogate across the shared memory plane is very hard.
The other thing to note is that claude usage inside anthropic is tiny compared to the customer's usage. even with uber agents at "mythos++" its going to be at best a few thousand servers. not like the massive fleet needed to serve the paying customer.
So using anthropic as some sort of rational target to base any kind of prediction is madness. Its like looking at lyons tea rooms and going yeah, every company is going to spin up an R&D arm to make a company specific computer: https://www.sciencemuseum.org.uk/objects-and-stories/meet-le...
ALSO this assumes that the current way of running LLMs is the way forward. Custom software is expensive (in both time and tokens) to look after, its much easier and cheaper to buy it in from SaaS companies and let them figure that shit out. (yes I know SaaS apocalypse, but you are paying for real world experience, and a packaged way of doing things, rather than experimenting your self, where in a lot of cases the company doing the experimentation doesn't know what its doing)
Garbage. You can't include training by the companies that develop an llm in the comparison against companies that merely use the same llm. Apples and potatoes.
Exactly, it's like saying Shell is spending a fortune on fuel compared to what they spend on employees, if you count oil extraction costs as 'fuel'.
So where are these training costs getting paid from?
It'll get paid from revenue, not by redirecting employee salaries. All that AI+compute is literally what customers pay Anthropic for.
Big AI labs are not software companies where payroll dominates expenses. They're capex-heavy industrial entities; it just so happens that the "machines" (whose output they sell) are nominally the same category as the devices that their knowledge worker employees use on their desks.
VC mostly, since Anthropic is not profitable.
That's about to change: https://www.wsj.com/tech/ai/mind-blowing-growth-is-about-to-...
Anthropic was profitable last quarter.
xAI is discounting the first few months of their rental to Anthropic, which will help them seem profitable for a bit longer. We'll need to see if that lasts.
I wonder if they ever will be. If the chinese open source models are only 3-6 months behind every major frontier model release, I can't see the business model. GLM-5.2 is supposedly on par to Opus depending on the case. And everybody and their mother can run that model in their datacenter and charge Dollars for tokens.
There is distillation going on where chinese providers give the model lots of outputs. We don't live in a world where chinese providers are not doing this so we can't compare the advantage of this distillation, but there is some advantage to it otherwise they wouldn't do it.
If Anthropic can block distillations somehow (which are fair game imo given that Anthropic et al did the same with the written works of mankind), then they might stop or slow down the chinese from catching up.
Chinese also have like 40% of the AI researchers of the world, plus they have access to a lot of cheap labour for writing training data. I'm sure an hour of training data creation from one of China's 162 million university educated people is much cheaper than an hour of work from one of US's 97 million. Probably still cheaper than someone from the grand area.
China is behind in AI chips/GPUs but they are catching up. One thing where they have a hard dependence on outside is their energy imports: they have to import a lot of stuff from third party countries. The US on the other hand is energy self sufficient.
I think the panic around distillation misses the fact that US labs also benefit heavily from Chinese breakthroughs like Deepseek's work on sparsity, MoE and training architecture
It may be that US labs use Chinese models for distillation but we'd ofc never know because they can host the models themselves
Stopping distilation is a condradiction to growth: The more active users Anthropic gets, the easier it will be for chinese companies to distile the model. Heck, I can see a paid browser extension being issued that does nothing but send copies of your AI chatbox prompts+results to China - with a "hidden" feature that creates distillation prompts every now and then. Give each 5$ a month for installing the browser extension, and you got an unstoppable distillation botnet.
I guess they should include tuition cost as well.
In a way in US it is - _IF_ ppl were rational economic agents and free market allocation worked student loans should reflect on the wages too.
Luckily we're not, because then your tip to a waiter/waitress would be dependent on their student loans remaining, especially considering how many expensive liberal arts majors struggle to find a sufficient career.
Apples and oranges, or chalk and cheese. Why would you say apples and potatoes?
Maybe an ESL thing? "Potatoes" are literally called "earth apples" in some languages (e.g. pommes de terre in French; Erdäpfel in some German dialects.)
I read it as trying to indicate that it's even more different than apples and oranges.
Not sure it succeeds in that, but I think that's the intent.
Grape and aspergus, we all get it
OpenAI and Anthropic aren't charities, so whatever cost they inccur for training will be passed down to the companies using the models. So you absolute should include it.
OpenAI and Anthropic aren't charities, so whatever cost they inccur for training will be passed down to the companies using the models
You should, but with two important caveats. First, you don't know what their amortization schedule is like so you don't know what the impact on the pricing will be (are they going to pass the cost on over 5 years or over 20 years?), and second they may go bust before paying the cost down so they may not get a chance to pass it all on. If someone buys the company then they'll get a discount on the value, which means the training costs are just eaten by the investors.
Well … one was a non profit and I still can’t figure out how it kept the donations the tax benefits and because a trillion for profit company
The problem is how it's framed:
Anthropic spends [...] about $2m of compute per employee per year against a likely all-in comp of $500k+.
The rest of the software market trails. The top 1% of companies spend $89k per engineer per year on AI
This framing makes no sense. The reason Anthropic spends so much on compute per employee is that they are building models. Anthropic employees aren't opening Claude Code and spending $2m in inference every year, so comparing it to other software companies, where AI expense is mostly inference, is completely incoherent.
Yes, the cost has to be passed down eventually, but it's not passed down to one company; it's passed down to all of Anthropic's customers, so the actual share of that money will be distributed among Anthropic's clients.
Look, I 100% agree with the idea that OpenAI and Anthropic are both unsustainable companies that have dug themselves so far into a debt hole that, most likely, the only way they'll be rescued is with government intervention, but this is still a terrible article.
> whatever cost they inccur for training will be passed down to the companies using the models
Assuming their investors win the bet they placed on them. Which isn't given.
Why can't we pass on the costs of OpenAI and Anthropic's training back to OpenAI and Anthropic?
Bandwidth isn't free, and all my life I've been told that piracy is theft.
Because the money that OpenAI and Anthropic have to pay those costs with comes from the users who pay for the service. There is no "passing the cost" of anything. The consumer always pays all costs.
Apples and potatoes are both something people will need to eat if we want to see it from the human utility perspective, and they both require some land space to be allocated for their culture (though one can of course conjugate both culture).
If you want to take the DDG LLM summary at fate value, apples are lower in calories and sugar but higher in fiber compared to potatoes, which are richer in vitamins and minerals like potassium and vitamin B6. Overall, apples provide more dietary fiber, while potatoes offer more protein and essential nutrients.
Comparison rarely lead to one obvious all superior option that discard every other considerations.
the saying "comparing x and y" implies that you compare something that one of them can't compete ; if people praise the softness of the skin first and foremost, comparing apples and potatoes won't lead interesting results
Yes, that’s certainly what people mean generally. Now if we consider perspectives like the one elaborated by Marcel Detienne in Comparer l'incomparable[1], we can go a bit further.
The comparison no longer starts with the goal to assess distinct objects in the frame of a given more or less established framework, and instead our attention is framed toward challenging ourself. That is, anchored toward finding what frameworks would allow to assess anything meaningful. And latter on, what does frameworks and framework creation reveals about ourself.
[1] https://archive.org/details/comparerlincompa0000deti
I don't know, compute is compute. Arguably making complex software with LLMs isn't all that different from training a model to do a thing. You're throwing a lot of compute at the problem and hoping for a stochastic solution. The distinction will become even blurrier with time.
Though I agree it might be informative to split it by industry sector.
AI training uses wildly more compute than most companies, who are generally building domain specific CRUD apps.
Compare AI costs per-engineer-salary-dollar, because more expensive engineers probably need more expensive AI.
> Compare AI costs per-engineer-salary-dollar, because more expensive engineers probably need more expensive AI.
Let's see how this works out in the long run. For a historical analog, more expensive engineers don't use more expensive computers (by and large).
Which is probably a backwards anti-pattern companies have built.
Your most expensive engineer's time is most valuable, so if you give them standard issue which is half the speed, you are throttling the value you can get from your engineer. Not to mention the mental drain of your cursor barely being able to move due to all the bloated virtual networking systemization.
It would seem to make sense to give more valuable employees faster equipment, so that their time isn't spent toiling with the slow machine, but rather actually producing value.
> more expensive engineers don't use more expensive computers
They don't? If you give your best engineers substandard hardware to work on, you're going to get worse output from them compared to if you give them more expensive computers to work with.
If you’re going to include AI training in costs, you should include education as part of the costs of an engineer …
And why only education? Everything the engineered needed so far should be included. Can’t have a dev that never eaten since they were born.
... and that actually shows - senior engineers have spent actual paid time to train juniors. Plus they used to spent time contributing to open source projects or Stack Overflow, all the stuff which every company benefits from.
why stop there? Count how long and how much energy it took for evolution to produce that 3 chimp brain that is then educated, and add how long it took culture to produce the knowledge in text books for said education to be possible.
Analogous statement:
Evian use 1.25 million litres of water per employee per year. When can we expect other non-bottled-water corporations to rise to this level of water usage?
A.I suffers from the last-mile problem. It can do 90% of the work in 20 minutes but then the remaining 10% ends up taking 20 million hours to actually finish. It frustrating to the point that I sometimes want to throw the whole thing out and start from scratch.
Some people would argue that this is the best way to use AI as it exists today. Generate a POC and then if that POC makes sense, then rebuild it from scratch by hand. Maybe you can still use AI for a bit of boilerplate generation, but you should write all of the business logic and verify it by hand.
Personally, I'm starting to lean more and more towards this approach.
Though, I have to admit, for a well defined bug ticket, AI can be super useful to knock those out.
I find AI useful for boilerplate stuff, very generic code like mappers etc.
For more complex stuff, I find that the best workflow is usually treating AI like a kind of stupid, but very motivated intern you're pair programming with. Nothing unsupervised and you might have to touch up/do manually the really critical parts, but it can help with a lot of the bitchwork.
That has been my experience at the beginning, but not anymore.
I have developed some intuition of how large tasks I can give it so that it will complete them well, probably erring on the conservative side.
I am using it daily for all my code writing and honestly don't remember the last time I had the feeling that I had to spend a lot of work to get the last few % done.
Open-weight models are going to completely shatter these forecasts. It takes a little more effort – right now, probably won’t be true in three months – but you can achieve the same at 1/10th of the cost.
> but you can achieve the same at 1/10th of the cost.
For some tasks, sure. But not for all tasks. And for some tasks, cost per token is irrelevant if it provides real benefits that are oom compared to what you had.
Local models are indeed becoming "good enough" for some tasks, but there are still tasks that they can't touch. There's a recent benchmark for kernel writing. Fable wrote a kernel that provides ~30% more throughput per unit of compute compared to the latest Opus max / gpt max. Does it matter how much that session cost in terms of one session if you can take that kernel, deploy it on your inference fleet and "magically" get 30% more tokens served to your clients? There are companies that would pay millions for such a "leap". Because they can make more millions down the line.
You're looking at the status quo and ignoring the trajectory. The best current open models are about as good as closed models from ~1.5 generations ago. The rate of improvement of all models is converging to zero. It follows that in a few generations, open models inferencing will be about as good as closed model inferencing.
The problem is going to become that there's no incentive for anyone to run the stupidly-expensive training phase. May God have mercy on the stock market.
> The rate of improvement of all models is converging to zero.
That's so obviously not true that I don't even think it's worth the energy to even debate it. It's been said for years, yet here we are, constantly improving. People really don't get RL / the bitter lesson, do they?
> It follows that in a few generations, open models inferencing will be about as good as closed model inferencing.
Not a chance. There's hundreds of billions of dollars on one side, and oom less on the other. There's also scaling laws and information theory. No matter how good, a 30B model will not be able to be better than a 3T+ model, all things being equal.
You are mistaking models becoming "good enough" for an increasingly number of tasks, which I agree is happening, with SotA models stagnating, hitting walls etc. That will not happen for many many years to come.
The question is: what proportion of tasks can not be handled by GLM5.2?
How many software developers were working on code like the one you describe?
On that aspect, I agree. Smaller / open models are becoming "good enough" at an increasing number of tasks. And that's great for us, consumers. But there will always be tasks that are "worth" pursuing with better models, and cost is irrelevant for those tasks. That was the point I was trying to make.
That’s true, there will always be demand for ultra-intelligent assistants, especially if they surpass what humans can achieve at similar cost. For the other 90%, the average frontier model will be good enough.
I don't disagree with you, but it's important to pay attention to where the money is. Cheap non frontier models is something that Anthropic and open AI could do too, but who's willing to pay a premium for using them? It will be like competing to sell rice, lots of demand at Rock bottom margins.
And 10x the headache. Money can be exchanged for goods and services, and people pay money to not have to deal with things. If you don't have the money for it, you pay for it in dealing-with-bullshit credits.
Working regularly with AI is like managing a small team of unbelievably knowledgeable, very smart, and occasionally crashingly naïve junior developers. Because they're so knowledgeable and smart, they can get a lot done very quickly. Because they make a proportion of howling errors, you have to keep a close eye on them -- or carefully train another agent to do it for you, in which case you now have to keep a close eye on that agent as well.
So, overall, you get more done that without AI, at the cost of spending almost all of your time writing specs and doing code review and almost none of it writing code.
Do you get 3.3x the work done? Probably not. Do you get 2x the work done? I think maybe, if you can hack the dynamics of the new job as a manager of eager robots. For me the jury's still out on the second point.
I get the feeling that either I'm using LLMs wrong, or everyone else is.
Outside of enthusiastic use of Tab and some one-off scripts, I don't really tell it to write code. Instead I ask vague questions about the codebase and its inner workings.
Reading other people's code has always been my Achilles' heel - particularly if it's a huge project and has a lot of undocumented conventions. LLMs are brilliant at explaining this sort of stuff.
Shit, I know people that don't code that are asking LLMs for advice about personal matters. As far as AI companies are concerned, as long as you're using their product and are paying them for it, who cares what you're actually asking it for.
Mr. Mark Zuckerberg is particularly not happy about these stats. He was promised something else and he has already fired like half of the company.
It is really crazy people didn't think this through.
The layoffs are irrelevant to the discourse. It's typically considered by management to be good, for mature companies, to periodically fire as many employees as they can sustain without visibly impacting operations, and then re-hire cheaper workers only where strictly necessary. This allows them to keep costs down, reduce risks of excessive worker entrenchment, and overcome the drawbacks of contingent hiring-sprees.
Excuses for these exercises will vary, AI is just the latest; but it's fundamentally just a labor-containment/efficiency-seeking strategy.
The thing AI has been the most useful at is showing without a doubt that the emperor does in fact not have any clothes.
It's exposed the incompetence, hubris and sheer out-of-touchness of the tech leadership caste, open for everyone to see.
They're not smarter than you, they don't have any great strategic insights. They're just rich kids that happened to be at the right place at the right time and now have a cadre of sycophants blowing smoke up their ass.
Mark Zuckerberg‘s and Meta’s incompetence should have been recognized after the metaverse debacle.
They were lucky with the ad empire he built and that‘s it.
I think its a fallacy to believe people like Zuckerberg or any other stupidly rich person aren't extremely calculative about this. I am very sure they have surrounded themselves by top tier engineers making very informed decisions while their top tier marketing teams make very calculated decisions on how its expressed to the public. The public generally is NOT in favor of AI outside of tech circles so it makes sense to communicate critique of AI to the public.
Executives are salivating at the prospect of AI being able to execute their plans instead of real humans. It is not even about human payroll cost, you can just tell that many higher up executives just complain about how hard it is to steer the ship and get people to work on the right things.
Unfortunately they also don't realize just how much decision-making real people do lower down the org-chart. Critical decisions are often done by the leaf nodes, often without even discussing it internally with the leaf-node team. AI will likely not be very good at this kind of decision making or realize any decision needs to be made at all.
At that level it becomes hard not to surround yourself with obsequious yes men who will instinctively agree with all your harebrained ideas about virtual reality or AI.
Are you really saying that Mark Zuckerberg, CEO of "Meta", can't make a massive miscalculation?
I am saying that none of this is just an "oopsie", a miscalculation, maybe, but definitely not as unexpected as it seems.
I'm currently interning and solely spending ~10-20x on AI than my monthly spend as a part of my job (we're getting training rigs too) probably not the best comparison but it's very real haha
I've not seen anyone yet implement a true cost to productivity assessment or guardrails for AI usage yet. Sure this is hard to do with people, but performance management is a well understood field with a hundred years of practice for knowledge workers.
We don't get unlimited hiring budget, so we also won't get unlimited token budgets, and we as the operators will be responsible for the productivity of our agents.
What does performance management for engineers look like when dollar token cost is included in reviews? I think it's going to change a lot of assumptions and a lot of strategy around AI use.
Garbage. You choose to pay that money, it doesn’t have to cost that much. You have a choice and choose the priciest option, “because shiny”.
The bear case being set at 40% of employee costs is still quite wild.
I think that token usage by engineers continues to increase, probably at a very high rate for many years (we are in the middle of the S curve of adoption and it isn’t yet clear where this will plateaux) but an increasing percentage of those tokens are cheap, because we use expensive models for goals and design and cheap models for implementations and workflows.
A missing thread by the author for how Anthropic's training expenses becomes expenses for employee workplace expenses. And this is before we start adding Anthropic engineer's ability to use it's tools/models for far less than market price.
Is not one or the other. AI is a tool for the Engineer. Costs more? Depends on how you use it. You can reduce AI costs in multiple ways, accepting the tradeoffs.
Even if the current generation of frontier models becomes 10x cheaper, companies will still end up spending much more per employee than they do today.
Lower prices will not reduce AI spend. They will simply increase usage.
There is no real ceiling on how much companies can delegate to AI. The only limit is the floor where spend too little, and you simply stop being competitive.
My company recently got a ton of AI credits on Linear and they are testing out this feature where whenever an issue is created in linear, it triggers an AI agent to automatically fix the issue. The idea is that when the dev gets to it, he will just check the preview-URL or run unit tests and rubber-stamp the PR.
It is unlikely this kind of agentic workflow will ever get cheaper. Agents get stuck in doom-loops quite often, just burning tokens without any value. Especially by prompts created by people unfamiliar with the codebase.
And it is becoming increasingly obvious that better models just use more tokens (and take longer to execute on prompts). So this kind of human-out-of-the-loop workflows will be forced to use cheaper models and be time-gated in order to not waste tokens. And then they will also produce worse results than a manual change or a more powerful model...
If tokens get cheaper you just put a better model for this kind of problem and let it run for longer.
But what is more insane is that we are using a ton of cloud VM time on top of a ton of tokens just to save a few minutes from a developer doing the same on his machine...
I don't think my company will keep this system once the free credits run out once they realize how much it actually costs.
Ignoring the bizarre inclusion of training compute for the AI company estimates, the other comparisons are still valid.
> The rest of the software market trails. The top 1% of companies spend $89k per engineer per year on AI, 40% of a fully-loaded $224k senior engineer salary. The median spends $137. That is the gap : ... 0.4x at the top of the market, near zero at the median.
So it's not more expensive than an engineer it's 40% as expensive, and for many companies use-cases the cost is virtually negligible.
Even here in Europe where developers are much cheaper than in the US, it still makes sense to pay for the LLM Enterprise subscriptions.
>it still makes sense to pay for the LLM Enterprise subscriptions.
Does it though? I do not see any advantages in my day to day job over using the cheaper models.
My company has a Claude Code and Codex one and I use Claude Code because I am more familiar with it. That said, I just use Opus for planning and Sonnet for implementation and it's pretty cheap. Codex seems decent too so I should try it out some more.
But you can get an awful lot done even with just like $200 a month at API pricing if you are careful not to waste a powerful model on an easy task, or carry around a bloated context window etc.
I think a lot of the 'tokenmaxxing' people spending thousands every month are simply using the tools ineffectively (like having loads of Opus agents doing tasks that Sonnet or even Haiku could do). I suspect this will only get worse now with the release of Fable, but Anthropic must love it.
When you say the cheaper models do you mean like Deepseek or GLM? I haven't tried those but they look interesting. It'd be nice to shift to open weights and not be tied to one company.
With cheaper models I really meant cheaper subscriptions but used the wrong vocabulary. We still use Claude Opus (if thats what 4.6 is?). We just have the 20 bucks subscription and I barely use up my token limits in my day to day work.
I often wonder what kinda features other devs implement compared to me, if they need that many tokens?
It kind of feels impractical to bloat up an app with features one barely understands? I've just been reading about these devs using x-amount of tokens, having that y-amount of steps perfected AI workflow, but none of them ever talk about what they actually implement all day...
Using Anthropic in the comparison is obviously bullshit. That's like mentioning that a local construction company spends more on concrete and timber than on workers, but framing it as if they were spending more on power tools than on their employees
But even ignoring that: if AI was making Engineers 10x more productive (bear with me), wouldn't spending 2x the engineers salary on AI be the rational thing to do. In effect, what we are seeing here is a crude proxy for the benefit each company sees in AI. Whether that benefit is real or only in manager's heads is a different thing these numbers can't tell us
bullshit and useless. Not even a proper comparison
What about productivity gains?
This post smells of LLM writing.
It's amazing how misleading and just flat out wrong this post is.
I'm not a VC guru but in my opinion you can't include the time and money it takes to grow a tree and mine the iron to compare the time it takes to hammer in a nail with a hammer versus using your fist.
That’s how policy makers and concentrated decision power class get completely disconnected from actual resources at stake and what actual constraints need to be weighted. If a job require to put a blindfold and a sound blocker headset preventing to hear the things people scream, people in the role will happily accelerate against the wall the are induced to ignore.
This is not even specific to capitalism or VC mind you. Look how PRC led to the Great Chinese Famine. That’s why actual democracies (not the inter-elected aristocraties ), despite all their downsides, are so damn interesting. Corruption, negligence, or mere error with catastrophic follows, is easily spread in a situation where small core of individuals monopolize greatest part of decision weight, but is logistically impossible to achieve in a system optimized for widespread and highly redundant power responsibilities.
https://en.wikipedia.org/wiki/Great_Chinese_Famine
> Anthropic spends 2.3x its payroll on compute.1 With ~5,000 employees & roughly $10b in inference & training spend in 2026, that works out to about $2m of compute per employee per year against a likely all-in comp of $500k+.2
> The rest of the software market trails.
This shows how VC firms see things and why we have such a lopsided market where grift rises to top easily.
Yes the rest of the software market trails in comparision to the compute costs at Anthropic if you including training the actual models. Like is this the insight? Biggest AI company spends a lot of money to make AI models?
Sure you can find anthropic's business model risky/not feseable but using this as your starting point shows a lack of basic understanding at best and malicious intent to make a stupid point at worst
Excellent, with stunning insight like this, you can see why this VC is earning the big bucks.
This is almost economics level of line projection.
It would be good to understand _why_ anthropics "AI" bill is so high. First, They are going to be renting a lot of inference compute just to service customers (Meta's Capex bill is about 2x its wage bill) It then also needs a huge amount of infra to both run training and experimentation. THats probably a third of the cost. (storage and physical infra to get the most out of storage and compute is hard. Then getting it reliable, so that shit state doesn't propogate across the shared memory plane is very hard.
The other thing to note is that claude usage inside anthropic is tiny compared to the customer's usage. even with uber agents at "mythos++" its going to be at best a few thousand servers. not like the massive fleet needed to serve the paying customer.
So using anthropic as some sort of rational target to base any kind of prediction is madness. Its like looking at lyons tea rooms and going yeah, every company is going to spin up an R&D arm to make a company specific computer: https://www.sciencemuseum.org.uk/objects-and-stories/meet-le...
ALSO this assumes that the current way of running LLMs is the way forward. Custom software is expensive (in both time and tokens) to look after, its much easier and cheaper to buy it in from SaaS companies and let them figure that shit out. (yes I know SaaS apocalypse, but you are paying for real world experience, and a packaged way of doing things, rather than experimenting your self, where in a lot of cases the company doing the experimentation doesn't know what its doing)