> I think it is fundamentally a crisis of trust. I think that ordinary people don’t trust companies, governments, or the tech industry and always suspect that we are cooking up some new way to screw them over.
[...]
> I don’t think that a glitzy marketing campaign with a positive spin (which some have advocated that Anthropic do) is the way to win back that trust — at this point, saying that AI will cure cancer is more a cliche than it is inspiring, and most people think it is deceptive. The thing that will work is actually curing cancer. I think by far the most accurate criticism of AI companies including Anthropic is that we haven’t yet delivered on our big promises to benefit the world. That is totally on us, and I think it’s the criticism you should be making, instead of all this stuff about messaging and marketing.
> We are however doing our best to fix this: Anthropic is ramping up its efforts very quickly in biology and medicine, and we hope to have incredible results in the coming years and some early glimmers in the coming months. When we’ve actually accomplished something real, the whole world will hear about it, as loudly as possible, you have my word on that.
Turns out "winning back trust" doesn't have anything to do with any actual concerns people may have re: employment, electricity prices, stock market bubble, intellectual property, scams, cybersecurity, environmental issues etc. Rather we'll just do all of that even harder and the miracles ("curing cancer", lol) we've so far failed to deliver are bound to arrive in short order!
This is actually crazy, because biochemistry is where LLMs are weakest. It's one of science's most fuzzy and unpredictable domains, in general, and a lot of published works which an LLM might take at face value might be unreplicable or subtly flawed. (See e.g. the entire history of Alzheimer's.) It is also where real-world lab and clinical trial work is most important.
So Anthropic are going to spin up a medicinal chemistry lab and start mouse experiments?
I mean, it would be nice, but I don't think that the guys at Anthropic know what they're talking about here, or what they might be getting themselves into. (If indeed this is more than just PR.)
The effective altruist and longtermist crowd are into it as a religion. They would happily sacrifice anything human to satisfy their dreams of AI. They care way more about what the AI needs than their what their fellow humans need. We will end up with an AGI having better living and working conditions than humans and the whole AI industry will be clapping
Not really. The problem is that the trust is violated here. What people fear is that government and large companies are going to use AI against them, and make the reverse impossible. Sadly, this is exactly what is happening.
And that's exactly what EU's AI regulation does. Of course, AI is being blamed for this happening, despite that last time I checked every EU parlementarian, every EU commission member is flesh and blood, the fact that it is 100% intentional government policy, humans, that are doing this to you using very un-AI methods.
You see, in this regulation, governments are allowed to use AI, and to approve uses of AI. So there is absolutely nothing in EU's AI directive at all that prevents government and large companies, with approval, from answering the phone, and all "support requests" with AI, and not with people. Also "if you agree" (just like you've agreed to selling your location data in your cell contract)
The watermarking algorithm itself has a VERY specific property that should have set off everyone's alarm bells: it is NOT the case that you can "check text for AI watermark". What it technically mandates is that if you provide access to a model, those people should be able to check if text is generated by that model. Not by any other model. It is NOT a general "is this AI?" check. So let's analyze how this works in 2 specific cases:
You get a mail from the government about taxes. You want to check if that text is generated by an AI and whether you even should ask to talk to a human. So you've got a piece of text, or an audio recording. Can you check if it's AI or not? NO YOU CAN'T. SynthID requires access to the model keys, and the legislation only mandates checks if you've got access to the model.
The government or some large company, however, puts in it's regulations that you're not allowed to use AI to communicate with them (we all know this is coming), and they get a piece of ChatGPT text (or from any public model) from you. They would like to enforce that they won't talk to AI can they do this? YES, since they have access to the model too, SynthID allows for this usecase AND the AI regulation makes this mandatory.
Is this in any way a problem with AI? No. Can anything be done about this by "fighting AI"? Well let's see ... will government institutions be forced to stop worsening services further if ChatGPT etc become inaccessible or obviously excluded by government and large company services? No.
What is being outlawed in the EU, in other words, is exactly ONE thing: that you use AI to help you in your dealings with governments and large companies. To assert your rights, to resolve issues faster, to make them wait rather than you, to ... THAT is being outlawed, nothing else. Only that YOU get help from AI.
This legislation will make it a big advantage to have access to "your own" model, because you'll have the SynthID keys, and the law behind you if you do that. And I don't mean local model, I mean your own model with your own SynthID keys, ie. not Google, not ChatGPT, not Claude, not Grok, and if the EU can make it happen, not Chinese models either. Finetuned models, however, will not have SynthID and so you'll never be able to prove text generated by them is AI, and you will not effectively have the power to refuse contracts or laws that specify they can use AI anyway.
It's the EU government making it clear they are using AI to avoid even having to talk to you at all while disallowing the use of AI by mere citizens to assert their rights, against them, or anything they have interests in, like phone or electricity companies.
The only factual point you made is solved on February second. All Model providers have to provide offline tooling to check for invisible watermarking...
> Overall my view is that AI is structurally a technology that tends to concentrate power, for reasons that have nothing to do with regulation (more to do with the extreme implications of the scaling laws). Open-weights do help some with this but are nowhere near a sufficient solution because they simply shift the concentration somewhat to those with the most compute and chips (which are roughly the frontier labs plus maybe hardware providers).
This is actually a good point. Open models are getting better and better, some of them might even be useful in consumer hardware now. But if AI performance is still correlated with compute power, then no doubt power will remain with the people owning the chips.
Arguably the question is whether it's economically feasible to self-host something similar.
Are you self-hosting Google or Bing? No, but we have quite a huge ecosystem of full-text search tools with PageRank, with options to scale to almost Google scale (if you have the money). After all LLM training starts with the same crawl mechanism.
As long as barriers to entry is not too high (ie. it makes sense to take the risk to start a business that provides something similar - usually for a niche) market forces work.
We have the classic empirical chart reproducing microeconomics.
And setting up a pharma plant is also very capital intensive.
Here the obvious barrier to entry is completely artificial. (Which provides an incentive to spend a lot of money on R&D -- though it naturally raises the question of Pareto efficiency.)
You're missing the point, what will happen is this:
1) in things like tax law, registering with city hall, dealings with the DMV, your phone subscription, insurance contract, ... you will find that one of the new fine prints in the contract will be that you're not allowed to use AI to communicate with them.
2) because of how SynthID works (you need the SynthID keys to verify, which are secret. So the only way to find if text is ChatGPT/Google/Anthropic watermarked is to ask ChatGPT/Google/Anthropic), government and large companies can enforce this against you. That is what the watermark is for. To end any insurance claim written by AI with "you're not allowed to submit AI written insurance claims" and refuse it outright there and then.
"Sorry your request was AI watermarked and pursuant to law 234 of 2025/03/11 chapter 3258 paragraph 33 decile 1299 we hereby close it without response"
3) when they reply, however, they use a custom model that also has custom SynthID keys. You will not even be able to tell their responses are AI written, or at least, you won't be able to prove it. You won't be able to enforce any AI-related rights (ie. the right to talk to a human) you have under the law against large companies.
In other words: this is to make sure that all the advantages AI provides are available to deny your unemployment claim, and to Verizon to charge you more, but completely inaccessible TO YOU when you want to change to a cheaper subscription. They can inundate YOU with AI-written requests BUT YOU CAN'T.
Self-hosting helps because it prevents them from verifying if your responses are AI written, because you can generate non-watermarked AI text and so there is a level playing field.
I think his point is hand wavy at best. It presupposes infinite scaling and ignores all the algorithmic efficiency wins that are being discovered. Ironically, many of which are being discovered with autoresearch style workflows, using the very LLMs that his company builds.
The #1 post on HN right now[1] is full of people jubilating about how they can run Qwen 3.8 27B on their > 5 year old GPUs. If that isn't democratization of AI, I don't know what is.
I'm sure he's smart enough to instantaneously realize this too, but as the famous Upton Sinclair quote goes, he won't mention it even if he does.
The very definition of (applied) technology is power amplification. Use a lever, move more weight than you could before, 1 person with the tool now wields the power of 3 without.
Making "tech" a career and a societal goal onto itself, without the adjoining understanding of and deep commitment to ethics and the responsible use of power, is why we're sliding into authoritarian rule by a small circle of techno-oligarchs.
We need less "move fast and break things" and more "plant trees you will not live to see bear fruit."
Called it! He was bound to squeal after the release of QWEN 3.8 in one way or another and here it is. I wasn't even a teenager but I'm getting so much Jobs/Ballmer flashbacks with internet explorer and microsoft office.
Edit: Does anyone else notice the switching between dashes and em-dashes between paragraphs? Tells you a lot about the man, doesn't it.
> Edit: Does anyone else notice the switching between dashes and em-dashes between paragraphs? Tells you a lot about the man, doesn't it.
That he doesn’t know how to use basic word processing, or even agent SKILLS.md or whatever it’s called now. Or maybe it’s just the next generation of vagueposting.
One thing Dario said is important reflecting on (but from another angle): where's the big deliverable from AI? If AI makes us 10x more productive, the 3 years since its popularization were enough for a product that would have taken 30 years to build without AI, for instance.
I'm an AI advocate but that question makes me feel that AI is simply "very useful" rather than being a historical game changer for humanity.
Anyone saying 10x as a serious claim is clearly using a round number and vibes; however, even if it were so, AI getting popular 3 years ago does not mean what you say.
The earliest used models in this category would easily 10x (what did just say) the creation of one-off short scripts, but you're not writing a 30-year app out of just a bunch of short scripts.
The METR time horizons graph suggests we can now get 10x (ahem) speedup on solving most coding problems that take us a few hours and about half of problems that take 2 days. Amdahl's law bites: even infinity speedup on half your problems is only 2x overall.
If you let an LLM lose, with a huge budget, what's the biggest artefact it can make before it drowns under the weight of bad decisions? The C compiler and web browser headlines a while back? The maths papers we see now that solve problems which stumped the maths world for decades?
But this is the other side of the same coin: teamwork. One person getting a thing made in twelve months vs a team of a hundred, you can scale up fast with money when you have a proof of concept, and an LLM can make a lot of proofs of a lot of concepts even in free accounts.
LLM-assisted software engineering seems to be very efficient if you have a deterministic target. (bun rewrite from zig to Rust, 100% Node.js compatibility, pnpm compatibility -- https://github.com/oven-sh/bun/pull/38333)
Can only speak for my own project, but can give an example:
After an aquisition earlier this year I got the task of doing an SAP-Integration for the new company, last time I did this 5 years ago it was a 6 month task, but with the experience and skills ive gained since I estimated it would be a 3 month project (with or without AI, most work is just logistics, AI cant help much there).
In those 3 months I was able to not only integrate SAP but also deliver a completely modernised user-facing software for that integration. While I could have written that software myself in a vacuum it would have never been worth it financially, since it would have delayed the launch of the integration by 6+ months. Building the software post-launch of the integration would have easily taken 2.5 years at minimum.
But this is also basically a "spherical cow in a vacuum" scenario, where I was essentially acting as a solo dev, in full operational control of the project, with deep domain knowledge of the topic and an allready fully set up codebase that I knew perfectly while working down ideas I've had in my backlog for 5+ years.
First, it's not three years since. The real improvements in programming ability arrived in the last 6-8 months.
Second, the Internet didn't show up much in GDP and similar measures either!
But your point stands. Where are the amazing digital products/stuff? I get that it might take time to arrive as we scale up compute and learn new paradigms. But so much infra already exists (deployment pipeliens, everyone reachable on a smartphone) that we should be seeing something.
What's moving the goalposts? I am very much amazed at what Fable can do. I push its code straight to prod.
But I am just as amazed with how little real life consequence it seems to have! Even software houses were hit more by interest rates than by this magical revolution.
If I couldn't directly observe Fable in action, I wouldn't believe in AI.
I think it's more that it shifts the thought process from "If this is going to take 30 years then we won't bother, because the investment can be spent on things that pay off sooner" to "If we can do this in 3 years then we'll make that investment because that's a good bet."
The way it changes the game is by lowering the cost of making radical bets so we end up trying more moonshots.
Right now we are in the golden age where we do the same and take time off. Employers have not yet fully caught up with the workforce. I can't think of people that are not putting less hours this year for the same salaries.
> If AI makes us 10x more productive, the 3 years since its popularization were enough for a product that would have taken 30 years to build without AI, for instance.
A product still requires a lot of handholding and human thinking, at least if one does not want everyone even throwing a glance at it to immediately be repulsed by the usual AI slop tells.
> I'm an AI advocate but that question makes me feel that AI is simply "very useful" rather than being a historical game changer for humanity.
It absolutely already is a historical game changer on par with the Industrial Revolution when it comes to the amount of jobs destroyed and economies screwed up - and the impact will be even worse in 10+ years as existing seniors retire but no new seniors rise as AI has destroyed entry level career paths.
Not disagreeing with you, but adding to my argument. Despite the human handholding, I feel that 3 years would have been enough for 18 months of thinking about the product plus 18 months where a team could get 3-5 years worth of coding/development done. Yet, we haven't seen anything big yet, like a new Youtube/Instragram, an amazing videogame, a major cure etc. Maybe they are coming, but every day that passes is an indication that the net productivity positive of the technology isn't as big as advertised. In my own personal use of AI, I experienced a 30-50% increase in productivity, but no more than that.
It's only been at most the past year where AI has been unambiguously helpful and not a hindrance. 3 years ago it gave the appearance of being helpful but it tended to be more of a hindrance.
Maybe so, but least for me in my personal coding projects, I'm going through my own task list noticeably faster. I had created this list before getting a ChatGPT Pro subscription and the number of bugs found in AI reviews and rate of closing tasks has certainly increased. I'm not saying my anecdote scales to teams or even other people, but I have no doubt about my personal productivity change. I wouldn't be paying for it otherwise.
> It absolutely already is a historical game changer on par with the Industrial Revolution when it comes to the amount of jobs destroyed and economies screwed up
Which economies are already screwed up?
IMO it can’t ever be on par with the Industrial Revolution because AI can only really affect the information economy. Things people do with their hands/bodies have either already been automated or can’t be with current tech. If you’d asked people decades ago they might say no one will ever work in factories by 2026 because they’ll all be automated. It didn’t work out that way. I think AI will go the same way: absolutely game changing to some industries (of which software engineering will be one) but a great many will still survive with less dramatic changes.
If anything it might result in more focus on the human aspects. How many people out there earn their stripes putting together slide decks? In a world where an AI can put together the snazziest presentation you’ve ever seen in a heartbeat it’s going to matter more how you stand at the front of the room and present those slides than it does today.
It will be interesting to see what kind of progress in biology and medicine they will preview during the fall. He does make a good point that most of the progress so far have not been material in the sense of providing real positive outcomes for ordinary people.
Or even for the HN crowd, when will we see e.g. "Mythos aided research discovers 10 new viable battery technologies"
When people talk about Qwen 3.8 being on a par with Fable, they're really talking about Qwen 3.8 Max aka Qwen3.8-2.4T-A95B. That's a 2.4 trillion parameter Mixture of Experts model with 95B active parameters. You need about 400GB of RAM to run it. No one is running that locally.
The distillations of Qwen 3.8 down to a 27B model are good, but they're not on a par with frontier models.
When Dario talks about open weights not being a solution this is what he means - if you don't have 400GB of VRAM lying around the fact that there's an open model like Qwen3.8-2.4T-A95B doesn't really help much. If we're not regulating how models are available, or making sure access is open, then RAM prices will mean everything concentrates on a few very rich companies.
Practically, no, the distill is great. It's fine to use it.
However, if you're having a discussion about access to frontier models, and using Qwen 3.8 as an example of how open weights is a solution, then you should be honest and accurate about what you're talking about. Making an argument like "People can run Qwen 3.8 at home. That shows open weights are great." is a bit disingenuous if you're not also making it clear that you're not talking about Qwen 3.8 Max (or that you have a beast of a PC at home :D ).
I agree 100%. IMO, it all started with ollama misrepresenting the Deepseek R1 distills as Deepseek R1, all for hype and marketing. I've had so many ostensibly technical people telling me: "I tried DeepSeek R1 and it was terrible", and every time when I probe further they'd tried the tiny 1.5B Qwen2.5 distill model that was further brain damaged by ollama's naive RTN quantization[1]. DeepSeek themselves were very forthright about it by naming it DeepSeek-R1-Distill-Qwen-1.5B[2].
I suppose the road to technical hell is paved with marketers and grifters. :)
> Overall my view is that AI is structurally a technology that tends to concentrate power
> Open-weights do help some with this but are nowhere near a sufficient solution
Which is exactly why Anthropic contributes nothing to, and actively pushes for roadblocks and regulations for open weight models. Can't allow any hope to the masses.
Only people with $$$$$ are allowed to touch Fable. Which of course we have have aggressive guardrails for in case you even try to use it for something dangerous like AI model developement. Plus we are going to retain all data submitted to it just in case someone is trying to be sneaky.
Why the comment submitted by jacquesm, who posted the link and it's not exactly an anonymous poster, that said "Apologies for linking to X but this is worth reading." has been flagged to death??
I'm more interested in why somebody would suddenly flag that post. It's not spam, bot, offensive or anything else. Maybe you could downvote it if you don't like the take on X - it still would be childish but at least would be proportionate.
> This is why Anthropic has always made its policy proposals very carefully. We try very hard to make proposals that disadvantage (slow down) frontier AI companies while advantaging smaller competitors.
> ...
> completely exempt any company below a certain amount of revenue or model training costs from being covered at all
One could argue that "Frontier AI" company know they have nothing to fear from company with less than XM$ revenue, and so their support for this type of regulation is still a way to force regulatory capture. In any case, whatever regulation you support, it's a regulation that you didn't have to handle when you were growing, but that incumbent will have to deal with.
Whatever it is Dario's intention or not does not matter. Capitalism push to consolidation and the eventual regulation that will need to be applied to mitigate the externality from a new industry, will mean that their will only be a handful of "very big" winner. Same story since the beginning of the industrial era. Even if that's not what Dario personally want, it's in the best interest of the shareholders, which will force Anthropic to do everything it can to be one of the big one.
> Overall my view is that AI is structurally a technology that tends to concentrate power, for reasons that have nothing to do with regulation (more to do with the extreme implications of the scaling laws).
The same can be said about a lot of other industry (aviation, oil, chip manufacturing, ...). Every country / union big enough will finance their own champion to try to keep a foot in the industry even if they are not the best.
> I know that there’s a sort of Silicon Valley shorthand where regulation = regulatory capture = concentration of power, but I’ve always found this to be an overly simplified picture of the world.
To paraphrase Sinclair "It's difficult to make a man realize the issues with regulatory capture, if they're to be the benefactor of regulatory capture".
> A crude analogy is that the formal court system can sometimes feel stuffy and elitist, but it does a much better job of defending the rights of vulnerable individuals than the alternative, mob justice.
Not so sure. After all lack of the latter did help establish the current tech overlord rule.
> "I know that there’s a sort of Silicon Valley shorthand where regulation = regulatory capture = concentration of power, but I’ve always found this to be an overly simplified picture of the world."
> wall of text follows
> doesnt proceed to clearly tell us what the actual picture of the world is, then
am i correct in summarizing that the line of reasoning is
- frontier llm access means you are at an economic advantage
- a big risk of this is ongoing wealth concentration
- the "open weights" approach can't solve the problem of wealth and llm access being linked; you need compute too, and compute is expensive, thus "open weights" still favors the wealthy
- instead we need "objective and fair institutional processes", as this will allow small labs cook up their stuff while frontier labs get regulated
why would i care about what these smaller players do, if economic advantage = frontier model access? also, the reasoning around "why bother with open weights because compute is expensive too" seems like the kind of logic a motivated 12 year old could work their way around in 30 seconds. things are not either/or dario, you said so yourself.
seems like a 400 word corpo misdirection essay. par for the course.
> I think that ordinary people don’t trust companies, governments, or the tech industry and always suspect that we are cooking up some new way to screw them over.
It is so tragicomic to see people whose lives are built around companies coming so close to realizing that everything they do is bad, and then at the last minute swerving aside to convince themselves that no, if they just do more of it and somehow do it "better", then it will all be okay. The reason people don't trust companies and believe they are cooking up some new way to screw them over is because that is what they are doing. If Dario or anyone else really wanted to dispel that perception there's an easy way: do a total 180 and start fighting against everything you've been pushing. But none of them will do that because they still fundamentally believe that what they are doing is good, and are unable to see that fundamentally it's bad.
I think that Dario is intelligent, incisive, has good judgement, and is well-intentioned. I hope he continues to wield influence. I think Sam is also most of those things, but I think power has a One Ring-like effect on him. My most contrarian view is that I think Elon’s problems are primarily appalling judgement when it comes to areas outside of tech, and has the emotional regulation of a child, although somewhat incredibly I believe he is fundamentally well intentioned. I don’t think any of those three really want tech feudalism, although I think the current US administration would press a “turn us into Russia” button the second they caught sight of it, which is to say, I think they have the worst of intentions.
> I think Sam is also most of those things, but I think power has a One Ring-like effect on him.
That's putting it mildly, very mildly. The guy all but wrecked the entire world's DRAM supply chain by using negotiation tactics that, if there were any semblance of regulatory authority left in the US, would lead to criminal charges for market manipulation.
Arrest Sam Altman, he is one of the most dangerous people on the planet. Not a single shred of respect for the 99% and the consequences his actions have on them. (And similar things apply to the other controversial figure you mentioned, Elon Musk actively enjoys hurting people by the looks of it.)
Dario's messaging has been extremely confusing and HE'S RESPONSIBLE for further increasing the negative sentiment of the public towards AI.
He's clearly focusing on being on the news to provoke emotions on people, preparing for the Anthropic big blockbuster IPO.
Dario, Sam Altman and others should instead be praying every night that this will take us to the Singularity VERY SOON, because if it doesn't, the entire country will blame them for absolutely destroying their retirement and the US' economy once this AI Datacenter bubble pops.
> I think it is fundamentally a crisis of trust. I think that ordinary people don’t trust companies, governments, or the tech industry and always suspect that we are cooking up some new way to screw them over.
[...]
> I don’t think that a glitzy marketing campaign with a positive spin (which some have advocated that Anthropic do) is the way to win back that trust — at this point, saying that AI will cure cancer is more a cliche than it is inspiring, and most people think it is deceptive. The thing that will work is actually curing cancer. I think by far the most accurate criticism of AI companies including Anthropic is that we haven’t yet delivered on our big promises to benefit the world. That is totally on us, and I think it’s the criticism you should be making, instead of all this stuff about messaging and marketing.
> We are however doing our best to fix this: Anthropic is ramping up its efforts very quickly in biology and medicine, and we hope to have incredible results in the coming years and some early glimmers in the coming months. When we’ve actually accomplished something real, the whole world will hear about it, as loudly as possible, you have my word on that.
Turns out "winning back trust" doesn't have anything to do with any actual concerns people may have re: employment, electricity prices, stock market bubble, intellectual property, scams, cybersecurity, environmental issues etc. Rather we'll just do all of that even harder and the miracles ("curing cancer", lol) we've so far failed to deliver are bound to arrive in short order!
This is actually crazy, because biochemistry is where LLMs are weakest. It's one of science's most fuzzy and unpredictable domains, in general, and a lot of published works which an LLM might take at face value might be unreplicable or subtly flawed. (See e.g. the entire history of Alzheimer's.) It is also where real-world lab and clinical trial work is most important.
So Anthropic are going to spin up a medicinal chemistry lab and start mouse experiments?
I mean, it would be nice, but I don't think that the guys at Anthropic know what they're talking about here, or what they might be getting themselves into. (If indeed this is more than just PR.)
See also this article by a genomics PhD about how intelligence is not the bottleneck in medical research.
https://www.writingruxandrabio.com/p/intelligence-is-not-the...
The effective altruist and longtermist crowd are into it as a religion. They would happily sacrifice anything human to satisfy their dreams of AI. They care way more about what the AI needs than their what their fellow humans need. We will end up with an AGI having better living and working conditions than humans and the whole AI industry will be clapping
rapture but for tech people
Not really. The problem is that the trust is violated here. What people fear is that government and large companies are going to use AI against them, and make the reverse impossible. Sadly, this is exactly what is happening.
And that's exactly what EU's AI regulation does. Of course, AI is being blamed for this happening, despite that last time I checked every EU parlementarian, every EU commission member is flesh and blood, the fact that it is 100% intentional government policy, humans, that are doing this to you using very un-AI methods.
You see, in this regulation, governments are allowed to use AI, and to approve uses of AI. So there is absolutely nothing in EU's AI directive at all that prevents government and large companies, with approval, from answering the phone, and all "support requests" with AI, and not with people. Also "if you agree" (just like you've agreed to selling your location data in your cell contract)
The watermarking algorithm itself has a VERY specific property that should have set off everyone's alarm bells: it is NOT the case that you can "check text for AI watermark". What it technically mandates is that if you provide access to a model, those people should be able to check if text is generated by that model. Not by any other model. It is NOT a general "is this AI?" check. So let's analyze how this works in 2 specific cases:
You get a mail from the government about taxes. You want to check if that text is generated by an AI and whether you even should ask to talk to a human. So you've got a piece of text, or an audio recording. Can you check if it's AI or not? NO YOU CAN'T. SynthID requires access to the model keys, and the legislation only mandates checks if you've got access to the model.
The government or some large company, however, puts in it's regulations that you're not allowed to use AI to communicate with them (we all know this is coming), and they get a piece of ChatGPT text (or from any public model) from you. They would like to enforce that they won't talk to AI can they do this? YES, since they have access to the model too, SynthID allows for this usecase AND the AI regulation makes this mandatory.
Is this in any way a problem with AI? No. Can anything be done about this by "fighting AI"? Well let's see ... will government institutions be forced to stop worsening services further if ChatGPT etc become inaccessible or obviously excluded by government and large company services? No.
What is being outlawed in the EU, in other words, is exactly ONE thing: that you use AI to help you in your dealings with governments and large companies. To assert your rights, to resolve issues faster, to make them wait rather than you, to ... THAT is being outlawed, nothing else. Only that YOU get help from AI.
This legislation will make it a big advantage to have access to "your own" model, because you'll have the SynthID keys, and the law behind you if you do that. And I don't mean local model, I mean your own model with your own SynthID keys, ie. not Google, not ChatGPT, not Claude, not Grok, and if the EU can make it happen, not Chinese models either. Finetuned models, however, will not have SynthID and so you'll never be able to prove text generated by them is AI, and you will not effectively have the power to refuse contracts or laws that specify they can use AI anyway.
It's the EU government making it clear they are using AI to avoid even having to talk to you at all while disallowing the use of AI by mere citizens to assert their rights, against them, or anything they have interests in, like phone or electricity companies.
The only factual point you made is solved on February second. All Model providers have to provide offline tooling to check for invisible watermarking...
> Overall my view is that AI is structurally a technology that tends to concentrate power, for reasons that have nothing to do with regulation (more to do with the extreme implications of the scaling laws). Open-weights do help some with this but are nowhere near a sufficient solution because they simply shift the concentration somewhat to those with the most compute and chips (which are roughly the frontier labs plus maybe hardware providers).
This is actually a good point. Open models are getting better and better, some of them might even be useful in consumer hardware now. But if AI performance is still correlated with compute power, then no doubt power will remain with the people owning the chips.
Arguably the question is whether it's economically feasible to self-host something similar.
Are you self-hosting Google or Bing? No, but we have quite a huge ecosystem of full-text search tools with PageRank, with options to scale to almost Google scale (if you have the money). After all LLM training starts with the same crawl mechanism.
As long as barriers to entry is not too high (ie. it makes sense to take the risk to start a business that provides something similar - usually for a niche) market forces work.
We have the classic empirical chart reproducing microeconomics.
https://www.fda.gov/about-fda/center-drug-evaluation-and-res...
And setting up a pharma plant is also very capital intensive.
Here the obvious barrier to entry is completely artificial. (Which provides an incentive to spend a lot of money on R&D -- though it naturally raises the question of Pareto efficiency.)
You're missing the point, what will happen is this:
1) in things like tax law, registering with city hall, dealings with the DMV, your phone subscription, insurance contract, ... you will find that one of the new fine prints in the contract will be that you're not allowed to use AI to communicate with them.
2) because of how SynthID works (you need the SynthID keys to verify, which are secret. So the only way to find if text is ChatGPT/Google/Anthropic watermarked is to ask ChatGPT/Google/Anthropic), government and large companies can enforce this against you. That is what the watermark is for. To end any insurance claim written by AI with "you're not allowed to submit AI written insurance claims" and refuse it outright there and then.
"Sorry your request was AI watermarked and pursuant to law 234 of 2025/03/11 chapter 3258 paragraph 33 decile 1299 we hereby close it without response"
3) when they reply, however, they use a custom model that also has custom SynthID keys. You will not even be able to tell their responses are AI written, or at least, you won't be able to prove it. You won't be able to enforce any AI-related rights (ie. the right to talk to a human) you have under the law against large companies.
In other words: this is to make sure that all the advantages AI provides are available to deny your unemployment claim, and to Verizon to charge you more, but completely inaccessible TO YOU when you want to change to a cheaper subscription. They can inundate YOU with AI-written requests BUT YOU CAN'T.
Self-hosting helps because it prevents them from verifying if your responses are AI written, because you can generate non-watermarked AI text and so there is a level playing field.
I think his point is hand wavy at best. It presupposes infinite scaling and ignores all the algorithmic efficiency wins that are being discovered. Ironically, many of which are being discovered with autoresearch style workflows, using the very LLMs that his company builds.
The #1 post on HN right now[1] is full of people jubilating about how they can run Qwen 3.8 27B on their > 5 year old GPUs. If that isn't democratization of AI, I don't know what is.
I'm sure he's smart enough to instantaneously realize this too, but as the famous Upton Sinclair quote goes, he won't mention it even if he does.
[1]: https://news.ycombinator.com/item?id=49324985
The very definition of (applied) technology is power amplification. Use a lever, move more weight than you could before, 1 person with the tool now wields the power of 3 without.
Making "tech" a career and a societal goal onto itself, without the adjoining understanding of and deep commitment to ethics and the responsible use of power, is why we're sliding into authoritarian rule by a small circle of techno-oligarchs.
We need less "move fast and break things" and more "plant trees you will not live to see bear fruit."
Called it! He was bound to squeal after the release of QWEN 3.8 in one way or another and here it is. I wasn't even a teenager but I'm getting so much Jobs/Ballmer flashbacks with internet explorer and microsoft office.
Edit: Does anyone else notice the switching between dashes and em-dashes between paragraphs? Tells you a lot about the man, doesn't it.
> Edit: Does anyone else notice the switching between dashes and em-dashes between paragraphs? Tells you a lot about the man, doesn't it.
That he doesn’t know how to use basic word processing, or even agent SKILLS.md or whatever it’s called now. Or maybe it’s just the next generation of vagueposting.
Tells you a lot about the man, doesn't it.
The CEO of Anthropic not using AI extensively would be news.
One thing Dario said is important reflecting on (but from another angle): where's the big deliverable from AI? If AI makes us 10x more productive, the 3 years since its popularization were enough for a product that would have taken 30 years to build without AI, for instance.
I'm an AI advocate but that question makes me feel that AI is simply "very useful" rather than being a historical game changer for humanity.
Anyone saying 10x as a serious claim is clearly using a round number and vibes; however, even if it were so, AI getting popular 3 years ago does not mean what you say.
The earliest used models in this category would easily 10x (what did just say) the creation of one-off short scripts, but you're not writing a 30-year app out of just a bunch of short scripts.
The METR time horizons graph suggests we can now get 10x (ahem) speedup on solving most coding problems that take us a few hours and about half of problems that take 2 days. Amdahl's law bites: even infinity speedup on half your problems is only 2x overall.
If you let an LLM lose, with a huge budget, what's the biggest artefact it can make before it drowns under the weight of bad decisions? The C compiler and web browser headlines a while back? The maths papers we see now that solve problems which stumped the maths world for decades?
But this is the other side of the same coin: teamwork. One person getting a thing made in twelve months vs a team of a hundred, you can scale up fast with money when you have a proof of concept, and an LLM can make a lot of proofs of a lot of concepts even in free accounts.
LLM-assisted software engineering seems to be very efficient if you have a deterministic target. (bun rewrite from zig to Rust, 100% Node.js compatibility, pnpm compatibility -- https://github.com/oven-sh/bun/pull/38333)
Can only speak for my own project, but can give an example:
After an aquisition earlier this year I got the task of doing an SAP-Integration for the new company, last time I did this 5 years ago it was a 6 month task, but with the experience and skills ive gained since I estimated it would be a 3 month project (with or without AI, most work is just logistics, AI cant help much there).
In those 3 months I was able to not only integrate SAP but also deliver a completely modernised user-facing software for that integration. While I could have written that software myself in a vacuum it would have never been worth it financially, since it would have delayed the launch of the integration by 6+ months. Building the software post-launch of the integration would have easily taken 2.5 years at minimum.
But this is also basically a "spherical cow in a vacuum" scenario, where I was essentially acting as a solo dev, in full operational control of the project, with deep domain knowledge of the topic and an allready fully set up codebase that I knew perfectly while working down ideas I've had in my backlog for 5+ years.
First, it's not three years since. The real improvements in programming ability arrived in the last 6-8 months.
Second, the Internet didn't show up much in GDP and similar measures either!
But your point stands. Where are the amazing digital products/stuff? I get that it might take time to arrive as we scale up compute and learn new paradigms. But so much infra already exists (deployment pipeliens, everyone reachable on a smartphone) that we should be seeing something.
I remember people moving the goalposts like that since, let me check... when did AI Dungeon go viral? Wow, 2019.
What's moving the goalposts? I am very much amazed at what Fable can do. I push its code straight to prod.
But I am just as amazed with how little real life consequence it seems to have! Even software houses were hit more by interest rates than by this magical revolution.
If I couldn't directly observe Fable in action, I wouldn't believe in AI.
> What's moving the goalposts? I am very much amazed at what Fable can do. I push its code straight to prod.
>> What's moving the goalposts? I am very much amazed at what Opus 4.8 can do. I push its code straight to prod.
>>> What's moving the goalposts? I am very much amazed at what Opus 4.6 can do. I push its code straight to prod.
>>>> What's moving the goalposts? I am very much amazed at what GPT5 can do. I push its code straight to prod.
>>>>>> What's moving the goalposts? I am very much amazed at what Opus 3.5 can do. I push its code straight to prod.
I think it's more that it shifts the thought process from "If this is going to take 30 years then we won't bother, because the investment can be spent on things that pay off sooner" to "If we can do this in 3 years then we'll make that investment because that's a good bet."
The way it changes the game is by lowering the cost of making radical bets so we end up trying more moonshots.
Do you have examples of such moonshots? Ideally not themselves AI related
Right now we are in the golden age where we do the same and take time off. Employers have not yet fully caught up with the workforce. I can't think of people that are not putting less hours this year for the same salaries.
> If AI makes us 10x more productive, the 3 years since its popularization were enough for a product that would have taken 30 years to build without AI, for instance.
A product still requires a lot of handholding and human thinking, at least if one does not want everyone even throwing a glance at it to immediately be repulsed by the usual AI slop tells.
> I'm an AI advocate but that question makes me feel that AI is simply "very useful" rather than being a historical game changer for humanity.
It absolutely already is a historical game changer on par with the Industrial Revolution when it comes to the amount of jobs destroyed and economies screwed up - and the impact will be even worse in 10+ years as existing seniors retire but no new seniors rise as AI has destroyed entry level career paths.
Not disagreeing with you, but adding to my argument. Despite the human handholding, I feel that 3 years would have been enough for 18 months of thinking about the product plus 18 months where a team could get 3-5 years worth of coding/development done. Yet, we haven't seen anything big yet, like a new Youtube/Instragram, an amazing videogame, a major cure etc. Maybe they are coming, but every day that passes is an indication that the net productivity positive of the technology isn't as big as advertised. In my own personal use of AI, I experienced a 30-50% increase in productivity, but no more than that.
It's only been at most the past year where AI has been unambiguously helpful and not a hindrance. 3 years ago it gave the appearance of being helpful but it tended to be more of a hindrance.
perhaps being 30-50% ‘busier’ is something quite different from being 30-50% more ‘productive’ and ppl are just bad at detecting the difference
Maybe so, but least for me in my personal coding projects, I'm going through my own task list noticeably faster. I had created this list before getting a ChatGPT Pro subscription and the number of bugs found in AI reviews and rate of closing tasks has certainly increased. I'm not saying my anecdote scales to teams or even other people, but I have no doubt about my personal productivity change. I wouldn't be paying for it otherwise.
> It absolutely already is a historical game changer on par with the Industrial Revolution when it comes to the amount of jobs destroyed and economies screwed up
Which economies are already screwed up?
IMO it can’t ever be on par with the Industrial Revolution because AI can only really affect the information economy. Things people do with their hands/bodies have either already been automated or can’t be with current tech. If you’d asked people decades ago they might say no one will ever work in factories by 2026 because they’ll all be automated. It didn’t work out that way. I think AI will go the same way: absolutely game changing to some industries (of which software engineering will be one) but a great many will still survive with less dramatic changes.
If anything it might result in more focus on the human aspects. How many people out there earn their stripes putting together slide decks? In a world where an AI can put together the snazziest presentation you’ve ever seen in a heartbeat it’s going to matter more how you stand at the front of the room and present those slides than it does today.
It will be interesting to see what kind of progress in biology and medicine they will preview during the fall. He does make a good point that most of the progress so far have not been material in the sense of providing real positive outcomes for ordinary people.
Or even for the HN crowd, when will we see e.g. "Mythos aided research discovers 10 new viable battery technologies"
None?
When people talk about Qwen 3.8 being on a par with Fable, they're really talking about Qwen 3.8 Max aka Qwen3.8-2.4T-A95B. That's a 2.4 trillion parameter Mixture of Experts model with 95B active parameters. You need about 400GB of RAM to run it. No one is running that locally.
The distillations of Qwen 3.8 down to a 27B model are good, but they're not on a par with frontier models.
When Dario talks about open weights not being a solution this is what he means - if you don't have 400GB of VRAM lying around the fact that there's an open model like Qwen3.8-2.4T-A95B doesn't really help much. If we're not regulating how models are available, or making sure access is open, then RAM prices will mean everything concentrates on a few very rich companies.
> The distillations of Qwen 3.8 down to a 27B model are good, but they're not on a par with frontier models.
Does it have to be? There are plenty of coding tasks, where it's good enough.
Practically, no, the distill is great. It's fine to use it.
However, if you're having a discussion about access to frontier models, and using Qwen 3.8 as an example of how open weights is a solution, then you should be honest and accurate about what you're talking about. Making an argument like "People can run Qwen 3.8 at home. That shows open weights are great." is a bit disingenuous if you're not also making it clear that you're not talking about Qwen 3.8 Max (or that you have a beast of a PC at home :D ).
I agree 100%. IMO, it all started with ollama misrepresenting the Deepseek R1 distills as Deepseek R1, all for hype and marketing. I've had so many ostensibly technical people telling me: "I tried DeepSeek R1 and it was terrible", and every time when I probe further they'd tried the tiny 1.5B Qwen2.5 distill model that was further brain damaged by ollama's naive RTN quantization[1]. DeepSeek themselves were very forthright about it by naming it DeepSeek-R1-Distill-Qwen-1.5B[2].
I suppose the road to technical hell is paved with marketers and grifters. :)
[1]: https://ollama.com/library/deepseek-r1:1.5b [2]: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-...
No one is running that locally because of the AI bubble consuming all the hardware in the industry. That won’t be the case long term though
> Overall my view is that AI is structurally a technology that tends to concentrate power
> Open-weights do help some with this but are nowhere near a sufficient solution
Which is exactly why Anthropic contributes nothing to, and actively pushes for roadblocks and regulations for open weight models. Can't allow any hope to the masses.
Only people with $$$$$ are allowed to touch Fable. Which of course we have have aggressive guardrails for in case you even try to use it for something dangerous like AI model developement. Plus we are going to retain all data submitted to it just in case someone is trying to be sneaky.
Why the comment submitted by jacquesm, who posted the link and it's not exactly an anonymous poster, that said "Apologies for linking to X but this is worth reading." has been flagged to death??
Vouched. (A few more might resurrect it)
Maybe because "apologies for linking to X" is needless gaudy virtue signalling.
Make it "Apologies for linking to Amodei" and we're talking.
You can 'vouch' for a comment, if you think it shouldn't be flagged.
I'm more interested in why somebody would suddenly flag that post. It's not spam, bot, offensive or anything else. Maybe you could downvote it if you don't like the take on X - it still would be childish but at least would be proportionate.
i was about to post the same thing. i vouched for the comment, but it's too dead to recover from my single vouching action.
This site's audience has been less of the technical curious kind and more of the pathological misalignment one
> This is why Anthropic has always made its policy proposals very carefully. We try very hard to make proposals that disadvantage (slow down) frontier AI companies while advantaging smaller competitors.
> ...
> completely exempt any company below a certain amount of revenue or model training costs from being covered at all
One could argue that "Frontier AI" company know they have nothing to fear from company with less than XM$ revenue, and so their support for this type of regulation is still a way to force regulatory capture. In any case, whatever regulation you support, it's a regulation that you didn't have to handle when you were growing, but that incumbent will have to deal with.
Whatever it is Dario's intention or not does not matter. Capitalism push to consolidation and the eventual regulation that will need to be applied to mitigate the externality from a new industry, will mean that their will only be a handful of "very big" winner. Same story since the beginning of the industrial era. Even if that's not what Dario personally want, it's in the best interest of the shareholders, which will force Anthropic to do everything it can to be one of the big one.
> Overall my view is that AI is structurally a technology that tends to concentrate power, for reasons that have nothing to do with regulation (more to do with the extreme implications of the scaling laws).
The same can be said about a lot of other industry (aviation, oil, chip manufacturing, ...). Every country / union big enough will finance their own champion to try to keep a foot in the industry even if they are not the best.
> I know that there’s a sort of Silicon Valley shorthand where regulation = regulatory capture = concentration of power, but I’ve always found this to be an overly simplified picture of the world.
To paraphrase Sinclair "It's difficult to make a man realize the issues with regulatory capture, if they're to be the benefactor of regulatory capture".
> A crude analogy is that the formal court system can sometimes feel stuffy and elitist, but it does a much better job of defending the rights of vulnerable individuals than the alternative, mob justice.
Not so sure. After all lack of the latter did help establish the current tech overlord rule.
- frontier llm access means you are at an economic advantage
- a big risk of this is ongoing wealth concentration
- the "open weights" approach can't solve the problem of wealth and llm access being linked; you need compute too, and compute is expensive, thus "open weights" still favors the wealthy
- instead we need "objective and fair institutional processes", as this will allow small labs cook up their stuff while frontier labs get regulated
why would i care about what these smaller players do, if economic advantage = frontier model access? also, the reasoning around "why bother with open weights because compute is expensive too" seems like the kind of logic a motivated 12 year old could work their way around in 30 seconds. things are not either/or dario, you said so yourself.
seems like a 400 word corpo misdirection essay. par for the course.
He seems to be grasping for a point but not providing argument for what it is exactly. And certainly not support that is logically coherent
So, what does a for-profit CEO, beholden to profit-seeking investors, have to say?
> I think that ordinary people don’t trust companies, governments, or the tech industry and always suspect that we are cooking up some new way to screw them over.
It is so tragicomic to see people whose lives are built around companies coming so close to realizing that everything they do is bad, and then at the last minute swerving aside to convince themselves that no, if they just do more of it and somehow do it "better", then it will all be okay. The reason people don't trust companies and believe they are cooking up some new way to screw them over is because that is what they are doing. If Dario or anyone else really wanted to dispel that perception there's an easy way: do a total 180 and start fighting against everything you've been pushing. But none of them will do that because they still fundamentally believe that what they are doing is good, and are unable to see that fundamentally it's bad.
I think that Dario is intelligent, incisive, has good judgement, and is well-intentioned. I hope he continues to wield influence. I think Sam is also most of those things, but I think power has a One Ring-like effect on him. My most contrarian view is that I think Elon’s problems are primarily appalling judgement when it comes to areas outside of tech, and has the emotional regulation of a child, although somewhat incredibly I believe he is fundamentally well intentioned. I don’t think any of those three really want tech feudalism, although I think the current US administration would press a “turn us into Russia” button the second they caught sight of it, which is to say, I think they have the worst of intentions.
> I think Sam is also most of those things, but I think power has a One Ring-like effect on him.
That's putting it mildly, very mildly. The guy all but wrecked the entire world's DRAM supply chain by using negotiation tactics that, if there were any semblance of regulatory authority left in the US, would lead to criminal charges for market manipulation.
Arrest Sam Altman, he is one of the most dangerous people on the planet. Not a single shred of respect for the 99% and the consequences his actions have on them. (And similar things apply to the other controversial figure you mentioned, Elon Musk actively enjoys hurting people by the looks of it.)
Dario's messaging has been extremely confusing and HE'S RESPONSIBLE for further increasing the negative sentiment of the public towards AI.
He's clearly focusing on being on the news to provoke emotions on people, preparing for the Anthropic big blockbuster IPO.
Dario, Sam Altman and others should instead be praying every night that this will take us to the Singularity VERY SOON, because if it doesn't, the entire country will blame them for absolutely destroying their retirement and the US' economy once this AI Datacenter bubble pops.
There are other countries.
* * *