Watching the latest Ezra Klein NYT video thinkpiece from today [1], it seems to me there's an alliance emerging: some degree of political control will handed to the party of the managerial class, the party that represents the threatened class of knowledge workers, in exchange for the regulatory capture the labs are after.
They've been raising the issue bi-monthly through mini-scandals that have until now been consistently slapped down by Jensen Huang, but it seems an alliance with Democrats, just prior to an election where they're poised to take power in the Senate and the House, might finally be how they crack their "problem."
It won't be long before a massive incident is blamed on an open model in the wild, not from inside the labs, and none of us will be able to leverage open models to run private business workflows for the cost of electricity and hardware.
It's worth noting as well that if the Democrats imagine they'll get a "slow down" to protect one of their main constituencies, the professional class of credentialed knowledge workers (or however you slice it), they're dreaming--the labs have stated openly again and again that their business model is to capture the 10T TAM that represents the sum of wages of that very class of workers.
This is an AI written post and the details are wrong (the description of the HF incident as involving Irregular is wrong and the description of the incident as only involving stealing public credentials is wrong, per the technical report the agents got access to internal HF infrastructure).
Your description of the HF attack as being merely "the elite task of discovering 14 Hugging Face API tokens that careless developers had committed to public GitHub repositories" does not match the description in the technical report[1].
The description reads "the elite task of discovering 14 Hugging Face API tokens that careless developers had committed to public GitHub repositories, and used them to try to get benchmark solutions from directly from Hugging Face by applying a template injection flaw that’s been known about since 2015[1]."
Chaining a public token to an 11-year-old Jinja2 template injection vuln shouldn't be dressed up as an unprecedented "alien intellect" that threatens human civilisation. (And HuggingFace should take some flack for having such a dated vulnerability exposed - if your Bank was compromised in this way, you'd be blaming your bank, not the attacker.)
One correction is fair though, the 14 tokens were in a public Hugging Face dataset not a public GitHub repository. I've updated the post to reflect that.
"Every single one of these catastrophic breakouts happened inside the testing environments of the exact same vendor."
This is incorrect, the HF incident for example (the most well known) had nothing to do with irregular. I know there has been a news site pushing inaccurate articles (effort.news) on this topic but these are the facts.
Thank you, you're correct. Effort.news was one of my research sources, but you're right that although OpenAI use Irregular, they were not involved in the specific HF incident (although the failure mode was otherwise identical). I've updated the post to make that clear.
> For Anthropic, Google, and Meta, the catastrophic breakouts happened inside the testing environments of the exact same contractor.
If this is the level of understanding you have of the relevant incidents, there's a lot of chutzpah in saying that other people are "selling garbage", carrying out an "extraordinary confidence trick", etc.
The failure mode was _not_ identical. The HF incident agents were not directly connected to the internet and had to compromise an internal package registry in order to access the internet.
Let them cry, I don't see anything changing unless they can somehow get China to agree. And I doubt China will drink any of that kool aid especially while they're being disadvantaged by export controls, so the ever-improving open weight models will continue to rain. This is something the US Big Tech oligarchs will NOT win.
It isn't about China. The Big AI wants regulatory relaxation. In particular anti-cartel relaxation. "Threepenny novel" explains it very well.
Currently we have a classic market race - the market forces both companies to provide more and more capable models with more and more value for the money for the customers while the suppliers (Nvidia, Micron, Dell) squeeze them from the other side. The end result would be one winner taking it all (and that is a very humongous "all") while the other falling into a very distant second position at best. Do their leadership and investors (bonus points - look at some OpenAI investors), some already worth tens of billions on paper, like the prospects of that "50% chances of even more riches, 50% - bust" outcome? I'd think - no.
The outcome they would like is both companies divvying up that huge market while jointly raising prices and providing less capable models (ie. cheaper to train and run) while squeezing their suppliers Wallmart style. How to get there? By breaking the anti-cartel limitations.
The typical tools to break anti-cartel limitations is for example perception of
national interests or perception of some imminent dire emergency.
Thus the "lets us collaborate or our AI will kill you all" scare campaign.
There is really is no excuse for Washington here. Even for the layperson+, it doesn't take much use of an LLM to figure out where they produce good and usable results and where it's of specious of value.
+ every layperson is an expert in 1 or more areas.
I dont have to be an expert to recognize when something is said with authority and confidence. Any time a model says something with conviction, my instinct is to double check.
Now if they taught them to express uncertainty and speak in terms of probability, I might be more likely to be blindly fooled by some kind of uncertain conviction.
Neither of you is wrong? Every lived experience does produce a unique expertise, not in "being" that person, but navigating their experiences. It would be very unusual that all of someone's experiences are completely worthless.
I suppose you've never had a unique perspective on something? Maybe that reflects on your self esteem? How unfortunate.
I thought the whole point of this discussion is that they don't have to care if it works. All that matters is that the majority believes it does, or at least doesn't make a fuss about the continued spending.
Watching the latest Ezra Klein NYT video thinkpiece from today [1], it seems to me there's an alliance emerging: some degree of political control will handed to the party of the managerial class, the party that represents the threatened class of knowledge workers, in exchange for the regulatory capture the labs are after.
They've been raising the issue bi-monthly through mini-scandals that have until now been consistently slapped down by Jensen Huang, but it seems an alliance with Democrats, just prior to an election where they're poised to take power in the Senate and the House, might finally be how they crack their "problem."
It won't be long before a massive incident is blamed on an open model in the wild, not from inside the labs, and none of us will be able to leverage open models to run private business workflows for the cost of electricity and hardware.
It's worth noting as well that if the Democrats imagine they'll get a "slow down" to protect one of their main constituencies, the professional class of credentialed knowledge workers (or however you slice it), they're dreaming--the labs have stated openly again and again that their business model is to capture the 10T TAM that represents the sum of wages of that very class of workers.
[1] https://www.youtube.com/watch?v=fjZ90V_JREk
This is one of the most lucid pieces of writing capturing the current state of play I’ve read. Who is the author?
This is an AI written post and the details are wrong (the description of the HF incident as involving Irregular is wrong and the description of the incident as only involving stealing public credentials is wrong, per the technical report the agents got access to internal HF infrastructure).
Please see below, one detail was incorrect and has been acknowledged and amended.
Your description of the HF attack as being merely "the elite task of discovering 14 Hugging Face API tokens that careless developers had committed to public GitHub repositories" does not match the description in the technical report[1].
[1] https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78... - page 9
The description reads "the elite task of discovering 14 Hugging Face API tokens that careless developers had committed to public GitHub repositories, and used them to try to get benchmark solutions from directly from Hugging Face by applying a template injection flaw that’s been known about since 2015[1]."
Chaining a public token to an 11-year-old Jinja2 template injection vuln shouldn't be dressed up as an unprecedented "alien intellect" that threatens human civilisation. (And HuggingFace should take some flack for having such a dated vulnerability exposed - if your Bank was compromised in this way, you'd be blaming your bank, not the attacker.)
One correction is fair though, the 14 tokens were in a public Hugging Face dataset not a public GitHub repository. I've updated the post to reflect that.
[1] https://blackhat.com/docs/us-15/materials/us-15-Kettle-Serve...
"Every single one of these catastrophic breakouts happened inside the testing environments of the exact same vendor."
This is incorrect, the HF incident for example (the most well known) had nothing to do with irregular. I know there has been a news site pushing inaccurate articles (effort.news) on this topic but these are the facts.
https://openai.com/index/hugging-face-incident-and-the-road-...
Thank you, you're correct. Effort.news was one of my research sources, but you're right that although OpenAI use Irregular, they were not involved in the specific HF incident (although the failure mode was otherwise identical). I've updated the post to make that clear.
As of writing it still says:
> For Anthropic, Google, and Meta, the catastrophic breakouts happened inside the testing environments of the exact same contractor.
If this is the level of understanding you have of the relevant incidents, there's a lot of chutzpah in saying that other people are "selling garbage", carrying out an "extraordinary confidence trick", etc.
The failure mode was _not_ identical. The HF incident agents were not directly connected to the internet and had to compromise an internal package registry in order to access the internet.
thank you for this reasonable reaction
Politicians aren’t “gullible”. They know the game.
Agreed, but they're getting paid with our money.
Let them cry, I don't see anything changing unless they can somehow get China to agree. And I doubt China will drink any of that kool aid especially while they're being disadvantaged by export controls, so the ever-improving open weight models will continue to rain. This is something the US Big Tech oligarchs will NOT win.
It isn't about China. The Big AI wants regulatory relaxation. In particular anti-cartel relaxation. "Threepenny novel" explains it very well.
Currently we have a classic market race - the market forces both companies to provide more and more capable models with more and more value for the money for the customers while the suppliers (Nvidia, Micron, Dell) squeeze them from the other side. The end result would be one winner taking it all (and that is a very humongous "all") while the other falling into a very distant second position at best. Do their leadership and investors (bonus points - look at some OpenAI investors), some already worth tens of billions on paper, like the prospects of that "50% chances of even more riches, 50% - bust" outcome? I'd think - no.
The outcome they would like is both companies divvying up that huge market while jointly raising prices and providing less capable models (ie. cheaper to train and run) while squeezing their suppliers Wallmart style. How to get there? By breaking the anti-cartel limitations.
The typical tools to break anti-cartel limitations is for example perception of national interests or perception of some imminent dire emergency.
Thus the "lets us collaborate or our AI will kill you all" scare campaign.
There is really is no excuse for Washington here. Even for the layperson+, it doesn't take much use of an LLM to figure out where they produce good and usable results and where it's of specious of value.
+ every layperson is an expert in 1 or more areas.
> every layperson is an expert in 1 or more areas.
This isn't even remotely true, unless you're willing to go as far as "being this person is an area of expertise".
I dont have to be an expert to recognize when something is said with authority and confidence. Any time a model says something with conviction, my instinct is to double check.
Now if they taught them to express uncertainty and speak in terms of probability, I might be more likely to be blindly fooled by some kind of uncertain conviction.
Neither of you is wrong? Every lived experience does produce a unique expertise, not in "being" that person, but navigating their experiences. It would be very unusual that all of someone's experiences are completely worthless.
I suppose you've never had a unique perspective on something? Maybe that reflects on your self esteem? How unfortunate.
I mean the military was buying dowsing rods as bomb detectors not that long ago so I don't have a ton of faith in them not being hoodwinked.
> + every layperson is an expert in 1 or more areas.
What do you think 70 years old career politicians are an expert in that would allow them to weigh whether a chatbot's answers are bullshit or not.
I thought the whole point of this discussion is that they don't have to care if it works. All that matters is that the majority believes it does, or at least doesn't make a fuss about the continued spending.
That's their expertise.
That is the entire business model of AI companies: selling garbage to people foolish enough to buy the hype.