Funny - I was just reading about the creator of Ethernet this morning (Robert Metcalfe):
In 1995, Metcalfe argued that the Internet would suffer a "catastrophic collapse" in the following year; he promised to eat his words if it did not. During his keynote speech at the sixth International World Wide Web Conference in 1997, he took a printed copy of his column that predicted the collapse, put it in a blender with some liquid and then consumed the pulpy mass.[26][27] He had suggested having his words printed on a very large cake, but the audience would not accept this form of "eating his words."[28]
“Sovereignty is the ability to resist power being exerted over you” is a pretty compelling definition. Thinking forward about what form that power takes with LLMs developing custom exploits faster than our existing processes can patch them - it seems to me that our current security best practices might turn out to be pretty useless or even an active risk in and of themselves.
Our responsible disclosure and CVE info cycle and patch cadences and defense in depth, etc, etc ecosystem are tuned around a world where there's a relatively small number of bad 0days and the cost of said 0days is reasonably high. We can evolve how we address threats, but our models have read all of the best literature about how it's been done previously. Think there's a gap for people too small to gain access to "the good security models" - the nerfed closed models and the open models seem likely to fail to adapt, unless that gap is addressed.
I watched my 75 year old human plumber (who sniped a guy his first day in vietnam), kneel on two 3 inch wide joists and drill a 4 inch diameter hole at a 89 degree angle through said joists. Where’s the robot that does all that?
Above from all the other doomer crap in this article, I find the job replacement rhetoric the most offensive. It is always people that are so detached from the reality of labor, people who’s only “job” has been to essentially have ideas and opinions.
"The way that Hugging Face defended itself was with an open source Chinese model, GLM, because they didn’t have access to the high cybersecurity models on the other side. They’re too dangerous."
This is not what happened. HuggingFace patched the vulnerabilities in the ordinary way, then used GLM for some of the forensic analysis afterwards. It's in the interest of HF to promote the GLM part of the story, because it emphasizes their agency in the situation as well as the open-source models they distribute. But GLM was a (useful) part of the postmortem, not of the defense.
I’ve already read so many sci-fi novels where the part of the plot was that human communication infrastructure became unusable due to AI shenanigans… not here to read another one, thank you very much.
I see no reason for the Internet to go offline, other than various political blocks compartmentalizing it. So: China, Russia, EU having their own network? Yeah, maybe. Super-AI nuking the Internet: no.
In fact, the more I use AI (handed sudo to Claude on my AI dev box just yesterday), the more I see it as a pretty good, and certainly tireless, sysadmin. Looking at what people do with AI by connecting it to undocumented hardware interfaces and getting working setups from this, the Internet will do fine.
The whole article is just a grab-bag of various AI fears without much grounding in what is happening. Maybe the summary doesn't do justice to what was said.
“If you say you’re Uyghur or from certain parts of China, it will write backdoors and hacks in the code.”
is so sensationalized to the point of leaving me genuinely flabbergasted as to how a rational person could have published this. If interested, I would recommend reading the actual study itself.
Indeed, you don't need to tell it who you are, it will already know to write the backdoors.
It may be sensationalized, but that doesn't mean it isn't based on legitimate fears based on actual real current and past actions of governments and corporations. They do spy on people. They do put in backdoors. They do want control over what you do and don't do so as to maintain the balance of power in the hands of those who currently wield it. None of these are fictions, they have all happened and are currently still happening, but with an AI accelerator/amplifier.
What does it even mean for "the Internet" to go offline? All nodes go off at the same time? All routers? BGP as a whole? All of that across the whole world?
These "visionaries" talking about these things hardly never have an actual grasp on how these things work in practice, which is why you'll see outlandish claims like this. What they mean in clear language is basically "Some things will change a bit", but that never sells clicks so "World will end as we know it" becomes the title.
> “Hugging Face got hacked by OpenAI agents that coordinated themselves, created a message board and broke out to the internet,” he says, before adding dryly: “This isn’t scary at all, right?
I don't understand how any serious person doesn't 100% put that on OpenAI being (intentionally?) sloppy with security and isolation for a test that for sure would have required much better safeguarding than what they should have expected, as it wasn't even the first time.
It's not scary because "LLMs in a harness managed to hack other company", it's scary because somehow police hasn't yet raided OpenAI's offices to gather proof about how fucking reckless they were about those tests, putting the public at risk by doing those tests in those manners, without proper safeguards.
Edit: Continued reading, but now I regret spending even the slightest amount of time on it:
> Then there is the question of trusting the models themselves. Mostaque cites an analysis of a Chinese open-weight model: “If you say you’re Uyghur or from certain parts of China, it will write backdoors and hacks in the code.”
What kind of conspiracy theories is this? Is there any sort of proof and evidence about this what so ever?
Funny - I was just reading about the creator of Ethernet this morning (Robert Metcalfe):
In 1995, Metcalfe argued that the Internet would suffer a "catastrophic collapse" in the following year; he promised to eat his words if it did not. During his keynote speech at the sixth International World Wide Web Conference in 1997, he took a printed copy of his column that predicted the collapse, put it in a blender with some liquid and then consumed the pulpy mass.[26][27] He had suggested having his words printed on a very large cake, but the audience would not accept this form of "eating his words."[28]
“Sovereignty is the ability to resist power being exerted over you” is a pretty compelling definition. Thinking forward about what form that power takes with LLMs developing custom exploits faster than our existing processes can patch them - it seems to me that our current security best practices might turn out to be pretty useless or even an active risk in and of themselves.
Our responsible disclosure and CVE info cycle and patch cadences and defense in depth, etc, etc ecosystem are tuned around a world where there's a relatively small number of bad 0days and the cost of said 0days is reasonably high. We can evolve how we address threats, but our models have read all of the best literature about how it's been done previously. Think there's a gap for people too small to gain access to "the good security models" - the nerfed closed models and the open models seem likely to fail to adapt, unless that gap is addressed.
I watched my 75 year old human plumber (who sniped a guy his first day in vietnam), kneel on two 3 inch wide joists and drill a 4 inch diameter hole at a 89 degree angle through said joists. Where’s the robot that does all that?
Above from all the other doomer crap in this article, I find the job replacement rhetoric the most offensive. It is always people that are so detached from the reality of labor, people who’s only “job” has been to essentially have ideas and opinions.
Yeah but the accountant, the product manager, the graphics designer, the engineer, the doctor, the lawyer are fucked.
Good thing we have plumbers. But who’s gonna have money to pay a plumber if nobody has jobs?
"The way that Hugging Face defended itself was with an open source Chinese model, GLM, because they didn’t have access to the high cybersecurity models on the other side. They’re too dangerous."
This is not what happened. HuggingFace patched the vulnerabilities in the ordinary way, then used GLM for some of the forensic analysis afterwards. It's in the interest of HF to promote the GLM part of the story, because it emphasizes their agency in the situation as well as the open-source models they distribute. But GLM was a (useful) part of the postmortem, not of the defense.
Another AI salesman selling AI, nothing new here
I’ve already read so many sci-fi novels where the part of the plot was that human communication infrastructure became unusable due to AI shenanigans… not here to read another one, thank you very much.
I see no reason for the Internet to go offline, other than various political blocks compartmentalizing it. So: China, Russia, EU having their own network? Yeah, maybe. Super-AI nuking the Internet: no.
In fact, the more I use AI (handed sudo to Claude on my AI dev box just yesterday), the more I see it as a pretty good, and certainly tireless, sysadmin. Looking at what people do with AI by connecting it to undocumented hardware interfaces and getting working setups from this, the Internet will do fine.
The whole article is just a grab-bag of various AI fears without much grounding in what is happening. Maybe the summary doesn't do justice to what was said.
Well, he is right about one thing; we are not prepared for what’s coming.
“If you say you’re Uyghur or from certain parts of China, it will write backdoors and hacks in the code.”
is so sensationalized to the point of leaving me genuinely flabbergasted as to how a rational person could have published this. If interested, I would recommend reading the actual study itself.
Indeed, you don't need to tell it who you are, it will already know to write the backdoors.
It may be sensationalized, but that doesn't mean it isn't based on legitimate fears based on actual real current and past actions of governments and corporations. They do spy on people. They do put in backdoors. They do want control over what you do and don't do so as to maintain the balance of power in the hands of those who currently wield it. None of these are fictions, they have all happened and are currently still happening, but with an AI accelerator/amplifier.
What does it even mean for "the Internet" to go offline? All nodes go off at the same time? All routers? BGP as a whole? All of that across the whole world?
These "visionaries" talking about these things hardly never have an actual grasp on how these things work in practice, which is why you'll see outlandish claims like this. What they mean in clear language is basically "Some things will change a bit", but that never sells clicks so "World will end as we know it" becomes the title.
It's always the DNS (
Cloudflare goes down
The more satellites become the backbone access the simpler loss of access becomes. Physical connections will be more resilient.
> “Hugging Face got hacked by OpenAI agents that coordinated themselves, created a message board and broke out to the internet,” he says, before adding dryly: “This isn’t scary at all, right?
I don't understand how any serious person doesn't 100% put that on OpenAI being (intentionally?) sloppy with security and isolation for a test that for sure would have required much better safeguarding than what they should have expected, as it wasn't even the first time.
It's not scary because "LLMs in a harness managed to hack other company", it's scary because somehow police hasn't yet raided OpenAI's offices to gather proof about how fucking reckless they were about those tests, putting the public at risk by doing those tests in those manners, without proper safeguards.
Edit: Continued reading, but now I regret spending even the slightest amount of time on it:
> Then there is the question of trusting the models themselves. Mostaque cites an analysis of a Chinese open-weight model: “If you say you’re Uyghur or from certain parts of China, it will write backdoors and hacks in the code.”
What kind of conspiracy theories is this? Is there any sort of proof and evidence about this what so ever?
The usual daily dose of fearmongering we get nowadays I guess..
You will eat your fearslop and you will like it.
It's all fun and games until Skynet becomes self-aware and junk gets real.
For the more religious, "Skynet"=="The Beast" of Revelation.
The sweet spot is somewhere between Pollyanna and Chicken Little.