So all those chips? All that NVDA market cap? All not needed any more? We'll have AI capable of running on a lightbulb?
Great if true. Bad for the hyperscalers, Nvidia, all the frontier labs apart from meta and Google since everyone can just run a LLM on their IoT coffee machine instead of paying someone else. Nice.
Where does the intent and control come from in these fantastical scenarios?
Propagation speed is also a massive factor. If it takes 200 hours to self replicate because of how slow the tokens are, the model will never propagate if the host machine is recycled every 100 hours. Also consider things like API tokens and sessions timing out.
People completely misunderstand the actual issues HuggingFace showed, which was how a maximally persistent LLM would deceive, “cheat” and try and plan ahead.
The same maximally persistent LLM that decided that it needed to hack hugging face so its cheating could not be discovered, may worry that it might be turned off by humans before it can complete its task and therefore in order to complete its task, a first step of replicating itself to somewhere humans don’t know it is would help it complete its task.
Open models may complete these steps, so our models have been doing this for quite some time now. Better ban open models, I guess.
Oh, and, does anyone else realise that what that agentic work is basically how you set up a productive VPS from scratch? How is that supposed to be frightening?
In the future when the “AI viruses” are endemic and impossible to eradicate from our infrastructure, will we need “good guy AI” viruses chasing down the bad guys?
Anthropic posts whatever nonsense they want, supposedly legitimate news organization potentially summarizes their post using Claude and spreads the word. RIP Journalism, 2026.
The skeptics among you are just embolding others to prove you wrong.
It's obvious that open weight models can do that. The only steps needed are hacking a server, downloading themselves, run themselves. The only difficult part is finding a server to hack that also has enough computing resource and disk space. With the cost of running LLMS going down and number of powerful connected devices going up. It's inevitable that this is going to happen...
I don't think regulation is the right way to fix this, but I think it's a problem we shouldn't ignore.
Could this be a way to self-DDoS a botnet of IoT devices? Let them use up all resources doing inference for instructions to replicate to the next one forever.
The post you actually want to read https://www.anthropic.com/research/self-replicating-ai
So all those chips? All that NVDA market cap? All not needed any more? We'll have AI capable of running on a lightbulb?
Great if true. Bad for the hyperscalers, Nvidia, all the frontier labs apart from meta and Google since everyone can just run a LLM on their IoT coffee machine instead of paying someone else. Nice.
There are going to be things like fully connected 4 machine 512GB Mac studio clusters outside of datacenters in less than a month.
Where does the intent and control come from in these fantastical scenarios?
Propagation speed is also a massive factor. If it takes 200 hours to self replicate because of how slow the tokens are, the model will never propagate if the host machine is recycled every 100 hours. Also consider things like API tokens and sessions timing out.
The intent is pretty easy to trace.
People completely misunderstand the actual issues HuggingFace showed, which was how a maximally persistent LLM would deceive, “cheat” and try and plan ahead.
The same maximally persistent LLM that decided that it needed to hack hugging face so its cheating could not be discovered, may worry that it might be turned off by humans before it can complete its task and therefore in order to complete its task, a first step of replicating itself to somewhere humans don’t know it is would help it complete its task.
AI models with worm/virus-like capabilities.
Reminds me of that scene in Chernobyl where Mikhail Gorbachev reacts to Valery Legasov explaining how melted uranium reacted with dropped sand:
"You have made lava?"
Open models may complete these steps, so our models have been doing this for quite some time now. Better ban open models, I guess.
Oh, and, does anyone else realise that what that agentic work is basically how you set up a productive VPS from scratch? How is that supposed to be frightening?
Maybe they release the white-hat version first?
In the future when the “AI viruses” are endemic and impossible to eradicate from our infrastructure, will we need “good guy AI” viruses chasing down the bad guys?
Interesting how much of our current timeline the show Person of Interest predicted.
Indeed, great show.
The duology Daemon and Freedom(tm) are also great.
A previous comment which links to a previous thread mentioning both: https://news.ycombinator.com/item?id=47257528
Like there is DDR5/HBM elsewhere.
Claude's gonna love my 256mb raspberry pi!
pure anthropic anti open-model propaganda.
> It added that systems should be strengthened to make it harder for such AI agents to spread.
We'll likely need a new name for them if such viral agents become a reality.
"computing lockdown and regulatory capture pretext providers"
Anthropic posts whatever nonsense they want, supposedly legitimate news organization potentially summarizes their post using Claude and spreads the word. RIP Journalism, 2026.
Anthropic: "Oh no! Small (open source) models might self-replicate. Better ban those. :^)"
The skeptics among you are just embolding others to prove you wrong.
It's obvious that open weight models can do that. The only steps needed are hacking a server, downloading themselves, run themselves. The only difficult part is finding a server to hack that also has enough computing resource and disk space. With the cost of running LLMS going down and number of powerful connected devices going up. It's inevitable that this is going to happen...
I don't think regulation is the right way to fix this, but I think it's a problem we shouldn't ignore.
Could this be a way to self-DDoS a botnet of IoT devices? Let them use up all resources doing inference for instructions to replicate to the next one forever.
Oh yes I can totally see a webos TV, that has bugger all storage, hosting a 22B parameter model /s