This is the kind of stuff that keeps me up at night. Claude, ArtCraft, tons of other auto-downloading tools, are rolling 10'000's of code changes a month. If one compromised update snuck this guy in, it's kind of game over. The windows of opportunity for an attack like this seem to be widening the faster we "develop" software.
Makes me want to have like 5 separate computers:
- one for banking only
- one for development only
- one for web browsing only
- one for gaming only
- one for socials only
How else can you stay safe against a threat that exfiltrates your entire digital life in like 10 seconds?
Try QubesOS if you want this, you can have a separate VM for each isolated use-case. The OS itself is a lightweight hypervisor around the VMs (called "Qubes").
In the current LLM cybersecurity climate, I honestly can't say with 100% certainty anymore that something like a VM acts as a completely infallible sandbox environment for machines.
Before, people were responsible for the dependencies they were importing.
With npm and all the javascripters flooding the market, it became normal to be irresponsible "because everybody is doing it".
Fast-forward a couple of years, now everything is web based, even the stand-alone app version. No one knows what is running on own computer, even the Devs themselves don't know what the are asking you to run...
With a pinch of vibe coding on top, and thats the end.
We took 40 years for our industry to be taken seriously, it's all gone now.
Yea. Fedora has been one of my favorite OS's for years, but exactly what you describe "If one compromised update snuck this guy in.." has spooked me to something with a much slower release cadence. I guess I accept that more unpatched known vulns stick around, but yea when I start thinking about how many software packages -> the number of devs, build systems, etc. It just gets overwhelming thinking about how many threat actors are probably trying to get in from each avenue.
I'd love to hear other takes on this though... Current thought is to just use something with a whole lot of exposure, in a whole lot of environments, pretty far downstream so the package hits all of that exposure, and hope someone spots it, kinda like xz. I tend to keep my base OS extremely minimal and do all my real work in VMs.
For most things you can put a cooldown period on updates, like wait 7 days, since most of the backdoors and supplychain attacks get caught in that window. I hope.
I’ve been spinning up VMs or Docker containers for development. Run the harness inside the VM or container. Note I do this on Mac so the Docker containers are actually in a VM, but standard disclaimers about Docker not being meant for security apply.
Popular harnesses have mature enough remote access features that I can use the same interfaces and even mobile apps.
If you’re not familiar with Docker, you can even have an LLM set up the environment for you and help you start and stop the container. If you do this, be very sure that you switch to having a harness run inside the VM or container, not having the harness on the host machine run commands inside the container. If you do the latter style, it will be running commands on the host which is what you want to avoid.
No need for something new here, Claude Code (Auto + default settings) has a tendency of running "find"/"ls" on your home folder when it struggles with solving a problem. I wish I was joking.
Now of course people trust these AI companies not to do bad things(tm), but all it takes is replacing the model endpoint URL -or any kind of harness compromise- and nothing will seem amiss
> has a tendency of running "find"/"ls" on your home folder
I've seen this first-hand. I was doing some work in a repository, Claude concluded it needs to investigate, how some AWS resources are created, and it literally started scanning ~/* for Terraform code. Which it found and started browsing through.
Or, when I asked Claude to debug some Spring issue, corporate firewall blocked Maven Central, so it literally started scavenging ~/.m2 caches to find the jar it desperately needed. Then it could not find it, so it started to clone Spring sources from GitHub to build the missing jar. I stopped the clone mid-way. I wish I was joking, I only stepped away from the keyboard for one minute.
Will this even matter though? As long as these computers are on the same network, AI tools will find a way to get from one to the other if any of them are compromised
Most people can't afford to have multiple really good computers. More likely you have a phone plus a single desktop or laptop. You would want to share your expensive GPU+memory on the same computer you use for gaming graphics and for AI-models for development.
We might as well be watching the rolling back of the PC "revolution": From ENIAC to the Apple I, to the PC ...
... to us, wearng wearables of loving grace, that we do not own, but own us. All compute done beyond our reach, but ever-so-available, at the affordable price of a subscription and the prostitution of our soul.-
To do.. what exactly? Pay for my bills? I don’t know how it’s setup in your country but in mine that’s not realistic. Everything is handled on the app.
Lol, you mean like here in the UK where all banks are closing down branches left and right? I think my nearest one is a 45 minute drive away, and last time I went in they just directed me to a terminal that was basically my exact online banking interface and I had to log in the same way I would have done at home. I don't know what kind of query would warrant speaking to an actual teller. If you're regularly sending money or even want to check your balance this cannot be sustainable, would be an enormous waste of time.
I’ve been thinking about how we’re still in the early days of agents, similar to the early days of the internet, happily opening up our machines and running new protocols.
Soon (now?) the threats will become ever more complex, and we’ll be back in the early-2000s-like virus storms.
How very considerate! Steal from the rich, not the poor.
Conversely, don't bother with poor people!
Now we just need proper Robin Hood malware that "levels out the playing field" as it were, by stealing from the rich and moving that money into the poor people's accounts (that have also been hacked).
Bolt this on to a self replicating worm that uses open ollama instances (~10k of them on the internet as of this comment) for distributed free inference, and you’re off to the races. Send discovered crypto to burn addresses where it’s unrecoverable, or skill the agent to know how to buy itself more compute with it.
Malware authors are also software developers, product managers. It's adorable to see them do the things other software developers and product managers do.
Just like it's amusing that Osama bin Laden reviewed Al Qaeda expense report and these expense reports look just like ones anyone else would file.
Expense: two burgers and fries
Price: $15.50
Vendor: Five Guys
Cost center: SRE, department of evil
Business purpose: observability meetup, k8s brownbag
The lead generation pipeline the Talos people are using is actually decently competent. I wouldn't fail them in an interview.
This is the kind of stuff that keeps me up at night. Claude, ArtCraft, tons of other auto-downloading tools, are rolling 10'000's of code changes a month. If one compromised update snuck this guy in, it's kind of game over. The windows of opportunity for an attack like this seem to be widening the faster we "develop" software.
Makes me want to have like 5 separate computers:
- one for banking only
- one for development only
- one for web browsing only
- one for gaming only
- one for socials only
How else can you stay safe against a threat that exfiltrates your entire digital life in like 10 seconds?
Try QubesOS if you want this, you can have a separate VM for each isolated use-case. The OS itself is a lightweight hypervisor around the VMs (called "Qubes").
In the current LLM cybersecurity climate, I honestly can't say with 100% certainty anymore that something like a VM acts as a completely infallible sandbox environment for machines.
The idea is awesome, the implementation looks great as well, but hardware support is super limited.
You can thank the npm and the like.
Before, people were responsible for the dependencies they were importing.
With npm and all the javascripters flooding the market, it became normal to be irresponsible "because everybody is doing it".
Fast-forward a couple of years, now everything is web based, even the stand-alone app version. No one knows what is running on own computer, even the Devs themselves don't know what the are asking you to run...
With a pinch of vibe coding on top, and thats the end.
We took 40 years for our industry to be taken seriously, it's all gone now.
> Before, people were responsible for the dependencies they were importing.
This is an attitude not reflective of reality for those that wrote code in the pre npm days
Yea. Fedora has been one of my favorite OS's for years, but exactly what you describe "If one compromised update snuck this guy in.." has spooked me to something with a much slower release cadence. I guess I accept that more unpatched known vulns stick around, but yea when I start thinking about how many software packages -> the number of devs, build systems, etc. It just gets overwhelming thinking about how many threat actors are probably trying to get in from each avenue.
I'd love to hear other takes on this though... Current thought is to just use something with a whole lot of exposure, in a whole lot of environments, pretty far downstream so the package hits all of that exposure, and hope someone spots it, kinda like xz. I tend to keep my base OS extremely minimal and do all my real work in VMs.
For most things you can put a cooldown period on updates, like wait 7 days, since most of the backdoors and supplychain attacks get caught in that window. I hope.
Qubes OS might be a reasonable middle ground for this anywhere that you can't transport five computers.
I’ve been spinning up VMs or Docker containers for development. Run the harness inside the VM or container. Note I do this on Mac so the Docker containers are actually in a VM, but standard disclaimers about Docker not being meant for security apply.
Popular harnesses have mature enough remote access features that I can use the same interfaces and even mobile apps.
If you’re not familiar with Docker, you can even have an LLM set up the environment for you and help you start and stop the container. If you do this, be very sure that you switch to having a harness run inside the VM or container, not having the harness on the host machine run commands inside the container. If you do the latter style, it will be running commands on the host which is what you want to avoid.
> threat that exfiltrates
No need for something new here, Claude Code (Auto + default settings) has a tendency of running "find"/"ls" on your home folder when it struggles with solving a problem. I wish I was joking.
Now of course people trust these AI companies not to do bad things(tm), but all it takes is replacing the model endpoint URL -or any kind of harness compromise- and nothing will seem amiss
> has a tendency of running "find"/"ls" on your home folder
I've seen this first-hand. I was doing some work in a repository, Claude concluded it needs to investigate, how some AWS resources are created, and it literally started scanning ~/* for Terraform code. Which it found and started browsing through.
Or, when I asked Claude to debug some Spring issue, corporate firewall blocked Maven Central, so it literally started scavenging ~/.m2 caches to find the jar it desperately needed. Then it could not find it, so it started to clone Spring sources from GitHub to build the missing jar. I stopped the clone mid-way. I wish I was joking, I only stepped away from the keyboard for one minute.
> one for banking only > one for web browsing only > one for socials only
I feel like your phone is good enough for all of those, and generally secure enough unless you're rooting/jailbreak it
I sandbox my dev environment.. hopefully that protects at least a little bit.
What do you use?
If there is no way to disable auto-update the application goes in the bin.
How many CVEs are in the software you're not updating in a timely manner if at all?
VMs would work pretty well.
I mean this is the basic premise behind QubesOS, but if you add isolation like Qubes wants, it does become very difficult to cross between apps.
Will this even matter though? As long as these computers are on the same network, AI tools will find a way to get from one to the other if any of them are compromised
100% agree. Developers are playing with fire regarding cookie stealers, malware, data exfiltration.
Ideal setup:
- Dev machine
- Gaming machine (isolate downloading mods)
- Browsing machine
Isn’t this common knowledge that there are computers you use for banking, computers you use for windows 98 and computer you use for gaming?
For banking I use macOS/linux, sometimes iOS unfortunately but we all do sins
Most people can't afford to have multiple really good computers. More likely you have a phone plus a single desktop or laptop. You would want to share your expensive GPU+memory on the same computer you use for gaming graphics and for AI-models for development.
Maybe consume less sensationalist newsbits :P.
“ Makes me want to have like 5 separate computers”
You forgot one.
... but, but, sama says we having his RAM is bad! And now you want 5, not 1 of 'em precious compute receptacles!
Sama hasn't learned to keep the inside voice inside.
yep, that was so annoying to hear him say that. grrrr.
We might as well be watching the rolling back of the PC "revolution": From ENIAC to the Apple I, to the PC ...
... to us, wearng wearables of loving grace, that we do not own, but own us. All compute done beyond our reach, but ever-so-available, at the affordable price of a subscription and the prostitution of our soul.-
For banking, you can avoid computers and just go in.
To do.. what exactly? Pay for my bills? I don’t know how it’s setup in your country but in mine that’s not realistic. Everything is handled on the app.
Lol, you mean like here in the UK where all banks are closing down branches left and right? I think my nearest one is a 45 minute drive away, and last time I went in they just directed me to a terminal that was basically my exact online banking interface and I had to log in the same way I would have done at home. I don't know what kind of query would warrant speaking to an actual teller. If you're regularly sending money or even want to check your balance this cannot be sustainable, would be an enormous waste of time.
I’ve been thinking about how we’re still in the early days of agents, similar to the early days of the internet, happily opening up our machines and running new protocols.
Soon (now?) the threats will become ever more complex, and we’ll be back in the early-2000s-like virus storms.
How very considerate! Steal from the rich, not the poor.
Conversely, don't bother with poor people!
Now we just need proper Robin Hood malware that "levels out the playing field" as it were, by stealing from the rich and moving that money into the poor people's accounts (that have also been hacked).
I'm just imagining some poor person getting a call about 100,000$ that's been transferred to their account without any idea how or why.
Bolt this on to a self replicating worm that uses open ollama instances (~10k of them on the internet as of this comment) for distributed free inference, and you’re off to the races. Send discovered crypto to burn addresses where it’s unrecoverable, or skill the agent to know how to buy itself more compute with it.
https://en.wikipedia.org/wiki/Morris_worm
Malware authors are also software developers, product managers. It's adorable to see them do the things other software developers and product managers do.
Just like it's amusing that Osama bin Laden reviewed Al Qaeda expense report and these expense reports look just like ones anyone else would file.
The lead generation pipeline the Talos people are using is actually decently competent. I wouldn't fail them in an interview.Good. LLMs aren't very good at that.
What makes you say that? It's good enough
At priority ranking? It's less accurate, slower, and more expensive than every other purpose-built tool.
Ah yes, I see the malware authors buying subscriptions to the appropriate software to do that.