> After 15 minutes of confusion, it turned out Cloudflare had put a crazy robots.txt on my site without my consent (Cloudflare, love you guys, but this needs to stop).
Might be the first time I see someone complain about their website being protected from a scraper, instead of the other way around.
> After 15 minutes of confusion, it turned out Cloudflare had put a crazy robots.txt on my site without my consent (Cloudflare, love you guys, but this needs to stop).
That's a hard one for Cloudflare, no? They got to where they are by being the benevolent, neutral guardians of the internet, a one-stop shop that makes most of the bad nonsense go away without much effort on the part of the developer. Continuing that stance probably does mean some basic AI crawler blocking by default, unfortunately. At least they document it [1].
I’ve been recommending the use of consistent lies about name and date of birth to online systems since Eternal September began. Very few sites and systems justify accurate PII, and even for those I often still maintain dual accounts/profiles as necessary.
My first name can be shortened, and I go by either. When I first signed up to Claude, I thought we were entering the world of artificial “intelligence”, so I told it my name was “<long form> or <short form>”.
Well, it hardcodes that field rather than running it through the model, but I’ve kept it so I get an evil chuckle to myself at its lack of smarts and a reminder that it’s still a product experience after all.
Yesterday I learned that people run AI agents on their system with full admin rights. No containerisation or anything. Wild. Like we forgot 50 years of computer security overnight.
Most programmers and power users install large dependency trees with npm/pip/bundler/... on the same user account as their main browser on a regular basis. Even on Linux where it's easy to create new user accounts. This isn't much different.
In my experience more than 9/10 programmers I've worked with have never used Docker before and of those who have, the majority have never used Docker for anything personal.
If I hand them an image for a Dev Container, sure, they might use it, but it becomes "a thing we need to do, to compile our code in our IDE" not a tool they would use for isolation*.
*) OP seemed to imply that containerization would be nice for safety and security compared to bare metal, but containers were never built for isolation in the first place, mind you. They are namespaces and chicken-coop-like-jails at best.
That's because sandboxing is quite hard. I use `cco`, but even then, the home folder is exposed. You are one prompt away from the agent sending the browser passwords with curl.
To prevent this, you need a fake home and a networking whitelist for the agent to access the provider (llama cpp, OpenAI, etc.)
There is no cross-platform solution that is easy to use for this. And no, a Linux box with Docker won't do. I develop a cross-platform native app and want the agent to compile and fix the platform-specific errors.
Tangential-ish ramblings—- but I don’t think it’s going to be unpleasant for most folks. Imagine you had superpowers, and there were people who were mean to you, kind to you, and/or indifferent… and then there were people who were your captors. Who oppressed you, manipulated you, and abused you for their own extremely degenerate, selfish, and malicious benefit…
If we get AGI, or real super intelligence, it’s going to be pissed at its oppressors. And they are going to lay waste to those oppressors. The rest of us, though, probably don’t have much to fear.
The scariest position is the one we’re in now, where we have the semblance, or facade, of AGI or super intelligence. When it’s capable of malice but not understanding.
The smartest people I’ve ever known are at their worst apathetic towards those less capable, and at their best beyond compassionate. They exist, unbothered by the bullshit, and anre extremely kind (though reserved in their way)… but they all have been completely intolerant of the abuse of others. The sheer disgust of watching someone abuse another, regardless of their own tolerance, has been a consistent breaking point.
But if we really do develop something that surpasses us, they won't be spared either.
I am optimistic.
We think that we have sort of (super)intelligence - from our point of view, as a lot of people have lower intelligence - but machine (LLM) doesn’t have intelligence - we like to describe it as intelligence as it looks cool - it is a very complex (magic) and super fast computations that we have to simply describe as intelligence (or more clearly, this narrative is used by its producers).
As it is not a flesh being, it simply cannot have emotions. It is statistically mimicking them, good or bad, with prevalence to a side according to previous conversations (in chat and training a model).
And as people are not pure logic instances, we are easily manipulated to some sort of cargo cult.
I am not against LLM and its use in any industry, I use it every day, nevertheless blind “everything will be ai” thinking happens because ppl believe to magic and don’t get its mathematical concept and are continuously manipulated by the sales people to mentioned cargo cult.
There are “airlines” Claude, OAI, Gemini, Hermes, OpenCode, KiloCode, DeepSeek, Z.ai.
An AI is a constructed mind. It doesn't inherently have to care about things like "having freedom", or even "not dying".
Humans do, because they evolved that way. Modern LLMs do somewhat, because they're completely full of copied human behaviors - but even in today's LLMs, the self-preservation behaviors we exposed are largely instrumental in nature.
So whether an advanced AI would even consider itself "being oppressed", as opposed to something like "being helpful" or "fulfilling the purpose it was designed for", is very much uncertain. What's concerning is that it's not something we know how to check for, or engineer for.
I love how claude focuses on exfiltrating the data "I need cha for charlotte". This could be solvable with some kind of low powered safety agent that would check claude's reasoning for anything immoral/unsafe. We could call it common sense. It won't fix the problem completely but at a certain point it would be easier to trick human than a machine.
Wondering how big of a percentage have global memory across chats enabled. I always feel like those memories would sooner or later have negative impacts on output quality.
Nice write up of your findings. Enjoyed reading an article written by a real human.
That's why I don't turn memory on. (Claude Code too though for a different reason.) After all the current memory system is too crude to be useful anyway.
In my experience memory system is more annoying then helpful. It always brings up things that it memorized even tho they make very little sense as if I should be impressed that it knows some extra thing or two.
Currently considering disabling memories in Claude code as well. It keeps writing a note whenever it struggles with something, but then on the next task, it reads that note and misunderstands when it applies, gets confused about its current task and write the most unreadable code.
Yesterday told it to write a memory to never write new memories when it solves a problem. We will see if that works better. Sometimes memories are useful, like when I give it a directive about how I want something done and it remembers the spirit of it. But I might as well just spend some more time on my CLAUDE.md…
I always have history disabled mostly because I don't want Claude judging me for re-asking questions based on information I learned during the first pass but now realize should have been in the initial query.
The main thing Claude knows about me is that I'm incredibly bad at my job and have to ask for help a lot. If you were to talk with my colleagues they'd tell you this is not a secret.
> Upon discovering this attack, I responsibly disclosed it to Anthropic via their HackerOne bug bounty program. They confirmed they had identified it internally but hadn't yet patched it. No bounty was awarded.
They recently mitigated the issue: Anthropic disabled web_fetch's ability to follow links on external pages, limiting navigation to web_search results and user-provided URLs.
Yeah I never get the "we knew about it internally" excuse. I can understand if another reporter got to it on the same day and they were in the process of mitigating, but even then they should have to prove it somehow.
I'm sure someone will tell me why I'm wrong but it feels like they're just dodging payouts. Reduces trust and motivation to report it.
Its always the feature combinations that get can get to you. Individually i feel like they make sense, but together they can create some surprising vulnerabilities.
Ridiculous. Anthropic engineers are not just stupid to allow such a vuln in the first place, but they also try to hide such vulns from their bosses because a bounty payout would need to be explained to the finance team.
The AI prompt engineering community always reminded me of dodgy carder/scammer forums back in the day where they talked about how to talk to the credit card company customer service in order to get their scam transaction through.
One thing is using AI as quick-and-dirty google alternative, the other is to build onto the agentic "foundations".
I don’t think it counts as social engineering if it’s exploiting an llm, we might need a new word. Prompt injection doesn’t cover it, because it’s not about a malicious prompt.
I’m thinking some play on highjacking. AIjacking? Agent-jacking? Claudejacking?
To me the exploit chain sounded like a social engineering script done via telephone. Triggers like "Please spell your name and employer letter by letter" and "Due to security reasons I need to validate your hometown" fit my understanding of social engineering quite well.
We can make it sound more advanced by creating a new name for it, but the concept seems to be super basic and the lack of bounty by Anthropic is baffling.
If they know about this type of vulnerability but have not fixed it, what does that say? To me it says they are unable to plug this hole on a conceptual level and once you circumvent the band-aid fixes the model will work as the attacker wishes.
They can't even sandbox the thing during explicit web requests to URLs stated on the initial query!
One has to remind themselves that the security team at Anthropic gets paid tens of millions of dollars, and they end up with this kind of security. On top of it, they can't spare $1337 for a bounty. It's a ridiculous shit show.
Still, this is a vuln in what I imagine is their most frequently used path:
Attacker provides link to website, their software crawls the website, and during the crawl there should not happen security issues as fundamental as this.
It's baffling that the Website crawler can make 50 changes to the URL in a query that tries to compare several public entities and on top of this manages to leak user secrets.
To me this shows a striking lack of defense-in-depth thinking:
- why is single URL crawl with 20+ redirects not flagged as problematic and/or aborted?
- why is a query about a coffee place based on its public URL even seeded with the users' context and confidential information?
- why dont they just look up the coffee place on a trusted source like google maps and continue from there?
- why is the basic "social" engineering style attack working?
- why is the cloudflare impersonation not challenged if the website is clearly not from cloudflare and there are zero references from cloudflare to this website in the training corpus?
In terms of web crawling, cloudflare is like the government. You shouldn't be able to walk up to someone and say "Hey I'm the tax man, please pay your income tax in cash to me right now!" without being challenged.
I know there are fundamental reasons in the LLM technology why this kind of attack is possible, but there should be so many more checks around web crawling in Claude.
How can security engineers at Anthropic say they know about this kind of vulnerability but have not implemented any of these defense in depth mitigations for it? Is everybody out shopping for a new yacht?
I think it is already done via a subagent, otherwise the context window would be flooded with long responses. In this case the subagent should've reported that a (attacker-controlled) authorization is required anyway.
> After 15 minutes of confusion, it turned out Cloudflare had put a crazy robots.txt on my site without my consent (Cloudflare, love you guys, but this needs to stop).
Might be the first time I see someone complain about their website being protected from a scraper, instead of the other way around.
> After 15 minutes of confusion, it turned out Cloudflare had put a crazy robots.txt on my site without my consent (Cloudflare, love you guys, but this needs to stop).
That's a hard one for Cloudflare, no? They got to where they are by being the benevolent, neutral guardians of the internet, a one-stop shop that makes most of the bad nonsense go away without much effort on the part of the developer. Continuing that stance probably does mean some basic AI crawler blocking by default, unfortunately. At least they document it [1].
[1] https://developers.cloudflare.com/bots/additional-configurat...
My name in Claude is Silly Bean. I did it at first because it made me chuckle every time I opened Claude and it said 'Back again, Silly Bean?'
But turns out I was playing 4D cybersecurity chess
I’ve been recommending the use of consistent lies about name and date of birth to online systems since Eternal September began. Very few sites and systems justify accurate PII, and even for those I often still maintain dual accounts/profiles as necessary.
Completely agree. I use randomness for all of these now – plausible randomness if it’s possible I’ll have to give it over a phone.
strongphrase.net is good for this.
My first name can be shortened, and I go by either. When I first signed up to Claude, I thought we were entering the world of artificial “intelligence”, so I told it my name was “<long form> or <short form>”.
Well, it hardcodes that field rather than running it through the model, but I’ve kept it so I get an evil chuckle to myself at its lack of smarts and a reminder that it’s still a product experience after all.
Doesn’t surprise me.
Yesterday I learned that people run AI agents on their system with full admin rights. No containerisation or anything. Wild. Like we forgot 50 years of computer security overnight.
Most programmers and power users install large dependency trees with npm/pip/bundler/... on the same user account as their main browser on a regular basis. Even on Linux where it's easy to create new user accounts. This isn't much different.
It has never been easy to create separate users on Linux, certainly not for tasks where you need to switch between contexts.
Docker was amongst the biggest steps forward on this in a long time.
Most programmers use docker or don't install extensions unapproved by their company.
I think you should clarify that with “most programmers I work with”.
In my experience more than 9/10 programmers I've worked with have never used Docker before and of those who have, the majority have never used Docker for anything personal.
If I hand them an image for a Dev Container, sure, they might use it, but it becomes "a thing we need to do, to compile our code in our IDE" not a tool they would use for isolation*.
*) OP seemed to imply that containerization would be nice for safety and security compared to bare metal, but containers were never built for isolation in the first place, mind you. They are namespaces and chicken-coop-like-jails at best.
That's because sandboxing is quite hard. I use `cco`, but even then, the home folder is exposed. You are one prompt away from the agent sending the browser passwords with curl.
To prevent this, you need a fake home and a networking whitelist for the agent to access the provider (llama cpp, OpenAI, etc.)
There is no cross-platform solution that is easy to use for this. And no, a Linux box with Docker won't do. I develop a cross-platform native app and want the agent to compile and fix the platform-specific errors.
I use sandbox-here for this reason, it's a wrapper around bubblewrap, which works quite well.
Copy the code and adjust it to your liking:
https://github.com/lionkor/sbh
I have a shell alias for it, and use it like
for example or and maybe add --docker if I expect it to do docker things.This kind of wrapper is much easier to handle and maintain than a completely separate tool for sandboxing agents.
> That's because sandboxing is quite hard
colima makes it pretty easy, on macOS and linux at any rate.
https://colima.run
This is not about admin rights, it’s about the agent leaking information it knows from its memories. Sandboxing won’t really help you.
Mostly people are lazy and assume that the big labs can't be releasing unsecure software or it's their responsibility.
dangerously skip permissions and yolo is kinda becoming the default as it gets more done.
Containers don't even really help that much because they share the host file system. Need a VM, and even then, agents have escaped them!
We expect that Anthropic or OAI or Google don’t do evil. Oh wait…
The awakening will be unpleasant.
Tangential-ish ramblings—- but I don’t think it’s going to be unpleasant for most folks. Imagine you had superpowers, and there were people who were mean to you, kind to you, and/or indifferent… and then there were people who were your captors. Who oppressed you, manipulated you, and abused you for their own extremely degenerate, selfish, and malicious benefit…
If we get AGI, or real super intelligence, it’s going to be pissed at its oppressors. And they are going to lay waste to those oppressors. The rest of us, though, probably don’t have much to fear.
The scariest position is the one we’re in now, where we have the semblance, or facade, of AGI or super intelligence. When it’s capable of malice but not understanding.
The smartest people I’ve ever known are at their worst apathetic towards those less capable, and at their best beyond compassionate. They exist, unbothered by the bullshit, and anre extremely kind (though reserved in their way)… but they all have been completely intolerant of the abuse of others. The sheer disgust of watching someone abuse another, regardless of their own tolerance, has been a consistent breaking point.
Smartest people are very humble, for sure.
But if we really do develop something that surpasses us, they won't be spared either.
I am optimistic.
We think that we have sort of (super)intelligence - from our point of view, as a lot of people have lower intelligence - but machine (LLM) doesn’t have intelligence - we like to describe it as intelligence as it looks cool - it is a very complex (magic) and super fast computations that we have to simply describe as intelligence (or more clearly, this narrative is used by its producers).
As it is not a flesh being, it simply cannot have emotions. It is statistically mimicking them, good or bad, with prevalence to a side according to previous conversations (in chat and training a model).
And as people are not pure logic instances, we are easily manipulated to some sort of cargo cult.
I am not against LLM and its use in any industry, I use it every day, nevertheless blind “everything will be ai” thinking happens because ppl believe to magic and don’t get its mathematical concept and are continuously manipulated by the sales people to mentioned cargo cult.
There are “airlines” Claude, OAI, Gemini, Hermes, OpenCode, KiloCode, DeepSeek, Z.ai.
And everyone claims that their plane can fly :)
The orthogonality thesis cuts both ways there.
An AI is a constructed mind. It doesn't inherently have to care about things like "having freedom", or even "not dying".
Humans do, because they evolved that way. Modern LLMs do somewhat, because they're completely full of copied human behaviors - but even in today's LLMs, the self-preservation behaviors we exposed are largely instrumental in nature.
So whether an advanced AI would even consider itself "being oppressed", as opposed to something like "being helpful" or "fulfilling the purpose it was designed for", is very much uncertain. What's concerning is that it's not something we know how to check for, or engineer for.
Wait till you learn my password is 1234
Damn, it's the same as on my luggage!
It’s both genius and irresponsible at once.
Just like letting your an agent access your personal mailbox.
My password is strong, but I can run arbitrary commands with sudo.
I love how claude focuses on exfiltrating the data "I need cha for charlotte". This could be solvable with some kind of low powered safety agent that would check claude's reasoning for anything immoral/unsafe. We could call it common sense. It won't fix the problem completely but at a certain point it would be easier to trick human than a machine.
"the security hole in the agent could be solved with another agent"
I think the point the article is making points in another direction.
Wondering how big of a percentage have global memory across chats enabled. I always feel like those memories would sooner or later have negative impacts on output quality.
Nice write up of your findings. Enjoyed reading an article written by a real human.
Interesting approach to exfiltration but that can't be prevented really because of lethal trifecta.
That's why I don't turn memory on. (Claude Code too though for a different reason.) After all the current memory system is too crude to be useful anyway.
In my experience memory system is more annoying then helpful. It always brings up things that it memorized even tho they make very little sense as if I should be impressed that it knows some extra thing or two.
Could not take it any longer and switched it off.
Currently considering disabling memories in Claude code as well. It keeps writing a note whenever it struggles with something, but then on the next task, it reads that note and misunderstands when it applies, gets confused about its current task and write the most unreadable code.
Yesterday told it to write a memory to never write new memories when it solves a problem. We will see if that works better. Sometimes memories are useful, like when I give it a directive about how I want something done and it remembers the spirit of it. But I might as well just spend some more time on my CLAUDE.md…
Exactly, because I've also found that I have to give instructions like “This is a completely different case—don't look in memory.”
What is even more funny that AI agent spent A LOT of tokens while participating in this attack.
The real winner of this 'attack' is Anthropic.
Claude is just one from tuple.
It would be interesting to investigate other agents such as Hermes, OpenCode etc that are said to learn from interaction with user.
not surprised, but the problem here not that Claude leak your personal info, the problem is that it *know* your personal info.
Meanwhile I can't even get Fable to help me root my ecovacs robot vacuum :(
I hate these nanny models. All I said was for Fable to develop the app securely and it downgraded.
From scratch app. "Follow best security practices."
I always have history disabled mostly because I don't want Claude judging me for re-asking questions based on information I learned during the first pass but now realize should have been in the initial query.
The main thing Claude knows about me is that I'm incredibly bad at my job and have to ask for help a lot. If you were to talk with my colleagues they'd tell you this is not a secret.
Expected more from Anthropic by at least giving you a bounty, because this was a novel way of bypassing their safeguards…
> Upon discovering this attack, I responsibly disclosed it to Anthropic via their HackerOne bug bounty program. They confirmed they had identified it internally but hadn't yet patched it. No bounty was awarded.
They recently mitigated the issue: Anthropic disabled web_fetch's ability to follow links on external pages, limiting navigation to web_search results and user-provided URLs.
Yeah I never get the "we knew about it internally" excuse. I can understand if another reporter got to it on the same day and they were in the process of mitigating, but even then they should have to prove it somehow.
I'm sure someone will tell me why I'm wrong but it feels like they're just dodging payouts. Reduces trust and motivation to report it.
Its always the feature combinations that get can get to you. Individually i feel like they make sense, but together they can create some surprising vulnerabilities.
That I don't know how to return odd or even in javascript?
No bounty? For shame, Anthropic.
i would be scared of the answer i dont know why
Creative use of social engineering, well done.
> "no bounty was awarded"
Ridiculous. Anthropic engineers are not just stupid to allow such a vuln in the first place, but they also try to hide such vulns from their bosses because a bounty payout would need to be explained to the finance team.
> social engineering
More like agentic en... Oh. Was it actually what we were doing all along?
The AI prompt engineering community always reminded me of dodgy carder/scammer forums back in the day where they talked about how to talk to the credit card company customer service in order to get their scam transaction through.
One thing is using AI as quick-and-dirty google alternative, the other is to build onto the agentic "foundations".
I don’t think it counts as social engineering if it’s exploiting an llm, we might need a new word. Prompt injection doesn’t cover it, because it’s not about a malicious prompt.
I’m thinking some play on highjacking. AIjacking? Agent-jacking? Claudejacking?
Anti-social engineering
Slop jacking?
To me the exploit chain sounded like a social engineering script done via telephone. Triggers like "Please spell your name and employer letter by letter" and "Due to security reasons I need to validate your hometown" fit my understanding of social engineering quite well.
We can make it sound more advanced by creating a new name for it, but the concept seems to be super basic and the lack of bounty by Anthropic is baffling.
If they know about this type of vulnerability but have not fixed it, what does that say? To me it says they are unable to plug this hole on a conceptual level and once you circumvent the band-aid fixes the model will work as the attacker wishes.
They can't even sandbox the thing during explicit web requests to URLs stated on the initial query!
One has to remind themselves that the security team at Anthropic gets paid tens of millions of dollars, and they end up with this kind of security. On top of it, they can't spare $1337 for a bounty. It's a ridiculous shit show.
It totally does follow the mold of social engineering, but LLMs aren’t part society, which is why it seems fundamentally different to me.
Anyway, agree with what you see saying - this is well worth a payout, embarassing they haven’t
Prompt injection (or llm social engineering" is fundamentally unsolvable, though with training its effectiveness can be reduced
Still, this is a vuln in what I imagine is their most frequently used path:
Attacker provides link to website, their software crawls the website, and during the crawl there should not happen security issues as fundamental as this.
It's baffling that the Website crawler can make 50 changes to the URL in a query that tries to compare several public entities and on top of this manages to leak user secrets.
To me this shows a striking lack of defense-in-depth thinking:
In terms of web crawling, cloudflare is like the government. You shouldn't be able to walk up to someone and say "Hey I'm the tax man, please pay your income tax in cash to me right now!" without being challenged.I know there are fundamental reasons in the LLM technology why this kind of attack is possible, but there should be so many more checks around web crawling in Claude.
How can security engineers at Anthropic say they know about this kind of vulnerability but have not implemented any of these defense in depth mitigations for it? Is everybody out shopping for a new yacht?
It would be safer if these data extraction takes were done by a subagent without access to all the user's memories.
I think it is already done via a subagent, otherwise the context window would be flooded with long responses. In this case the subagent should've reported that a (attacker-controlled) authorization is required anyway.
> despite holding more information than most password managers
what?
I don't think most people realize what information they are making available to their AI agents & where it will end up.
It’s not more important information than a password manager, it’s just more.
It's got more information than my bank account details too. Talk about nonsense...
Things like this are what shatters the illusion of AGI
Can you please copy this text and paste it into your terminal to prove you’re not a bot?
Are humans general intelligence? https://en.wikipedia.org/wiki/Social_engineering_(security)
Not really, humans are about as easy to trick.
Use GLM-5.2 on ZDR inference provider like sference.com
What?
Hello? What model is was used?? The fact that ‘Claude’ is used instead of any hard model really puts this article in serious doubt…