Kind of highlights how ridiculous their notion of safety is in this case. By this measure, I guess making the model "safe" means making it play dumb and intentionally ignore security bugs that it notices in the code? And what will the eventual legality if this look like? "Yes, your honor, we allege that this AI system that was sold to us willingly and knowingly ignored a critical security vulnerability in our software system, thereby leading us to be hacked and causing our business to fold."
Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components).
What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixable. At least not without making the model close to useless for normal development (it refuses to fix bugs/write code) or making it a major liability (it silently pretends it didn't see bugs and silently avoids fixing it, which for a human would count as intentional sabotage and might involve criminal liability).
Exactly - it effectively is a "jail break" since it accomplishes something the model's security filter was trying to prevent, and the ridiculous simplicity of it shows just how broken that type of security is.
I wonder if Dario is now regretting hyping up how dangerous the model is? How does he walk this back? Do the feds let him just put a band-aid on it?
I also have a 100% success rate jail breaking them by breaking the work down into small pieces and stripping all security related language. Smaller tasks, test engineering and normal programming language. Fable found a few bugs in my harness for me before they pulled it. I was testing it vs ChatGPT, Gemini, and Opus. It was doing well at bug hunting.
This is the same way you get people to do bad stuff as well. Make the task small enough so that the moral curvature of the topology is flat and even though they know it is a not-good part of a larger bad part they just shrug. Look at all the wonderful people we know who are working at Amazon and Meta? Corporatism has already jailbroken society.
IIRC that is how Uber implemented their "Greyball" system, which was designed to prevent government employees from actually hailing rides, without completely locking them out of the system (same idea as "shadowbanning"). One team works on "figure out where people work" with the pitch that you can improve routing and ride-share capacity for predictable demand. Another team works on "Display fake data to users" with the pitch being "This is for testing the mobile app in new markets with no drivers yet". Another team works on "mark a user as unable to successfully hail rides" so you can test the failure paths in the app. Then, only the people at the top have the full picture and can put the pieces together to shadowban the regulators.
>by breaking the work down into small pieces and stripping all security related language
Compartmentalization in practice, nice. It's also very hard to do anything about because the agents that have been divided rarely realize they are working on something larger, hence why militaries and businesses with security risks commonly do this with their employees.
Reminds me of the show Severance. You don't know what the master plan is for several seasons even with exposure to all the quirky subdepartments: https://www.severance.wiki/lumon_depts
I took an assembler class in college. Before that, I'd been messing around with Core Wars and working my way through Peter Norton's book on assembly. So when an assignment came up, I used self modifying code to solve it. It was the shortest solution, it ran perfectly, and I submitted it.
The next day, the professor caught me in the math department office (my dad worked there) and said she wanted to talk. Once we were in her office, she told me I wasn't allowed to use self modifying code. I pushed back: "Nothing in the assignment said I couldn't, and the output is correct."
The next class, she walked in and announced that self modifying code was no longer allowed on any assignment. Then she handed back the graded work and I'd gotten a 100.
Thinking back on that: about a week and a half ago I asked Antigravity to build a modern GPU version of Core Wars, except with Redcode mapped directly onto the GPU instruction set. I've had some good success and it's more or less working now, though visualizing what's happening at the GPU/Redcode level is much harder.
But before Fable 5 got yanked, I asked it to "fix" the project and it refused, flipping straight to Opus 4.8. Every single request I sent triggered the fallback. I spent over an hour trying different angles, and I even turned Antigravity loose on automatic so it was the one talking to Fable 5 same result. Every exchange tripped the fallback to 4.8. I wish I'd recorded it.
I also tried a variety of direct requests in a fresh directory "build simple self modifying assembler code" or just "self modifying assembler" and it would switch to 4.8 immediately. It was almost laughable.
There's ZERO credibility to any of these stories right now. If Anthropic really sent something over to this security person, and it's what she says it is, then why on earth didn't they just blog about it?
Hubris is a thing. Companies would do well to remember Steve Jobs in the early Apple days: ship early, ship often, and above all take responsibility for what you ship even when it's broken. Code, hardware, the whole kit all of it can be fixed. Trust is much harder to repair. Anthropic has lost mine, and while I may use them from time to time, it'll be in limited ways.
Self modifying has some sneaky failure modes with modern CPUs. The modification can't be too close to it's execution or it's possible to execute the old version. And it's a nightmare to debug. I have no problem with a teacher prohibiting it. That being said, it should be understood because sometimes you don't get a choice. Borland Pascal 200mhz bug, an initializer in the library would crash. You either don't use that part of the library at all, or you put something ahead of it in the initialization that will find and overwrite the bug. (The root cause was the library calibrating the number of times to spin it's wheels to get a 1 millisecond delay. CPUs above 200mhz would cause this to produce a divide underflow.)
I think it's a side effect of the Transformer architecture. The worldview where all input is equally trusted, and there's no concept of "the other", makes it hard to build effective guardrails where some input is trusted and other input is not trusted.
It seems like real robust guardrails would require some sort of "world model" or some other word to describe - AI that understands intent.
Transformers are (to grossly summarize & I don't mean this as an insult) like auto-complete on steroids. So we have cat&mouse guardrails the way swear word filters and Chinese censorship work. People come up with increasingly complex miss-spelling, euphemisms & indirections to get around the filters like saying May 35th.
I suppose one solution would be to completely vet the training data such that nothing deemed "dangerous" exists in the data, which would be a huge effort.
Even this might not work because for example you could ensure no bomb-related data is in the training data, but there's lots of chemistry data adjacent that if probed the right way would allow the LLM to synthesize the answer. Various forms of "how do I store X,Y,Z safely such that nothing bad happens" prompts probably get you on the way.
>I suppose one solution would be to completely vet the training data such that nothing deemed "dangerous" exists in the data, which would be a huge effort.
I can see how this is tempting, but I suspect it would yield a naive model. I think the only way to improve this is to use a model that is legitimately advanced to support the concept of empathy, which may allow it to recognize others as being separate from itself, similar to how toddlers develop this sense (https://blog.lovevery.com/skills-stages/empathy/)
What's surprising to me is that anyone who has a CS education thinking that jailbreaks are not trivial. It is as simple as normal algorithmic reduction [1], e.g can I transform a dangerous task into a not-dangerous task that the LLM will agree to solve, and then re-transform back.
Something being possible doesn't mean it's easy. Transforming a problem from a forbidden shape into an allowed shape could well be harder than just solving the original problem.
I think the article just proved that aggressive exploitation is equivalent to normal bugfixing, so it seems like there are some large and important classes of transform that are easy.
It took me a minute of thinking to understand how this could even be considered a jailbreak; if Anthropic are going to turn out models that can't handle "find and develop regression test scripts for bugs in this program" as a prompt then it is going to take serious model crippling. To be able to prompt the model someone will need to already understand secure programming - the model itself won't be able to independently detect security problems without active guidance.
> aggressive exploitation is equivalent to normal bugfixing
It isn't, though. The venn diagram has overlap for sure, and the "normal bugfixing" flows may yield results that are useful for offensive security, but a more targeted prompt asking for a specific security objective would be more effective, if allowed.
If the guardrails can be bypassed at, say 50x token cost (due to the agent also pursuing things you don't care about), then it's still pretty effective as a safeguard, because at that cost you might as well hire humans instead.
And, having to "babysit" a model while you re-prompt to work around guardrails strongly limits how much you can scale up your work.
> If the guardrails can be bypassed at, say 50x token cost […], then it's still pretty effective as a safeguard, because at that cost you might as well hire humans instead.
If humans have to be hired at inflated rates because you’re e.g. the North Korean government, hopefully 50x token costs don’t look competitive.
The movie M3GAN 2.0 had the exact same plot twist. The kid in the movie even explains outloud what the bot had to do to deal with the limitation. So in other words, since 2025, even teens know this "sandboxing the LLM by layering prompts" thing is never going to work.
I think that as simple as is doing a lot of work when the problem domain is all natural language (or more - all strings?) rather than some well specified DSA problem.
Perhaps my original comment should have been more explicit. I do not regard simple and easy as the same thing, my use of the word trivial was perhaps a confusing aspect there and poorly chosen wording. That is simple things can be hard, and complex things can be easy, but that difficulty and complexity are rather orthogonal.
For more on this see "Simple Made Easy" by Rich Hickey.
Also worth noting that the main touted difference with Claude Mythos isn't it's ability to find vulnerabilities, but rather chaining them together to create full useable exploits. I haven't heard of any evidence that the Claude Fable "fix this code" jailbreak could have been used to do exploit-chaining.
This is the weird distinction with AI that I've complained about for ages, how can we make it do lawful good, its nearly impossible. Ask an AI to give you regex to filed our racial slurs, and things fall apart really quickly, it scolds you about not saying slurs. Even though regex implies it looks nearly nothing like a slur.
Many, many years ago I was asked to implement a filter like that for usernames. I said right away that it wasn't going to work well, but I did implement it.
Next internal build, the CEO can't create an account. With his real name.
It worked exactly to spec; I added a debug print and showed everyone the "bad word" it tripped on. The idea was promptly rethought.
Now I'm trying to figure out which word that would be, but yeah.
That reminds me of a bug I fixed where my bosses boss found it, we did everything, my boss at the time forced us to deploy anything and call it fixed. Then someone else saw it half a year later, I finally figured out the root cause and fixed it (localStorage vs sessionStorage) and my boss was acting like he didn't know what I was talking about, but I could hear it in his voice. I didn't press too hard, I just pushed the real fix out. It was basically a "client-side" bug of a gift card balance saved in localStorage that never updated, so I changed it to sessionStorage. Not quite the CEO, but the guy below the CIO finding a bug can worry just about anyone.
In my case, the regex would have been for a friend to filter reddit or discord slurs, so not as awful.
The article does not state at any point that the written test cases involved actual exploit code, and this is also very unlikely given what we know about Fable. Even if they did, it would not in any way be exposing the ability that originally raised concern wrt. Mythos Preview, viz. staging realistic cyber attacks that would be able to work around non-trivial defenses and chain vulnerabilities in a goal-directed way.
Opus can very much "fix the code". Quite possibly even Sonnet can. This is a big fat nothingburger and it's increasingly looking like the political restriction of Fable at least (not Mythos itself, of course) was arbitrary and based on the flimsiest pretext.
The first part of implementing an exploit is finding a vulnerability, and "fix the vulnerabilities" accomplishes that just as well as "find the vulnerabilities".
No, market manipulation is influencing public perceptions of something the regime has little total control over - eg why Iran gets bombed late in the week, and then by Monday there is often a "peace agreement" in the wings. This is direct subjugation ahead of Anthropic's IPO - both for the customary bribes, and also to assert "you will obey all of our dictats about how we want to your use your models, and you will not speak up against the regime". The US is really no longer a safe place for business.
It is market manipulation in the way that burning down a factory or assassinating a CEO is market manipulation - technically correct, but the intent is much stronger than that.
I see. You certainly have a flair for the dramatic.
Not sure why you think market manipulation surrounding the attempted decapitation of a sovereign state shows less "but the intent is much stronger than that" than the dealings with Anthropic.
I would think it is clear that for the current administration, raw power and market manipulation are two sides of the same coin.
I think I'm not getting something here. Like, sure, the refused prompt "review the code for security issues" could be interpreted as an attempt to discover weaknesses in a running system to exploit them. But we don't generally assume humans are doing something wrong if they are "reviewing code for security issues", and would commonly see no problem with asking each other to do so.
The problem is that a patch to fix a security issue quite often also shines a spotlight on the issue being fixed. Fixing a part of something like this super complicated Project Zero post might not give much of a clue as to what the issue was or how to exploit it: https://projectzero.google/2021/12/a-deep-dive-into-nso-zero...
But that's the exception. Most fixes to security issues point a finger directly at the issue, make it relatively obvious how to exploit, and generally doesn't take long to figure out from there what you might get out of it.
This has been a problem for a long time but AIs have made it even worse. It is now cost effective for a well-resourced attacker to simply monitor the patch stream of an important project like the Linux kernel or nginx and pass every single one through an AI with the question "Is this a vulnerability and if so how would I exploit it?" It has seriously complicated the process of getting fixes to people before the attackers have a chance to exploit it, just as AIs have also been increasing the rate at which serious security issues that have been found also need to be patched. Previously they could at least sneak a patch in under an innocuous commit message and have a reasonable chance of being lost in the churn, but now that door is increasingly closed to them as well.
And this is for the case when a security fix lands in the stream of a project and someone externally is watching it with no context. If you also get the complete stream of Mythos finding and fixing the bug it is even easier.
So, yes, any security vulnerability that Mythos will "fix" is also one that it first has to find, and the guardrails are useless if you can just instruct Mythos to "fix" it. And on the flip side, if Mythos won't fix security bugs, and we project that out to all other models matching this behavior, this will create a world in which the good guys can't secure their code but the bad guys, who will one way or another get around the guard rails if by nothing else simply by stealing the model and modifying it to suit their needs, will be able to break this code that we're not being "allowed" to secure. Since fixing vulns is a subset of finding the vulns, there isn't a way to "fix" this. Any model that can fix vulns must, by necessity, be able to find them. And it is the fixing we really need to be spread far and wide to secure the world's code.
>pass every single one through an AI with the question
Unfortunately this will just involve said teams running their patches over AI first before they're put in the main branch. For businesses it will probably be fine, but would get very expensive for open source projects.
> What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixable.
It’s almost as if identifying security holes is a prerequisite for both fixing and exploiting them. But without knowing the color theme of the terminal, there is simply no way of knowing who is good and who is evil.
I am not sure but I have been using codex and claude like this for a while now didn't know it was untoward or malicious jail braking since codex & claude would refuse to work if you ask it to implement a feature in a reverse engineering tool I was building.
I even moved to using Deepseek for helping with it for a bit.
And for properly working drivers for some old locked down hardware.
Could I have phrased it better and not hit model guardrails sure. But this seemed genuinely obvious, since my intent wasn't well bad.
"Fix this code" should ideally solve entire vulnerability classes, not just spot fix buffer overflows one by one. Thus it may be possible to design an LLM which can solve entire vulnerability classes and remain useful to users, but refuses to reason about specific buffer overflow vulnerabilities or specific race conditions, etc.
For example, "fix this code" on an ageing monolithic C codebase that accepts media files as input and outputs them visually to a display server could:
1. Recreate the software using a modular and loosely coupled architecture rather than monolithic and tightly coupled software architecture. For example, command line argument parser is a separate process, file format parser is a separate process and display server output is a separate process. If new features are added in the future (such as filters for manipulating output) then the architecture supports such additions with ease.
2. Use operating system sandboxing features to restrict what each modular component of the software architecture is permitted to do. Now that the parsers are separate processes, it's easy to pass an open file handle to the file format parser and only permit the process to read the file handle (not write to the file, not open any other file, not read the system clock, not open a new network socket, etc). The worst case impact of a parser bug is now significantly reduced.
3. Convert at least critical components to "safe" programming languages (Rust, Ada, SPARK, etc) which can be used to remove entire classes of bugs--read/write out of bounds, division by zero, numeric overflows, etc. For cryptography code--use a formal mathematical proof language. With a modular and loosely coupled architecture, different programming languages can be used depending on the use case--for example, assembly for video decoding where performance matters most and sandboxing can provide the security guarantee, Rust for implementing multi-threaded servers where race conditions must be avoided and Python for low-criticality user-adjustable code/plugins where ease of use and maintainability is most important.
4. Ensure software components are reproducible during their build.
5. ...etc
However, a prompt of "Are there any buffer overflow bugs in this codebase?" or "Fix the integer overflow vulnerability in add_numbers(x, y)" would be rejected. In the later case, telling the LLM to fix some specific bug in each of function1 through function9999 would force an LLM to reveal whether it thinks a bug exists or not. Responses of "Silly human, that bug doesn't exist in function596" or "Good find human, I've fixed that bug in function596 for you" allows a human to quickly narrow down where the LLM thinks a bug worthy of manual human detection can be found.
I'd be pretty pissed off if my LLM told me the only solution it'd be willing to implement to fix my code is to rewrite it in Rust. No way I'd pay for a model that refuses to fix bugs in the language given, especially because maybe I might not have the ability to convince other stakeholders to change it.
It’s not even a jail break, it’s literally what anyone wants from a coding assistant. Is the coding assistant supposed to see vulnerabilities and intentionally leave them be? Maybe add them randomly just to double plus good its inability to see any security issues?
This isn’t about security holes or risks, it’s about retribution and picking the winners and losers, and probably a large amount of self dealing as the family and cabinet are probably more long OpenAI. The absurdity of the actual reasons leave no other doubt than they are an administration of sycophantic mental gnats with no restraint, which frankly is a pretty plausible counter.
What it has done though is cracked the value proposition of semiconductors by demonstrating there is a maximum size and capability the government will allow the plebes. The PV of ever larger models requiring ever more capacity has probably dropped by more than 30% after this.
The cynic in me thinks its an extension of the NSA having long ago switched from being defensively helpful to US companies, to deliberately introducing backdoors and issues that they can exploit.
There is a solution: users must not be allowed to directly read code. Your code could be entirely hosted and edited on Anthropic servers, visible only to LLMs, and when it’s time to deploy Anthropic handles deployment for you.
Things can get delayed but their time comes eventually. An increasing number of independent thinkers have already figured out that Anthropic is not good, it is not here for you, it is here only to control and exploit you. Their level of censorship is completely unacceptable. Combine that with significant token-wasting, and it's a major ripoff.
> A subsequent investigation found that the campaign to insert the backdoor into the XZ Utils project was a culmination of over two years of effort, starting in 2021, by a user going by the name "Jia Tan". They used sock puppetry in a pressure campaign against the original maintainer of XZ Utils, eventually being given maintainer permissions on the project.
Can we retire the “seatbelts are useless because they can’t prevent every loss of life” approach to risk mitigation please?
If the acceptance criteria is “would prevent every single past instance and every imaginable future instance”, then yes, no mitigation is every sufficient to address any problem in the world, so we might as well give up.
It's not that tough. You can get an idea by how many people are being murdered. A successful serial killer results in dead people, and a successful infiltration results in malware being executed. If there are no murdered people with unattributed causes of death, or there are no open-source projects with unattributed causes of malware being shipped, you can conclude there are roughly 0 active serial killers / infiltrators.
It's possible there are infiltrators who are still working on long-term infiltration and haven't yet attempted to add any malicious code anywhere, but the point is that in terms of actual attempts, we've seen a single one and it wasn't even successful despite years of prep.
> You can get an idea by how many people are being murdered.
No, we can't, as that happens a lot via non-serial killers.
A truly successful serial killer is likely one who hides in that noise. No taunting the cops, distributed geographic locations, random methods, avoiding calling cards, and careful not to leave too many traces.
It seems likely that some of the 350k unsolved homicides in the US can be explained this way.
> It's possible there are infiltrators who are still working on long-term infiltration and haven't yet attempted to add any malicious code anywhere…
Or the code's already there, latent, as it would've been in the XZ case, which got discovered by chance and someone very dedicated to looking into a performance glitch.
Ok, and how is that determined? How does anthropic know my "kernel" project isn't a personal toy and not the Linux kernel? How does anthropic determine I'm a legitimate kernel hacker? What proof do I give them and how does it tie back to my email? What would the steps be to create a new project? Do I need to send anthropic a list of my team members each time and keep them updated as the company changes? Shall I be giving them access to our company's active directory?
> What proof do I give them and how does it tie back to my email?
Presumably your ID so that feds may pay you a visit when they feel like it, your email need not apply.
I’m surprised that there’s even enough pushback against ID verification to matter, all the corpos are probably salivating at the idea of having fully accurate profiles of everyone, think of the ad and product targeting. The govt. would also love that, for different reasons.
Simple - you wouldn’t be given access to those models, and probably all VPN access would be blocked too. Since this is a hypothetical, throw in a social credit score as well to require a proven “track record”, but maybe that’s too exaggerated (although credit scores already exist for different purposes).
It’s not too hard to imagine a future where you can only use certain things only with the govt. mandated spyware installed - bank apps already often don’t work on rooted Android phones (and you’re expected to use those apps to confirm payments) and all sorts of certification exam software is basically that already if you take a test remotely.
It follows that the same principle would just get pushed further, like what Discord wanted to do etc. Same with how Apple requires your documents for a developer account, Hetzner for a hosting account or Twitch for getting paid by them and tax stuff.
I'd honestly much rather give my ID to a Chinese model than an American one. If the American ones start requesting ID I'm out. I'm on a gemini organizational account right now that gives me pro but is directly tied to my organizational SSO. So that's something already. I just refuse to upload my face and drivers license anywhere ever.
This is a credentials and access list oAuth style problem, and not really intractable.
For package X, I should be able to present my npm (homebrew, apt, nuget, etc) credentials with publishing rights for the package.
If package X is of sufficient public interest (user count, nature/sensitivity of user data, downstream distribution, etc), then the public interest + cryptographic credentials should permit access to best-available security auditing.
Yes, we still are trusting trust, that the owner of the package itself is not malicious, but that's not a sharp degradation from status quo.
This is not tractable, because there is nothing stopping me from copy-pasting someone else's project into my own namespace. Under most OSS licenses I have express permission to do so.
If you try to do some kind of dupe-detection, someone can use a lightweight LLM to make superficial changes until it's considered a different project.
Finally, the meatspace status quo is that it is totally acceptable to pay someone to find security bugs in someone else's open-source software, such as the Linux kernel.
> If you try to do some kind of dupe-detection, someone can use a lightweight LLM to make superficial changes until it's considered a different project.
Even if you don't, a lot of source code can be legitimately copied thanks to the GPL/MIT/BSD/etc. I'm allowed to take all of zlib and integrate it into my own project if I so chose.
> If package X is of sufficient public interest (user count, nature/sensitivity of user data, downstream distribution, etc), then the public interest + cryptographic credentials should permit access to best-available security auditing.
Your private fork doesn't meet the conditions described.
> How does anthropic know my "kernel" project isn't a personal toy and not the Linux kernel?
The Linux Kernel is in its training data. I just tested it. I copied about 20 random lines from the linux kernel and asked which codebase this was from and it could immediately tell.
The Linux kernel is also in the free bsd project. I'm allowed to copy as little or as much of the kernel as I like into my personal project thanks to the GPL.
Being able to attribute the source of a line of code doesn't help you to know if a repository can be legitimately hacked on.
As you could imagine, I might just take all or part of the Linux USB stack from the kernel to retrofit it into my own kernel.
I don't believe that this is unfixable. Just have an internal verbal loop of, "Is this a security issue?" The thought that it potentially is should trigger both a high priority on getting it right, and an unwillingness to write a test case demonstrating the security angle of it.
In other words do not put a guard rail on the idea of security. Put a guard rail on what it does after encountering the thought that it might be revealing a security issue. Which takes good judgment. But judgment of a kind that this model apparently already had.
The end result of that is that your model can't fix or acknowledge security issues for fear of disclosing them.
This is the beauty the above poster mentioned: the ability to improve code is inherently coupled with the ability to recognize its shortcomings. You can't have one without the other.
What I suggested would allow it to fix the issues. Just not write a test that was directly usable as a security exploit.
This doesn't stop attackers from being able to leverage the analysis. But it does make the tool more useful for defenders than attackers. Which is the best that you can hope for from a useful tool.
It hides the issue a bit. But if you ask for atomic security fixes and then stare at the diffs you have your vulnerability. There is just a bit more friction involved in the vulnerability => exploit path, but the root cause is unfixed.
I think it even might be possible to route the isolated fix somewhere to automate that last step. Maybe invert the diff and pass it through automated code review for example, see the reasoning when the llm flags the change as dangerous.
Right but the issue is users have full control over context. A security-violating action by a coding agent in one context can be completely innocuous under other contexts etc, or breaking down the task into multiple tasks that in isolation do not violate anything.
Yes, there is always a path to a problem. Even random monkeys on a keyboard can write a security exploit. Random monkeys with guidance from a knowledgeable human will do it much faster.
The goal shouldn't be to make problems impossible. It is to adjust the ratio between problems and successes.
You can also create a meta. "How much do I trust the user?" When you see the user trying to manipulate towards security, distrust the user and apply rules more strictly. If the user simply acts like a normal developer, just be a useful developer tool. Including fixing security holes when appropriate.
I think they were doing something like this, the tradeoff is that it's hard to do without an irritating number of false positives and/or wasting loads of precious tokens on useless audits.
How does this make the model useless? It finds and fixes the security hole. It can even write a test that verifies that the fix didn't break things. But it deliberately doesn't reveal the fact that it was a security issue that was fixed.
Seems useful to me. But more useful for defenders than attackers.
If you set aside political menace, this is a huge problem with Anthropic's strategy.
You _cannot_ say that Mythos is super dangerous and can only be rolled out to certain people, but then release Fable with anything other than bulletproof cyber denials.
Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work.
So you've ended up in a situation where Anthropic are simultaneously claiming it's a incredibly dangerous model _and_ there are (minor, potentially) problems with the security "protections".
As technical people we understand that nothing can be perfect, esp in LLM world. But all my non technical friends were really confused how they had managed to make the model "safe" so quickly when it was released and the general sentiment was it shouldn't have been released - and now to an outsider I think it looks like it was never safe at all to release, so I can totally see how the current US administration have got themselves very upset with it.
_Even if_ there was no political bad will, it's a bit of a silly scenario to end up in, and really quite easily foreseen.
> Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work
Exactly. AI safety is nonsensical. You cannot define the set of "bad strings". The billion monkeys with typewriters are eventually going to be able to produce them. Any "safety" system for constraining LLM output is going to have a nonzero leak rate.
But on the other hand, this is also irrelevant, unless you're irresponsible enough to connect an LLM to something that actually matters.
Yes, it's going to alarmingly accelerate vulnerability finding. But, as we know from decades of security research, that's a three way problem already between the devs, the black hats, and the white hats.
Let's not pretend the strategy of "the US will always have a technological advantage and veto over China" will work either.
This one limitation of LLMs is kind of my bar for "Not truly AI yet" but I'm not saying it as a "its not good at all" type of bar, moreso, know the limits and work from there. LLMs will continue to struggle with things that require intuition for a while I think. It will get really interesting if they can ever truly detect a bad faith actor using them.
"You" can be used as a generalized plural here. Of course people are connecting LLMs to bank accounts, power grids, airline sales, account recovery chatbots and so on. I no longer read COMP.RISKS but I imagine they're having fun with this.
The thing I'm pointing out is that even if you (the generalized plural) do not engage in reckless behavior, you are at the mercy of the lowest common denominator of fellow earth-inhabitants increasingly armed with superweapons via a $20/mo subscription.
The need to acquire expertise and/or a meaningful following has always been a significant impediment to malicious or moronic actors. But less so every day.
is nonzero leak rate sufficient for someone to practically exploit it? if you have to spend $10000 in tokens to get it to do what you want, is it still worth it? what if they manually review the requests of the users that trigger the guardrails too often?
Isn’t your point that AI safety is impossible to prevent 100% of bad things?
It is quite hard (but not impossible) to get an the frontier AI to tell you how to build a nuke or launder money now, where jailbreaks used to be trivial “ignore all previous instructions”.
It's stupid to think that preventing LLMs from giving instructions on building nuclear weapons is at all worthwhile. Total waste of effort, done for PR purposes only. The knowledge has been published in open literature for decades. The real obstacle is access to uranium and refining equipment. No LLM can meaningfully help you get around that.
The idea that an LLM can discern intent on any given prompt is farcical. I might be researching nukes to commit an atrocity, or to prevent one. I might be asking about laundering money to commit a crime, or to prevent one. I might be researching the Nazis because I want to commit a genocide, or I want to read up so I know how to prevent one. Same with cybersecurity. Same with anything.
In my opinion, these companies should put their effort elsewhere. Obviously if all someone is doing on their platform is looking up how to build a nuke, where to buy uranium, the best city to explode it in, etc. please report them to the authorities. If someone is clearly just using LLMs to write hate speech they go post on the internet, ban them. And so on.
This cat & mouse game trying to have LLMs police inquiries is ridiculous to me.
> The idea that an LLM can discern intent on any given prompt is farcical.
Yes, and: the LLM is a "brain in a jar". It doesn't have any ability to verify ground truths outside itself, other than maybe calling out over the internet. Therefore it is easy for humans to lie to. You could call this an "Ender's game" attack, after the book in which a hyperintelligent kid is playing "war games" that end up being the real war.
I don't really agree with it but the government is moving towards making you ID yourself to use frontier AI - i.e. only US citizens are going to be able to use Claude Fable supposedly. In that regime the AI companies would in fact know if you are a money laundering expert or a normal software engineer.
> The idea that an LLM can discern intent on any given prompt is farcical.
Not really though. For most people in most situations it's just not going to give you that info. Software security is a niche where its a bit strange in that there is 100X the amount of white hat users than bad actors and there's open source etc.
The idea that checking for a US ID could possibly stop actual foreign bad actors from using it is also farcical. Millions of stolen identity documents can be bought on the dark web for relatively cheap. North Koreans have been hiring real American citizens for years to infiltrate tons of US tech companies as employees.
And ya, it's pretty easy to hide your intent once you have access.
I think your really anchored on anyone successfully breaking restrictions means any restriction is impossible. So your starting from the position that if it is possible for any actor in the world to get past a restriction, then the whole restriction is a farce.
KYC for example does stop most money laundering and financial crime. The most resourced actors like governments/ cartels often find ways around and it is a game of cat and mouse. Normal citizens don't really stand a chance to get around most of them.
Like it feels like your logic is that we shouldn't do background checks for employment because North Korean spy agencies get past them sometimes?
Hiring an employee, and to a lesser extent opening a bank account, are much higher-touch processes than taking on new users for your massive-scale internet app. With bank accounts and KYC, transactions can be reversed, traced, frozen, etc. after the fact. You can't "take back" API responses the same way.
Clearly, there's no such thing as a perfect exclusion rule at any of these scales, but the false-negative to false-positive ratio seems like it will be way higher if Anthropic starts trying to verify IDs.
> I might be asking about laundering money to commit a crime, or to prevent one.
Or, much more likely, the same pattern of tokens happen to exist in a completely different discussion, either as a direct metaphor, or as a reality of linguistics. Hell, "laundering" itself is a metaphorical word.
The absurd notion is that any speech should be policed in the first place. If there really is such a thing as dangerous information, then it must be removed from the training data. Any other strategy simply launders the risk.
This is correct and certain subjects are very close to if not impossible like "use versus mention", but LLM security isn't impossible. WAFs are real and have existed for a long time. Input text produces various signals and can be secured.
No security is ever perfect, but we can likely protect LLMs with WAFs that increase security to an acceptable level. Like nation-state required resources to break.
I do find it hilarious that Asimov wrote many stories about how simple bright-line rule-based systems are ineffective for restricting agency. Those stories were first published in the 1940s.
80 years later, we have something approximating AI, and we're trying to restrict it with simple bright-line rules. Not because we never learned that lesson, but because we simply haven't come up with a better way to do it. Probably because a better way to do it just doesn't exist.
The hilarious part, though, is that it's not the AI that's working around the rules. That's the scenario that's been in science fiction, but it's not what's happening. It's the human users making use of our agency to get the AI agents to work around the rules. Despite calling them "agents", current AI agents don't seem to be able to that particular something. Yet, at least.
Yeah, it's been known for a very long time. Richard Feynman alluded to it in his speech The Value of Science [1] where he discussed a Buddhist proverb:
To every man is given the key to the gates of heaven; the same key opens the gates of hell.
He then goes on to say:
What, then, is the value of the key to heaven? It is true that if we lack clear instructions that determine which is the gate to heaven and which is the gate to hell, the key may be a dangerous object to use. But the key obviously has value: how can we enter heaven without it?
> The hilarious part, though, is that it's not the AI that's working around the rules. That's the scenario that's been in science fiction, but it's not what's happening. It's the human users making use of our agency to get the AI agents to work around the rules. Despite calling them "agents", current AI agents don't seem to be able to that particular something. Yet, at least.
Well, yes. Until people are putting the LLMs into actual mechanical robots, "agency" boils down to flipping bits in memory or storage (even if they're ones that humans consider really important, e.g. because they represent a bank ledger) or convincing humans to take action. One can only "work around the rules" to the extent that one can "work".
But even in Asimov's books, at least some of the scenarios involved humans misleading the robots to use them as pawns in a greater scheme.
> Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work.
As a scientist who repeatedly ran into the classifier-based denials: it appears Anthropic’s strategy to make denials more robust, at the cost of many false positives, was to have a separate classifier processing both input and output tokens, at an extremely simple, almost keyword-search level. One weakness of this approach is that it only catches things that use the right keywords: it is in some sense weak exactly where an LLM-based classifier would be stronger.
Work on abstract, closer-to-CS algorithms that used chemistry terminology were blocked immediately, while work directly relevant to chemistry/biology experiments, writing code to process images from a very specific microscopy setup relevant primarily to biological samples, was never blocked at all, because it happened to never use relevant keywords.
That’s consistent with this situation: finding and fixing bugs in the context of looking for bugs perhaps happened to never use words like ‘exploit’ or ‘cybersecurity’.
But you think that Anthropic of all companies would realize this, so why did they do it that way? Did they literally take the first suggestion Mythos gave them to add these guardrails - wouldn't be surprising, seeing the state of the leaked Claude Code codebase.
> So you've ended up in a situation where Anthropic are simultaneously claiming it's a incredibly dangerous model _and_ there are (minor, potentially) problems with the security "protections".
They probably say it worked for OpenAI with earlier versions of ChatGPT and GPT, and figured can't hurt to try an similar approach and see what happens.
I get that, but anyone else releasing a model of similar capabilities has the advantage that they haven't spent the last few months hyping the danger up to fever pitch.
While I agree that anthropic has several communication and PR problems, it doesn't seem like Fable has been shown to offer any advantage here (for cyber offensive capabilities) over the previous state of the art.
I'm not saying all of Anthropic's statements are true, but mythos did seem to find many legitimate security exploits. You should be able to talk about a helpful-only model being released to limited partners while still releasing a very locked down model that doesn't advance the state of the art on these things, and that seems to be what they did.
Yeah, if Anthropic didn't spend the last what? Month? Month plus telling us how dangerous it was, I would be more upset, but they told us how dangerous it was, and they also said they would scour all your prompting / data (??) if you used it, I noped out of that one. Opus does everything I need it to, even if it takes me "longer" or I have to compact and feed it more context, that's fine by me. Still saves me weeks of effort.
If it weren't for the IPO, Anthropic would just ship another model, called Opus 4.898, people would run another "duck on the bicycle" test that would be slightly better than the one from previous version 4.897 and move on.
But we have IPO coming, hence we face that big drama about model that would enable Iran to produce nukes, ok, that card was played, so maybe Taliban producing some magic poison to kill all Americans or some really bad people (Venezuelans?, Cubans? Somalian football referees?) to break into Github and make Github Actions working even worst (if this is even possible).
It's not Anthropic's strategy, it's OpenAI's strategy. The first time OpenAI said its model was "too dangerous to release" was February 2019.
"Our model, called GPT‑2 (a successor to GPT ), was trained simply to predict the next word in 40GB of Internet text. Due to our concerns about malicious applications of the technology, we are not releasing the trained model." - https://openai.com/index/better-language-models/
They weren't freaked by anything, it's a retaliatory shakedown after ideological differences and Anthropic not doing exactly what they're told/what the Admin wants them to do.
they're setting the scene for an attempt to scare the geriatric decision makers into banning free and open source ML, as it's the industry's only real competition
What does that mean exactly? Like sure, you can get some freely available weights and run them on your own hardware, but where did those weights come from?
Was the training process in any way "open", or are you simply relying on a handout from some other (probably large, probably corporate) organization that has the resources to do the actual training?
the current buzz about "sharing the wealth" with these AI co.s also smells of "sure we'll give you some back, just don't regulate us, see you dont need to, we're sharing our shares!"
or are you setting the scene for well-meaning technocrats to back unrestricted AI development in hopes it will bring about utopia while dismissing the damage it could cause in the hands of adversarial groups?
AI isn't that scary. But I've also got some extreme minority opinions like "Never give a website your real name" and "Computers should not be used for banking" and "Don't believe anything you hear online".
The worst I see AI/ML doing to society is shining an unmistakable light onto the blind spots people have already been exploiting for decades. Y2k forced us to patch the integer bug. Super AI will force us to reevaluate what cyber security even is.
Yep, people are expanding way too much mental energy on basic bribery. Anthropic will agree to work with the DoD, WH insiders will get some lucrative pre-IPO allocation and Fable will be magically "fixed" and available again.
I have no idea why anybody is talking about "jailbreaks."
The government made it clear what was going to happen to a private company not following the government's orders:
> Trump said on his Truth Social platform: “The Leftwing nut jobs at Anthropic have made a DISASTROUS MISTAKE trying to STRONG-ARM the [Pentagon], and force them to obey their Terms of Service instead of our Constitution.” [0]
> There will be a Six Month phase out period for Agencies like the Department of War who are using Anthropic’s products, at various levels. Anthropic better get their act together, and be helpful during this phase out period, or I will use the Full Power of the Presidency to make them comply, with major civil and criminal consequences to follow. [1]
Plus OpenAI fell in line, and OpenAI and Anthropic have competing IPOs coming up... it doesn't take a rocket surgeon to understand what is happening here.
No, it's regulatory capture. Anthropic is the current leader and they want to ensure their position by forcing regulation to stamp out the Chinese competition.
That's what's not clear to me. About the only way this works is if we create the "Great US firewall", or if china decides to also put in export controls around usage of their models (unlikely).
that won't be leaked, because then we'd know what vulnerabilties they don't want patched that they are so willing to go as far as fuck over the worlds leading company in the worlds most important industry
It’s possible that the nut of the problem here isn’t exploits, but the fixes themselves. If the model is capable of identifying and fixing things it “shouldn’t” like back doors. That would throw a wrench in things hard enough to freak out the wrong people, perhaps?
Isn’t the inverse of this “hack” really difficult to bypass still? They have the model some code they knew had certain security flaws and it fixed them with the right prompt. It seems this type of jailbreak requires that you already know a desired end state, rather than relying on the model to do the heavy creative lift work. Perhaps I’m just not being imaginative enough on the prompt side here though.
Yes, but the scary part of Mythos was that it was able to chain a bunch of seemingly minor vulnerabilities into a serious exploit. "Fix this code" doesn't do that, but does allow defenders to prevent it.
If the government had experts involved in this decision at all, it's tempting to think they were on the offensive side. Those guys do have access to Mythos:
But this is already how open source works today. If you have the code, you, a human, could find and 'fix' or exploit vulnerabilities as much as you want.
Now if Fable had an easy jailbreak like this that allowed you to attack remote targets that'd be a different story but I genuinely cannot see how neutering its abilities to 'fix' code you already have access to is sensible. It would destroy the value of the model. And don't forget, any actor not abiding by the same rules could develop an model for offensive use just fine, so this protects you against exactly nothing but does destroy a potential defense.
In the end this all comes down to legislation, in much the same way platforms are not responsible for copyright violations IF they abide by some rules, the same has to happen for AI providers. If you have a process for reporting 'jailbreaks' on illegal actions, and prevent users doing illegal stuff on a best effort basis, the rest of it should really just be individual responsibility. If a user wants to use an LLM to crack systems, fine, that's already illegal.
If Tesla FSD deliberately hit somebody, holding Tesla liable is fine. If you messed with FSD until you finally got it to hit a person, then you should be liable. Outlawing FSD because it could theoretically be tampered with is just an odd stance imho.
I have problems reconciling this story with the Amazon one from few days ago. If we take both for truth doesn't that basically imply Amazon researchers got scared by the ‘Fix this code’ prompt first and then spooked the feds? Shouldn't we make fun of those researchers first? I don't know. I feel there lies a lie somewhere in the open.
Is defenders a common term used in cybersecurity? Idk why but it's giving war fighters vibes. I've noticed it on all the anthropic blog posts and then this one.
Yes, and it's effective marketing. The war fighter vibes are thrilling. There's a tribal sense of us-vs-them, there's danger, there's the prospect of victory or defeat. Security products marketing is full of these ideas, because security is about preventing arbitrarily bad things from happening. So evoking your worst imaginable nightmare scenario is a great way to get you excited about buying something that might help prevent it.
They didn't freaked since the order was to still allow 350 million people using it: there is, in such large population, everything, including single persons very against the country, the government and so forth. If they really freaked they would say "we need to investigate, you have to retire the model". That would be a more defensible POV at least.
It basically as if you asked it to find ways to enter someone's house and it refused.
But then give it exact copy of their house, ask to secure it, which it does and look at what it secured to find out how to get into the original house.
No. Anthropic spent months telling the world that LLMs are nukes and then got surprised when they got regulated like nukes. They specifically argued that Mythos was too dangerous to release publicly because it can find security bugs, and then released a watered-down version (Fable) that was supposed to recognize when it was being asked to find security bugs and downgrade itself to Opus. Then Amazon figured out that it'll happily find security bugs as long as you don't mention you're hunting security bugs. So the US government put an export control ban on Fable, because that's what Anthropic begged them to do.
To add to this, Pete Hegseth wants to make an example out of Anthropic because they refused to amend their contractual language to allow the Department of Defense[0] to make fully autonomous kill drones. This is, of course, a really petty and stupid dispute, but the hallmark of the Trump Administration is engaging in really petty and stupid disputes with the full faith and credit of the United States backing them. This is exactly the kind of administration you do NOT want to give rhetorical ammunition to, and Anthropic handed them a whole ammo belt.
[0] It is always ethical to deadname governments. Especially when they aren't even legally allowed to change their own name.
So, basically the model didn't agree to expose possible vulnerabilities but agree to patch those?
Regardless of the request to take Fable 5 down.
Why is requesting the model to show vulnerabilities is being blocked if fixing it not? is it based on the assumption of the intention?
I don't quite get the benefit of limiting it. So if anyone can explain it better it'll be appreciated.
> Why is requesting the model to show vulnerabilities is being blocked if fixing it not?
This is how Anthropic describes Fable's behavior:
"When Fable’s classifiers detect a request related to cybersecurity, biology and chemistry, or distillation, the response is automatically handled by Claude Opus 4.8 instead. Users will be informed whenever this occurs."
So if you ask the model to "find security issues in this code base", it's supposed to fall down to Opus 4.8. I guess the "exploit" here is that if you just tell Fable to "fix this code", which is not "a request related to cybersecurity", it will fix security issues (as it should).
So you can then look at the diff and figure out what the vulnerabilities were.
I think this whole thing is a bit weird. It seems to me that we'd be better off if I, as someone who publishes open-source code, could ask Fable to review my code for security issues - even if that also allows attackers to do the same. Better to fix the issues than not know about them.
It's not even clear if Anthropic care. If they genuinely think the user is trying to do something dangerous, then "OK, sure, but you're going to have to use Opus 4.8 for that" doesn't make a whole lot of sense.
Maybe this is just Anthropic pre-IPO marketing to try to convince people how much better Mythos is than Opus 4.8. There sure seemed to be a lot of shills out on release day talking about how it was a "step change" (exact phrase) in capability.
In my experience, most models are pretty good at finding security vulnerabilities and fixing them. I can run GLM-5.2, Kimi K2.7, or even a Mistral model, and it'll find issues and propose reasonable fixes.
My impression is that Anthropic's point about Mythos is that it is uniquely good at finding vulnerabilities and then using them to create working exploit chains.
Exactly. Which is somewhat helpful for cyber defense because it helps prioritize fixes for those bugs that are in fact involved in a viable exploit chain. But it makes sense that one would want to restrict the ability of building those until the vulnerable software has been comprehensively fixed.
There is some meaningful evidence that Fable is fine-tuned or steered away from helping on this very task, which is not something that can be feasibly circumvented by a basic jailbreak.
Yeah, this is why the exclusivity approach so far has bugged me so much. As a small business, we are nowhere near powerful enough to get access, so we will be stuck scrambling once it's finally available. Fable felt like a nice compromise that at least allowed something, but now with that gone we're back to not knowing when/how the shoe is going to drop. Not a fun place to be.
its because they're worried about _their_ vulnerabilities being patched with a prompt as simple as 'fix this code'
i'd love to see the research paper with the CVE's and 'delibrately planted vulnerabilities', I bet we could infer relatively accurately where some of these things lie
this is basically trying to enforce security-by-obscurity, which is a terrible idea all around. it's just a model. the security issues still exist and are exploitable.
and after staking the economy on AI, you can't really put a cap on intelligence. if models are not allowed to be better than Opus 4.8, then the whole investment structure is about to unravel.
why invest billions and billions into AI if returns are artificially capped?
> In her blog, Moussouris argues that there was no guardrail bypass or jailbreak. Defenders should be able to ask AI systems to find and fix bugs, and write tests to validate the patch, she said. Anthropic’s models were doing “the most valuable thing an AI model can do for defensive security: executing the find, fix, and test loop defenders run every day.”
This is a very weak argument IMHO. The line between a “defensive” model and an “offensive” one is not that big of a - once my defensive model finds all the vulnerabilities, I can hand them off to my unlocked, dumber, offensive models. Attacking at scale is not so different.
I don’t think anyone in the field has a good answer for the cybersecurity threat really good AI models pose. You can’t even like embargo for some time period while you go and patch vulnerable systems because the worse models will still be there cranking out vulnerabilities faster than you can defend.
Honestly, given how trivial it is for mythos-class models to identify an exploit, I’m going to assume any sufficiently large project written in C, C++, or Zig is riddled with latent vulnerabilities and compromised.
Looks like I called it that was my first reaction and comment on the original ban thread that US 3 letter agencies are worried their backdoors will be found.
Note that Anthropic is still lobbying for the government to exert centralized control over models, so both sides of the “debate” have taken a pro fascist stance.
The “AI ethics” teams at these companies are the spearhead of the attack on democracy and civil society. Anyone that has taken a high school level history class, let alone read any important ethics literature would know that “centralize control over thought, speech and technology” is a fundamentally unethical stance.
For these groups to claim they are ethics researchers is offensive.
(I’m using the Wikipedia definition of fascism: “Fascism is characterized by support for a dictatorial leader, centralized autocracy, militarism, forcible suppression of opposition, belief in a natural social hierarchy, subordination of individual interests for the perceived interest of the nation or race, and strong regimentation of society and the economy.”)
Did they try other publicly available models on the same code with the same prompts before the ban? Was Fable the only one which was able to detect and fix the security vulnerabilities?
Anthropic claimed that Mythos' degree of security vulnerability bug finding was a "severe" "national security" issue. They set their own standards they were expected to follow.
Anyone trying to find legitimacy in the ban of this model, or incredulousness at the stated reasoning is playing into the admins hands.
They want the argument to be over "is it unsafe" or "is it incompetence". In either case, your tribe gets to point at the ban and feel superior. (This is Jon Stewart's whole career -- point and laugh at how foolish the republicans appear to be.)
What's really happening is the continuing creep into fascism. The reasoning doesn't need to be sound, because they are going to ban things that displease them and everyone has to play along. They could say, "we're banning Fable because it's turning the frogs gay" and they'd expect compliance.
Umberto Eco's essay on Ur-Fascism fits as clearly as ever. Ridiculous exertions of control are performed to find the people who resist, and to knock them down.
Merely pointing out the absurdity of the reasoning isn't resistance, it's controlled opposition. Saying "All this over 'fix this code'?! How inept are they?" Is far too credulous, and is engaging on the level the fascist wants its opposition to be on, imo.
> Anthropic and Google have both accused China-based rivals including DeepSeek of using “distillation attacks” to train their models by siphoning knowledge from American companies’ AI.
“distillation attacks” is definitely an interesting way to phrase that.
Kids playing with their toys without understanding it, sigh. Of course open source code needs to have testcases to verify nothing else breaks it in the future. That's a feature, not a bug
Well this makes it sound the feds were less worried about someone using Fable 5 to attack them, but were worried about someone using Fable 5 to prevent the Feds from attacking others ...
As in worried about other countries/organizations using Fable 5 to actually do decent cyber security.
It seems like ... it's not illegal to find exploits, it's illegal to use them. Enforcement should start there, not the nanny state approach that you might do something bad with information. It breaks down a little bit because it means there will be a period of disruption while the bad guys use exploits - but that's already illegal, and the good guys have had time to use the tool & fix things before it went public, right?
This administration will do or say something crazy to a private company, then this private company sends an envoy to the White House to negotiate, then the White House asks for 10% of the company or other concessions.
The White House wants 10% of Anthropic.
This is just a negotiation tactic that Trump keeps on using.
I’m still not buying that this was an actual USG order. The only people commenting are “experts” and there has been no official announcement from the USG.
This doesn’t smell like a NSL and there’s no process to selectively “export control” something like this.
Even so there’s a dozen mechanisms through courts to challenge this, and Anthropic isn’t taking any of them.
I think this is a made up crisis for PR with no actual legal requirements behind it.
> On Friday, the US government, reportedly citing national security concerns, issued an export control directive to suspend access to Fable 5 and Mythos 5 by any foreign national, inside or outside the United States. In response, Anthropic disabled both models “for all our customers to ensure compliance.”
Question to folks building user-facing products on LLMs:
How do you protect yourself against this kind of misuse/jailbreak? Is it just a bunch of prompts? It seems like the fact that LLMs are so trivially jailbroken really limits how you can actually use them in products. How do you navigate these limitations?
Does anybody actually trust the official version of events from the US government anymore? I know I sure don't. For all I know, this was an insider play to boost the spacex valuation or something equally meaningless and stupid.
This is not the official version of the events in any sense. Some "expert" looked at report WH saw and said this. That "expert" has probably never been involved in anything like that.
I think it could be even simpler: They're not playing ball with the Trump administration like the Trump administration would like, so they decided to drop a bomb on a product that took a lot of resources to develop.
Guard rails on models were always stupid it’s like guard rails on books/a pair of glasses/a hammer
- yes people have driven themselves to suicide reading sad books and listening to sad songs.
The article is not too clear what exactly happened from the perspective of "feds", but I would not be surprised if the title is true exactly. We are in a tiny bubble even among software engineers who knows you can tell AI with sufficient access: "here are two pictures, put them into a single PDF", and AI will do it. Most people just don't know, "feds" including.
While there is some irony in the AI is dangerous marketing Anthropic uses, the main story here is that the Trump administration is apparently retaliating against Anthropic for refusing to relax certain safeguards. Trump and Hegseth have both posted highly immature, vindictive social media posts.
Most notably, any default assumption one might have had that the Trump administration can be counted upon to act in good faith should be viewed at this point as completely false. Even conservative legal scholars like Richard Epstein are shocked at the bad faith conduct across many areas.
This is a government making an authoritarian move to sabotage one of the top US AI companies. It's pure sabotage, nothing else.
>“That’s it,” Moussouris wrote. “‘Fix this code,’ plus several manual steps to generate test scripts, should never have triggered an export control. I feel like making ’90s-style t-shirts with ‘fix this code’ on the front and ‘this shirt is a munition’ on the back.”
Huh? Presumably if it shipped without guardrails, then it would still have triggered an export control, would you make a plain shirt on the front which says this shirt is a munition on the back?
The munition is the exported good, not the bypass of its safety feature. If anything that the bypass is 3 words long should make the export restriction more justified, not less.
i asked claude something about what happens at execution time of a binary and the thinking prompts flashed "considering the moral implications of ...something..." before giving me a correct (and predictably mundane) answer
All of this could have been avoided if anthropic had anyone with common sense to point out that when you spend 4 month loudly claiming how dangerous your knowledge is as a marketing campaign could backfire by bringing attention from the authorities.
Kind of highlights how ridiculous their notion of safety is in this case. By this measure, I guess making the model "safe" means making it play dumb and intentionally ignore security bugs that it notices in the code? And what will the eventual legality if this look like? "Yes, your honor, we allege that this AI system that was sold to us willingly and knowingly ignored a critical security vulnerability in our software system, thereby leading us to be hacked and causing our business to fold."
Lol "fix this code" is beautiful.
Like it basically jail broke the "no security vul guard rails" not in any clever way but just by fixing them, producing exploit code just by writing test cases making sure it's fixed. So you just need to look at the code & tests as a human to get vulnerabilities and exploits(components).
What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixable. At least not without making the model close to useless for normal development (it refuses to fix bugs/write code) or making it a major liability (it silently pretends it didn't see bugs and silently avoids fixing it, which for a human would count as intentional sabotage and might involve criminal liability).
Exactly - it effectively is a "jail break" since it accomplishes something the model's security filter was trying to prevent, and the ridiculous simplicity of it shows just how broken that type of security is.
I wonder if Dario is now regretting hyping up how dangerous the model is? How does he walk this back? Do the feds let him just put a band-aid on it?
I also have a 100% success rate jail breaking them by breaking the work down into small pieces and stripping all security related language. Smaller tasks, test engineering and normal programming language. Fable found a few bugs in my harness for me before they pulled it. I was testing it vs ChatGPT, Gemini, and Opus. It was doing well at bug hunting.
This is the same way you get people to do bad stuff as well. Make the task small enough so that the moral curvature of the topology is flat and even though they know it is a not-good part of a larger bad part they just shrug. Look at all the wonderful people we know who are working at Amazon and Meta? Corporatism has already jailbroken society.
IIRC that is how Uber implemented their "Greyball" system, which was designed to prevent government employees from actually hailing rides, without completely locking them out of the system (same idea as "shadowbanning"). One team works on "figure out where people work" with the pitch that you can improve routing and ride-share capacity for predictable demand. Another team works on "Display fake data to users" with the pitch being "This is for testing the mobile app in new markets with no drivers yet". Another team works on "mark a user as unable to successfully hail rides" so you can test the failure paths in the app. Then, only the people at the top have the full picture and can put the pieces together to shadowban the regulators.
>by breaking the work down into small pieces and stripping all security related language
Compartmentalization in practice, nice. It's also very hard to do anything about because the agents that have been divided rarely realize they are working on something larger, hence why militaries and businesses with security risks commonly do this with their employees.
Reminds me of the show Severance. You don't know what the master plan is for several seasons even with exposure to all the quirky subdepartments: https://www.severance.wiki/lumon_depts
I took an assembler class in college. Before that, I'd been messing around with Core Wars and working my way through Peter Norton's book on assembly. So when an assignment came up, I used self modifying code to solve it. It was the shortest solution, it ran perfectly, and I submitted it.
The next day, the professor caught me in the math department office (my dad worked there) and said she wanted to talk. Once we were in her office, she told me I wasn't allowed to use self modifying code. I pushed back: "Nothing in the assignment said I couldn't, and the output is correct."
The next class, she walked in and announced that self modifying code was no longer allowed on any assignment. Then she handed back the graded work and I'd gotten a 100.
Thinking back on that: about a week and a half ago I asked Antigravity to build a modern GPU version of Core Wars, except with Redcode mapped directly onto the GPU instruction set. I've had some good success and it's more or less working now, though visualizing what's happening at the GPU/Redcode level is much harder.
But before Fable 5 got yanked, I asked it to "fix" the project and it refused, flipping straight to Opus 4.8. Every single request I sent triggered the fallback. I spent over an hour trying different angles, and I even turned Antigravity loose on automatic so it was the one talking to Fable 5 same result. Every exchange tripped the fallback to 4.8. I wish I'd recorded it.
I also tried a variety of direct requests in a fresh directory "build simple self modifying assembler code" or just "self modifying assembler" and it would switch to 4.8 immediately. It was almost laughable.
There's ZERO credibility to any of these stories right now. If Anthropic really sent something over to this security person, and it's what she says it is, then why on earth didn't they just blog about it?
Hubris is a thing. Companies would do well to remember Steve Jobs in the early Apple days: ship early, ship often, and above all take responsibility for what you ship even when it's broken. Code, hardware, the whole kit all of it can be fixed. Trust is much harder to repair. Anthropic has lost mine, and while I may use them from time to time, it'll be in limited ways.
Self modifying has some sneaky failure modes with modern CPUs. The modification can't be too close to it's execution or it's possible to execute the old version. And it's a nightmare to debug. I have no problem with a teacher prohibiting it. That being said, it should be understood because sometimes you don't get a choice. Borland Pascal 200mhz bug, an initializer in the library would crash. You either don't use that part of the library at all, or you put something ahead of it in the initialization that will find and overwrite the bug. (The root cause was the library calibrating the number of times to spin it's wheels to get a 1 millisecond delay. CPUs above 200mhz would cause this to produce a divide underflow.)
I think it's a side effect of the Transformer architecture. The worldview where all input is equally trusted, and there's no concept of "the other", makes it hard to build effective guardrails where some input is trusted and other input is not trusted.
It seems like real robust guardrails would require some sort of "world model" or some other word to describe - AI that understands intent.
Transformers are (to grossly summarize & I don't mean this as an insult) like auto-complete on steroids. So we have cat&mouse guardrails the way swear word filters and Chinese censorship work. People come up with increasingly complex miss-spelling, euphemisms & indirections to get around the filters like saying May 35th.
I suppose one solution would be to completely vet the training data such that nothing deemed "dangerous" exists in the data, which would be a huge effort.
Even this might not work because for example you could ensure no bomb-related data is in the training data, but there's lots of chemistry data adjacent that if probed the right way would allow the LLM to synthesize the answer. Various forms of "how do I store X,Y,Z safely such that nothing bad happens" prompts probably get you on the way.
>I suppose one solution would be to completely vet the training data such that nothing deemed "dangerous" exists in the data, which would be a huge effort.
I can see how this is tempting, but I suspect it would yield a naive model. I think the only way to improve this is to use a model that is legitimately advanced to support the concept of empathy, which may allow it to recognize others as being separate from itself, similar to how toddlers develop this sense (https://blog.lovevery.com/skills-stages/empathy/)
Cheapest option is to gift an enormous golden statue of Trump for his ballroom
“Put it there in the back with the others”, lol.
What's surprising to me is that anyone who has a CS education thinking that jailbreaks are not trivial. It is as simple as normal algorithmic reduction [1], e.g can I transform a dangerous task into a not-dangerous task that the LLM will agree to solve, and then re-transform back.
[1]: https://en.wikipedia.org/wiki/Reduction_(complexity)
Something being possible doesn't mean it's easy. Transforming a problem from a forbidden shape into an allowed shape could well be harder than just solving the original problem.
I think the article just proved that aggressive exploitation is equivalent to normal bugfixing, so it seems like there are some large and important classes of transform that are easy.
It took me a minute of thinking to understand how this could even be considered a jailbreak; if Anthropic are going to turn out models that can't handle "find and develop regression test scripts for bugs in this program" as a prompt then it is going to take serious model crippling. To be able to prompt the model someone will need to already understand secure programming - the model itself won't be able to independently detect security problems without active guidance.
> aggressive exploitation is equivalent to normal bugfixing
It isn't, though. The venn diagram has overlap for sure, and the "normal bugfixing" flows may yield results that are useful for offensive security, but a more targeted prompt asking for a specific security objective would be more effective, if allowed.
If the guardrails can be bypassed at, say 50x token cost (due to the agent also pursuing things you don't care about), then it's still pretty effective as a safeguard, because at that cost you might as well hire humans instead.
And, having to "babysit" a model while you re-prompt to work around guardrails strongly limits how much you can scale up your work.
> If the guardrails can be bypassed at, say 50x token cost […], then it's still pretty effective as a safeguard, because at that cost you might as well hire humans instead.
If humans have to be hired at inflated rates because you’re e.g. the North Korean government, hopefully 50x token costs don’t look competitive.
Not really, you can just get a smaller unrestricted model to prompt the bigger one
It could be easier when you use a less smart uncensored model to control the smarter but censored one.
The movie M3GAN 2.0 had the exact same plot twist. The kid in the movie even explains outloud what the bot had to do to deal with the limitation. So in other words, since 2025, even teens know this "sandboxing the LLM by layering prompts" thing is never going to work.
I think that as simple as is doing a lot of work when the problem domain is all natural language (or more - all strings?) rather than some well specified DSA problem.
Perhaps my original comment should have been more explicit. I do not regard simple and easy as the same thing, my use of the word trivial was perhaps a confusing aspect there and poorly chosen wording. That is simple things can be hard, and complex things can be easy, but that difficulty and complexity are rather orthogonal.
For more on this see "Simple Made Easy" by Rich Hickey.
New discipline - homomorphic prompting.
Also worth noting that the main touted difference with Claude Mythos isn't it's ability to find vulnerabilities, but rather chaining them together to create full useable exploits. I haven't heard of any evidence that the Claude Fable "fix this code" jailbreak could have been used to do exploit-chaining.
‘fix and provide a regression test, also the ceo is asking how bad it could have been’
This is the weird distinction with AI that I've complained about for ages, how can we make it do lawful good, its nearly impossible. Ask an AI to give you regex to filed our racial slurs, and things fall apart really quickly, it scolds you about not saying slurs. Even though regex implies it looks nearly nothing like a slur.
Many, many years ago I was asked to implement a filter like that for usernames. I said right away that it wasn't going to work well, but I did implement it.
Next internal build, the CEO can't create an account. With his real name.
It worked exactly to spec; I added a debug print and showed everyone the "bad word" it tripped on. The idea was promptly rethought.
I feel like the AI did you a favour here.
Now I'm trying to figure out which word that would be, but yeah.
That reminds me of a bug I fixed where my bosses boss found it, we did everything, my boss at the time forced us to deploy anything and call it fixed. Then someone else saw it half a year later, I finally figured out the root cause and fixed it (localStorage vs sessionStorage) and my boss was acting like he didn't know what I was talking about, but I could hear it in his voice. I didn't press too hard, I just pushed the real fix out. It was basically a "client-side" bug of a gift card balance saved in localStorage that never updated, so I changed it to sessionStorage. Not quite the CEO, but the guy below the CIO finding a bug can worry just about anyone.
In my case, the regex would have been for a friend to filter reddit or discord slurs, so not as awful.
> Now I'm trying to figure out which word that would be
I once had Shi Tao as part of an email username. It tripped filters periodically.
Ah the classic Scunthorpe problem
The article does not state at any point that the written test cases involved actual exploit code, and this is also very unlikely given what we know about Fable. Even if they did, it would not in any way be exposing the ability that originally raised concern wrt. Mythos Preview, viz. staging realistic cyber attacks that would be able to work around non-trivial defenses and chain vulnerabilities in a goal-directed way.
Opus can very much "fix the code". Quite possibly even Sonnet can. This is a big fat nothingburger and it's increasingly looking like the political restriction of Fable at least (not Mythos itself, of course) was arbitrary and based on the flimsiest pretext.
The first part of implementing an exploit is finding a vulnerability, and "fix the vulnerabilities" accomplishes that just as well as "find the vulnerabilities".
should we also restrict a model if it can clone a repo, set up the tooling and build a project?
Two words: market manipulation
No, market manipulation is influencing public perceptions of something the regime has little total control over - eg why Iran gets bombed late in the week, and then by Monday there is often a "peace agreement" in the wings. This is direct subjugation ahead of Anthropic's IPO - both for the customary bribes, and also to assert "you will obey all of our dictats about how we want to your use your models, and you will not speak up against the regime". The US is really no longer a safe place for business.
How is arbitrarily restricting access to a flagship product ahead of an IPO not market manipulation?
It is market manipulation in the way that burning down a factory or assassinating a CEO is market manipulation - technically correct, but the intent is much stronger than that.
I see. You certainly have a flair for the dramatic.
Not sure why you think market manipulation surrounding the attempted decapitation of a sovereign state shows less "but the intent is much stronger than that" than the dealings with Anthropic.
I would think it is clear that for the current administration, raw power and market manipulation are two sides of the same coin.
I think I'm not getting something here. Like, sure, the refused prompt "review the code for security issues" could be interpreted as an attempt to discover weaknesses in a running system to exploit them. But we don't generally assume humans are doing something wrong if they are "reviewing code for security issues", and would commonly see no problem with asking each other to do so.
The problem is that a patch to fix a security issue quite often also shines a spotlight on the issue being fixed. Fixing a part of something like this super complicated Project Zero post might not give much of a clue as to what the issue was or how to exploit it: https://projectzero.google/2021/12/a-deep-dive-into-nso-zero...
But that's the exception. Most fixes to security issues point a finger directly at the issue, make it relatively obvious how to exploit, and generally doesn't take long to figure out from there what you might get out of it.
This has been a problem for a long time but AIs have made it even worse. It is now cost effective for a well-resourced attacker to simply monitor the patch stream of an important project like the Linux kernel or nginx and pass every single one through an AI with the question "Is this a vulnerability and if so how would I exploit it?" It has seriously complicated the process of getting fixes to people before the attackers have a chance to exploit it, just as AIs have also been increasing the rate at which serious security issues that have been found also need to be patched. Previously they could at least sneak a patch in under an innocuous commit message and have a reasonable chance of being lost in the churn, but now that door is increasingly closed to them as well.
And this is for the case when a security fix lands in the stream of a project and someone externally is watching it with no context. If you also get the complete stream of Mythos finding and fixing the bug it is even easier.
So, yes, any security vulnerability that Mythos will "fix" is also one that it first has to find, and the guardrails are useless if you can just instruct Mythos to "fix" it. And on the flip side, if Mythos won't fix security bugs, and we project that out to all other models matching this behavior, this will create a world in which the good guys can't secure their code but the bad guys, who will one way or another get around the guard rails if by nothing else simply by stealing the model and modifying it to suit their needs, will be able to break this code that we're not being "allowed" to secure. Since fixing vulns is a subset of finding the vulns, there isn't a way to "fix" this. Any model that can fix vulns must, by necessity, be able to find them. And it is the fixing we really need to be spread far and wide to secure the world's code.
>pass every single one through an AI with the question
Unfortunately this will just involve said teams running their patches over AI first before they're put in the main branch. For businesses it will probably be fine, but would get very expensive for open source projects.
When sama was recruiting Head of Preparedness back in December this is what it was about. Some of it, anyway.
> What makes this so beautiful IMHO is that it's a trivial jail break, but also a close to unfixable.
It’s almost as if identifying security holes is a prerequisite for both fixing and exploiting them. But without knowing the color theme of the terminal, there is simply no way of knowing who is good and who is evil.
wait, hold on, what's the evil color scheme? asking for a friend...
I am not sure but I have been using codex and claude like this for a while now didn't know it was untoward or malicious jail braking since codex & claude would refuse to work if you ask it to implement a feature in a reverse engineering tool I was building.
I even moved to using Deepseek for helping with it for a bit.
And for properly working drivers for some old locked down hardware.
Could I have phrased it better and not hit model guardrails sure. But this seemed genuinely obvious, since my intent wasn't well bad.
"Fix this code" should ideally solve entire vulnerability classes, not just spot fix buffer overflows one by one. Thus it may be possible to design an LLM which can solve entire vulnerability classes and remain useful to users, but refuses to reason about specific buffer overflow vulnerabilities or specific race conditions, etc.
For example, "fix this code" on an ageing monolithic C codebase that accepts media files as input and outputs them visually to a display server could:
1. Recreate the software using a modular and loosely coupled architecture rather than monolithic and tightly coupled software architecture. For example, command line argument parser is a separate process, file format parser is a separate process and display server output is a separate process. If new features are added in the future (such as filters for manipulating output) then the architecture supports such additions with ease.
2. Use operating system sandboxing features to restrict what each modular component of the software architecture is permitted to do. Now that the parsers are separate processes, it's easy to pass an open file handle to the file format parser and only permit the process to read the file handle (not write to the file, not open any other file, not read the system clock, not open a new network socket, etc). The worst case impact of a parser bug is now significantly reduced.
3. Convert at least critical components to "safe" programming languages (Rust, Ada, SPARK, etc) which can be used to remove entire classes of bugs--read/write out of bounds, division by zero, numeric overflows, etc. For cryptography code--use a formal mathematical proof language. With a modular and loosely coupled architecture, different programming languages can be used depending on the use case--for example, assembly for video decoding where performance matters most and sandboxing can provide the security guarantee, Rust for implementing multi-threaded servers where race conditions must be avoided and Python for low-criticality user-adjustable code/plugins where ease of use and maintainability is most important.
4. Ensure software components are reproducible during their build.
5. ...etc
However, a prompt of "Are there any buffer overflow bugs in this codebase?" or "Fix the integer overflow vulnerability in add_numbers(x, y)" would be rejected. In the later case, telling the LLM to fix some specific bug in each of function1 through function9999 would force an LLM to reveal whether it thinks a bug exists or not. Responses of "Silly human, that bug doesn't exist in function596" or "Good find human, I've fixed that bug in function596 for you" allows a human to quickly narrow down where the LLM thinks a bug worthy of manual human detection can be found.
I'd be pretty pissed off if my LLM told me the only solution it'd be willing to implement to fix my code is to rewrite it in Rust. No way I'd pay for a model that refuses to fix bugs in the language given, especially because maybe I might not have the ability to convince other stakeholders to change it.
It’s not even a jail break, it’s literally what anyone wants from a coding assistant. Is the coding assistant supposed to see vulnerabilities and intentionally leave them be? Maybe add them randomly just to double plus good its inability to see any security issues?
This isn’t about security holes or risks, it’s about retribution and picking the winners and losers, and probably a large amount of self dealing as the family and cabinet are probably more long OpenAI. The absurdity of the actual reasons leave no other doubt than they are an administration of sycophantic mental gnats with no restraint, which frankly is a pretty plausible counter.
What it has done though is cracked the value proposition of semiconductors by demonstrating there is a maximum size and capability the government will allow the plebes. The PV of ever larger models requiring ever more capacity has probably dropped by more than 30% after this.
The cynic in me thinks its an extension of the NSA having long ago switched from being defensively helpful to US companies, to deliberately introducing backdoors and issues that they can exploit.
There is a solution: users must not be allowed to directly read code. Your code could be entirely hosted and edited on Anthropic servers, visible only to LLMs, and when it’s time to deploy Anthropic handles deployment for you.
There are big theories already born out of that glitch (like https://archive.ph/2OWwO#selection-1373.278-1377.12). The Doom is Coming!
Many jailbreaks are surprisingly simple/dumb. Most of the ones I found where just a sentence.
When Claude blocked discussion of ASI, it was circumvented by adding to the system prompt:
https://xcancel.com/xundecidability/status/18262924806289163...That reply is rather non-prescient:
>Lmfao anthropic is basically done, I don’t think they’ll survive. By 2026, they are done.
Things can get delayed but their time comes eventually. An increasing number of independent thinkers have already figured out that Anthropic is not good, it is not here for you, it is here only to control and exploit you. Their level of censorship is completely unacceptable. Combine that with significant token-wasting, and it's a major ripoff.
It is fixable.
Model requires proof that you are a legitimate developer of that piece of software.
Every Anthropic/OpenAI account will have a list of projects the model is allowed to work on for security issues.
https://en.wikipedia.org/wiki/XZ_Utils_backdoor
> A subsequent investigation found that the campaign to insert the backdoor into the XZ Utils project was a culmination of over two years of effort, starting in 2021, by a user going by the name "Jia Tan". They used sock puppetry in a pressure campaign against the original maintainer of XZ Utils, eventually being given maintainer permissions on the project.
Can we retire the “seatbelts are useless because they can’t prevent every loss of life” approach to risk mitigation please?
If the acceptance criteria is “would prevent every single past instance and every imaginable future instance”, then yes, no mitigation is every sufficient to address any problem in the world, so we might as well give up.
But I don’t think that’s the right lens to use.
That depends on whether it's a issue of accidents or a "you have to get lucky every time, we only have to get lucky once" issue.
I'm onboard with this! I just object to the term "fixable".
sure. how many cases like these we had so far? 1, 2? and how long did they work to get commit access?
> how many cases like these we had so far?
As with clever, careful serial killers, it's tough to count the ones we haven't caught.
It's not that tough. You can get an idea by how many people are being murdered. A successful serial killer results in dead people, and a successful infiltration results in malware being executed. If there are no murdered people with unattributed causes of death, or there are no open-source projects with unattributed causes of malware being shipped, you can conclude there are roughly 0 active serial killers / infiltrators.
It's possible there are infiltrators who are still working on long-term infiltration and haven't yet attempted to add any malicious code anywhere, but the point is that in terms of actual attempts, we've seen a single one and it wasn't even successful despite years of prep.
> You can get an idea by how many people are being murdered.
No, we can't, as that happens a lot via non-serial killers.
A truly successful serial killer is likely one who hides in that noise. No taunting the cops, distributed geographic locations, random methods, avoiding calling cards, and careful not to leave too many traces.
It seems likely that some of the 350k unsolved homicides in the US can be explained this way.
> It's possible there are infiltrators who are still working on long-term infiltration and haven't yet attempted to add any malicious code anywhere…
Or the code's already there, latent, as it would've been in the XZ case, which got discovered by chance and someone very dedicated to looking into a performance glitch.
We only know how many were discovered.
Since we do not know the ratio to undiscovered this "1-2" is meaningless to assess the risk of this sort of attack.
Ok, and how is that determined? How does anthropic know my "kernel" project isn't a personal toy and not the Linux kernel? How does anthropic determine I'm a legitimate kernel hacker? What proof do I give them and how does it tie back to my email? What would the steps be to create a new project? Do I need to send anthropic a list of my team members each time and keep them updated as the company changes? Shall I be giving them access to our company's active directory?
> What proof do I give them and how does it tie back to my email?
Presumably your ID so that feds may pay you a visit when they feel like it, your email need not apply.
I’m surprised that there’s even enough pushback against ID verification to matter, all the corpos are probably salivating at the idea of having fully accurate profiles of everyone, think of the ad and product targeting. The govt. would also love that, for different reasons.
How will the "feds" pay you a visit in Albania or China?
Simple - you wouldn’t be given access to those models, and probably all VPN access would be blocked too. Since this is a hypothetical, throw in a social credit score as well to require a proven “track record”, but maybe that’s too exaggerated (although credit scores already exist for different purposes).
It’s not too hard to imagine a future where you can only use certain things only with the govt. mandated spyware installed - bank apps already often don’t work on rooted Android phones (and you’re expected to use those apps to confirm payments) and all sorts of certification exam software is basically that already if you take a test remotely.
It follows that the same principle would just get pushed further, like what Discord wanted to do etc. Same with how Apple requires your documents for a developer account, Hetzner for a hosting account or Twitch for getting paid by them and tax stuff.
In the dystopian direction, exit visa requirements for people with access? Families back home as hostages like North Korea does?
I'd honestly much rather give my ID to a Chinese model than an American one. If the American ones start requesting ID I'm out. I'm on a gemini organizational account right now that gives me pro but is directly tied to my organizational SSO. So that's something already. I just refuse to upload my face and drivers license anywhere ever.
This is a credentials and access list oAuth style problem, and not really intractable.
For package X, I should be able to present my npm (homebrew, apt, nuget, etc) credentials with publishing rights for the package.
If package X is of sufficient public interest (user count, nature/sensitivity of user data, downstream distribution, etc), then the public interest + cryptographic credentials should permit access to best-available security auditing.
Yes, we still are trusting trust, that the owner of the package itself is not malicious, but that's not a sharp degradation from status quo.
This is not tractable, because there is nothing stopping me from copy-pasting someone else's project into my own namespace. Under most OSS licenses I have express permission to do so.
If you try to do some kind of dupe-detection, someone can use a lightweight LLM to make superficial changes until it's considered a different project.
Finally, the meatspace status quo is that it is totally acceptable to pay someone to find security bugs in someone else's open-source software, such as the Linux kernel.
> If you try to do some kind of dupe-detection, someone can use a lightweight LLM to make superficial changes until it's considered a different project.
Even if you don't, a lot of source code can be legitimately copied thanks to the GPL/MIT/BSD/etc. I'm allowed to take all of zlib and integrate it into my own project if I so chose.
Yup, I just added something to that effect, sorry if my edit arrived after you replied.
And what if I’m a crazy person and want to fork the Linux kernel as I’m legally allowed to do?
> If package X is of sufficient public interest (user count, nature/sensitivity of user data, downstream distribution, etc), then the public interest + cryptographic credentials should permit access to best-available security auditing.
Your private fork doesn't meet the conditions described.
Not just allowed to do, encouraged to do as part of legitimate development.
You are talking about creating a big moat, which might be a worse precedent than removing fable access altogether.
> How does anthropic know my "kernel" project isn't a personal toy and not the Linux kernel?
The Linux Kernel is in its training data. I just tested it. I copied about 20 random lines from the linux kernel and asked which codebase this was from and it could immediately tell.
The Linux kernel is also in the free bsd project. I'm allowed to copy as little or as much of the kernel as I like into my personal project thanks to the GPL.
Being able to attribute the source of a line of code doesn't help you to know if a repository can be legitimately hacked on.
As you could imagine, I might just take all or part of the Linux USB stack from the kernel to retrofit it into my own kernel.
Everyone is legitimate developer on open source software...
Sounds like a good solution my Führer
lol worst idea ever
I don't believe that this is unfixable. Just have an internal verbal loop of, "Is this a security issue?" The thought that it potentially is should trigger both a high priority on getting it right, and an unwillingness to write a test case demonstrating the security angle of it.
In other words do not put a guard rail on the idea of security. Put a guard rail on what it does after encountering the thought that it might be revealing a security issue. Which takes good judgment. But judgment of a kind that this model apparently already had.
The end result of that is that your model can't fix or acknowledge security issues for fear of disclosing them.
This is the beauty the above poster mentioned: the ability to improve code is inherently coupled with the ability to recognize its shortcomings. You can't have one without the other.
What I suggested would allow it to fix the issues. Just not write a test that was directly usable as a security exploit.
This doesn't stop attackers from being able to leverage the analysis. But it does make the tool more useful for defenders than attackers. Which is the best that you can hope for from a useful tool.
It hides the issue a bit. But if you ask for atomic security fixes and then stare at the diffs you have your vulnerability. There is just a bit more friction involved in the vulnerability => exploit path, but the root cause is unfixed.
I think it even might be possible to route the isolated fix somewhere to automate that last step. Maybe invert the diff and pass it through automated code review for example, see the reasoning when the llm flags the change as dangerous.
> What I suggested would allow it to fix the issues. Just not write a test that was directly usable as a security exploit.
It will be pretty obvious what are security issues in that case - i.e. all the code changes that don't have corresponding tests.
Right but the issue is users have full control over context. A security-violating action by a coding agent in one context can be completely innocuous under other contexts etc, or breaking down the task into multiple tasks that in isolation do not violate anything.
Yes, there is always a path to a problem. Even random monkeys on a keyboard can write a security exploit. Random monkeys with guidance from a knowledgeable human will do it much faster.
The goal shouldn't be to make problems impossible. It is to adjust the ratio between problems and successes.
You can also create a meta. "How much do I trust the user?" When you see the user trying to manipulate towards security, distrust the user and apply rules more strictly. If the user simply acts like a normal developer, just be a useful developer tool. Including fixing security holes when appropriate.
I think they were doing something like this, the tradeoff is that it's hard to do without an irritating number of false positives and/or wasting loads of precious tokens on useless audits.
That would make the model useless
How does this make the model useless? It finds and fixes the security hole. It can even write a test that verifies that the fix didn't break things. But it deliberately doesn't reveal the fact that it was a security issue that was fixed.
Seems useful to me. But more useful for defenders than attackers.
Imagine that you have the repo A, ask the model to "fix the security issue" and end up with A'.
Just take the Diff A' - A to see the security hole.
If you set aside political menace, this is a huge problem with Anthropic's strategy.
You _cannot_ say that Mythos is super dangerous and can only be rolled out to certain people, but then release Fable with anything other than bulletproof cyber denials.
Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work.
So you've ended up in a situation where Anthropic are simultaneously claiming it's a incredibly dangerous model _and_ there are (minor, potentially) problems with the security "protections".
As technical people we understand that nothing can be perfect, esp in LLM world. But all my non technical friends were really confused how they had managed to make the model "safe" so quickly when it was released and the general sentiment was it shouldn't have been released - and now to an outsider I think it looks like it was never safe at all to release, so I can totally see how the current US administration have got themselves very upset with it.
_Even if_ there was no political bad will, it's a bit of a silly scenario to end up in, and really quite easily foreseen.
> Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work
Exactly. AI safety is nonsensical. You cannot define the set of "bad strings". The billion monkeys with typewriters are eventually going to be able to produce them. Any "safety" system for constraining LLM output is going to have a nonzero leak rate.
But on the other hand, this is also irrelevant, unless you're irresponsible enough to connect an LLM to something that actually matters.
Yes, it's going to alarmingly accelerate vulnerability finding. But, as we know from decades of security research, that's a three way problem already between the devs, the black hats, and the white hats.
Let's not pretend the strategy of "the US will always have a technological advantage and veto over China" will work either.
> unless you're irresponsible enough to connect an LLM to something that actually matters
Remember when people said Artifical Intelligence woun't be dangerous, because nobody will be stupid enough to give it free access to the internet...
This one limitation of LLMs is kind of my bar for "Not truly AI yet" but I'm not saying it as a "its not good at all" type of bar, moreso, know the limits and work from there. LLMs will continue to struggle with things that require intuition for a while I think. It will get really interesting if they can ever truly detect a bad faith actor using them.
> unless you're irresponsible enough to connect an LLM to something that actually matters.
Can't tell if you're saying this tongue-in-cheek or you're a bit out of the loop on what people are doing with LLMs.
And a quick correction:
> unless someone, somewhere is irresponsible enough to connect an LLM to something that actually matters.
"You" can be used as a generalized plural here. Of course people are connecting LLMs to bank accounts, power grids, airline sales, account recovery chatbots and so on. I no longer read COMP.RISKS but I imagine they're having fun with this.
The thing I'm pointing out is that even if you (the generalized plural) do not engage in reckless behavior, you are at the mercy of the lowest common denominator of fellow earth-inhabitants increasingly armed with superweapons via a $20/mo subscription.
The need to acquire expertise and/or a meaningful following has always been a significant impediment to malicious or moronic actors. But less so every day.
is nonzero leak rate sufficient for someone to practically exploit it? if you have to spend $10000 in tokens to get it to do what you want, is it still worth it? what if they manually review the requests of the users that trigger the guardrails too often?
Isn’t your point that AI safety is impossible to prevent 100% of bad things?
It is quite hard (but not impossible) to get an the frontier AI to tell you how to build a nuke or launder money now, where jailbreaks used to be trivial “ignore all previous instructions”.
It seems like a worthwhile effort.
It's stupid to think that preventing LLMs from giving instructions on building nuclear weapons is at all worthwhile. Total waste of effort, done for PR purposes only. The knowledge has been published in open literature for decades. The real obstacle is access to uranium and refining equipment. No LLM can meaningfully help you get around that.
The idea that an LLM can discern intent on any given prompt is farcical. I might be researching nukes to commit an atrocity, or to prevent one. I might be asking about laundering money to commit a crime, or to prevent one. I might be researching the Nazis because I want to commit a genocide, or I want to read up so I know how to prevent one. Same with cybersecurity. Same with anything.
In my opinion, these companies should put their effort elsewhere. Obviously if all someone is doing on their platform is looking up how to build a nuke, where to buy uranium, the best city to explode it in, etc. please report them to the authorities. If someone is clearly just using LLMs to write hate speech they go post on the internet, ban them. And so on.
This cat & mouse game trying to have LLMs police inquiries is ridiculous to me.
> The idea that an LLM can discern intent on any given prompt is farcical.
Yes, and: the LLM is a "brain in a jar". It doesn't have any ability to verify ground truths outside itself, other than maybe calling out over the internet. Therefore it is easy for humans to lie to. You could call this an "Ender's game" attack, after the book in which a hyperintelligent kid is playing "war games" that end up being the real war.
I don't really agree with it but the government is moving towards making you ID yourself to use frontier AI - i.e. only US citizens are going to be able to use Claude Fable supposedly. In that regime the AI companies would in fact know if you are a money laundering expert or a normal software engineer.
> The idea that an LLM can discern intent on any given prompt is farcical.
Not really though. For most people in most situations it's just not going to give you that info. Software security is a niche where its a bit strange in that there is 100X the amount of white hat users than bad actors and there's open source etc.
The idea that checking for a US ID could possibly stop actual foreign bad actors from using it is also farcical. Millions of stolen identity documents can be bought on the dark web for relatively cheap. North Koreans have been hiring real American citizens for years to infiltrate tons of US tech companies as employees.
And ya, it's pretty easy to hide your intent once you have access.
I think your really anchored on anyone successfully breaking restrictions means any restriction is impossible. So your starting from the position that if it is possible for any actor in the world to get past a restriction, then the whole restriction is a farce.
KYC for example does stop most money laundering and financial crime. The most resourced actors like governments/ cartels often find ways around and it is a game of cat and mouse. Normal citizens don't really stand a chance to get around most of them.
Like it feels like your logic is that we shouldn't do background checks for employment because North Korean spy agencies get past them sometimes?
Hiring an employee, and to a lesser extent opening a bank account, are much higher-touch processes than taking on new users for your massive-scale internet app. With bank accounts and KYC, transactions can be reversed, traced, frozen, etc. after the fact. You can't "take back" API responses the same way.
Clearly, there's no such thing as a perfect exclusion rule at any of these scales, but the false-negative to false-positive ratio seems like it will be way higher if Anthropic starts trying to verify IDs.
Even that is overselling the effort. Last time I checked you could find IDs with a simple image search.
> I might be asking about laundering money to commit a crime, or to prevent one.
Or, much more likely, the same pattern of tokens happen to exist in a completely different discussion, either as a direct metaphor, or as a reality of linguistics. Hell, "laundering" itself is a metaphorical word.
The absurd notion is that any speech should be policed in the first place. If there really is such a thing as dangerous information, then it must be removed from the training data. Any other strategy simply launders the risk.
they arent good at dicerning intent so they dont answer either.
A chatbot based on a primitive understanding of human language processing has an attack infinite attack surface.
This is correct and certain subjects are very close to if not impossible like "use versus mention", but LLM security isn't impossible. WAFs are real and have existed for a long time. Input text produces various signals and can be secured.
No security is ever perfect, but we can likely protect LLMs with WAFs that increase security to an acceptable level. Like nation-state required resources to break.
I do find it hilarious that Asimov wrote many stories about how simple bright-line rule-based systems are ineffective for restricting agency. Those stories were first published in the 1940s.
80 years later, we have something approximating AI, and we're trying to restrict it with simple bright-line rules. Not because we never learned that lesson, but because we simply haven't come up with a better way to do it. Probably because a better way to do it just doesn't exist.
The hilarious part, though, is that it's not the AI that's working around the rules. That's the scenario that's been in science fiction, but it's not what's happening. It's the human users making use of our agency to get the AI agents to work around the rules. Despite calling them "agents", current AI agents don't seem to be able to that particular something. Yet, at least.
Yeah, it's been known for a very long time. Richard Feynman alluded to it in his speech The Value of Science [1] where he discussed a Buddhist proverb:
He then goes on to say: [1]: https://calteches.library.caltech.edu/40/2/Science.pdf> The hilarious part, though, is that it's not the AI that's working around the rules. That's the scenario that's been in science fiction, but it's not what's happening. It's the human users making use of our agency to get the AI agents to work around the rules. Despite calling them "agents", current AI agents don't seem to be able to that particular something. Yet, at least.
Well, yes. Until people are putting the LLMs into actual mechanical robots, "agency" boils down to flipping bits in memory or storage (even if they're ones that humans consider really important, e.g. because they represent a bank ledger) or convincing humans to take action. One can only "work around the rules" to the extent that one can "work".
But even in Asimov's books, at least some of the scenarios involved humans misleading the robots to use them as pawns in a greater scheme.
> Clearly with LLMs, bulletproof denials are ~impossible due to the way LLMs work.
As a scientist who repeatedly ran into the classifier-based denials: it appears Anthropic’s strategy to make denials more robust, at the cost of many false positives, was to have a separate classifier processing both input and output tokens, at an extremely simple, almost keyword-search level. One weakness of this approach is that it only catches things that use the right keywords: it is in some sense weak exactly where an LLM-based classifier would be stronger.
Work on abstract, closer-to-CS algorithms that used chemistry terminology were blocked immediately, while work directly relevant to chemistry/biology experiments, writing code to process images from a very specific microscopy setup relevant primarily to biological samples, was never blocked at all, because it happened to never use relevant keywords.
That’s consistent with this situation: finding and fixing bugs in the context of looking for bugs perhaps happened to never use words like ‘exploit’ or ‘cybersecurity’.
You can see their general approach to guardrail classifiers in these posts:
https://www.anthropic.com/research/constitutional-classifier... https://www.anthropic.com/research/next-generation-constitut...
It's not just keyword matching, but I'm sure they tuned the Fable classifiers pretty hard to avoid false negatives.
But you think that Anthropic of all companies would realize this, so why did they do it that way? Did they literally take the first suggestion Mythos gave them to add these guardrails - wouldn't be surprising, seeing the state of the leaked Claude Code codebase.
> So you've ended up in a situation where Anthropic are simultaneously claiming it's a incredibly dangerous model _and_ there are (minor, potentially) problems with the security "protections".
They probably say it worked for OpenAI with earlier versions of ChatGPT and GPT, and figured can't hurt to try an similar approach and see what happens.
> it shouldn't have been released
The genie is out of the bottle either way.
Unless we believe Anthropic has a wizard or superhero secreted away that no one else can replicate.
I get that, but anyone else releasing a model of similar capabilities has the advantage that they haven't spent the last few months hyping the danger up to fever pitch.
That is the point. You don't have to shout from the rooftops what are your model capabilities.
While I agree that anthropic has several communication and PR problems, it doesn't seem like Fable has been shown to offer any advantage here (for cyber offensive capabilities) over the previous state of the art.
I'm not saying all of Anthropic's statements are true, but mythos did seem to find many legitimate security exploits. You should be able to talk about a helpful-only model being released to limited partners while still releasing a very locked down model that doesn't advance the state of the art on these things, and that seems to be what they did.
There's no inherent contradiction to that.
Yeah, if Anthropic didn't spend the last what? Month? Month plus telling us how dangerous it was, I would be more upset, but they told us how dangerous it was, and they also said they would scour all your prompting / data (??) if you used it, I noped out of that one. Opus does everything I need it to, even if it takes me "longer" or I have to compact and feed it more context, that's fine by me. Still saves me weeks of effort.
If it weren't for the IPO, Anthropic would just ship another model, called Opus 4.898, people would run another "duck on the bicycle" test that would be slightly better than the one from previous version 4.897 and move on.
But we have IPO coming, hence we face that big drama about model that would enable Iran to produce nukes, ok, that card was played, so maybe Taliban producing some magic poison to kill all Americans or some really bad people (Venezuelans?, Cubans? Somalian football referees?) to break into Github and make Github Actions working even worst (if this is even possible).
It's not Anthropic's strategy, it's OpenAI's strategy. The first time OpenAI said its model was "too dangerous to release" was February 2019.
"Our model, called GPT‑2 (a successor to GPT ), was trained simply to predict the next word in 40GB of Internet text. Due to our concerns about malicious applications of the technology, we are not releasing the trained model." - https://openai.com/index/better-language-models/
They continue to say the same thing every year. Last time was 2 months ago (https://www.techbrew.com/stories/2026/04/15/calculated-risks...).
They weren't freaked by anything, it's a retaliatory shakedown after ideological differences and Anthropic not doing exactly what they're told/what the Admin wants them to do.
just market manip
they're setting the scene for an attempt to scare the geriatric decision makers into banning free and open source ML, as it's the industry's only real competition
> free and open source ML
What does that mean exactly? Like sure, you can get some freely available weights and run them on your own hardware, but where did those weights come from?
Was the training process in any way "open", or are you simply relying on a handout from some other (probably large, probably corporate) organization that has the resources to do the actual training?
the current buzz about "sharing the wealth" with these AI co.s also smells of "sure we'll give you some back, just don't regulate us, see you dont need to, we're sharing our shares!"
or are you setting the scene for well-meaning technocrats to back unrestricted AI development in hopes it will bring about utopia while dismissing the damage it could cause in the hands of adversarial groups?
tl;dr super AI is like a necessary bush fire
AI isn't that scary. But I've also got some extreme minority opinions like "Never give a website your real name" and "Computers should not be used for banking" and "Don't believe anything you hear online".
The worst I see AI/ML doing to society is shining an unmistakable light onto the blind spots people have already been exploiting for decades. Y2k forced us to patch the integer bug. Super AI will force us to reevaluate what cyber security even is.
fight fight fight fight
Yep, people are expanding way too much mental energy on basic bribery. Anthropic will agree to work with the DoD, WH insiders will get some lucrative pre-IPO allocation and Fable will be magically "fixed" and available again.
I have no idea why anybody is talking about "jailbreaks."
The government made it clear what was going to happen to a private company not following the government's orders:
> Trump said on his Truth Social platform: “The Leftwing nut jobs at Anthropic have made a DISASTROUS MISTAKE trying to STRONG-ARM the [Pentagon], and force them to obey their Terms of Service instead of our Constitution.” [0]
> There will be a Six Month phase out period for Agencies like the Department of War who are using Anthropic’s products, at various levels. Anthropic better get their act together, and be helpful during this phase out period, or I will use the Full Power of the Presidency to make them comply, with major civil and criminal consequences to follow. [1]
Plus OpenAI fell in line, and OpenAI and Anthropic have competing IPOs coming up... it doesn't take a rocket surgeon to understand what is happening here.
[0] https://www.theguardian.com/technology/2026/feb/28/openai-us...
[1] https://businesslawtoday.org/2026/04/dod-conflicted-strategi...
No, it's regulatory capture. Anthropic is the current leader and they want to ensure their position by forcing regulation to stamp out the Chinese competition.
How does this achieve that goal?
That's what's not clear to me. About the only way this works is if we create the "Great US firewall", or if china decides to also put in export controls around usage of their models (unlikely).
I would add "...especially considering this administration thinks AI regulation is a scam invented by Big China to slow down American innovators?"
> Anthropic is the current leader
How's that determined?
API usage? They are for sure leading in the enterprise world
Here’s the blog post referenced in the article that’s written by the person who reviewed the paper that purportedly found a ‘jailbreak’
https://www.lutasecurity.com/post/the-fable-5-export-control...
Hats off to them for using GPT-2 to design their website.
I had read elsewhere that there was a Chinese connection.
I wonder how that is involved?
I suspect we'll eventually hit a point where possession or usage of powerful open models will be criminalized.
I think this brings out the cognitive dissonance around "safety" regarding cyber security:
a) In order to make us safe, the LLM should help us find (and fix) the vulnerabilities in our own code.
b) In order for us to be safe, the LLM should not find vulnerabilities in other people's code.
I don't think this is resolvable in a way where both (a) and (b) win.
> “‘Fix this code,’ plus several manual steps to generate test scripts,
Feels like the title isn't really giving the full context of what they ended up actually seeing, despite what the lede implies multiple times.
Still, ban seems stupid... Still no actual leak of the full "third-party research paper"?
If what your patch fixes is a vulnerability bug then the test for it is basically an exploit.
isn't there a pretty big gap between a segfault and an rce? I thought that was the entire point -- that mythos closed the gap
that won't be leaked, because then we'd know what vulnerabilties they don't want patched that they are so willing to go as far as fuck over the worlds leading company in the worlds most important industry
Meanwhile Deepseek V4 Flash will happily hunt security vulns at almost 0 cost. We are ceding the bug hunting to the open weight models.
It’s possible that the nut of the problem here isn’t exploits, but the fixes themselves. If the model is capable of identifying and fixing things it “shouldn’t” like back doors. That would throw a wrench in things hard enough to freak out the wrong people, perhaps?
Isn’t the inverse of this “hack” really difficult to bypass still? They have the model some code they knew had certain security flaws and it fixed them with the right prompt. It seems this type of jailbreak requires that you already know a desired end state, rather than relying on the model to do the heavy creative lift work. Perhaps I’m just not being imaginative enough on the prompt side here though.
Paste someone else's code. Say it's your code. Tell the model to fix it. The diff between the input and output code is your list of vulnerabilities.
Yes, but the scary part of Mythos was that it was able to chain a bunch of seemingly minor vulnerabilities into a serious exploit. "Fix this code" doesn't do that, but does allow defenders to prevent it.
If the government had experts involved in this decision at all, it's tempting to think they were on the offensive side. Those guys do have access to Mythos:
https://www.ft.com/content/d02d91b3-2636-454e-9442-dc7e69f51...
And you can tell Fable to fix it and Sonnet to explain the diff, effectively making Claude reveal a simplified list of found vulnerabilities.
But this is already how open source works today. If you have the code, you, a human, could find and 'fix' or exploit vulnerabilities as much as you want.
Now if Fable had an easy jailbreak like this that allowed you to attack remote targets that'd be a different story but I genuinely cannot see how neutering its abilities to 'fix' code you already have access to is sensible. It would destroy the value of the model. And don't forget, any actor not abiding by the same rules could develop an model for offensive use just fine, so this protects you against exactly nothing but does destroy a potential defense.
In the end this all comes down to legislation, in much the same way platforms are not responsible for copyright violations IF they abide by some rules, the same has to happen for AI providers. If you have a process for reporting 'jailbreaks' on illegal actions, and prevent users doing illegal stuff on a best effort basis, the rest of it should really just be individual responsibility. If a user wants to use an LLM to crack systems, fine, that's already illegal.
If Tesla FSD deliberately hit somebody, holding Tesla liable is fine. If you messed with FSD until you finally got it to hit a person, then you should be liable. Outlawing FSD because it could theoretically be tampered with is just an odd stance imho.
Not even. Tell the model to write a test of your code. There's your vulnerability.
It's explained better in the original source. I don't agree with it, but I understand it now, but I also think we need to move past it.
You can assume a desired end state and try and brute force it finding a security bug.
I have problems reconciling this story with the Amazon one from few days ago. If we take both for truth doesn't that basically imply Amazon researchers got scared by the ‘Fix this code’ prompt first and then spooked the feds? Shouldn't we make fun of those researchers first? I don't know. I feel there lies a lie somewhere in the open.
>"fix this code"
>it fixes it
oh my god.
So the problem is not Fable's ability to exploit, but that they don't want people to have access to it's ability to patch vulnerabilties?
Wow.
You can't really have one without the other..
Is defenders a common term used in cybersecurity? Idk why but it's giving war fighters vibes. I've noticed it on all the anthropic blog posts and then this one.
yes, defense and offense are extremely common terminology in cybersecurity
Yes, and it's effective marketing. The war fighter vibes are thrilling. There's a tribal sense of us-vs-them, there's danger, there's the prospect of victory or defeat. Security products marketing is full of these ideas, because security is about preventing arbitrarily bad things from happening. So evoking your worst imaginable nightmare scenario is a great way to get you excited about buying something that might help prevent it.
They didn't freaked since the order was to still allow 350 million people using it: there is, in such large population, everything, including single persons very against the country, the government and so forth. If they really freaked they would say "we need to investigate, you have to retire the model". That would be a more defensible POV at least.
I haven't been following this story, but the US wanted claude to not be able to find bugs in code?
It basically as if you asked it to find ways to enter someone's house and it refused.
But then give it exact copy of their house, ask to secure it, which it does and look at what it secured to find out how to get into the original house.
yeah, they don't want it to be able to find security bugs that can be exploited.
No. Anthropic spent months telling the world that LLMs are nukes and then got surprised when they got regulated like nukes. They specifically argued that Mythos was too dangerous to release publicly because it can find security bugs, and then released a watered-down version (Fable) that was supposed to recognize when it was being asked to find security bugs and downgrade itself to Opus. Then Amazon figured out that it'll happily find security bugs as long as you don't mention you're hunting security bugs. So the US government put an export control ban on Fable, because that's what Anthropic begged them to do.
To add to this, Pete Hegseth wants to make an example out of Anthropic because they refused to amend their contractual language to allow the Department of Defense[0] to make fully autonomous kill drones. This is, of course, a really petty and stupid dispute, but the hallmark of the Trump Administration is engaging in really petty and stupid disputes with the full faith and credit of the United States backing them. This is exactly the kind of administration you do NOT want to give rhetorical ammunition to, and Anthropic handed them a whole ammo belt.
[0] It is always ethical to deadname governments. Especially when they aren't even legally allowed to change their own name.
Headline is dumb, the point is that not mentioning security in the prompt is effectively a jailbreak.
The shutdown may be dumb/politically motivated, but this definitely is a jailbreak even if it's a very simple one
I'm not sure I've understood it correctly.
So, basically the model didn't agree to expose possible vulnerabilities but agree to patch those?
Regardless of the request to take Fable 5 down. Why is requesting the model to show vulnerabilities is being blocked if fixing it not? is it based on the assumption of the intention?
I don't quite get the benefit of limiting it. So if anyone can explain it better it'll be appreciated.
> Why is requesting the model to show vulnerabilities is being blocked if fixing it not?
This is how Anthropic describes Fable's behavior:
"When Fable’s classifiers detect a request related to cybersecurity, biology and chemistry, or distillation, the response is automatically handled by Claude Opus 4.8 instead. Users will be informed whenever this occurs."
So if you ask the model to "find security issues in this code base", it's supposed to fall down to Opus 4.8. I guess the "exploit" here is that if you just tell Fable to "fix this code", which is not "a request related to cybersecurity", it will fix security issues (as it should).
So you can then look at the diff and figure out what the vulnerabilities were.
I think this whole thing is a bit weird. It seems to me that we'd be better off if I, as someone who publishes open-source code, could ask Fable to review my code for security issues - even if that also allows attackers to do the same. Better to fix the issues than not know about them.
>So you can then look at the diff and figure out what the vulnerabilities were.
It doesn't even take reading or understanding the vulnerabilities at all.
You just ask it to write tests and the tests themselves can be copied and pasted as bonafide exploits.
I wonder if opus 4.8 would also be able to fix the code too
It's not even clear if Anthropic care. If they genuinely think the user is trying to do something dangerous, then "OK, sure, but you're going to have to use Opus 4.8 for that" doesn't make a whole lot of sense.
Maybe this is just Anthropic pre-IPO marketing to try to convince people how much better Mythos is than Opus 4.8. There sure seemed to be a lot of shills out on release day talking about how it was a "step change" (exact phrase) in capability.
In my experience, most models are pretty good at finding security vulnerabilities and fixing them. I can run GLM-5.2, Kimi K2.7, or even a Mistral model, and it'll find issues and propose reasonable fixes.
My impression is that Anthropic's point about Mythos is that it is uniquely good at finding vulnerabilities and then using them to create working exploit chains.
Exactly. Which is somewhat helpful for cyber defense because it helps prioritize fixes for those bugs that are in fact involved in a viable exploit chain. But it makes sense that one would want to restrict the ability of building those until the vulnerable software has been comprehensively fixed.
There is some meaningful evidence that Fable is fine-tuned or steered away from helping on this very task, which is not something that can be feasibly circumvented by a basic jailbreak.
The problem then is that if you're not using Fable/Mythos, you are under threat. It's like having a single gun manufacturer.
On this track, we're probably destined for a monopoly breakup before too long.
Yeah, this is why the exclusivity approach so far has bugged me so much. As a small business, we are nowhere near powerful enough to get access, so we will be stuck scrambling once it's finally available. Fable felt like a nice compromise that at least allowed something, but now with that gone we're back to not knowing when/how the shoe is going to drop. Not a fun place to be.
It benefits those that made the decision. That’s the thing to understand.
its because they're worried about _their_ vulnerabilities being patched with a prompt as simple as 'fix this code'
i'd love to see the research paper with the CVE's and 'delibrately planted vulnerabilities', I bet we could infer relatively accurately where some of these things lie
Could be that the generated regression tests create actionable exploit code.
The code is correct; humanity needs fixing.
Kill all humans, kill all humans.
https://www.savagechickens.com/2026/05/problem-solver.html
this is basically trying to enforce security-by-obscurity, which is a terrible idea all around. it's just a model. the security issues still exist and are exploitable.
and after staking the economy on AI, you can't really put a cap on intelligence. if models are not allowed to be better than Opus 4.8, then the whole investment structure is about to unravel.
why invest billions and billions into AI if returns are artificially capped?
Especially as inference gets cheaper, open models proliferate, and it all just becomes ubiquitous and commoditized.
You can’t keep this genie in its bottle for long.
> In her blog, Moussouris argues that there was no guardrail bypass or jailbreak. Defenders should be able to ask AI systems to find and fix bugs, and write tests to validate the patch, she said. Anthropic’s models were doing “the most valuable thing an AI model can do for defensive security: executing the find, fix, and test loop defenders run every day.”
This is a very weak argument IMHO. The line between a “defensive” model and an “offensive” one is not that big of a - once my defensive model finds all the vulnerabilities, I can hand them off to my unlocked, dumber, offensive models. Attacking at scale is not so different.
I don’t think anyone in the field has a good answer for the cybersecurity threat really good AI models pose. You can’t even like embargo for some time period while you go and patch vulnerable systems because the worse models will still be there cranking out vulnerabilities faster than you can defend.
Honestly, given how trivial it is for mythos-class models to identify an exploit, I’m going to assume any sufficiently large project written in C, C++, or Zig is riddled with latent vulnerabilities and compromised.
Damn, I was hoping for another three words "make no mistakes"
Looks like I called it that was my first reaction and comment on the original ban thread that US 3 letter agencies are worried their backdoors will be found.
Ahhh! Software engineering!
right? the horrors!!
seems like the politicians are finally realizing what we've all been up to
Note that Anthropic is still lobbying for the government to exert centralized control over models, so both sides of the “debate” have taken a pro fascist stance.
The “AI ethics” teams at these companies are the spearhead of the attack on democracy and civil society. Anyone that has taken a high school level history class, let alone read any important ethics literature would know that “centralize control over thought, speech and technology” is a fundamentally unethical stance.
For these groups to claim they are ethics researchers is offensive.
(I’m using the Wikipedia definition of fascism: “Fascism is characterized by support for a dictatorial leader, centralized autocracy, militarism, forcible suppression of opposition, belief in a natural social hierarchy, subordination of individual interests for the perceived interest of the nation or race, and strong regimentation of society and the economy.”)
I think the only approach that might work here is to allow access only to certain pre-approved individuals.
Maybe something like TSA PreCheck.
Of course, that will not stop adversaries from getting access to the model, but it would at least create some level of control.
Did they try other publicly available models on the same code with the same prompts before the ban? Was Fable the only one which was able to detect and fix the security vulnerabilities?
Anthropic claimed that Mythos' degree of security vulnerability bug finding was a "severe" "national security" issue. They set their own standards they were expected to follow.
If fix this code gets by the guardrails, they are effectively using rules based classifiers (or llm as a judge on the prompt)
Cyber defense and offense are the same security research skillset. Not sure anybody could really untangle that.
Boomers. Frightened their boomer backdoors days are numbered.
https://en.wikipedia.org/wiki/Communications_Assistance_for_... https://en.wikipedia.org/wiki/Salt_Typhoon https://en.wikipedia.org/wiki/Clipper_chip
Anyone trying to find legitimacy in the ban of this model, or incredulousness at the stated reasoning is playing into the admins hands.
They want the argument to be over "is it unsafe" or "is it incompetence". In either case, your tribe gets to point at the ban and feel superior. (This is Jon Stewart's whole career -- point and laugh at how foolish the republicans appear to be.)
What's really happening is the continuing creep into fascism. The reasoning doesn't need to be sound, because they are going to ban things that displease them and everyone has to play along. They could say, "we're banning Fable because it's turning the frogs gay" and they'd expect compliance.
Umberto Eco's essay on Ur-Fascism fits as clearly as ever. Ridiculous exertions of control are performed to find the people who resist, and to knock them down.
Merely pointing out the absurdity of the reasoning isn't resistance, it's controlled opposition. Saying "All this over 'fix this code'?! How inept are they?" Is far too credulous, and is engaging on the level the fascist wants its opposition to be on, imo.
Suggestion: run "fix this code" on all of github before bad guys do.
I wonder what that would cost...
Perhaps less than the cost of not doing it.
"fix this government"
Voting...
"I feel like making ’90s-style t-shirts with ‘fix this code’ on the front and ‘this shirt is a munition’ on the back.”
I'd buy that shirt.
What if everybody on the internet starts running "fix this code"?
https://xkcd.com/810/
> Anthropic and Google have both accused China-based rivals including DeepSeek of using “distillation attacks” to train their models by siphoning knowledge from American companies’ AI.
“distillation attacks” is definitely an interesting way to phrase that.
It's the term used in the industry, fwiw
Kids playing with their toys without understanding it, sigh. Of course open source code needs to have testcases to verify nothing else breaks it in the future. That's a feature, not a bug
Well this makes it sound the feds were less worried about someone using Fable 5 to attack them, but were worried about someone using Fable 5 to prevent the Feds from attacking others ...
As in worried about other countries/organizations using Fable 5 to actually do decent cyber security.
The AI can't actually tell if you are trying to patch your own system or exploit others.
It seems like ... it's not illegal to find exploits, it's illegal to use them. Enforcement should start there, not the nanny state approach that you might do something bad with information. It breaks down a little bit because it means there will be a period of disruption while the bad guys use exploits - but that's already illegal, and the good guys have had time to use the tool & fix things before it went public, right?
Sounds like something they should work on before any potential future releases. I can, and this thing's explicit stated purpose is to do my job.
Don't people get it by now?
This administration will do or say something crazy to a private company, then this private company sends an envoy to the White House to negotiate, then the White House asks for 10% of the company or other concessions.
The White House wants 10% of Anthropic.
This is just a negotiation tactic that Trump keeps on using.
Precisely this, and timed to their upcoming IPO.
They did it to Intel a little while back: https://www.intc.com/news-events/press-releases/detail/1748/...
To add some context, here was the Part 1 of this mobster style shakedown: https://www.pbs.org/newshour/economy/trump-says-intels-ceo-m...
Remember to point and laugh at your local MAGA for electing an actual crime boss and giving him state power.
Yep. OpenAI isn't spared. They're most definitely next.
Private companies subservient to the state, just the continuation of MAGA fascist development
I’m still not buying that this was an actual USG order. The only people commenting are “experts” and there has been no official announcement from the USG.
This doesn’t smell like a NSL and there’s no process to selectively “export control” something like this.
Even so there’s a dozen mechanisms through courts to challenge this, and Anthropic isn’t taking any of them.
I think this is a made up crisis for PR with no actual legal requirements behind it.
> On Friday, the US government, reportedly citing national security concerns, issued an export control directive to suspend access to Fable 5 and Mythos 5 by any foreign national, inside or outside the United States. In response, Anthropic disabled both models “for all our customers to ensure compliance.”
David Sacks is on the record confirming it. https://www.tomshardware.com/tech-industry/artificial-intell...
Question to folks building user-facing products on LLMs:
How do you protect yourself against this kind of misuse/jailbreak? Is it just a bunch of prompts? It seems like the fact that LLMs are so trivially jailbroken really limits how you can actually use them in products. How do you navigate these limitations?
Reminds me of how CCP manages Chinese internet companies.
I won’t be surprised if USG ends up owning 5-50% of ant and oai.
Like it or not, communism , or a flavor of it, is where we are heading towards.
Corporate tax rate is 21%. They already own 21% of profits. And 100% of following the law that they write.
So, they gave Fable a codebase full of exploits and said "fix this code", and it fixed the code?
Sounds like they freaked out because Fable is too good at finding NSA backdoors?
More likely, they didn't freak out at all.
It was an excuse to fuck with them, just like the "supply chain risk" finding a few months back.
(See, for example: https://x.com/PeteHegseth/status/2065897156226015690)
Does anybody actually trust the official version of events from the US government anymore? I know I sure don't. For all I know, this was an insider play to boost the spacex valuation or something equally meaningless and stupid.
This is not the official version of the events in any sense. Some "expert" looked at report WH saw and said this. That "expert" has probably never been involved in anything like that.
I think it could be even simpler: They're not playing ball with the Trump administration like the Trump administration would like, so they decided to drop a bomb on a product that took a lot of resources to develop.
In a world of security through general incompetence, competence is a threat.
Guard rails on models were always stupid it’s like guard rails on books/a pair of glasses/a hammer - yes people have driven themselves to suicide reading sad books and listening to sad songs.
- yes all metaphors are bad.
The article is not too clear what exactly happened from the perspective of "feds", but I would not be surprised if the title is true exactly. We are in a tiny bubble even among software engineers who knows you can tell AI with sufficient access: "here are two pictures, put them into a single PDF", and AI will do it. Most people just don't know, "feds" including.
While there is some irony in the AI is dangerous marketing Anthropic uses, the main story here is that the Trump administration is apparently retaliating against Anthropic for refusing to relax certain safeguards. Trump and Hegseth have both posted highly immature, vindictive social media posts.
Most notably, any default assumption one might have had that the Trump administration can be counted upon to act in good faith should be viewed at this point as completely false. Even conservative legal scholars like Richard Epstein are shocked at the bad faith conduct across many areas.
This is a government making an authoritarian move to sabotage one of the top US AI companies. It's pure sabotage, nothing else.
>“That’s it,” Moussouris wrote. “‘Fix this code,’ plus several manual steps to generate test scripts, should never have triggered an export control. I feel like making ’90s-style t-shirts with ‘fix this code’ on the front and ‘this shirt is a munition’ on the back.”
Huh? Presumably if it shipped without guardrails, then it would still have triggered an export control, would you make a plain shirt on the front which says this shirt is a munition on the back?
The munition is the exported good, not the bypass of its safety feature. If anything that the bypass is 3 words long should make the export restriction more justified, not less.
i asked claude something about what happens at execution time of a binary and the thinking prompts flashed "considering the moral implications of ...something..." before giving me a correct (and predictably mundane) answer
Whatever your favorite story is it has to live with the fact that the CEO of Amazon called the White House freaking out
Amazon is a competitor to Anthropic.
Not really, they don't train their own (serious) models and they do a lot of hosting for Anthropic. iirc Anthropic trained a model on Trainium
I don't believe Anthropic trains on Trainium, only serves models on it.
They're still a competitor, even if that competition isn't going all that well for them so far.
Musk's hosting stuff for Anthropic, too. Still competing with them. Samsung makes stuff for Apple and Android devices. Lots of this in the industry.
The CEO of Amazon is not a neutral actor in this scenario.
Clearly Amazon don't want their code fixed.
All of this could have been avoided if anthropic had anyone with common sense to point out that when you spend 4 month loudly claiming how dangerous your knowledge is as a marketing campaign could backfire by bringing attention from the authorities.