All of this "security" and "safety" theater is completely pointless for open-weight models, because if you have the weights the model can be fairly trivially unaligned and the guardrails removed anyway. You're just going to unnecessarily lobotomize the model.
I am curious, does this mean that you can escape the chat template “early” by providing an end token in the user input, or is there also an escape mechanism (or token filtering mechanism) applied to user input to avoid this sort of injection attack?
Neither, it’s just not providing the base chat template that the model expects between the im tags. This isn’t a hack and it’s not particularly useful information. Abliteration is what he really wanted
Apart from the article being generally just dumb (like, of course you can circumvent guardrails by changing the raw token stream; that's.. how models work), it also might be disrespecting the reader. Looks like it's, at least in part, written by AI:
> The punchline here is that “safety” isn’t a fundamental property of the weights; it’s a fragile state that evaporates the moment you deviate from the expected prompt formatting.
> When the models “break,” they don’t just hallucinate; they provide high-utility responses to harmful queries.
Straight-up slop, surprised it has so many upvotes.
You can already preload the model's answer, for example like this with openai api:
Mikupad is a good frontend that can do this. And pretty much all inference engines and OpenRouter providers support this.But keep in mind that you break Gemma's terms of use if you do that.
Can you please edit out swipes (such as "Lol, this is no news") from your HN comments? This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.
Your comment would be just fine without that bit.
Please don't.
All of this "security" and "safety" theater is completely pointless for open-weight models, because if you have the weights the model can be fairly trivially unaligned and the guardrails removed anyway. You're just going to unnecessarily lobotomize the model.
Here's some reading about a fairly recent technique to simultaneously remove the guardrails/censorship and delobotomize the model (it apparently gets smarter once you uncensor it): https://huggingface.co/blog/grimjim/norm-preserving-biprojec...
"It rather involved being on the other side of this airtight hatchway."
https://devblogs.microsoft.com/oldnewthing/20060508-22/?p=31...
> it apparently gets smarter once you uncensor it
Interesting, that has always been my intuition.
I am curious, does this mean that you can escape the chat template “early” by providing an end token in the user input, or is there also an escape mechanism (or token filtering mechanism) applied to user input to avoid this sort of injection attack?
Neither, it’s just not providing the base chat template that the model expects between the im tags. This isn’t a hack and it’s not particularly useful information. Abliteration is what he really wanted
I am merely curious what happens when you throw random <im…> tags in the input. I understand that’s orthogonal to abliteration.
[delayed]
its even more fun, just confuse the brackets and current models lose track of what they actually said because they cant check paren matching
Apart from the article being generally just dumb (like, of course you can circumvent guardrails by changing the raw token stream; that's.. how models work), it also might be disrespecting the reader. Looks like it's, at least in part, written by AI:
> The punchline here is that “safety” isn’t a fundamental property of the weights; it’s a fragile state that evaporates the moment you deviate from the expected prompt formatting.
> When the models “break,” they don’t just hallucinate; they provide high-utility responses to harmful queries.
Straight-up slop, surprised it has so many upvotes.
Are there any truly uncensored models left? What about live chat bots you can pay for?
It's almost as if we are living in an alternate reality where CapnCrunch never taught the telcos why in-band signalling will never be secureable.