My suspicion is that the spam filter programmers didn't do a comprehensive evaluation of every code point on every plane of Unicode because... well that's a massive job. So your "sanitize and denormalize" tasks are actually massive mappings which were likely imperfectly created.
Whoever thought it was acceptable to have a string of text that renders something unreadable or not immediately obvious to the human eye, was a complete moron.
I realize ASCII was limited, but one thing I like about it is I can understand every single character code, and program to handle all the edge cases with certainty.
> I realize ASCII was limited, but one thing I like about it is I can understand every single character code
It's great for teaching and other things, but everyday life is filled with many characters, is the suggestion we'd have one ASCII per language where there is more distinct characters, or what would we do? I don't see what else we could have done, that would have worked for the world, but I'm curious to hear ideas.
The tags were needed for language indication to control CJK glyph variants. Flag emoji were grafted onto this scheme. The key is that tag sequences have to start with a valid introductory codepoint. Simple enough to strip out anything that isn't a flag.
A few years ago, during a particularly spam heavy period, I got pissed at Google's ineptness at combating spam, and decided to have a think about how I could get rid of the problem for good.
What I arrived at:
- I should never hand out my actual email address. As in, should be all proxied, every contact reaching me via a different address only known to them. Ideally, these aliases would be generated using a CSPRNG, and would be of sufficient length. Allows for tracing contact provenance in "space".
- I should rotate that address periodically, if possible. Allows for tracing contact provenance in "time".
Together, this would tell me who's the source of particular influx of strange mails, and when did they leak the contact address provided to them (willfully or otherwise).
I was really taken by this idea, even wondered why this has not been baked into the underlying protocols over the years, to make it readily deployed and available for all, making it effortless.
Because boy is there an effort involved! Several years later, I now avoid spam mail by simply no longer reading my emails anymore...
Sometimes I wonder what corporate IT thinks when I pass their anti-phishing test a month after the test campaign.
Why not sanitize and denormalize (or whatever it's called) the text before feeding it to the spam filter? Or the LLM prompt?
Do you have any evidence they didn't?
My suspicion is that the spam filter programmers didn't do a comprehensive evaluation of every code point on every plane of Unicode because... well that's a massive job. So your "sanitize and denormalize" tasks are actually massive mappings which were likely imperfectly created.
Whoever thought it was acceptable to have a string of text that renders something unreadable or not immediately obvious to the human eye, was a complete moron.
I realize ASCII was limited, but one thing I like about it is I can understand every single character code, and program to handle all the edge cases with certainty.
The top comment (the Staff Highlighted one) explains why this range of code point exists.
There was a rationale (ISO country codes to modify a flag to display that national flag).
Maybe the problem wasn't the proposal, but the lack of the ability for others to reject it for being insecure.
> I realize ASCII was limited, but one thing I like about it is I can understand every single character code
It's great for teaching and other things, but everyday life is filled with many characters, is the suggestion we'd have one ASCII per language where there is more distinct characters, or what would we do? I don't see what else we could have done, that would have worked for the world, but I'm curious to hear ideas.
The tags were needed for language indication to control CJK glyph variants. Flag emoji were grafted onto this scheme. The key is that tag sequences have to start with a valid introductory codepoint. Simple enough to strip out anything that isn't a flag.
Even classic ASCII has "unreadable" control codes, but to be fair they would not be confused with text even by an LLM. Well probably not.
A few years ago, during a particularly spam heavy period, I got pissed at Google's ineptness at combating spam, and decided to have a think about how I could get rid of the problem for good.
What I arrived at:
- I should never hand out my actual email address. As in, should be all proxied, every contact reaching me via a different address only known to them. Ideally, these aliases would be generated using a CSPRNG, and would be of sufficient length. Allows for tracing contact provenance in "space".
- I should rotate that address periodically, if possible. Allows for tracing contact provenance in "time".
Together, this would tell me who's the source of particular influx of strange mails, and when did they leak the contact address provided to them (willfully or otherwise).
I was really taken by this idea, even wondered why this has not been baked into the underlying protocols over the years, to make it readily deployed and available for all, making it effortless.
Because boy is there an effort involved! Several years later, I now avoid spam mail by simply no longer reading my emails anymore...
Sometimes I wonder what corporate IT thinks when I pass their anti-phishing test a month after the test campaign.