Graydon Hoare, the creator of the Rust programming language, wrote his seminal piece on text in 2014. In it, he said "text is the most powerful, useful, effective communication technology ever, period."[1] Text is durable.
[1] https://archive.ph/FhG5L (the original either got deleted or login-walled, here is the archived version)
This is the public dividend of a standard finally winning. The article gives credit to Unicode, but it is the fact that ASCII unambiguously won that gives plain text its portability and longevity. It looks like Unicode is on its way to winning in the same way, but it is not there yet. Most text files I write are still pure ASCII because that's the only way to avoid unexpected glitches [0].
“Windows” newlines are also the standard in many communication protocols like HTTP and SMTP. That’s not because of Windows, it’s because it was the standard for teletypes which needed bot CR and LF. It’s arguably systems like Unix that deviated from that standard.
I agree that beyond ASCII there is a slope from well-supported to less-supported and quirky if not problematic areas of Unicode. For example, HN filters many Unicode text elements like combining characters (Zalgo text) and emojis.
Even within ASCII, most control characters don’t have a portable meaning. So it’s really just the printable subset of ASCII, and not even that, given that there are regional variants of ASCII, such as the Japanese one where backslash becomes the Yen sign.
Ahh, plain text, wherein "plain" does some heavy lifting. If plain text had something as simple as a 4 byte signature it could have been soooo much better.
As it is programs have to guess the following;
A) encoding. If ANSI which code page? If unicode which encoding? In the case of utf-16 big or little endian?
B) line endings? CR? LF? CRLF? I guess no-one uses LFCR... right?
C) number formats? 100.000 or 100,000? Date formats? is that mm-dd-yyyy or dd-mm-yyyy?
D) what human-language is it in?
E) CSV? Don't get me started...
Yes. Text is a lot easier to load, parse, make guesses about than say XLS. Yes all of the above things can be "guessed" to a greater or lesser extent.
Yes BOM exists to at least try solving the encoding question. Of course most files don't use them. Lots of tools don't support them. And they don't solve any of the other issues.
But sure, ASCII files in English with US date formats, and Windows line endings....no problems at all...
ASCII is just a raw binary format. It isn't just for text.
The line endings problem really isn't a problem. Pretty much every text editor out there can handle different line endings.
I don’t think number nor date formats are relevant here. For example you could have that same problem entering text into MS Word. That’s really more of an issue if you want to use text as a database rather than a document format, which isn’t really something that even plain text advocates would generally recommend.
As for human language detection, that’s a much easier problem to solve than decoding a proprietary binary blob.
CSV definitely has its warts. But it’s not like that’s the only plain text option for serialising data. (JSON, jsonlines, YAML, XML, etc). Or you could use the actual ASCII codes reserved for records, if you really wanted something that didn’t require quoting and escaping in plain text. It’s actually a pity nobody does this.
I work on scientific data repositories [1] and pushed for plain text files as our archiving file format two decades ago as the best long term format. We implemented it on our data repositories. Very human readable also. The datasets are small and table based, so this works well. We latter started archiving 2-D image data and that creates very large files. We are still looking for an elegant solutions for those datasets.
I love plain text, but the calculus of being able to access the content in 50 years is much more interesting in the age of agents. They can infer the meaning of structured data without schemas, decompressed archives, find embedded files, etc. It's not perfect, but the durability of binary formats is better now than it has ever been.
It's not baked-in to the weights of any model, but agents can write their own tools to work with arbitrary binary formats and get many of the benefits of off-the-shelf unix utilities.
I love that the Godot game engine has a textual scene description. That's a stark contrast to Unreal Engine's binary format for blueprints (visual scripting).
What, no mention of our old friend, ASCII-armored Base64? For shame! Everything can be text with Base64, including things that have absolutely no business being text! Best of all, in light of popular widespread abuse of every available resource, Base64 is comparatively efficient! Bring on the petabytes! Yay and I'm not being completely sarcastic
Fun timing on this plaintext conversation. I built a little browser based plaintext playwriting app this weekend for a little weekend project. Found myself really hating 1. how clunk screenwriting software can be and 2. How unsharable the files are.
With plaintext you can hand the file off to anyone on any device--the caveat being absolutely no one wants to be handed a plaintext script. The software I used ten years ago to write plays is long since deprecated and those files are basically unopenable. Plaintext however remains.
John August has also talked about using plain text for movie/television screenplays, specifying a format based on markdown and hollywood standard: https://fountain.io/
Sharing aside, I imagine a plain text format would also be helpful for version control.
ASCII (7-Bit) is the only widely understood charset there is. Everything beyond this point depends on the loaded charset.
The fact that unicode maps the lower 7 bits to its own character set is a nice touch but none of the unicode sets are plain text.
Unicode are multibyte characters with variable byte length and endianess at play. If you read it wrong or guess the length wrong your results might be anything but useful.
You seemed to be deeply confused about encodings and character sets.
> The fact that unicode maps the lower 7 bits to its own character set is a nice touch but none of the unicode sets are plain text.
That is only true for the mapping of Unicode character (code point to be exact) to UTF-8, which is an encoding of Unicode characters.
> Unicode are multibyte characters with variable byte length and endianess at play.
That is only true of the UTF-16 encoding. UTF-8 does not have endianess, UTF-32 does not have variable length per code point.
None of that is true for Unicode, because it is abstracted away from any byte representation. Furthermore, getting back to the start:
> ASCII (7-Bit) is the only widely understood charset there is. Everything beyond this point depends on the loaded charset.
ASCII also depends on how you try to decode your text. If you interpret two bytes as one character, ASCII will never be correct. There is nothing magical about one byte mapping to one character. I'd even argue that the only reason ASCII support is so universal internationally is because of UTF-8. Otherwise many countries would default to encodings where ASCII is not a subset (as they did before UTF-8 became common). So IMHO, UTF-8 and Unicode are the only widely understood encoding and character set.
> The Unicode Standard defines plain text essentially as a sequence of character codes, without the additional formatting information associated with rich text. Fonts, colors, layout, and similar presentation details belong somewhere else.
Nope. Skintones are part of Unicode. It also has 33 control characters from ASCII including one that rings a bell... There are also numerous characters added by Unicode that are literally called "layout controls".
If you want a format that provides purely semantic information, then "plaintext" doesn't fit the bill.
RSS was removed from most sites (unfortunately) due to business, not technical reasons. Just like Web 2.0 era "mashup" friendly APIs, business started locking down their data despite it being an extremely user-hostile move.
I feel obligated to mention ledger[0] and hledger[1], those are plain text accounting software, well, as you can guess from their names, they allow you to do personal accounting in plain text.
Tangential to the actual point of the post, but the talk "Plain text? Really?" by Dylan Beattie[1] is one of my favorite talks. It does a great job capturing the problems with something that "seems" so simple. I think the author of the blog is well aware of these, though :)
Many things I knew, and a bunch I didn't. Thanks for that.
I wish he'd spent a little more time on the fundamental issue of CR vs CR/LF vs LF. That's been a "plain" text nightmare since before Unicode and many of the other complexities existed.
There was a challenge awhile back to describe a picture in 1000 words can't remember where I saw it. The winner just made a encoder/decoder to turn jpeg into words. The image was somewhat compressed but contained more visual data then you could accurately represent in the same way a raw description could. I thought it was so cool.
Graydon Hoare, the creator of the Rust programming language, wrote his seminal piece on text in 2014. In it, he said "text is the most powerful, useful, effective communication technology ever, period."[1] Text is durable.
[1] https://archive.ph/FhG5L (the original either got deleted or login-walled, here is the archived version)
[2] https://hn.algolia.com/?q=always+bet+on+text
[3] https://news.ycombinator.com/item?id=26164001
[4] https://news.ycombinator.com/item?id=8451271
[5] https://news.ycombinator.com/item?id=10284202
[6] https://news.ycombinator.com/item?id=12815829
I like plain text. I tolerate markdown. Markdown was designed as a shorthand for html. It gets stretched to do other stuff.
I created Brashtag [1]. It is simpler than markdown.
[1] https://github.com/PratikDeoghare/brashtag
This is the public dividend of a standard finally winning. The article gives credit to Unicode, but it is the fact that ASCII unambiguously won that gives plain text its portability and longevity. It looks like Unicode is on its way to winning in the same way, but it is not there yet. Most text files I write are still pure ASCII because that's the only way to avoid unexpected glitches [0].
[0] Windows newlines not withstanding.
> Windows newlines not withstanding
“Windows” newlines are also the standard in many communication protocols like HTTP and SMTP. That’s not because of Windows, it’s because it was the standard for teletypes which needed bot CR and LF. It’s arguably systems like Unix that deviated from that standard.
I agree that beyond ASCII there is a slope from well-supported to less-supported and quirky if not problematic areas of Unicode. For example, HN filters many Unicode text elements like combining characters (Zalgo text) and emojis.
Even within ASCII, most control characters don’t have a portable meaning. So it’s really just the printable subset of ASCII, and not even that, given that there are regional variants of ASCII, such as the Japanese one where backslash becomes the Yen sign.
Ahh, plain text, wherein "plain" does some heavy lifting. If plain text had something as simple as a 4 byte signature it could have been soooo much better.
As it is programs have to guess the following;
A) encoding. If ANSI which code page? If unicode which encoding? In the case of utf-16 big or little endian?
B) line endings? CR? LF? CRLF? I guess no-one uses LFCR... right?
C) number formats? 100.000 or 100,000? Date formats? is that mm-dd-yyyy or dd-mm-yyyy?
D) what human-language is it in?
E) CSV? Don't get me started...
Yes. Text is a lot easier to load, parse, make guesses about than say XLS. Yes all of the above things can be "guessed" to a greater or lesser extent.
Yes BOM exists to at least try solving the encoding question. Of course most files don't use them. Lots of tools don't support them. And they don't solve any of the other issues.
But sure, ASCII files in English with US date formats, and Windows line endings....no problems at all...
ASCII is just a raw binary format. It isn't just for text.
The line endings problem really isn't a problem. Pretty much every text editor out there can handle different line endings.
I don’t think number nor date formats are relevant here. For example you could have that same problem entering text into MS Word. That’s really more of an issue if you want to use text as a database rather than a document format, which isn’t really something that even plain text advocates would generally recommend.
As for human language detection, that’s a much easier problem to solve than decoding a proprietary binary blob.
CSV definitely has its warts. But it’s not like that’s the only plain text option for serialising data. (JSON, jsonlines, YAML, XML, etc). Or you could use the actual ASCII codes reserved for records, if you really wanted something that didn’t require quoting and escaping in plain text. It’s actually a pity nobody does this.
I've made a language called CSTML to do some of this: https://docs.bablr.org/guides/cstml
Basically it's a text-based language for embedding semantic metadata into some kind of underlying text stream. Is this interesting to you?
And file endings..
If you write files in ASCII you’re already writing in UTF-8.
That’s true, though if you’re writing UTF-8 you have to handle the corner cases when reading UTF-8, of which there are many.
Fortunately they’re easy to test for and most languages have standard libraries that make this painless.
ASCII has, in principal, infinitely many superset.
And in practice, still a rather large number of them, going back to the 1970's - https://en.wikipedia.org/wiki/Extended_ASCII
I work on scientific data repositories [1] and pushed for plain text files as our archiving file format two decades ago as the best long term format. We implemented it on our data repositories. Very human readable also. The datasets are small and table based, so this works well. We latter started archiving 2-D image data and that creates very large files. We are still looking for an elegant solutions for those datasets.
[1] https://earthref.org/FIESTA/
I love plain text, but the calculus of being able to access the content in 50 years is much more interesting in the age of agents. They can infer the meaning of structured data without schemas, decompressed archives, find embedded files, etc. It's not perfect, but the durability of binary formats is better now than it has ever been.
It's not baked-in to the weights of any model, but agents can write their own tools to work with arbitrary binary formats and get many of the benefits of off-the-shelf unix utilities.
I love that the Godot game engine has a textual scene description. That's a stark contrast to Unreal Engine's binary format for blueprints (visual scripting).
What, no mention of our old friend, ASCII-armored Base64? For shame! Everything can be text with Base64, including things that have absolutely no business being text! Best of all, in light of popular widespread abuse of every available resource, Base64 is comparatively efficient! Bring on the petabytes! Yay and I'm not being completely sarcastic
And then you add RFC2045 headers so you know what the encoding is...
Text headers, mind you! Ah, text text text
Don't ask me what my point is
Fun timing on this plaintext conversation. I built a little browser based plaintext playwriting app this weekend for a little weekend project. Found myself really hating 1. how clunk screenwriting software can be and 2. How unsharable the files are.
With plaintext you can hand the file off to anyone on any device--the caveat being absolutely no one wants to be handed a plaintext script. The software I used ten years ago to write plays is long since deprecated and those files are basically unopenable. Plaintext however remains.
John August has also talked about using plain text for movie/television screenplays, specifying a format based on markdown and hollywood standard: https://fountain.io/
Sharing aside, I imagine a plain text format would also be helpful for version control.
ASCII (7-Bit) is the only widely understood charset there is. Everything beyond this point depends on the loaded charset.
The fact that unicode maps the lower 7 bits to its own character set is a nice touch but none of the unicode sets are plain text.
Unicode are multibyte characters with variable byte length and endianess at play. If you read it wrong or guess the length wrong your results might be anything but useful.
You seemed to be deeply confused about encodings and character sets.
> The fact that unicode maps the lower 7 bits to its own character set is a nice touch but none of the unicode sets are plain text.
That is only true for the mapping of Unicode character (code point to be exact) to UTF-8, which is an encoding of Unicode characters.
> Unicode are multibyte characters with variable byte length and endianess at play.
That is only true of the UTF-16 encoding. UTF-8 does not have endianess, UTF-32 does not have variable length per code point.
None of that is true for Unicode, because it is abstracted away from any byte representation. Furthermore, getting back to the start:
> ASCII (7-Bit) is the only widely understood charset there is. Everything beyond this point depends on the loaded charset.
ASCII also depends on how you try to decode your text. If you interpret two bytes as one character, ASCII will never be correct. There is nothing magical about one byte mapping to one character. I'd even argue that the only reason ASCII support is so universal internationally is because of UTF-8. Otherwise many countries would default to encodings where ASCII is not a subset (as they did before UTF-8 became common). So IMHO, UTF-8 and Unicode are the only widely understood encoding and character set.
ASCII was amazingly efficient for conveying English text. Then we needed to encode multi languages and emojis, the resulting Unicode is just a mess.
Unicode was mostly OK when it wanted to encode multi languages. It started to get a mess when thought it needed to do more.
> The Unicode Standard defines plain text essentially as a sequence of character codes, without the additional formatting information associated with rich text. Fonts, colors, layout, and similar presentation details belong somewhere else.
Nope. Skintones are part of Unicode. It also has 33 control characters from ASCII including one that rings a bell... There are also numerous characters added by Unicode that are literally called "layout controls".
If you want a format that provides purely semantic information, then "plaintext" doesn't fit the bill.
Probably because plain text can encode any form of distilled data. Technically our DNA can be plain text lol
Prediction: RSS is coming back stronger than ever
RSS was removed from most sites (unfortunately) due to business, not technical reasons. Just like Web 2.0 era "mashup" friendly APIs, business started locking down their data despite it being an extremely user-hostile move.
.plan is really simpler syndication.
I think text is getting a big boost from the age of ai.
I feel obligated to mention ledger[0] and hledger[1], those are plain text accounting software, well, as you can guess from their names, they allow you to do personal accounting in plain text.
0: https://ledger-cli.org
1: https://hledger.org
I prefer the simpler beancount: https://github.com/beancount/beancount
All of these use slightly incompatible formats, the irony of which will not escape the educated reader.
Tangential to the actual point of the post, but the talk "Plain text? Really?" by Dylan Beattie[1] is one of my favorite talks. It does a great job capturing the problems with something that "seems" so simple. I think the author of the blog is well aware of these, though :)
[1] https://www.youtube.com/watch?v=_mZBa3sqTrI
Many things I knew, and a bunch I didn't. Thanks for that.
I wish he'd spent a little more time on the fundamental issue of CR vs CR/LF vs LF. That's been a "plain" text nightmare since before Unicode and many of the other complexities existed.
that's why gno.land contracts render to markdown as the standard. you can browse the world through your terminal.
imagine the world wide web but markdown (and decentralized). gno.land is that.
"A picture is worth a thousand words"
There was a challenge awhile back to describe a picture in 1000 words can't remember where I saw it. The winner just made a encoder/decoder to turn jpeg into words. The image was somewhat compressed but contained more visual data then you could accurately represent in the same way a raw description could. I thought it was so cool.
Sometimes I'd rather have ten unambiguous words.