(ass. sports.txt politics.txt and business.txt are text docs pertaining from the sports, politics and business domains, respectively, and have equal size)
The test file belongs to the topic with the smallest size *.gz file.
Witten's group at Waikato uni were perhaps the first to work on this.
Also check out the Hutter prize if you are interested in this.
That requires you to decide what a "word" is, which is not trivial (if you think that ignoring punctuation gets you to a clean "letters surrounded by spaces" you will get lots of issues with various Asian languages)
Also some languages have a lot of prefixes and suffixes on their verbs or even nouns, which dilutes your list of 1000 words by just adding the same common words over and over again with different suffixes designating grammatical tense, grammatical gender, etc.
The gzip version sounds more general and more obviously correct
By looking at mutual information from different authors on the same topic vs same author on different topics. As I recall, it convincingly disproved the hypothesis.
For anyone wanting an introductory text for information theory & that explores some of these connections & applications, it's worth checking out the late David MacKay's 2003 textbook Information Theory, Inference & Learning Algorithms https://www.inference.org.uk/itila/
That's a bit of a stretch. The LZ77 algorithm merely finds repeated items within a sliding window and encodes them into Huffman codes. A basic model should at least be capable of classification and prediction, like very early-stage machine models. The former is a fixed, quantifiable algorithm, whereas the latter becomes difficult to quantify and understand once the scale of neurons grows large. It's not very reasonable to regard the two as the same thing.
Nitpick: Doing it exactly like this is flawed because you let the compressibility of your references taint the result; what you would prefer is the compressed size of testfile given sports.txt/... as a dictionary without accounting for the compressed size of that, no?
While I also like 7z a lot and use it all the time, it's really not a better format that RAR. Higher compression ratios? Sure, marginally. But for archival RAR is still the better choice as it supports recovery records, which 7z still does not. I also found it more difficult to restore data from a partial 7z file (i.e. trailer data is missing) compared to an incomplete RAR file.
zstd is a good default for e.g. filesystem compression, but if you're making an archive file, presumably you're looking for higher compression and LZMA would be a better fit.
Yeah, zstd is awesome. I built a webapp that uses it via wasm and the decompression speed is incredible, so much so that I store everything in zstd and decompress it on the app load. My wasm binary also does advanced search and tag insertion and stuff in addition to zstd but it's only 38kb. I wish zstd was supported natively by web browsers. The 8kb implementation of zstd is only half as slow as wasm, so even that is still viable.
On a more realistic note: few years back, I've added zstd compression to our log subsystem (hand written direct buffers, native code, in-process, java). For the same CPU utilization if provides twice dense compression compared to regular [-6] gzip (the topic in the title). Zstd is =much= faster on decompression as well, and it this case - unparalleledly better as it uses twice less disk.
zstd is 'silicon valley' (the tv show) - life imitates fiction, except entirely open source
There is 2 softwares I will buy for myself if I still use Windows: WinRAR and Internet Download Manager. In fact, I will buy them in the future regardless the Windows usage.
The original author and also creator of the RAR format, Eugene Roshal, is Russian. I think he moved to the US way back in the day, but not sure about that. He has been very private with his life.
The company doing the software distribution, is located in Berlin. The Managing Directors for that company seem to have Turkish names, but I don't know if they're Turkish.
BTW, Looking for some info I just found a website [0], clearly AI generated (but not necessarily meaning the content is false) claiming Eugene Roshal had severe kidney failure this past month, and he's waiting for surgery. They're asking for donations. There are some names on who's theoretically behind it [1] but they don't link to any LinkedIn profile or personal site. I can't find any other references. The BTC wallet they're using for donations hasn't seen any traffic ever. BE WARY, SMELLS FISHY.
WHOIS information for the domain shows it's registered in the Philippines. The name of the owner does not show up in any of the about section people. So I think those are fake.
give it a normal text prompt, and it
continues that prompt by searching
for the byte sequences that compress
best.
One moment, how are we supposed to know how well that search was done? There is no way to search a meaningful part of the search space.
So the result only gives us some lower bound of how well gzip works as a "plausibility tester" of a continuation of a text. The space of possible sequences is many orders of magnitude larger than what was searched. So there might be sequences in there that compress much better.
The text mentions beamsearch, but I don't see a discussion about how well beamsearch performs in finding the global optima when it comes to gzip compressibility of a text?
Read to the end: they aren't actually looking for the best-compressing output, because this quickly devolves into aaaaaaaaaaa. They keep a sliding window over a small portion of recent text and use that.
Basically I think the entire premise falls apart due to that choice--they forced an interesting-looking outcome by adjusting the algorithm until gzip started picking random slabs of letters instead of ever-larger repeating runs.
That's a fair question. Suppose we have a way to find a byte sequence x that globally minimises len(gzip(context + prompt + x)) over all sequences x of length n. Here + denotes string concatenation.
It's unclear if this is very useful.
The reason it may not be very useful is that one of Deflate's ingredients is a pass that replaces repeated substrings with backreferences to the earlier occurrence in the plaintext input stream.
E.g. suppose we want to find an n=200 byte sequence x that minimises len(gzip(context+prompt+x)).
If there exists any 200 byte sequence y such that prompt+y is a substring of context, then Deflate can encode prompt+y as a backreference to that earlier sequence - it needs to store a match-length & a distance-length, encoded using its Huffman trees. This candidate solution y may not be a global minima to our stated objective function, but if not, it's probably going to be a very good near-optimal approximate solution.
Taking a step back, repeating huge chunks of the input context produces something that's great for minimising compressed output size but doesn't seem particularly helpful as a generative model.
edit:
Yep, I tried it out by running an experiment. Searching for the prompt in the context & then copying the following text as the solution produces solutions that are much better, in the sense of minimising the compressed output length, than beam search, while also being unhelpful as a generative tool.
With the same example as the blog post:
context: first 30,000 bytes of tinyshakespeare.txt
prompt: 'MENENIUS:\n'
Let x denote a solution, x is a string of length 200.
Let L(x) denote len(gzip(context+prompt+x)), our objective function
Let's call the proposed search method of searching for the prompt in the input rfind (after python's str.rfind).
Then we have
search method soln soln length feasible? objective value search time (wall clock, s)
------------- ---- ----------- --------- --------------- ---------------------------
emptystring "" 0 no 13,023 0.04s
gzipt beam search see blog post 200 yes 13,051 11.93s
rfind see below 200 yes 13,026 0.04s
So 'rfind' is finding a solution that does a better job of minimising the objective function -- it only takes 3 bytes more to encode than the infeasible emptystring solution, and costs 25 fewer bytes than the solution found by the beam search implemented by gzipt per the blog post.
Here's the solution 'generated' by rfind copying and pasting from the input context, starting from the rightmost occurrence of "MENENIUS:"
MENENIUS:
O, true-bred!
First Senator:
Your company to the Capitol; where, I know,
Our greatest friends attend us.
TITUS:
COMINIUS:
Noble Marcius!
First Senator:
MARCIUS:
Nay, let them follow:
The Volsces
Here's the code for 'rfind' - our complete 'generative algorithm':
def find_candidate_solution_from_context(context, prompt, length):
n = len(context)
i = context.rfind(prompt, 0, n-length)
if i < 0:
return b''
i += len(prompt)
return context[i:i+length]
Can hook it into gzipt.py by adding this line after out is defined, but before the beam search begins
out += find_candidate_solution_from_context(corpus_window, prompt, length)
This is fun, but historically people have gone a bit overboard with saying that models like this, or n-gram language models, are anywhere close to large neural network models. There is certainly a connection though.
Yes, but it is a useful insight that both methods try to solve the same mathematical problem. It's better than thinking of LLMs as magic.
When you say "cross-entropy loss" people without stats background go to Wikipedia, take a glance, and adjust their mental model to "inscrutable magic".
Thinking of the main difference as the trade-off in how much CPU, memory and storage is allowed is not really wrong.
The part that is wrong is to think of gzip as a method that might reach similar complexity or generalization. And more importantly, to ignore the advanced way how training data gets curated or generated for (instructed, chain-of-thought) LLMs. But even then. The mental model that the LLM's goal is text compression is not wrong. The question to ask next is what kind of text it is expecting to compress.
Yep agree, good details, and this doesn't contradict my point above, about people going "overboard" with the comparison.
I do think when making these comparisons, it is worth emphasising that neural nets are really different. E.g. I used to see people equating LLMs to n-gram models, etc. which is overly simplistic, (especially in the early days when the models weren't as good).
Would this work with video compression? Video codecs encode a lot of meaning; they use motion vectors to track the movement of objects on screen, for example.
I think language itself is compression, so the arxiv paper tracks for me.
Viz. if Language is compression (of thought / culture / the tacit je ne sait quois of being-to-being communication etc.), then definitionally, Language Modelling must also be Compression.
Except, language is an arbitrarily lossy compressor, who's "compression-prediction equivalence" is indeterminate and unstable, because Language co-evolves constantly; both as a function of or response to culture, as well as an influencer of culture.
So, the subjective-objective goodness of Language Models (of any kind of language) would be, at best, upper-bounded by the compression-prediction equivalence of the Languages corpus itself. And that is assuming the language corpus is perfect in every way---it captures all knowledge expressible by language and it is always in-sync with live evolution of all language expression and evolution (i.e. LLM training is not a batch job, but a real-time present continuous process).
For example, to my layperson eyes, the mathematical language of proofs actively weeds out ambiguity of subjective interpretation. Ideally, a proof ought to lead to the exact same conclusion on every single reading by any reader who can follow the steps. A proof also holds only if the rest of the formal, explicit, inviolable, internally-consistent set of axioms and results holds.
So it stands to reason that mathematical prose of proofs, being optimised as mechanical procedure of taking an open question to a deterministically closed solution, has better odds of approximating the tacit aspects of mathematical derivation.
Which makes an LLM able to construct a mathematical proof, which is mind-melting to say the least.
However, I wonder, can LLMs dream of mathematical sheep?
Language is compression of a sort. A dictionary is a decompressor. You look up one word, and you may get a paragraph about its meaning. An encyclopedia can be thought of as roughly the same with more detail for some nouns.
It makes a lot of sense why dictionary-based compression is named the way it is. A shorter symbol is used to store information that would take more symbols in the uncompressed corpus, if the shorter symbol hadn't been assigned to represent it. That's in a way just what an actual dictionary on your English professor's shelf does. The big difference is your compressor is coining new short symbols all the time.
I've been pondering on something related: can an LLM be a chat?
Some models are reproducible, in that the same prompt will generate the same output. Say that we could wire up such a model to generate some code.
In that case, we could create a prompt that generates, say, an entire codebase, or a large piece of text. The prompt (or really, the tokens) would then be the compressed version of the codebase or the text.
I am not talking about an "AI agent", but really a model that we call in a reproducible manner. Preferably one call, with one prompt. An agent could just run `git clone` to "decompress" a codebase, which conflates the idea of compression. If that were compression, then the "compressed version of the git kernel" would be a single line of text: `git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...`. I am really talking about having an LLM re-generate text based on a prompt.
Does that make sense? I can imagine that this is highly impractical and inefficient. But would this count as "compression" at all?
A large language model itself (the network) give you the probabilities for the next token given some prefix of tokens so far. You can use arithmetic coding to go from these probabilities to a deterministic compression / decompression algorithm.
When you use an LLM to generate text, you sample from that probability distribution. You can use a true random sample. Or you can make it trivially deterministic by using a seeded pseudo-random-number-generator or you just pick the highest probability each time. But that's all a red herring; really, what you want is arithmetic coding.
>An LLM is just as deterministic as any other computer program. For identical inputs (which includes the PRNG seed) it produces identical outputs.
This is not really true in practice because of multi-threading and out-of-order execution. Mathematically equivalent orderings of operations are not equivalent when dealing with floating point values, so most practical LLM implementations end up being non-deterministic.
Sounds like an interesting way for future OS included apps to be distributed.
Like when you click the Calculator button on your android, it wouldn't actually exist yet, your click actually prompts it into existence. But naively that has problems because you don't want a different UI every time. There's something to your idea.
It makes a lot of sense. I thought about it in the context of pull requests or change sets: if the text-to-code process is reliable, why don't you give me prompts instead of code? Code becomes just an intermediate representation.
R. Hendricks, D. Chugtai, and J. Dunn, "Lossless compression via optimized middle-out bitstream processing," Pied Piper Inc., Palo Alto, CA, Tech. Rep. 42, Apr. 2014.
I was curious to see how this would work with bzip2 and zstd. The source is public at https://github.com/nathanrs/gzipt, and I asked MiMo-V2.6-Flash to fork and modify it in a straightforward way. The answer is that bzip2 produces sequences that don't resemble human language:
Zstandard produces whitespace with the occasional letter thrown in. To quote MiMo: "As you can see, zstd does not speak Shakespeare. ... zstd encodes a run of one repeated byte as a near-free run-length sequence, and space and newline are the cheapest literals in the corpus: ten newlines cost about the same to append ten bytes of genuine corpus text and less than nonsense does."
I did. I read the code to make sure the quality of MiMo's work matched mine for a quick experiment, though not that the code was free from subtle bugs.
This was the main change for bzip2:
@@ -33,19 +34,16 @@ def candidate_lengths(
level: int = 9,
pool: ThreadPoolExecutor | None = None,
) -> list[int]:
- """Compressed length of ``context + seq`` for each seq, sharing the context.
+ """Compressed length of ``context + seq`` for each seq.
- Compresses ``context`` once into a ``compressobj``, then clones its encoder
- state per candidate and feeds only that candidate. Identical to
- ``len(zlib.compress(context + seq, level))`` for each seq, but the expensive
- match search over ``context`` happens a single time.
+ Unlike ``zlib``'s ``compressobj``, Python's ``BZ2Compressor`` cannot be
+ snapshotted mid-stream, and bzip2's move-to-front + Huffman stages see the
+ whole block, so every candidate recompresses the full context. Threads
+ still scale because ``bz2`` releases the GIL.
"""
- base = zlib.compressobj(level)
- head = len(base.compress(context))
def length_for(seq: bytes) -> int:
- clone = base.copy()
- return head + len(clone.compress(seq) + clone.flush(zlib.Z_FINISH))
+ return len(bz2.compress(context + seq, level))
if pool is not None:
return list(pool.map(length_for, sequences))
Is the fact that the original did [compress base]+[compress seq] rather than [compress [bytes + seq]] not important?
(I honestly don’t know is gzip does something different when presented with two chunks as opposed to one, or, if it does, if bz2 has equivalent behaviour - but the difference in the code did stand out to me, and it does seem related to ‘extending the token sequence’)
It's an interesting experiment, though the examples in the article didn't really seem that coherent, so I'm not sure that the answer to the title is "yes". Along the same lines, a few years ago I made a very tiny transformer demonstration in javascript, also trained on Shakespeare, but it was too small to produce text like Shakespeare. You can try it here:
A seemingly simple sentence like "I had lunch" has enormous amount of information compressed inside it.
The word lunch is a compressed form of "having food at noon" while "noon" in turn is a compressed form of "Sun's position against Earth's rotation" and so on and so forth.
Every sentence has layers of compressed sentences. How many layers one chooses to decompress is up to the person.
Yay, another mostly AI authored piece with vibe-coded aesthetics.
Some will say that I should 'judge the idea, not the form'.
But if the author didn't find enough strength to write alone a short ~700 words summary about his work, it means he himself isn't that interested or enthusiastic about it. Why should others bother then? Particularly since low-effort like that signals possibility the whole work is superficial and derivative.
Your claim that suspected AI assistance proves the author isn't interested -- and therefore that the work is probably superficial -- is unsupported.
The article presents a working experiment, explains why naive decoding fails, describes the beam-search fix, and links the code.
Dismissing all that with presumptuous personal speculation and banal boilerplate drive-by anti-AI snark adds absolutely nothing to the conversation -- and that is intrinsically poor form.
You couldn't even find enough strength to criticize anything beyond the form, while your own form is lackluster.
Ironically, an LLM could have written your comment and improved its form without losing anything distinctive.
> I'm judging the ideas in your post, not its form.
and simultaneously you write that 'my form is lackluster' and that 'an LLM could have written your comment and improved its form without losing anything distinctive'. We are having ourselves a small contradiction, aren't we.
Be my guest, enjoy chatbot writing and drowning in slop. But don't encroach upon my freedom to protest it.
Ironic, isn't it? Yet you continue your lackluster form devoid any criticism of ideas. All you can criticise is form, yet my criticism of your ideas still stands:
Your claim that suspected AI assistance proves the author isn't interested -- and therefore that the work is probably superficial -- is unsupported.
So I'm encroaching but you're only protesting, huh? I also have the freedom to ironically protest the poor form of your inability to criticize ideas, as well as your poorly formulated unsupportable ideas.
Will Wright once described the same connection from the opposite direction: compression as procedural content generation.
In the 2023 discussion of "Demoscene accepted as UNESCO cultural heritage in The Netherlands" I posted a transcript from a video of Will Wright discussing the demo scene:
>You can take any piece of content in the game, and imagine an algorithmic solution to it. Or also, you know, a way that the player could customize that object of thing.
>There's this group in Europe called the Demoscene that make these very elaborate demos for a computer that fit into very tiny little memory blocks, you know like 64K of memory, and you run the thing, and in fact it algorithmically generates about 100 megabytes worth of data, you know these rich 3D environment, generated music, generated wave files, generated animation.
>And they're developing techniques to generate, you know, huge amounts of interesting data, with very very simple, elegant, compression algorithms.
>And this is a skill that game developers used to have, back in the 8-bit days. That was the only ways to do a game like Karateka(?), was to find all these little tips and tricks to compress things and generate them algorithmically.
>But since the CD-ROM came out, and very cheap hard drives, storage is cheap, so basically we've lost that skill set, and now we attack all those problems with brute force. I think we've lost something by dropping that skill set.
[...] Here's a simple low-tech pre-LLM example that shows the equivalence of compression and procedural content generation:
Take a huge text file of HN postings, and compress it with gzip or compress or some other robust compression algorithm. The better the algorithm, the more the output will look like random noise. Then slice the compressed file in half, and replace the second half with random numbers. Then uncompress it. You'll find that at the point you sliced it, it keeps on writing out almost plausible text for a while, consisting of highly probably snippets of commonly encountered words and phrases, then goes downhill towards incoherence. It's not as coherent or confident as an LLM, but the point is to show how low the bar is for using compression for procedural content generation.
LLMs are essentially a form of compression of the world's knowledge or whatever they're trained on, not just word frequencies or pixel patterns, but also concepts and ideas. [...]
Not without attention or something approximating it.
The fact that gzip is relatively fast should be your first clue that something important is missing.
Gzip is great at predicting the next token for one very specific narrative. LLMs can predict next tokens for entire universes of narratives. Searching for the correct next token across this space scales ~quadratically with the input size. Gzip scales linearly. I can gzip a one terabyte file. Imagine feeding that much into an LLM. These are wildly different animals that happen to overlap in a very small way. Equating compression to intelligence looks increasingly silly to me.
If we must compare language models to compression, they are much more like jpeg and mp3 than they are gzip and flac. I can go fuck with a jpeg file pretty severely at the bitstream level and still have something resembling performance on the other side. Gzip cannot remotely approach this.
> Gzip scales linearly. I can gzip a one terabyte file.
In part because gzip only has a 32KiB window size, and I think it'd be at least quadratic within that window if you were going for optimal compression.
I'll concede the window part, but Gzip runs within the physical confines of a single cpu core and is typically entirely resident in local caches. The point is not just the quadratic scaling but also what it scales with.
Show me an LLM that can run at 300 megabytes per second. Even dedicated ASICs with weights burned in will never move this fast.
> if LLMs are used as compressors, how well is that expected to work
Quite well. This project[1], by Fabrice Bellard of ffmpeg fame, is quite old in AI years and uses an ancient LLM, but still beats xz by a solid margin.
A challenge as I understand it is reproducibility.
Normal LLM runtimes aren't typically fully reproducible even with same random seeds for distribution sampling, due to floating-point numbers, batching and such.
Though averaging over many runs could alleviate that I suppose.
While it would measure some aspects of intelligence, I'd argue it fails to capture other, more creative aspects.
I think not. it's true that a large language model compresses knowledge and allows us to decompress knowledge. but gzip compresses data and not knowledge. with a LM you can decompress various forms of knowledge from the same data. gzip is a 1 to 1 kind of decompression where as an LM is a 1 to infinity kind of decompression.
Fun topic but generated article text and then not even actually using gzip? Rubs me in a weird way.
Yes: you can classify a test file by topic with gzip as follows:
(ass. sports.txt politics.txt and business.txt are text docs pertaining from the sports, politics and business domains, respectively, and have equal size)The test file belongs to the topic with the smallest size *.gz file.
Witten's group at Waikato uni were perhaps the first to work on this.
Also check out the Hutter prize if you are interested in this.
Back in the day - maybe two decades ago - I implemented language detection like this.
I seeded gzip compressors’ dictionaries with Wikipedia articles in different languages.
I would then try to use said dictionaries on any random text, and the one that was best able to compress it, was the correct language.
Absolutely totally not the best approach, but very fast and super simple to implement.
Or maybe make a list of the most used 1000 words in each language. And see which list has the most occurrences.
That requires you to decide what a "word" is, which is not trivial (if you think that ignoring punctuation gets you to a clean "letters surrounded by spaces" you will get lots of issues with various Asian languages)
Also some languages have a lot of prefixes and suffixes on their verbs or even nouns, which dilutes your list of 1000 words by just adding the same common words over and over again with different suffixes designating grammatical tense, grammatical gender, etc.
The gzip version sounds more general and more obviously correct
Sounds like the best approach. :)
We used a similar technique for a class project (N decades ago) to test this:
https://en.wikipedia.org/wiki/Baconian_theory_of_Shakespeare...
By looking at mutual information from different authors on the same topic vs same author on different topics. As I recall, it convincingly disproved the hypothesis.
There are some deep connections between machine learning, compression, and cryptography with information theory as a common thread.
Also, I’ve never seen “ass.” Used to shorten “aside” — I typically use N.B. but perhaps only for important ones.
For anyone wanting an introductory text for information theory & that explores some of these connections & applications, it's worth checking out the late David MacKay's 2003 textbook Information Theory, Inference & Learning Algorithms https://www.inference.org.uk/itila/
The "ass." more likely stands for "assuming".
Yeah, I read it as "assuming" too.
3Blue1Brown has a good video about this: https://www.youtube.com/watch?v=l6DKRf-fAAM
That's a bit of a stretch. The LZ77 algorithm merely finds repeated items within a sliding window and encodes them into Huffman codes. A basic model should at least be capable of classification and prediction, like very early-stage machine models. The former is a fixed, quantifiable algorithm, whereas the latter becomes difficult to quantify and understand once the scale of neurons grows large. It's not very reasonable to regard the two as the same thing.
Nitpick: Doing it exactly like this is flawed because you let the compressibility of your references taint the result; what you would prefer is the compressed size of testfile given sports.txt/... as a dictionary without accounting for the compressed size of that, no?
Really interesting approach though.
I seem to remember it being shown for character recognition via JBIG. Maybe in Managing Gigabytes?
There would be some overlap with business sports analogies - eg team huddle.
aka Normalized compression distance (NCD). Its close cousin: Normalized Google distance (NGD) is also super interesting!
https://en.wikipedia.org/wiki/Normalized_compression_distanc...
This tracks perfectly with Winrar being more profitable than OpenAI... coincidence? I think not!
winrar is profitable? sure? well, on the other hand, they sure don't make losses
They are a German GmbH and must publicly state their financials: https://www.northdata.de/win%C2%B7rar%20GmbH,%20Berlin/Amtsg...
Looks pretty profitable to me.
They're one of the few companies that actually manage to sell "boxed software" (i.e. has not changed much in years but new customers keep buying it)
That said, Windows users should use 7-Zip. Better compression format, unpacks more kinds of archives
While I also like 7z a lot and use it all the time, it's really not a better format that RAR. Higher compression ratios? Sure, marginally. But for archival RAR is still the better choice as it supports recovery records, which 7z still does not. I also found it more difficult to restore data from a partial 7z file (i.e. trailer data is missing) compared to an incomplete RAR file.
> Windows users should use 7-Zip
Please no - no native zstd support. NanaZip is the better option (it's a different build of 7-zip) and it's available at windows store.
> Better compression format, unpacks more kinds of archives
winrar has supported zstd for 5 years[0]
In short - Everyone should be using zstd, and 7-zip does not support it.
[0]: https://www.win-rar.com/singlenewsview.html?&L=0&tx_ttnews%5...
zstd is a good default for e.g. filesystem compression, but if you're making an archive file, presumably you're looking for higher compression and LZMA would be a better fit.
Yeah, zstd is awesome. I built a webapp that uses it via wasm and the decompression speed is incredible, so much so that I store everything in zstd and decompress it on the app load. My wasm binary also does advanced search and tag insertion and stuff in addition to zstd but it's only 38kb. I wish zstd was supported natively by web browsers. The 8kb implementation of zstd is only half as slow as wasm, so even that is still viable.
Lzma. I did a simple test, same payload repeated with a gap. Lzma ruled it.
|gap |gzip |bz2 |lzma | |---------|----------|-----|--------| |0 |2.7% |18.9%|*0.9%*| |8 KB |2.4% |17.8%|0.9% | |*40 KB*|*94.4%* |17.7%|0.4% | |1 MB |*104.3%*|19.7%|*0.9%*|
HN doesn’t support Markdown tables. You can prepend two spaces to show it in monospace though.
I don’t see zstd in your comparison?
dont know the point of the table, but here it is:
> Everyone should be using zstd
Why?
It's just the best general purpose compression algorithm, in terms of compression ratio to CPU used, for the vast majority of use cases
It's just this good.
On a more realistic note: few years back, I've added zstd compression to our log subsystem (hand written direct buffers, native code, in-process, java). For the same CPU utilization if provides twice dense compression compared to regular [-6] gzip (the topic in the title). Zstd is =much= faster on decompression as well, and it this case - unparalleledly better as it uses twice less disk.
zstd is 'silicon valley' (the tv show) - life imitates fiction, except entirely open source
Huh. I guess the warez kids grew up and have money, now?
There is 2 softwares I will buy for myself if I still use Windows: WinRAR and Internet Download Manager. In fact, I will buy them in the future regardless the Windows usage.
That’s surprising. Seems there is a niche for everything.
You need to visit Japan...
TIL they're Turkish
The original author and also creator of the RAR format, Eugene Roshal, is Russian. I think he moved to the US way back in the day, but not sure about that. He has been very private with his life.
The company doing the software distribution, is located in Berlin. The Managing Directors for that company seem to have Turkish names, but I don't know if they're Turkish.
BTW, Looking for some info I just found a website [0], clearly AI generated (but not necessarily meaning the content is false) claiming Eugene Roshal had severe kidney failure this past month, and he's waiting for surgery. They're asking for donations. There are some names on who's theoretically behind it [1] but they don't link to any LinkedIn profile or personal site. I can't find any other references. The BTC wallet they're using for donations hasn't seen any traffic ever. BE WARY, SMELLS FISHY.
--
WHOIS information for the domain shows it's registered in the Philippines. The name of the owner does not show up in any of the about section people. So I think those are fake.
So the result only gives us some lower bound of how well gzip works as a "plausibility tester" of a continuation of a text. The space of possible sequences is many orders of magnitude larger than what was searched. So there might be sequences in there that compress much better.
The text mentions beamsearch, but I don't see a discussion about how well beamsearch performs in finding the global optima when it comes to gzip compressibility of a text?
Read to the end: they aren't actually looking for the best-compressing output, because this quickly devolves into aaaaaaaaaaa. They keep a sliding window over a small portion of recent text and use that.
Basically I think the entire premise falls apart due to that choice--they forced an interesting-looking outcome by adjusting the algorithm until gzip started picking random slabs of letters instead of ever-larger repeating runs.
Meh. That’s nothing compared to the amount of curation and tuning the LLMs are coerced with.
No--even a really tiny, underpowered model like GPT-2 with no system prompt produces coherent (though not necessarily desirable or correct) responses.
That's a fair question. Suppose we have a way to find a byte sequence x that globally minimises len(gzip(context + prompt + x)) over all sequences x of length n. Here + denotes string concatenation.
It's unclear if this is very useful.
The reason it may not be very useful is that one of Deflate's ingredients is a pass that replaces repeated substrings with backreferences to the earlier occurrence in the plaintext input stream.
E.g. suppose we want to find an n=200 byte sequence x that minimises len(gzip(context+prompt+x)).
If there exists any 200 byte sequence y such that prompt+y is a substring of context, then Deflate can encode prompt+y as a backreference to that earlier sequence - it needs to store a match-length & a distance-length, encoded using its Huffman trees. This candidate solution y may not be a global minima to our stated objective function, but if not, it's probably going to be a very good near-optimal approximate solution.
Taking a step back, repeating huge chunks of the input context produces something that's great for minimising compressed output size but doesn't seem particularly helpful as a generative model.
edit:
Yep, I tried it out by running an experiment. Searching for the prompt in the context & then copying the following text as the solution produces solutions that are much better, in the sense of minimising the compressed output length, than beam search, while also being unhelpful as a generative tool.
With the same example as the blog post:
Let x denote a solution, x is a string of length 200.Let L(x) denote len(gzip(context+prompt+x)), our objective function
Let's call the proposed search method of searching for the prompt in the input rfind (after python's str.rfind).
Then we have
So 'rfind' is finding a solution that does a better job of minimising the objective function -- it only takes 3 bytes more to encode than the infeasible emptystring solution, and costs 25 fewer bytes than the solution found by the beam search implemented by gzipt per the blog post.Here's the solution 'generated' by rfind copying and pasting from the input context, starting from the rightmost occurrence of "MENENIUS:"
Here's the code for 'rfind' - our complete 'generative algorithm': Can hook it into gzipt.py by adding this line after out is defined, but before the beam search begins3blue1brown did a series on this topic: https://www.youtube.com/watch?v=l6DKRf-fAAM https://www.youtube.com/watch?v=GlYgs6v2YfU (i think one more is yet to release)
This is fun, but historically people have gone a bit overboard with saying that models like this, or n-gram language models, are anywhere close to large neural network models. There is certainly a connection though.
Yes, but it is a useful insight that both methods try to solve the same mathematical problem. It's better than thinking of LLMs as magic.
When you say "cross-entropy loss" people without stats background go to Wikipedia, take a glance, and adjust their mental model to "inscrutable magic".
Thinking of the main difference as the trade-off in how much CPU, memory and storage is allowed is not really wrong.
The part that is wrong is to think of gzip as a method that might reach similar complexity or generalization. And more importantly, to ignore the advanced way how training data gets curated or generated for (instructed, chain-of-thought) LLMs. But even then. The mental model that the LLM's goal is text compression is not wrong. The question to ask next is what kind of text it is expecting to compress.
Yep agree, good details, and this doesn't contradict my point above, about people going "overboard" with the comparison.
I do think when making these comparisons, it is worth emphasising that neural nets are really different. E.g. I used to see people equating LLMs to n-gram models, etc. which is overly simplistic, (especially in the early days when the models weren't as good).
Would this work with video compression? Video codecs encode a lot of meaning; they use motion vectors to track the movement of objects on screen, for example.
I think language itself is compression, so the arxiv paper tracks for me.
Viz. if Language is compression (of thought / culture / the tacit je ne sait quois of being-to-being communication etc.), then definitionally, Language Modelling must also be Compression.
Except, language is an arbitrarily lossy compressor, who's "compression-prediction equivalence" is indeterminate and unstable, because Language co-evolves constantly; both as a function of or response to culture, as well as an influencer of culture.
So, the subjective-objective goodness of Language Models (of any kind of language) would be, at best, upper-bounded by the compression-prediction equivalence of the Languages corpus itself. And that is assuming the language corpus is perfect in every way---it captures all knowledge expressible by language and it is always in-sync with live evolution of all language expression and evolution (i.e. LLM training is not a batch job, but a real-time present continuous process).
For example, to my layperson eyes, the mathematical language of proofs actively weeds out ambiguity of subjective interpretation. Ideally, a proof ought to lead to the exact same conclusion on every single reading by any reader who can follow the steps. A proof also holds only if the rest of the formal, explicit, inviolable, internally-consistent set of axioms and results holds.
So it stands to reason that mathematical prose of proofs, being optimised as mechanical procedure of taking an open question to a deterministically closed solution, has better odds of approximating the tacit aspects of mathematical derivation.
Which makes an LLM able to construct a mathematical proof, which is mind-melting to say the least.
However, I wonder, can LLMs dream of mathematical sheep?
Language is compression of a sort. A dictionary is a decompressor. You look up one word, and you may get a paragraph about its meaning. An encyclopedia can be thought of as roughly the same with more detail for some nouns.
It makes a lot of sense why dictionary-based compression is named the way it is. A shorter symbol is used to store information that would take more symbols in the uncompressed corpus, if the shorter symbol hadn't been assigned to represent it. That's in a way just what an actual dictionary on your English professor's shelf does. The big difference is your compressor is coining new short symbols all the time.
I've been pondering on something related: can an LLM be a chat?
Some models are reproducible, in that the same prompt will generate the same output. Say that we could wire up such a model to generate some code.
In that case, we could create a prompt that generates, say, an entire codebase, or a large piece of text. The prompt (or really, the tokens) would then be the compressed version of the codebase or the text.
I am not talking about an "AI agent", but really a model that we call in a reproducible manner. Preferably one call, with one prompt. An agent could just run `git clone` to "decompress" a codebase, which conflates the idea of compression. If that were compression, then the "compressed version of the git kernel" would be a single line of text: `git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...`. I am really talking about having an LLM re-generate text based on a prompt.
Does that make sense? I can imagine that this is highly impractical and inefficient. But would this count as "compression" at all?
You're describing https://bellard.org/ts_zip/ ("Text Compression using Large Language Models") ?
Man,did that page load fast. It made me realize how slow the rest the (my) web is.
You sound a bit confused.
A large language model itself (the network) give you the probabilities for the next token given some prefix of tokens so far. You can use arithmetic coding to go from these probabilities to a deterministic compression / decompression algorithm.
When you use an LLM to generate text, you sample from that probability distribution. You can use a true random sample. Or you can make it trivially deterministic by using a seeded pseudo-random-number-generator or you just pick the highest probability each time. But that's all a red herring; really, what you want is arithmetic coding.
https://en.wikipedia.org/wiki/Arithmetic_coding
>I've been pondering on something related: can an LLM be a chat?
A chat?
>I am not talking about an "AI agent", but really a model that we call in a reproducible manner.
An LLM is just as deterministic as any other computer program. For identical inputs (which includes the PRNG seed) it produces identical outputs.
>compressed version of the git kernel
The git kernel, got it.
>But would this count as "compression" at all?
Yes. The decompressor is several tens of gigabytes though.
>An LLM is just as deterministic as any other computer program. For identical inputs (which includes the PRNG seed) it produces identical outputs.
This is not really true in practice because of multi-threading and out-of-order execution. Mathematically equivalent orderings of operations are not equivalent when dealing with floating point values, so most practical LLM implementations end up being non-deterministic.
Sounds like an interesting way for future OS included apps to be distributed.
Like when you click the Calculator button on your android, it wouldn't actually exist yet, your click actually prompts it into existence. But naively that has problems because you don't want a different UI every time. There's something to your idea.
Someone did it!
https://youtu.be/7NfyZhV1dKM?is=YOUXCHuFUiPdlD0p
This was tried many times in the past for images, even before LLMs.
https://imalogic.com/blog/2024/06/03/image-compression-decom...
It makes a lot of sense. I thought about it in the context of pull requests or change sets: if the text-to-code process is reliable, why don't you give me prompts instead of code? Code becomes just an intermediate representation.
Previous discussions of that specific page https://news.ycombinator.com/item?id=48557691
I'm more interested in the converse question: how well does an LLM perform as a compressor, compared to gzip (ignoring its insanely lower speed)?
Top contestant in the Hutter Prize uses a neural network for compression. So fair to say, LLMs would perform pretty well compared to gzip.
Even ignoring speed per GP, the Hutter Prize's metric includes the size of the decompressor. LLMs would be disqualified for being larger than 1GB.
And the hutter prize disallows GPU's. If you allow use of a powerful GPU, you can do quite a bit better.
hallucinations are lossy compression artefacts
lossless vs lossy is the question.
How lossy? Because I can lossy compress anything into 0 bits.
0 bits is just a very small bloom filter.
R. Hendricks, D. Chugtai, and J. Dunn, "Lossless compression via optimized middle-out bitstream processing," Pied Piper Inc., Palo Alto, CA, Tech. Rep. 42, Apr. 2014.
See also the Hutter Prize [1]. Compression is a subset of Intelligence.
[1] https://en.wikipedia.org/wiki/Hutter_Prize
I was curious to see how this would work with bzip2 and zstd. The source is public at https://github.com/nathanrs/gzipt, and I asked MiMo-V2.6-Flash to fork and modify it in a straightforward way. The answer is that bzip2 produces sequences that don't resemble human language:
Line breaks added. This looks roughly optimized for the most repetitive Burrows-Wheeler transform (https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transf...). Why are they runs of alternating symbols and not one symbol?Zstandard produces whitespace with the occasional letter thrown in. To quote MiMo: "As you can see, zstd does not speak Shakespeare. ... zstd encodes a run of one repeated byte as a near-free run-length sequence, and space and newline are the cheapest literals in the corpus: ten newlines cost about the same to append ten bytes of genuine corpus text and less than nonsense does."
Did you check MiMo correctly performed this unfamiliar task before posting this comment?
I did. I read the code to make sure the quality of MiMo's work matched mine for a quick experiment, though not that the code was free from subtle bugs.
This was the main change for bzip2:
Is the fact that the original did [compress base]+[compress seq] rather than [compress [bytes + seq]] not important?
(I honestly don’t know is gzip does something different when presented with two chunks as opposed to one, or, if it does, if bz2 has equivalent behaviour - but the difference in the code did stand out to me, and it does seem related to ‘extending the token sequence’)
So, you had an AI write code you don't understand, then posted output you don't understand in a comment on the internet for other humans to read?
Interesting approach, I wonder how this could be used as a classifier. :)
Check out "Text classification with Python 3.14's zstd module" (https://maxhalford.github.io/blog/text-classification-zstd/, https://news.ycombinator.com/item?id=46942864). I wanted to link it somewhere in the comments. :-)
I've used LZO as a spam classifier on chat. Spam tends to be very content-less and repetitive...
3blue did a series of videos on this
https://www.youtube.com/watch?v=l6DKRf-fAAM
It's an interesting experiment, though the examples in the article didn't really seem that coherent, so I'm not sure that the answer to the title is "yes". Along the same lines, a few years ago I made a very tiny transformer demonstration in javascript, also trained on Shakespeare, but it was too small to produce text like Shakespeare. You can try it here:
https://taonexus.com/mini-transformer-in-js.html
I guess the results are similar to the article, maybe a little more coherent since the tokens are words.
Good article.
Compression is a property of language.
A seemingly simple sentence like "I had lunch" has enormous amount of information compressed inside it.
The word lunch is a compressed form of "having food at noon" while "noon" in turn is a compressed form of "Sun's position against Earth's rotation" and so on and so forth.
Every sentence has layers of compressed sentences. How many layers one chooses to decompress is up to the person.
Yay, another mostly AI authored piece with vibe-coded aesthetics.
Some will say that I should 'judge the idea, not the form'.
But if the author didn't find enough strength to write alone a short ~700 words summary about his work, it means he himself isn't that interested or enthusiastic about it. Why should others bother then? Particularly since low-effort like that signals possibility the whole work is superficial and derivative.
I thought it was interesting.
so did I, that's why I clicked.
I'm judging the ideas in your post, not its form.
Your claim that suspected AI assistance proves the author isn't interested -- and therefore that the work is probably superficial -- is unsupported.
The article presents a working experiment, explains why naive decoding fails, describes the beam-search fix, and links the code.
Dismissing all that with presumptuous personal speculation and banal boilerplate drive-by anti-AI snark adds absolutely nothing to the conversation -- and that is intrinsically poor form.
You couldn't even find enough strength to criticize anything beyond the form, while your own form is lackluster.
Ironically, an LLM could have written your comment and improved its form without losing anything distinctive.
> I'm judging the ideas in your post, not its form.
and simultaneously you write that 'my form is lackluster' and that 'an LLM could have written your comment and improved its form without losing anything distinctive'. We are having ourselves a small contradiction, aren't we.
Be my guest, enjoy chatbot writing and drowning in slop. But don't encroach upon my freedom to protest it.
Ironic, isn't it? Yet you continue your lackluster form devoid any criticism of ideas. All you can criticise is form, yet my criticism of your ideas still stands:
Your claim that suspected AI assistance proves the author isn't interested -- and therefore that the work is probably superficial -- is unsupported.
So I'm encroaching but you're only protesting, huh? I also have the freedom to ironically protest the poor form of your inability to criticize ideas, as well as your poorly formulated unsupportable ideas.
> Ironic, isn't it? Yet you continue your lackluster form devoid any criticism of ideas.
Pointing out someone's contradiction is now being 'critical of form'? And someone's contradicting themselves is 'ironic'?
Now I'm not even sure you know the meaning of words you use. EOT from me.
Only if it can run DOOM.
Will Wright once described the same connection from the opposite direction: compression as procedural content generation.
In the 2023 discussion of "Demoscene accepted as UNESCO cultural heritage in The Netherlands" I posted a transcript from a video of Will Wright discussing the demo scene:
https://news.ycombinator.com/item?id=36599415
Will Wright Discusses the Demoscene:
https://www.youtube.com/watch?v=m7iuFVmTJus
>You can take any piece of content in the game, and imagine an algorithmic solution to it. Or also, you know, a way that the player could customize that object of thing.
>There's this group in Europe called the Demoscene that make these very elaborate demos for a computer that fit into very tiny little memory blocks, you know like 64K of memory, and you run the thing, and in fact it algorithmically generates about 100 megabytes worth of data, you know these rich 3D environment, generated music, generated wave files, generated animation.
>And they're developing techniques to generate, you know, huge amounts of interesting data, with very very simple, elegant, compression algorithms.
>And this is a skill that game developers used to have, back in the 8-bit days. That was the only ways to do a game like Karateka(?), was to find all these little tips and tricks to compress things and generate them algorithmically.
>But since the CD-ROM came out, and very cheap hard drives, storage is cheap, so basically we've lost that skill set, and now we attack all those problems with brute force. I think we've lost something by dropping that skill set.
[...]
https://news.ycombinator.com/item?id=36613058
[...] Here's a simple low-tech pre-LLM example that shows the equivalence of compression and procedural content generation:
Take a huge text file of HN postings, and compress it with gzip or compress or some other robust compression algorithm. The better the algorithm, the more the output will look like random noise. Then slice the compressed file in half, and replace the second half with random numbers. Then uncompress it. You'll find that at the point you sliced it, it keeps on writing out almost plausible text for a while, consisting of highly probably snippets of commonly encountered words and phrases, then goes downhill towards incoherence. It's not as coherent or confident as an LLM, but the point is to show how low the bar is for using compression for procedural content generation.
LLMs are essentially a form of compression of the world's knowledge or whatever they're trained on, not just word frequencies or pixel patterns, but also concepts and ideas. [...]
Not without attention or something approximating it.
The fact that gzip is relatively fast should be your first clue that something important is missing.
Gzip is great at predicting the next token for one very specific narrative. LLMs can predict next tokens for entire universes of narratives. Searching for the correct next token across this space scales ~quadratically with the input size. Gzip scales linearly. I can gzip a one terabyte file. Imagine feeding that much into an LLM. These are wildly different animals that happen to overlap in a very small way. Equating compression to intelligence looks increasingly silly to me.
If we must compare language models to compression, they are much more like jpeg and mp3 than they are gzip and flac. I can go fuck with a jpeg file pretty severely at the bitstream level and still have something resembling performance on the other side. Gzip cannot remotely approach this.
> Gzip scales linearly. I can gzip a one terabyte file.
In part because gzip only has a 32KiB window size, and I think it'd be at least quadratic within that window if you were going for optimal compression.
I'll concede the window part, but Gzip runs within the physical confines of a single cpu core and is typically entirely resident in local caches. The point is not just the quadratic scaling but also what it scales with.
Show me an LLM that can run at 300 megabytes per second. Even dedicated ASICs with weights burned in will never move this fast.
It won't move as fast as gzip provided the same resources, but to say that LLM's can never run at 300mbps on any hardware is ... a bold bet.
Match-finding does not need to be quadratic. However, truly optimal gzip block splitting is very slow, indeed.
Perhaps a better question is if LLMs are used as compressors, how well is that expected to work.
> if LLMs are used as compressors, how well is that expected to work
Quite well. This project[1], by Fabrice Bellard of ffmpeg fame, is quite old in AI years and uses an ancient LLM, but still beats xz by a solid margin.
[1]: https://bellard.org/ts_zip/
Makes me wonder if compression ratio can be used as a measure for intelligence. Any benchmarks using it?
> Any benchmarks using it?
A challenge as I understand it is reproducibility.
Normal LLM runtimes aren't typically fully reproducible even with same random seeds for distribution sampling, due to floating-point numbers, batching and such.
Though averaging over many runs could alleviate that I suppose.
While it would measure some aspects of intelligence, I'd argue it fails to capture other, more creative aspects.
Extremely well, aside from speed.
> aside from speed.
And energy consumption.
I wonder if, on an information theoretical level, compression radiates heat, and decompression absorbs it?
I agree, but also equating LLMs with intelligence is wrong.
I think not. it's true that a large language model compresses knowledge and allows us to decompress knowledge. but gzip compresses data and not knowledge. with a LM you can decompress various forms of knowledge from the same data. gzip is a 1 to 1 kind of decompression where as an LM is a 1 to infinity kind of decompression.