I would compress it all, and then selectively recompress at higher compression levels depending on the link, read frequency, diversity and capabilities of the clients.
Zstd 3 to 5 is nearly free in terms of not bottlenecking disk or network. Zstd 12 to 19 gives amazing compression results and still result in speedups when reading from disk. It really is a wonderful all purpose compressor.
One of the nice things about Zstd is if you try to compress an already compressed stream, it short circuits. So even if you are given say HVEC MP4 and run zstd -19 on it, it will "compress" immediately and not DOS your pipeline.
I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.
I assume the entire resource needs to be decompressed first, then indexed into, served, and discarded. Well, actually, you could just decompress up to the end of the range.
Which would have terrible performance for range requests starting late in a large file. For files that are frequently accessed that way, this could be prohibitive.
You could split the file into independently compressed blocks as well. But that'd reduce compression rate and require adding some kind of index for seeking.
Or they have an upper size limit for the file size they compress, since large files are rarely compressible text.
In any case it is something that needs the be handled before going live with a compressed cache. But the article sounds like they simply didn't implement compressed caching for those cases, which makes no sense.
Not with zstd, you could still support range requests. https://en.wikipedia.org/wiki/Zstd this whole subthread should take 10 minutes and glance over the spec and the capabilities. It would end a lot of wasted premature pontificating.
Why not serving files compressed if the client supports it even though the origin served an uncompressed file?
I was thinking this. Zstd is widely supported in browsers, over 80% right now and will increase over time:
https://caniuse.com/?search=zstd
> We initially considered limiting transcoding to popular content
Weird, I would have compressed cold content instead, if the goal was to save on CPU time during decode.
I would compress it all, and then selectively recompress at higher compression levels depending on the link, read frequency, diversity and capabilities of the clients.
Zstd 3 to 5 is nearly free in terms of not bottlenecking disk or network. Zstd 12 to 19 gives amazing compression results and still result in speedups when reading from disk. It really is a wonderful all purpose compressor.
One of the nice things about Zstd is if you try to compress an already compressed stream, it short circuits. So even if you are given say HVEC MP4 and run zstd -19 on it, it will "compress" immediately and not DOS your pipeline.
It has mostly become my go-to as well. Just wish it wasn't a Facebook product.
I'm confused by how this affects range requests. Without compression, those can be easily satisfied by reading the relevant part of the cached complete file. But how are they handled now? The article claims "range requests remain unchanged", but I don't see how that's possible if the cache no longer stores the uncompressed data.
I assume the entire resource needs to be decompressed first, then indexed into, served, and discarded. Well, actually, you could just decompress up to the end of the range.
Which would have terrible performance for range requests starting late in a large file. For files that are frequently accessed that way, this could be prohibitive.
You could split the file into independently compressed blocks as well. But that'd reduce compression rate and require adding some kind of index for seeking.
Or they have an upper size limit for the file size they compress, since large files are rarely compressible text.
In any case it is something that needs the be handled before going live with a compressed cache. But the article sounds like they simply didn't implement compressed caching for those cases, which makes no sense.
Not with zstd, you could still support range requests. https://en.wikipedia.org/wiki/Zstd this whole subthread should take 10 minutes and glance over the spec and the capabilities. It would end a lot of wasted premature pontificating.
Yes we can always trade space for time and time for space.