[haiku-development] Re: hpkgs and compression

  • From: Ingo Weinhold <ingo_weinhold@xxxxxx>
  • To: haiku-development@xxxxxxxxxxxxx
  • Date: Thu, 20 Mar 2014 22:39:44 +0100

On 16.03.2014 17:06, Jonathan Schleifer wrote:
Am 16.03.2014 um 16:48 schrieb Julian Harnath
<julian.harnath@xxxxxxxxxxxxxx>:

Another option would be switching the hpkg compression algorithm
to LZ4, which is optimized for performance and many times faster
than DEFLATE. It's the commonly used compression algorithm in ZFS.
See benchmark comparison here: http://code.google.com/p/lz4/ (in
Web+ that table looks messed up though)

I agree that this would be better for storing it on disk, or maybe
LZO. The problem is: That means larger files to transfer over the
net, as compressing something compressed is inefficient. That's
exactly the whole point: We should not use the same compression for
transfering an hpkg and storing an hpkg. That was my idea: Leave the
hpkg itself uncompressed, and compress it on the file system (e.g.
using LZO, LZ4) and compress it during transfer (e.g. LZMA). That'll
give the best of both worlds.

*The* file system for Haiku is BFS and I don't see that change anytime soon. Since it doesn't do any compression, not compressing the packages themselves means they wouldn't be stored compressed at all.

Regarding the algorithm, the package format was designed such that new compression algorithms can be added easily. [1] suggests that there are algorithm implementations that need only a few ns per byte to decompress (it's for text data -- not sure, if that makes a significant difference), so even with SSDs that should be beneficial, or at least not detrimental.

CU, Ingo

[1] http://mattmahoney.net/dc/text.html

Other related posts: