[haiku-depot-web] HPKG/HPKR file format change(s)

  • From: Ingo Weinhold <ingo_weinhold@xxxxxx>
  • To: haiku-depot-web@xxxxxxxxxxxxx
  • Date: Sun, 13 Jul 2014 00:02:27 +0200


just a heads up that the HPKG/HPKR file format has changed slightly, which may be relevant for the web app's parser.

Formerly the heap compression type B_HPKG_COMPRESSION_NONE was never used (only B_HPKG_COMPRESSION_ZLIB). I've changed the package kit and the package tool can now create files with this compression type and I've changed the file format for this case slightly: The chunk size table at the end of the heap is redundant (all entries would be 65535) and it is therefore omitted. [1]

This results in minor size savings, but more importantly now there's no reason for the parser to read the table from the end of the file before accessing the heap. This means recompressing an uncompressed package file can be done from a sequential stream (the package tool still needs to be adjusted for that), which is relevant for some intended package/repository changes:

1. Since the compression we use is not very space efficient, it would be nice to use a better algorithm for packages in the repository. The package file heap wouldn't use compression, but the whole file would be compressed e.g. via xz. On Haiku the downloaded package would then be extracted and recompressed to be suitable for use ("xzcat foo.hpkg.xz | package recompress - foo.hpkg").

2. To deal with the different compressions over the lifetime of a pacakge, the checksumming of packages needs to be canonicalized. The simplest way is to always checksum the uncompressed format ("package recompress foo.hpkg - | sha256sum" or likely "package checksum foo.hpkg").

3. I'd also like to introduce support for a new compression algorithm, LZ4 [2], since zlib is rather slow.

As usual, I can't say when I'll get to implementing these.

CU, Ingo

[1] https://dev.haiku-os.org/wiki/PackageManagement/FileFormat#Heap
[2] http://code.google.com/p/lz4/

Other related posts: