[haiku-development] Re: Package Building and Beta1
- From: Michael Lotz <mmlr@xxxxxxxx>
- To: haiku-development@xxxxxxxxxxxxx
- Date: Tue, 1 Aug 2017 10:42:41 +0200
Hi
On 30.07.2017 18:53, waddlesplash wrote:
Under the new model (buildmaster, or kitchen) there will be no such
list of packages in master (or if there is it'll be pretty different,
I think) and so the server must preserve all the old repositories so
that old commits can continue to be built as before. (If we want to
keep the list of packages in the tree, I think the thing to do would
be to cut out all but the packages that Haiku uses, and then instead
of transparently generating a repo file -- which is what is done now
-- just download those packages from the specified branch directly.
Which might be a good idea, anyway.)
I can see need for keeping around packages for the sake of being able to
build older hrevs. At some point we have to decide how long we're going
to keep them, and which hrevs and packages are going to be kept, due to
storage.
One problem with that premise is that you would already need a set of
packages to build the hrev, while you need the hrev to be built to build
the packages for it. With incompatible changes like the time_t change,
packages need to be built for that hrev after the Haiku package is
available. So to make a nightly available the process would have to be:
1. Commit changes to Haiku
2. Build Haiku packages with existing set of dependency packages
3. Rebuild packages for that new hrev
4. Build nightly with all new packages
However this presents another problem: The recipes do not change, but
the packages do. Deduplication based on file names would then fall flat.
We could require all recipes to be bumped in such a case, ideally
marking them with a dependency on the new hrev.
I guess that's roughly what currently happens manually and is similar to
what a bootstrap would entail. It will require some coordination between
recipe changes, package building and buildbots in any case.
The other reason to keep all the old repositories around is to keep
the present behavior we have, that each Haiku nightly is not shipped
with "current" repositories, but with the repositories for their
specific hrev. And the repository files are a few hundered KB; so,
there's really no reason not to keep them.
I don't think this is going to work indefinitely due to storage.
Certainly the repo files are small, but the packages aren't. Of course
they can be shared across hrev repos, but at some point we have to
lifecycle them and therefore need to know which of the packages are
still referenced and which aren't. Hardlinks would seem appropriate in
this instance.
Symlinking would probably work for this, however this makes things like
package obsoletion and general management less safe. Requesting the any arch
on one of the instances, or making an any arch instance would possibly be
saner.
package_repo does not allow creation of 'any' arch repos. I asked why
this was (on the mailing list, I think) in 2015, and I seem to recall
getting an answer that was more or less satisfactory. I don't remember
what it was, though.
Yeah, I didn't mean making an any arch repo, just having a separate
buildmaster instance that only concerns itself with building any arch
packages and making them available to be used in the other repos.
As was mentioned, the any arch package set is different across archs and
I agree that it would be cleaner to not have any arch packages available
on repos where they can't really be installed (or aren't useful) due to
their corresponding packages being unavailable. It also means that the
above can't really work (an any arch instance) as there would otherwise
have to be one per architecture, defeating the purpose.
So in the end, yes, the only real option is to use a common packages
directory and then producing repos from the actual set of packages
available for the respective arch. This avoids duplicated builds of any
arch packages while still producing repos only referencing the relevant
packages of the arch. This is how createrepo.sh currently works (i.e. it
uses HaikuPorter to enumerate the packages that are relevant and should
go into the repo), so this should already work.
Well, that's what we presently use, so, slightly messy as it is, it
does work. Moving the "ARCH" folder to be before the
"packages|repository" fork in the URL scheme sounds fine to me,
though. (I'm not sure why it "doesn't scale"?)
Doesn't scale as in "have you listed one of these directories lately?".
With thousands of packages, this process becomes a lot slower. Since
HaikuPorter needs to go through the set of available packages during
dependency resolution, this will become a performance bottleneck at some
point.
I hope all this makes sense; as I said, the stuff above about the URL
scheme I discussed with kallisti5 and jessicah on IRC some weeks ago
(although in much less detail). I could dig out our conversation if
you think it might prove additionally helpful.
It does indeed make sense, thanks for taking the time to explain.
As for status:
Yesterday I brought up my local x86_gcc2 builder and started making
packages and set up the buildmaster web frontend for that arch.
But shortly after that the server started to collapse in on itself.
Everything was being OOM killed, even though the processes all used only
reasonable amounts of RAM. The kernel messages also showed that there
seems to be irregular soft lockups on that VM:
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s!
They seem to happen from time to time. There were abrt-dump-journal
processes running that consumed quite a bit of memory, so both issues
might be related.
This morning the VM seems entirely unavailable, so I can't really
continue. Who can look into these issues?
Regards,
Michael
Other related posts:
