[haiku-development] Re: Package Building and Beta1
- From: waddlesplash <waddlesplash@xxxxxxxxx>
- To: Haiku Development ML <haiku-development@xxxxxxxxxxxxx>
- Date: Sun, 30 Jul 2017 12:53:18 -0400
On Sun, Jul 30, 2017 at 3:22 AM, Michael Lotz <mmlr@xxxxxxxx> wrote:
Well, buildmaster doesn't create repos (because it is only concerned with
packages), create_repo.sh does.
Ah, OK. kallisti5 said that buildmaster didn't have a way to make
repos, I guess he was wrong then.
Can you elaborate on what you came up with?
Sure (see below).
Are the for_hrev meant to hold packages built
with this Haiku hrev as a basis or are these just sets that are compatible
with that hrev?
Both -- i.e. all packages in that repo are built on or before that
hrev and are thus compatible with that hrev.
Is there supposed to be a repo for each hrev then?
Not precisely (see below for more on this).
How do
release branches factor into this, isn't there going to be just one repo for
the whole beta1 line?
No, there will not be. So here's how this goes:
The URL scheme is: /TREE/BRANCH/packages|repository/ARCH/for_HREV|current/
TREE will almost always be 'haikuports'; although in the future there
may be overlays, alternate repos, etc. etc. So that's just there for
future-proofing.
BRANCH is the branch that the repository came from. So, right now
there is "master"; but as soon as we "git branch" R1beta1 (or just R1,
or whatever) in HaikuPorts, there will then be "master" and a
"r1beta1" folders there.
There is one 'packages' directory shared between all arches for 2 reasons:
* Allows for easy 'any-arch' package sharing
* Allows for easy rsync'ing to mirrors (as the 'packages' directories
inside the individual repository directories -- which are what
packagekit actually looks in to download packages -- are just
symlinks.)
ARCH is, well, self-explanatory.
The last node is either "for_hrevXXXXX" or "current" (where "current"
is just a symlink to the highest "for_hrevXXXXX".) As stated above,
the hrev number comes from the hrev of the machine the packages were
built on (although for Buildmaster which stores the base haiku_
packages on the server, this should be even easier to do). The reason
for storing all the old repositories (it is assumed that the server
will mostly keep in sync with Haiku's 'master', so the numbers will
increment regularly) is for reproducible builds on Haiku's end.
At present, we have pseudo-reproducible builds, as the repository
files are generated from a list of packages presently stored in the
tree; and only the packages themselves are stored online. So when you
"git bisect", jam will download the old packages that commit had, etc.
So any one commit will always get the same package set, no matter what
has changed in "current" in the meantime.
Under the new model (buildmaster, or kitchen) there will be no such
list of packages in master (or if there is it'll be pretty different,
I think) and so the server must preserve all the old repositories so
that old commits can continue to be built as before. (If we want to
keep the list of packages in the tree, I think the thing to do would
be to cut out all but the packages that Haiku uses, and then instead
of transparently generating a repo file -- which is what is done now
-- just download those packages from the specified branch directly.
Which might be a good idea, anyway.)
The other reason to keep all the old repositories around is to keep
the present behavior we have, that each Haiku nightly is not shipped
with "current" repositories, but with the repositories for their
specific hrev. And the repository files are a few hundered KB; so,
there's really no reason not to keep them.
Symlinking would probably work for this, however this makes things like
package obsoletion and general management less safe. Requesting the any arch
on one of the instances, or making an any arch instance would possibly be
saner.
package_repo does not allow creation of 'any' arch repos. I asked why
this was (on the mailing list, I think) in 2015, and I seem to recall
getting an answer that was more or less satisfactory. I don't remember
what it was, though.
I find it rather messy to have all architecture packages mixed together for
serving. It doesn't really scale well either. A sub-directory per
architecture seems reasonable.
Well, that's what we presently use, so, slightly messy as it is, it
does work. Moving the "ARCH" folder to be before the
"packages|repository" fork in the URL scheme sounds fine to me,
though. (I'm not sure why it "doesn't scale"?)
I hope all this makes sense; as I said, the stuff above about the URL
scheme I discussed with kallisti5 and jessicah on IRC some weeks ago
(although in much less detail). I could dig out our conversation if
you think it might prove additionally helpful.
-waddlesplash
Other related posts:
