[openbeos-build-team] Re: Goals for the new build system

  • From: "Michael Phipps" <mphipps1@xxxxxxxxxxxxxxxx>
  • To: openbeos-build-team@xxxxxxxxxxxxx
  • Date: Thu, 04 Jul 2002 14:02:20 -0400

:-) Actually, I want the whole project done yesterday. ;-)
I want R3 by tomorrow. ;-)

I think that the 2.4 work and the looking at the content of the rules is good.
Sure, going and changing everything right now is kind of dumb, since the 
will all get moved around and require another touch. But remember that that is 
all it is -
we aren't actually changing the structure of the source code. Just the org of 
the directories.

I am very much a believer in the concept that the build system should "just 
I (as a developer) don't want to spend a lot of time making build stuff work. 
Honestly, as we
have talked about (not sure if it was here or on the admin list), I would love 
to have a build system
that uses heuristics to figure out how to build, possibly with a little 
configuration necessary if we want
to do something complicated later.

And Jam seems to be a lot of the way there, as it was 2 months ago, for 
everything but the kernel.
I looked at some of the non-kernel stuff and it was like "ok - my target is an 
application called foo and 
my source files are foo.c and bar.c". That was it. And this is all it should 
need to be. This .GLOB and an exception list
sound like a really good thing. Is there a way to take that a step further? 
Assume that, if there is no Jamfile, that we are
building an application, that the name of the app is the same as the current 
directory and to compile all .C and .,c files?

Now, I know that the kernel stuff is somewhat more complicated by its nature. 
And some of it was a hack job
by people like David and myself who don't know Jam (and don't want to). ;-) I 
am thrilled that you guys have a passion to
clean it up. I (and I presume David feels the same way) don't have any 
particular pride or interest in the build stuff that 
we did. Whatever is cleaner and works better makes me happy. ;-)

As far as the seperate objects directory, I *strongly* prefer that objects not 
be stored in the same directory as the code. We store
everything inline at my "real" job and I hate it. 
The biggest reasons for that are these:
1) When I am working in a directory, I want to see my source, not a bunch of 
intermediate files that the compiler needs. Yes, I could type: ls *.[Cch].
2) If I want to add a directory, I have to specify which files to add, instead 
of cvs -z8 add *.
3) If I run low on HD space, I can cd to the objects dir and type rm -rf *. 
Instead of something like find . -name "*.o" -exec rm {} \;
4) The objects can be blown away easily after a build.
5) It is just cleaner. Sorry, Ryan. ;-)

Finally, I want to thank you guys for doing this. It isn't a glory hound job, 
but believe me, it makes the developers lives better and it will make our 
ultimate product a better thing. And I appreciate that.

>I know you all wanted the new build system done yesterday 
>(metaphorically speaking), but I think something like this requires 
>more than just a few days of trial and error hacking.  Plus I hope you 
>would all agree that it would be a waste of effort to totally revamp 
>the build system in our current CVS tree when Erik is working on a new 
>tree and we would just have to spend extra time "porting" the new build 
>system over to that tree.
>So, given that, I've decided to explain what I feel are some of the 
>goals for the new build system.  Here we go:
>1. Building needs to be as easy as possible.  All someone should need 
>to do is "cd current; ./configure; jam" to get the whole tree built.
>We pretty much already have this now, so I don't think this goal is a 
>difficult one.  The main thing to do is make sure we get everything in 
>the tree to build correctly (which doesn't happen now) and also to get 
>rid of some compilation warnings.  Of course a lot of this is coding 
>related, and while it might be better if the authors of the modules 
>giving warnings fixed them, it might just be the case that my build 
>team fixes them for the sake of expediency.
>2. The Jam files should be as simple and easy to maintain as possible.  
>That way it won't take legions of build team members to maintain the 
>tree, but just a few newsletter articles from myself explaining Jam and 
>our build system to the masses so that they themselves can learn how to 
>maintain the tree.
>In my opinion we definitely don't have this now.  No disrespect to the 
>current Jamfile/Jamrules authors, but even I can barely figure them out 
>;)  I can't know for sure if this goal is possible given our complex 
>build requirements, but I think it at least deserves to be tried.  
>Also, mostly what I mean by "maintaining the tree" is that new people 
>should be able to easily add their modules with proper Jamfiles, so 
>that the build team doesn't have to go back in later to fix them. 
>3. We should reduce the hard-coding of source files in Jamfiles as much 
>as possible.  This is especially true in the kernel Jamfiles where we 
>have (for example) all the libc .c source file listed out in each libc 
>sub-directory, and then in the higher level kernel Jamfile, we have the 
>libc .o files listed out, with GRIST and all, so they can be build into 
>a static lib.
>That is not how we should do this.  I will address this more in my next 
>point, but basically if one is to hardcode (for example) bcopy.? 
>anywhere, it should only be once, and it should be bcopy.c in the libc/
>string sub-directory.  But in fact, I'm not entirely convinced we even 
>need the hard-coding at all.  Since we are using Jam 2.4 now, which has 
>the GLOB rule, we could just use KernelObjects [ GLOB . *.c ] ; in each 
>kernel sub-directory.  Or, if we also need the $(SOURCE_GRIST) appended 
>to each source (which must be the case given the current Jamfiles), we 
>could write a rule to do this using GLOB and a for loop, so that again 
>we have avoided the hard-coding.  I say if we can use the power of Jam 
>to make our build files simpler, we should.
>The only problem with GLOB is that if we have .c files in the current 
>directory that we don't want built (for some reason) they would 
>"accidentally" be added to the list.  Though I would really only 
>imagine this happening during development on someone's local tree, it 
>might happen.  In that case it would be trivial to add a second 
>optional parameter to the rule I described above that would be a list 
>of files to ignore.  Within the for loop there would be an if statement 
>seeing if the current file was in the ignore list, and if it was, it 
>wouldn't be added to the final list.  Doesn't that sound more elegant 
>than hard-coding all the file names?
>4. Each sub-directory of the tree should be self-contained, and higher 
>level directories should only make use of the "biggest" targets in each 
>of their sub-directories.  This means I should be able to go into any 
>directory of the tree, type jam, and get the targets in that directory 
>and lower built.
>It seems we pretty much already have this, except for my second point.  
>As I mentioned in point 3, the kernel Jamfile actually builds the 
>static libraries for libc and libm, as well as the kernel itself.  In 
>my opinion, the only thing the kernel Jamfile should be doing is 
>putting together all the various high-level targets from each sub-
>directory into the kernel.  For example, libc.a should be built by the 
>Jamfile in the libc directory, and then just linked in with the kernel 
>by the kernel Jamfile.  Basically I'm just advocating low-coupling and 
>high-cohesion in our build system.
>5. We will have new pseudo-targets for most major functions of the 
>build.  To create a distribution, once will be able to do jam dist.  To 
>install things on your local machine, one will be able to do jam 
>install.  To build the unit tests, one will be able to do jam tests.  
>To run those tests, one will be able to do jam runtests.
>Of course right now we have none of this, and I'm sure you all agree 
>they need to be added.
>6. One final point, though it isn't really a requirement of the build 
>system: could you guys explain the reasoning behind the separate 
>objects directory?  In my opinion (which may not be the best since I've 
>done very little C/C++ development), outputting the object files to a 
>separate directory doesn't give that much benefit, but does have 
>several drawbacks.  It makes the Jamfiles harder to write and it 
>"hides" the object files such that if someone is interested in looking 
>at them, they have to cd into another entire hierarchy to find them.  
>Also, why use x86.R5 directory in the objects directory?  I would 
>imagine that until we have a system for cross-compiling (which I doubt 
>we will), it is pretty redundant to list the platform something was 
>built on, since one tends to already know that already (i.e. you can be 
>pretty sure those objects files are for x86 if you are building on 
>The reason I say this is that it just seems more "clean" and organized 
>to me to keep the objects and other targets for each directory and 
>Jamfile in the same directory as the sources and Jamfile.  If you feel 
>that having all those .o files with the .c files makes things "messy", 
>how about an objects directory in each source directory?  Hmmm, though 
>I'm not sure about that as far as Jam is concerned.
>I guess I'm sort of on the fence on this issue, but I would like to 
>read opinions as to why the whole separate objects tree should be used 
>(and not just "it makes the source tree cleaner".)  Because keep in 
>mind that eventually we will be adding more pseudo-targets like I said 
>above, and I have a feeling having this separate object tree will make 
>those a lot more complicated.  Also keep in mind that "jam clean" works 
>very well and is completely automatic, so it is not like putting the 
>objects with the sources makes that any more complicated.
>So let me know what you think about this as well as any other views or 
>input on the new build system.
>Ryan Leavengood
>OpenBeOS Build Team Lead

Other related posts: