[haiku-development] A modest (FatELF) proposal
- From: Landon Fuller <landonf@xxxxxxxxxxxxxx>
- To: haiku-development@xxxxxxxxxxxxx
- Date: Sun, 18 Nov 2012 13:02:25 -0500
Hello All --
This would be my first post to the list, and while I'd intended to not send
anything until I sat with patches in-hand, I was convinced on IRC that I ought
to float my idea of introducing FatELF before spending too much time on it, on
the chance that it proves to be non-viable for inclusion. My hope is that any
controversy can wait until I have a least a proof-of-concept working, and that
this posting will instead elicit some suggestions or advice from those more
knowledgable than I.
With the combination of x86-gcc2, x86-gcc4, x86-64, and potentially ARM, there
are a plethora of architectures that Haiku currently does -- or will --
support. In addition, certain architectures -- such as ARM -- are evolving
relatively rapidly compared to x86, with new functionality introduced across
rapidly deployed revisions of the processors. On iPhones, we've seen the rapid
introduction of various iterations of VFP and NEON, THUMB-2, etc. On Linux,
they've struggled with weighing the costs of breaking ABI with the advantages
of adopting new functionality available in new processors.
These ABI shifts have the potential to introduce significant maintenance burden
on just about everyone -- the user has to choose between different binary
versions, the OS vendor has to decide at which point to break ABI (if at all),
and the developer has to rebuild their application individually for each ABI,
and provide separate downloads or packages. We see from the Linux, BSD, and
Windows worlds that even on x86, this results in completely distinct OS
distributions and packages. Transitioning from x86-32 to x86-64 requires a
complete reinstall, as would (for example) transitioning from armv7 to apple's
armv7s -- a small iterative improvement that adds VFPv4, and one that hardly
justifies a full OS reinstall.
In terms of solutions, the issue of ABI compatibility isn't just one of
selecting the right libraries at runtime, either, and can't be solved just by
dynamically providing the correct library ABI on demand. Applications that host
plugins must have an ABI that matches that of the plugins they load. This
introduces some interesting issues during transitionary periods, as not all
plugins will be updated immediately, and as such, it proves impossible to allow
for a gradual transition by third party plugins to a new architecture -- it
must be all or nothing.
Fat binaries ease this transitions considerably; they were leveraged by NeXT
when they were moving beyond m68k, by Apple when they were transitioning from
m68k to Intel, and further leveraged heavily during the Mac OS X transitions
from PPC to PPC64, PPC to x86, and mixed x86/x86-64 installations, in addition
to the iOS transitions from ARMv6->ARMv7->ARMv7s.
Fat binaries make it easy for the OS vendor, user, and software developer to
support new architectures through gradual migration, by pushing the cost of
doing so onto the development toolchain. Rather than compile and distribute
something N times for N different architectures, your build process can run the
compiler with the appropriate -arch flags, and compile a binary that runs on
all the available architectures. When hosting plugins, the hosting application
can automatically 'downgrade' to the lowest common denominator ABI by
relaunching itself, thus allowing for gradual transitions between old ABIs and
new ABIs -- this was done by Apple in System Preferences when the entire OS
switched over to x86-64, allowing System Preferences to switch back to i386 in
the case where the user still had legacy plugins installed.
From the user's perspective, there's nothing to think about. There's a single
binary to download, and a single installation CD/DVD to run, and the right ABI
is used depending on their host environment. From the software developer's
perspective, things are equally easy -- by adding new -arch flags to their
compiler invocations, they can automatically target multiple architectures/ABIs
straight from the same build, and provide a single binary to run. Transitions
from one architecture to another can occur gradually, without every single
third-party developer and OS vendor supplied application making the transition
at once.
The downsides are minimal; most of an application's size is consumed by
resources, rather than code, and it is entirely possible to strip out unused
architectures for the particularly space-concious user. In reality, this hasn't
been an issue, even on iOS where application downloads are performed over
cellular, capped at 20MB, and have to fit on devices with as 'little' as 8GB of
space.
Apple's solution is based on Mach-O, and involves changes to their runtime
linker, kernel loading, a compiler driver that sits in front of the actual
cross-compilers, binutils changes to support linking, gdb and lldb support for
parsing 'universal' Mach-O binaries. For ELF systems, Ryan Gordon has gone to
the considerable lengths of designing a formal specification, a set of portable
tools to work with fat ELF files, and patches for glibc, the Linux kernel,
file(1), binutils, gdb, etc: http://icculus.org/fatelf/ His work appears to be
particularly solid, and as a proof of concept, I'd like to bring it over to
Haiku.
All that said, FatELF was met with staunch objection in the Linux world.
Despite the advantages of fat binaries demonstrated across 20 years of
architectural transitions and ABI shifts, there was considerable pushback from
Linux developers that believed that packaging negates the need for fat
binaries, or that fat binaries are simply not useful for users or developers at
all. This, despite evidence of painful ABI updates, the inability to easily and
iteratively transition from x86-32 to x86-64 operating systems without a
complete reinstall, and considerable evidence that fat binaries greatly
simplify architectural transitions for users and developers alike, including
commercial developers targeting a platform.
I think it's the case that fat binaries can augment package management, as well
as work entirely without it: fat binaries make it easier for a developer to
produce multi-architecture packages, which can either be thinned server-side
into single-architecture packages, or thinned client-side, depending on
requirements. However, not shipping actual fat packages makes transitions such
as that of x86-32 to x86-64 much more difficult (generally requiring reinstall,
or lockstep update of all components), and I think that shipping fat binaries
makes sense.
Additionally, I've been told that there's no need to handle multiple
architectures this at the fat binary layer, and packaging could do this at a
higher level. That may be true, but the advantage of doing this with binaries
is in simplicity for both user, developer, and implementation. The developer
doesn't need to build multiple binaries and merge them into a package, they can
simply apply -arch flags to their build configuration, linking against the fat
libraries available on their system. The user doesn't need to 'install' a
package, they can run a binary in-place, wherever it is, and the right
architecture will be selected and run. This is something my company leverages
in our build systems, where we can check in external tools as binaries (such as
protobuf's protoc), and then simply run them in place, whether or the user is
on a PPC, i386, or x86-64 system. The runtime implementation itself is
extremely simple, requiring only minimal parsing of the file header before
selecting the section of the file to map -- most of the cost is placed on the
toolchain pieces responsible for linking.
TL;DR In conclusion, I wanted to float the idea of FatELF, since it has a broad
impact on tooling and distribution issues. That said, I don't think the
question of inclusion has to be decided up-front without even a
proof-of-concept implementation -- I'm working on bringing over the FatELF
tools now, and would like to get to the point that a proof-of-concept gcc2/gcc4
fat hybrid is running. However, I also don't want to waste my time on it if,
like the Linux community, the response is a resounding and absolute "no, never
ever!" from all parties involved. I haven't been here long, but so far I
haven't met any Ulrich Dreppers in Haiku:
http://www.redhat.com/archives/fedora-devel-list/2009-October/msg01118.html
Thanks for making it through all of the above (or at least reading the tl;dr) :)
-landonf
Other related posts:
