[cldp] Re: libraries wrapping incompatible interfaces

  • From: "Benjamin Rich" <benjamin.rich@xxxxxxxxx>
  • To: cldp@xxxxxxxxxxxxx
  • Date: Tue, 18 Apr 2006 15:17:10 +1000


On 4/18/06, Karol Pietrzak <kap4020@xxxxxxx> wrote:
> On Monday 17 April 2006 19:51, Benjamin Rich wrote:
> > Karol,
> > > Also, do you know, offhand, of recent distribution which doesn't use
> > > CUPS? This is part of the research I'll have to conduct to make the
> > > Printing Layer usable.
> >
> > I don't know offhand, but I think if you went to
> > http://distrowatch.com/ and just checked out the spoolers used in the
> > top 15 distros, that would give you what about 95% of linux users
> > have.
> Looks like CUPS is the default in SuSE, Mandrive, and Ubuntu.  That covers a
> vast majority.
> I wonder whether the API is stable, or whether it's going to be an adventure
> wrapping that.
> On a semi-related note, KDEPrint used non-public functions of CUPS that won't
> exist anymore in the next big stable CUPS release.
> http://www.kdedevelopers.org/node/1901

Looks like CUPS is the go then. I'd say it's API is pretty stable,
given it wants to be the common unix printing system.

> > > Apart from keeping it in a file somewhere on my computer, is there a
> > > Wiki page or something I could put it on?
> >
> > Haven't setup a wiki yet, but I might, what did you mean?
> Well, I was thinking about storing all the information and research we've been
> gathering in a central location for others to view, contribute, and update.
> The notes I keep on a file on my computer aren't doing anyone but me any good.
> Ideally, I'd like a full-fledged bug tracker, Wiki, etc. (Possibly Trac).
> Should we open up a project at GNA or Sourceforge or something?

Totally agree. At the moment I've just had lots of things to do, not
the least being fleshing out how the whole CLDP will work so we have
something half decent to present.

Bug tracking => bugzilla seems to be the defact, I'll set this up as
soon as we have some code, and some bugs =)

As for a public wiki, I've never set one up before. I dabbled with the
mediawiki software (base for wikipedia etc.) but it seemed large and
complex and frightening. I don't know if public contribution is what
we want so much as a good site to succinctly present the CLDP. But as
far as user-interaction, I was thinking maybe an interactive FAQ?

An irc channel and a sourceforge site, and all the usual trappings of
a good OSS project I think would be a good idea, I just haven't had
time to set them up yet.

> > > I see... I don't know what the KDE equivalent of gnome-volume-manager is,
> > > but the Volume Layer will have to wrap them.  For example, if an ISV
> > > specifies that RealPlayer can now play DVDs, both GNOME and KDE should
> > > know about it.
> >
> > I'm still against too much wrapping in general. This is a job that I
> > think should be done by an intermediatory component, like the xdg
> > tools - ie. gnome, kde, and the cldp use these tools to register
> > associations or default actions for the volume managers; not, cldp
> > wraps an ever-growing collection of incompatible desktop systems to
> > provide a small extra bit of efficiency. Then again we're at the mercy
> > of the distros and the desktop makers.
> >
> > Maybe there's a tool which already does this? Or we should suggest
> > this to the xdg list?
> Yeah, this definitely sounds like something for freedesktop.org to
> standardize.  But, on the other hand, this is more complicated than opening
> up a URL (i.e., xdg-open) and the desktop environment may find the minimal
> XDG wrapping to be insufficient for their needs.
> Or, if the freedesktop folks don't want to or lack the resources, we could
> write it and submit it _to_ them.
> Surely, I don't mean wrapping to sound as the be-all, end-all solution.
> Layers upon layers, like an onion, can get complicated.
> However, the situation is equally complicated.  Take the following into
> consideration:
> - FreeDesktop.org has a XML MIME specification out, but KDE3 does not use it.
> KDE4 is set to come out by the end of the summer (I believe...), but I
> imagine KDE3 users will outnumber KDE4 users until _next_ April.
> - GNOME versions prior to 2.8 use their own MIME spec as well
> The Autopackage folks actually already have a working solution to these
> issues, TODAY.
> http://autopackage.org/docs/devguide/ch08s02.html
> I feel that's just a small example of why we'll need to wrap many, many
> things, despite our utmost desire not to.  A quick browse through the
> Autopackage forums explains all the inconsistencies and incompatibilities
> present in today's Linux distributions.

It is true that cohesion amongst desktops is a huge quagmire. Given
the above, I suppose we should add in functionality to cover legacy
systems (eg. everything made up until last week =P ) and propose all
of these things to freedesktop to be standardized.

I really haven't looked deeply enough into the specific of xdg et al
and how all this stuff with mimetypes and menu shortcuts is
implemented yet, for which I'm regretful, but I've spent most of the
last week or two making the site and adding to the CLDP generals. As
of today I've just come down with a cold, so it doesn't look like I'll
be able to process all of this for a couple of days. Leave it with me
and I'll see how we can get this into bricklayer.

As for setting default policies in a chosen volume manager, I think we
should (or you if you have something already in mind) send this in as
a proposal to freedesktop.org. I would love to suggest something good
for it at this point, but my head is rapidly filling with stupidity
and fluids =P

> Another thing I'm thinking of in the back of my mind is Unixes other than
> Linux, which is something that the FreeDesktop folks are definitely keen on.
> After all, KDE and GNOME run on WAY more platforms than just Linux.  Older
> version of Solaris still abound, and if we could support a host of other
> Unixes (including the older ones), we would have a killer product!  Until
> then, we'll just offer a library to do things that a lot of developers can do
> by themselves anyway.

I agree, though the problem that the CLDP really solves (or is meant
to) is running desktop applications on Linux, specifically. As for
making it the common unix desktop platform, I don't think we'll be
able to until we have quite a body of code and can see where to go
next. But definitely something to think about =)

> Without the large breadth of support for even small players such as FVWM, I
> feel most developers would rather just:
> if(GNOME)
> else if(KDE)
> ...which is not the optimum solution.
> > Are you making a separate component which wraps some of
> > gnome-volume-manager to add extra functionality, or do you want to
> > wrap g-v-m to provide all functionality? I think we should make our
> > own simple component.
> I'm not quite sure.  I'll have to delve more deeply into the issue, but
> "[making] our own simple component" sounds good so far.  We must be careful
> to KISS, while still providing developers will everything they want to do.

Agreed, simple = win.

> > I'd say off the top of my head that it would all be GPL. AFAIK,
> > project utopia is hal/udev, driver-on-demand (now defunct), and the
> > love patches for the kernel (which everyone's stopped using because he
> > pissed off all the kernel and distro people a while ago for being
> > outspoken).
> Oh, I wasn't familiar with driver-on-demand.
> http://driverondemand.sourceforge.net/
> Could you briefly explain what happened to driver-on-demand and the "love
> patches for the kernel" and what they are?  I would be grateful.  I couldn't
> find much good information online.

Neither could I, but here's how I imagine the scenario:
Somebody, probably Love comes up with the idea, suggests it on LKML,
and is shot down by grumpy kernel hackers who instantly recognise and
highlight 4,000 potential implementation holes; Love creates own
project, sets up website; base of support is not available,
frustration with Linux never recognising anything except legacy IDE
RAID cards and 3-button mice still hasn't reached boiling point;
nobody helps out; project dies.

It seemed to rely on an infrastructure that just never got put in
place, having all these XML spec files lying around with drivers and
in databases. The essential problem is, I think, that there just
aren't enough drivers for linux or companies bothering to port drivers
to it, and the reason for that is not a lack of driver-on-demand.

But, it's something I want to integrate into the LDML, when I can get
around to it.

> > This is a good idea, though again I would recommend a more generic
> > approach. For example, all of these detection systems no doubt rely on
> > HAL. If SuSE's doesn't, then we should write in compatibility for it's
> > proprietary system since it's a special case and SuSE has like 30% of
> > the linux desktp market; but in general, I think we should try and
> > rely on the most underlying element. This means we only have to worry
> > about changes to that component (eg. hal/dbus), not any of the
> > distro-specific ones. Otherwise, we will be constantly updating for
> > the shifting sands of the distros, and will once again be at their
> > mercy.
> I see your point, and I've been convinced, heh.  Wrapping only that underlying
> component will save us a lot of time and effort.
> If we can pull this off, it would be grand.  I'm thinking of a few transitions
> that have happened in the not too distance past that would have killed ISVs,
> and now keeps them away.  devfs/udev and dnotify/inotify come to mind.  The
> latter would be easy to wrap, it seems to me.

I think, as always in OSS, it's just a matter of 'go with the
majority' - and the more control that majority gets, the more that
majority approches standardisation. inotify will eventuallywin out
over dnotify, because people will migrate, and because stuff like
beagle uses it, so devs will follow suit. udev is becoming the defacto
now; and so on.

This is partially why I made CLDP a toolkit/platform, it started out
as being concepts for an entire new linux distro, but then I realised
this would just be shoveling one more incompatible system on the heap,
that nobody would ever use because it wasn't Debian or whatever. If
you want to change 99% of the distros, I think you need to work with
them, not against them. Look at SkyOS - a totally POSIX compliant OS
built from the ground up, runs firefox etc., much more stable than
linux, cheap, effective - who uses it? Nobody. It doesn't have the
userbase and appeal of linux; and windows people will have never heard
of it.

I think we're lucky too because we've chosen a time when all of this
sort of thing is coming together on Linux - people seeing Linux's
potential for a desktop system, and finally shedding the
developer-only 'if you want it then make it yourself in raw c and stop
bothering me' mentality. I don't think any of this would have been
possible before freedesktop.org/xdg/LSB and apps like beagle etc.

> > > I'm not familiar with the Berkley license.  Are you referring to the BSD
> > > License?  http://www.opensource.org/licenses/bsd-license.php?
> >
> > Probably that one - it basically says code can be used for any reason,
> > any time, no stipulations. eg commercially too - NT 4.0 had some BSD
> > code in it for the TCP/IP stack, and windows probably still does to
> > this day (it sure as hell has DOS 5.0 code in it =)
> Heh, actually, I heard Microsoft re-wrote the netcode entirely somewhere
> around the Windows 2000 mark.
> > > FYI, here's a link to the zlib license:
> > > http://www.gzip.org/zlib/zlib_license.html
> >
> > Looks like the BSD license. *Exactly* like the BSD license. Seems
> > fine, though I think for the sake of getting our stuff accepted into
> > Linux, and since there are no commercial, closed-source Linux distros,
> > we'll want to make virtually everything GPL.
> Well, we can't make everything GPL because no closed source application will
> be able to use it.  LGPL is better, but I'm all for my code being under the
> zlib license, if not the public domain.  Please feel free to enlighten me,
> but for what we're doing, the less restrictions the better (at least for the
> code).

Here's the way I look at it. CLDP is a platform, a set of tools, for
all distros - this is because this is the only way, I felt, that it
would ever be used or accepted. Likewise, the only way people in the
OSS community will ever use it or contribute to it, is if it's GPL. We
also want to maintain some control over the code - since it's a
platform, it's one that has to be kept stable.

Nonetheless, if you really think that people will want to just
copy+paste the printing/device layers into the code initially, I don't
see a problem with licensing those components zlib/bsd. But why not
make them into loadable libraries? Then they can be GPL and commercial
developers can use them without opening their code. What are the
performance drawbacks of this approach?

> > What are the disadvantages of a shared library?
> >
> > Acceptance-wise I thought this approach would be good because it means
> > distros have to have versions of the cldp which complement their
> > systems, and so customisation gets done for us. BrickLayer, at least,
> > has to be installed for appfolder packages to work.
> Hmm, well, distribution inclusion is a different beast.  I was hoping we would
> have some usable code by this summer.  If we just leave it up to the
> distributions to customize, then the earliest we could make it is probably
> next year, me thinks.  Do you know the specifics of requesting a piece of

For distro inclusion, what I meant was that distros would be happy to
take it in and customise it to their own systems if it passed the
three-point oss developer political checklist, which is:
-is it GPL'd?
-does it use XML somehow?
-will it be compatible with GTK/deb/massively popular standard or technology?

I suppose zlib/bsd isn't really a drawback from this.

Also I didn't mean we'd rely on the distros to pick it up, we would
make versions of the CLDP tools for ubuntu, mandarke, suse, etc.
ourselves, but then hopefully some people would be interested and
offer to maintain the distro specific versions of, say, the LDBK and
some others things themselves.

> > But I agree, things probably won't get picked up immediately, so no
> > doubt our code will be cut+pasted to begin with. If we *can't* zlib
> > license everything though, and I think not using the GPL might be
> > hard, then commercial developers won't take on our code unless it is
> > in a dyanmic library, because this is the only way they can link to it
> > without opensourcing their code.
> What makes you feel that not using the GPL would be difficult?  As I see it,
> the less restrictive the license, the more exposure the software gets and the
> more people will use it.
> zlib and libpng have some of the largest penetration rates of any open source
> libraries, and neither of them is GPL.  The Apache webserver is licensed
> under the Apache License.
> I might be missing something, in which case, please feel free to correct me.

Okay, you've sold me =)
I suppose being an OSS person I hadn't imagined there was any other
license than the GPL =P

> > The the LD** scheme could be getting a little ubiquitous though, I was
> > thinking we should name each component? Like with sally and norwood,
> > we could have names for the ldml etc. - I was only ever intending 2 or
> > 3 components, so they seemed unique at the time.
> Hmm... marketing is not my thing.  I'm not sure what would work best.  We
> might very well end up with more than three components.  I like the full
> names (e.g., Volume Layer, etc.) you've chosen though. :)

Well if you can tell the difference then this gives me hope that
they're not too monotonous. I might come up with some code-names for
all of the projects, but mainly so we can refer to them quickly in
internal discussion, and also because I wanted to name them after
characters and things from battlestar galactica, the 2nd series of
which I just finished watching =P

But then, okay, for the moment we'll keep them as LD** and I'll change
them on the website accordingly.

> > Having a look now. Seems like something we could sponsor and use,
> > except for some things I've noticed:
> > "The klik server calculates dependencies either from entries in the
> > klik database or by using the debian package database.
> > ...
> > This means that the server needs to assume a certain set of
> > applications and libraries to be present in the base system. klik here
> > assumes the "least common denominator" of multiple distributions to be
> > present in the base system. "
> >
> > Great, except you can't assume anything. I think a local LDBK would be
> > much better here. You grab the app, it has meta-dependency info, you
> > get the distro to download and install it using it's local package
> > tree.
> >
> > Though how they've implemented things inside an image and so forth is
> > clever - but then how does a user run the app? A wrapper could be put
> > in /usr/bin which mounts the image and runs the default app I
> > suppose...
> >
> > This is just one more great system which I think still isn't
> > compatible with ours ideas. I'm going over all these though, I will be
> > updating the bricklayer page soon. Food for thought.
> Awesome, awesome.  I'll keep refreshing it on a regular basis.

Well I've come to the conclusion that basically I don't think we can
use any of these systems. All of them handle dependencies themselves -
autopackage downloads autopackaged versions of depdencies; klik does
some incredibly complex sounding server-side computation to try and
merge dependencies into the final app, possibly linking them
statically; and zeroinstall handles it's own dependencies outside the
system's package manager.

I don't think any of these will work, at least not unless I abandon
the entire concept I have so far, or we alter one of these projects
beyond recognition and come up with effectively a different system
anyway. The idea of the CLDP is that it doesn't handle dependencies at
all - it lets the system do it, and provides an interface for just
requesting a package, not a specific version (other than a major
version), etc.

I know creating our own new standard seems like death, but it would
seem to me that autopackage is the only one that actually poses any
sort of competition, the others are completely mickey-mouse, and
autopackage cant' be used.

*BUT*. I will take another look at autopackage and it's code, and see
if it might be possible to adapt it to allow us to install appfolder
style apps, and then grab depdencies from the LDBK. This way, instead
of it downloading other autopackages that fill depdencies, it can use
bricklayer/ldbk as an interface for getting deps from the system. I
have a friend who's expert in perl, I'll get him to have a look over
it with me.

> > > Also, do you think it's a good idea to create another package in the
> > > Linux world?  We already have incompatible RPMs, DEBs, klik packages, and
> > > Autopackages hoping to be distribution neutral.
> >
> > Yeah but we really only have debs, rpms and ebuilds. Klik and
> > autopackage have miniature userbases. Not until something can be
> > integrated into every distro, will one of these formats take over for
> > application installation - and that's just it, one set of package
> > standards is for system stuff, one is for desktop apps.
> >
> > Autopackage we simply can't use, because it installs everything to
> > multiple paths like rpm/apt. Klik looks promising. Rox and zeroinstall
> > I'll take a closer look at.
> Hmm... it would be very interesting to see appfolders develop for Linux.  It
> seems like quite the technical feat so far for them to work well.
> I wonder if Mac OS X has some code for us to use...

In a word, no.

Appfolders aren't really that hard - you just install all resources to
one place. You can also include system resources, like massive
libraries like gtk, with the appfolder too - the problem here is, that
on linux, there are about 7trillion different version of gtk being
built against any one time. apps that rely on gtk 2.6.1, 2.6.2, 2.6.3
and 2.6.4 can all use, say, gtk-2.6.8 which is installed by the
distribution. when they request functions, they'll be present in 2.6.8
and all these apps can share that library in memory.

but if you included those four minor versions with those four apps,
you'd end up with 4 different versions of gtk running in memory, and
the results are major slowdowns. in OSX, all the libraries are very
stable and  released on a slow cycle, so you don't have these
problems. Plus, they probably resitrict which versions can be used,
whereas we can't, without some sort of political backlash.

> > > There's immense technical challenges with app folders, I believe.  For
> > > example, .desktop files must be installed into a directory in
> > > $XDG_DATA_DIRS. So, the appfolder runtime will have to install the
> > > .desktop file into a directory in order for the application to appear in
> > > the window manager menus.
> >
> > Or, bricklayer will have to - which is fine. It can use the xdg tools
> > to do this.
> And then there will have to be an elaborate piece of software to realize when
> the user deletes the .app file and remove all the related .desktop, .mime,
> and what not files.
> I believe it will be very worthwhile posting to and reading the documentation
> of Autopackage, klik, etc. to see the technical challenges that they faced.
> I know for a fact that the Autopackage team considered making appfolders but
> just deemed it too difficult to implement and execute properly.

Well, klik, autopackage et al all rely on either a user uninstalled
via a script, or using a program to uninstall - any system on linux
can be broken by a user deleting some files. Autopackage decided not
to do full appfolders for the reasons above, with library duplication.

> > Good point, well made, will wrap xdg tools.
> Thank you... unfortunately, I feel lots of stuff will need wrappers.
> Fortunately though, once the wrappers are in place, it will allow the
> underlying mechanism to change without the API.  This, I feel, is one of the
> two biggest problems facing desktop Linux.  Binaries just can't run after a
> year or two because of the underlying changes.  To name a few:
> devfs -> udev/hal
> dnotify -> inotify
> Linux 2.4 -> Linux 2.6
> The large desktop environments (namely, KDE and GNOME) hide some of these
> changes, but not all.  And even different major versions of them are
> sometimes incompatible (see MIME spec, and .desktop files, etc.).
> > [....snip about kernerl API/ABI incompatibility....]
> > The stability problems seem to come from the ABI being broken all the
> > time. The APIs themselves, I don't think actually change that much -
> > so if we abstract portions of them, that's all we need. Plus the
> > library can put code between the kernel and the ldml APIs, so we can
> > retrofit old functions too. In any case, I haven't looked into it
> > deeply yet, but I don't think it'll be impossible.
> If you come across any interesting documents, please let me know.  A Wiki
> would be perfect for this. ;)
> Anyways, take care!  Hope to hear from you soon.

Me too, might be offline for a few days, but you'll see me in any
website updates.

Talk to you soon,

> --
> Karl Pietrzak
> kap4020@xxxxxxx

Other related posts: