[cad-linux-dev] Re: uber-converter
- From: "cr88192" <cr88192@xxxxxxxxxxx>
- To: <cad-linux-dev@xxxxxxxxxxxxx>
- Date: Tue, 8 Jun 2004 10:11:47 +1000
----- Original Message -----
From: "Eric Wilhelm" <ewilhelm@xxxxxxxxxxxxx>
To: <cad-linux-dev@xxxxxxxxxxxxx>
Sent: Tuesday, June 08, 2004 6:01 AM
Subject: [cad-linux-dev] Re: uber-converter
> # The following was supposedly scribed by
> # cr88192
> # on Sunday 06 June 2004 09:19 pm:
>
> >> Transparency is the primary goal of the textual format, and I don't
think
> >>XML really gives that.
> >
> >I believe it is flexibility...
> >for me flexibility implies the ability to tack on new features as needed
> >without breaking anything...
>
> Well, I'd call that extensibility. A setup might be flexible within a
given
> boundary, but if it can't go beyond that, then it isn't extensible.
>
understood.
I like things to be heavily extensible then (even if the extension mechanism
can make things look really crufty).
> >I am thinking yaml might be a bit overly structured, and this might
> >interfere with the ability to easily tack things on...
> >
> >I believe the core should be minimalistic and extensible, the rest should
be
> >extensions...
>
> An extension to data? The idea behind the hub-spoke model is that each
format
> which is available on a spoke is fully representable by the core. Without
> that, we can't have a core.
>
I believe the core should be minimalistic and extensible (itself sort of a
hub for new features).
if all the core features are defined up front and no real extension
mechanism is provided, then one will run into problems once there is
something missing.
of course, a lot could be defined in the core, which could at least avoid
things looking too crufty.
> Note that I'm not talking about a geometry engine or anything of the sort.
> Simply a core data format which can represent the union of all of the
other
> formats.
>
understood. I believe the best way to allow "a union of all other formats"
is to allow a strong extensibility mechanism, so if anything is missing it
can be added...
> Also, remember that if XML is not suitable for the core, that doesn't mean
> that you can't use the thing to make xml files. It just means that the
XML
> part needs to be a spoke rather than the hub.
>
maybe...
I don't think xml would be too big of a problem though (the main issue would
likely be breaking things up so that it is tolerably fast, but not so that
it is wasting craploads of hd space due to fs overhead...).
I had before spec'ed out some ideas for "random access archives", sadly I
have just never gotten to implementing them.
the idea I had considered was basically about like fat+compression (eg:
deflate)+end packing.
other features could be useful as well.
an issue I had wondered about was how to handle compression, basically if I
should compress enough data to fill a certain buffer (possibly problematic)
or if I should compress blocks of data and grab enough archive blocks to
hold the compressed block (eg: the file is compressed in 16 or 64 kB
chunks), with the end being padded.
this could be problematic as it would be waseful of space due to internal
fragmentation, and wouldn't be succeptible to end packing.
yet another approach could be to treat the file space sort of like a mm
heap, using either a bitmap or a "chunk chain" to locate free space (or
maybe a run-list, which is more common to file formats, and also makes
"repacking" fairly easy).
yet another approach is not to "really" allow random io on files, but
instead to fake it occasionally writing out the file in single bursts.
this could help in choosing whether to store or compress files.
if the whole file is written at once then best compression could be
achieved, at the cost that it would be a little more difficult to fake
random io (or at least not without using a lot of memory in the case of
larger files...).
even with no compression, however, just the fact that I am using 512 byte
blocks will win over the host fs (fat32 in my case), in that I wont be
using, eg, 4kB blocks...
hmm, I don't know.
as of yet I have just been using os files though...
all for now.
Other related posts:
