[openbeosstorage] Re: Logging/RAID

> > > If the boot partition is not affected by the changes, the whole
> > > process
> > > can be simplified by logging to a file on the boot partition. For 
> > > 2.
> > > there are some more options. E.g. resizing a partition doesn't 
> > > need
> > > external logging, if the FS is journalled anyway.
> >
> > That last part I'm not so sure about. I for some reason imagined we
> > would be doing complete logging (i.e. meta data AND user data).
> 
> I'm not sure, if I understand, what you mean by meta data and user 
> data.
> User data = FS contents, i.e. dirs and files, while
> Meta data = partition layout and the like?

No, I meant:
meta data = file and dir inodes, etc
user data = actual file data

IIRC, only the former is journalled by BFS.

> Resizing a partition has two phases, resizing the partition (changing 
> the
> partition map) and resizing the FS. In this order when the partition 
> is
> enlarged, the other way around when being shrunk. If the FS supports
> journalling, the FS should be able to use it's own log for the FS
> resizing. That may even be easier to implement for the FS add-on
> implementor.

Yes. I'm just concerned about whether or not the data is also journaled during 
a resize (perhaps this is an unnecessary concern...). There's always the 
chance that, say, kernel_intel would need to be relocated during a resize, and 
should the power be cut halfway, part of the data could be lost.

> For the other phase logging to a file on the boot partition should 
> usually
> work just fine (preferrably a special one made of a contiguous chunk 
> of
> memory).

Agreed.

> [...]
> > > It may be a bit of a challenge to generalize such a process to 
> > > work
> > > with arbitrary partitioning systems.
> >
> > Yes. It would probably behoove us to read up on other partitioning
> > systems and how they interact with the boot process. Unless either 
> > of
> > you are well versed in that already. :-)
> 
> A short specification of the Apple partitioning system can be found 
> here:
> http://developer.apple.com/techpubs/mac/Devices/Devices-121.html#MARKER-2-169
> (Integers are 2 bytes and LongInt 4 bytes, I suspect.)
> It is not exactly complex. :-)

Thanks. :-)

> > So, I would like to regroup here, if you don't mind, as I keep 
> > feeling
> > either slightly lost, or like points are being made about differeing
> > problems/systems (though it could be just me :-). We have the 
> > following
> > needs, correct?:
> >
> > 1. Somewhere to write information about currently processing
> > BDiskDeviceJobs to allow the system to start up said jobs again if
> > interrupted. How about calling this the job log for ease of 
> > reference?
> >
> > 2. Somewhere to store any code needed to reinitiate and finish
> > interrupted BDiskDeviceJobs, i.e. the job completion code.
> >
> > 3. To have the main (i.e. boot) partition be a RAID partition, a
> > special partition containing enough info and code to get the RAID
> > subsystem going must be available. All remaining RAID configuration 
> > can
> > be stored on the boot partition (whether that partition is a RAID or
> > not).
> >
> > Have I missed anything or gotten anything wrong?
> 
> Looks good so far.

Phew. :-)

> > Assuming not (or thereabouts :-): To my mind, 1 and 2 need have 
> > nothing
> > to do with 3 if no boot RAID partition is involved, and something 
> > along
> > the line of Ingo's proposal above would probably work nicely. If a 
> > boot
> > RAID partition *is* part of the changes, the data for 1 and 2 could 
> > be
> > placed somewhere in the necessary partition for 3, and all is well
> > again.
> 
> Yep.
> 
> > There also could be a need for:
> >
> > 4. A place to log filesystem changes during repartitioning ops.
> >
> > But I really like the idea of leaving that problem to the fs itself.
> 
> If a FS itself doesn't support journalling (e.g. FAT), the system must
> provide it with a space for a resizing (or moving in case of absolute
> addressing) log.

So here I wonder about the repititious overhead of implementing resizing 
journalling for any non-journaled filesystem. Would it be possible to provide 
a generic API for use in these instances that would work in the manner of 
"here are a bunch of disk block writes, add them to the log and don't remove 
them until all of them have actually occurred", rather than simply presenting 
each fs with a chunk of disk space for a log and saying "here, implement your 
own journaling system".

> > So, thoughts, corrections?
> 
> What might be tricky, if the job log is dynamically allocated when 
> needed
> -- beside the alloction, which might be difficult itself -- is 
> parallel
> processing of jobs. They are by definition working in disjoint 
> branches of
> the partition hierarchy, but they might compete for space for their 
> job
> log. Or even worse, the one job's log might get into the way of the
> other's actions (e.g. if placed in a spare region where a new 
> partition
> shall be created).

Plus, there would need to be a way to tie all the job logs together so they 
all could be found in the event of an interruption. Perhaps it would be easier 
to have just one job log, with all parallel job queues logged to it, anyway 
(for ease of reinitiating after an interruption). Separate logs could be 
allocated by each process as necessary for external management logs (i.e. 
during partition resizing, FAT resizing, etc).

As to the dynamic allocation, if previously empty partition area space is 
consumed for the log, perhaps the partition system should create a temporary 
BPartition typed as a "temporary partition management job log" or something to 
that degree to officially hold the space until the job is done.

-Tyler

Other related posts: