FreeLists / openbeosstorage / [openbeosstorage] Re: Registrar-Based Notification Mechanism

["Axel Dörfler" <axeld@xxxxxxxxxxxxxxxx>]

Ingo Weinhold <bonefish@xxxxxxxxxxxxxxx> wrote:
> > Just keep in mind that having the node monitor mechanism for the 
> > kernel
> > itself might also be a very nice idea, and that shouldn't be 
> > dropped so
> > easily. With a kernel delivering all those nice things, why 
> > shouldn't
> > it be able to benefit itself from these=3D3F
> Yes, I briefly thought about that too, but didn't follow that idea. 
> One of
> the problems is, how the kernel entities should be notified. For 
> userland
> application this is trival, since you have messenging, but for the
> kernel...=3F

You could also have messaging in the kernel as well, but I think 
probably the best way would be to be able to specify a notification 
hook that will be called when the event occurs.

> Discriminating active and passive entities -- i.e. those that run an 
> own
> thread (e.g. daemons) and those that don't (e.g. file systems) -- I
> could imagine different approaches:
> 
> 1) Event queues: Each subscriber has a queue the events are pushed 
> into.
> An event queue may have a counter semaphore released with each event
> pushed into it. So the (active) entity can wait for events.
> 
> 2) Ports: The event notifications are written to a port. This doesn't 
> work
> very well with passive entities, I suspect.

They could, of course, spawn a thread for that purpose.

> 3) A combination of 1) and 2): The subscriber can optionally provide 
> a
> port to which only a `there is a new event in the queue' message is
> written.
> 
> 4) Callbacks: The subscribers supply a function to be called when 
> events
> occur.
> 
> Certainly 4) is the most flexible of the approaches. The others could 
> even
> be implemented on top of it without performance issues. Of course, it
> could be misused by subscribers by implementing time consuming 
> callback
> functions which will hit the performance of the entity issuing the 
> events
> -- one would need to be very careful.

That's the standard problem in the kernel anyway :-))

> BTW, when using callbacks the mechanism for userland notifications 
> could
> hook in as just another subscriber (at the cost of one memcpy, I 
> think).

Even without the memcpy, I think - the notify=5Flistener() function would 
just call the hook function with the same parameters.
And that one could then write the message to the registrar (using 
shared memory).

> If there is the desire to unify the interfaces of the different 
> watching
> services (node monitoring, mounting, disk=5Fscanner stuff,...) a bit, I
> could imagine something like this:
> 
> struct notification=5Fsubscriber {
>   bool (*notify)(const notification=5Fsubscriber*,
>                  const notification=5Fevent*);
>   void *service=5Fparams;
> };
> 
> Where, e.g. in case of the node watching service, service=5Fparams 
> could
> point to:
> 
> struct node=5Fwatching=5Fparameters {
>   ino=5Ft      node;
>   uint32     event=5Fmask;
>   ...
> };
> 
> The subscription/unsubscription functions would look like:
> 
> status=5Ft xyz=5Fsubscribe(const notification=5Fsubscriber *subscriber);
> status=5Ft xyz=5Funsubscribe(const notification=5Fsubscriber *subscriber);
> 
> One could even provide:
> 
> status=5Ft subscribe(uint32 service,
>                    const notification=5Fsubscriber *subscriber);
> status=5Ft unsubscribe(uint32 service,
>                      const notification=5Fsubscriber *subscriber);
> 
> Anyway, for sake of performance optimization the structure holding 
> the
> subscribers for a certain service would need to be managed by the 
> service
> itself.

Something like this could be nice, although I would also be fine with a 
less generic approach, i.e. a parameter list like the 
send=5Fnotification() call.

> > Depends on the implementation - the shared memory buffer could also 
> > get
> > used up completely, and what would you like to do then=3D3F Just wait 
> > until
> > the buffer is free again=3D3F
> Seems like you read the answer later. :-)

Yep, I did ;-)

> > The registrar thread collecting the messages should probably run on 
> > a
> > high priority :-)
> That's not a bad idea.

Since that thread doesn't have much to do, it probably is.

> > Anyway, how does the Registrar know about the target of the message
> > =3D3F
> > The kernel knows this, but the registrar (currently) cannot. The 
> > only
> > way would be double house-keeping, I think.
> Partially. All the kernel needs to know is whether there is anyone at 
> all
> listening for an event. Only the registrar would know the concrete
> targets.

Right. I will think about a possible hook-scheme implementation in the 
kernel - the changes to the current mechanisms would probably be very 
small (and worth the effort).

> > > Possible negative effects:
> > > * Higher `event occurence -> arriving of notification' latency.
> > Yes, there are more context switchs needed to deliver the message, 
> > I
> > don't consider this a big problem though, as those notifications 
> > aren't
> > that time critical, are they=3D3F
> That's what I think too.

BTW with the hook model discussed above, we could even "stick" in 
different message delivering mechanisms, and have the possibility to 
compare them directly (memory overhead, etc.) - and it would also 
enable us to "design" the system as needed (i.e. different mechanisms 
for embedded devices or large servers or whatever [not that I am 
thinking "large server" already ;-))]).

> > That might be a bigger problem, but nowadays not sooo important,
> > although we should prevent doing too many things to let the kernel 
> > run
> > with a low amount of memory.
> Well, yes with some reasonable policy, the mechanism I proposed 
> shouldn't
> eat up that much memory. Some for the ring buffer (I bet 10 KB would 
> be
> sufficient) and some for pending messages. But the latter should be 
> rather
> harmless when some sanity limits are set.

Probably - one notification will have a maximum of about 280 bytes (256 
bytes for the filename, and the rest for additional data).
So a 12 kB buffer could hold a minimum of 43 messages - should be okay, 
but more wouldn't hurt as well (but we can play with those values 
later).

[...]
> > If we had always more than one CPU, that would probably better, 
> > though
> > :-)
> Well, more CPUs are always good to have. :-)

Hehe ;-))

> > In that case, why should the subscription functions in the 
> > libroot.so
> > then=3D3F
> Er, personally I'd rather want them in libbe. And now I think about 
> it,
> actually only open=5Flive=5Fquery() (or whatever the exact name was) is 
> not.

For whatever reason - I have never understood that call. It's not 
really usable from C anyway. I could only use it to check if the BeOS 
kernel would send BMessages directly ;-)

> > And not simply in libbe.so (there is no requirement to have only
> > libroot.so make syscalls, although it might be the cleaner design 
> > to do
> > so :)=3D3F
> Syscalls weren't my concern. Those living in libbe can directly send 
> a
> message to the registrar (as currently done for roster and clipboard
> watching). But in libroot, we do, of course, want to avoid dealing 
> with
> messages.

Indeed.

[...]
> > If we already have a syscall that does the notification, the kernel
> > could maintain its list independently. I kinda don't like the 
> > situation
> > that the kernel is dependend on a userspace application, even if it 
> > is
> > the registrar :-)
> > In theory, every application should be able to pick up this service
> > provided by the kernel :)
> Mmh, I wouldn't say that the kernel would depend on the registrar. 
> It's
> more that userland notifications depend on it. If there is no 
> registrar,
> then there won't be a ring buffer and the kernel doesn't have to do
> anything regarding notifications.

In that case, of course, there would be no problem.

[...]
> > Perhaps we should have a libroot function where a team can register 
> > to
> > provide that service, and once it goes down (i.e. crashes) that
> > position is free in the kernel again, and the kernel knows that it
> > doesn't have to care about maintaining those notifications anymore
> > (therefore, the buffer won't overflow).
> That is pretty much, what I intended anyway. As I wrote, the 
> registrar
> tells the kernel via a syscall when it is ready. Or more precisely, 
> it
> will pass the IDs of the locking semaphore and the area for shared 
> memory
> to the kernel. In theory any application could do that. When going to 
> die
> the registrar should invoke another syscall.
> 
> Regarding crashes of the registrar, I thought, that it would be 
> enough,
> that the locking semaphore is deleted with its death. But you're 
> right,
> some special handling is required, since otherwise things can end up
> really bad, if the registrar crashes just after the kernel has 
> acquired
> the lock.

Also, the shared memory area must be cloned in the kernel as well, as 
it must be accessible from kernel context.
And the queue doesn't have to be maintained anymore in the case there 
is no Registrar.

> > Also, with that ring buffer, we could also move the registrar 
> > thread(s)
> > which provide(s) that service into the kernel, too, saving one 
> > context
> > switch for a message delivery (but having the (or a) BMessage in 
> > the
> > kernel again).
> Yep, that's basically the tradeoff, BMessage in the kernel vs.
> notification delivery in userland. Letting the registrar deliver the
> messages has the additional advantage, that this effort can be joined
> with that of the other event watching services, which are all 
> implemented
> there.

Yep.

> > > BTW, there is quite some potential for optimization, e.g. by
> > > flattening a
> > > notification message very early (i.e. directly after setting it 
> > > up)
> > > and
> > > writing the flattened message to the target ports instead of 
> > > using
> > > the
> > > high-level API for sending BMessages, which would cause the 
> > > message
> > > to be
> > > flattened each time it is tried to be sent.
> > That's something I would do anyway, and not regard as an 
> > optimization :
> > -))
> Well, right. Currently it can't be implemented though, since it needs 
> some
> support from BMessage. And considering the unfortunately slow 
> BMessage
> progress, I suspect, we will be done with implementing the 
> notification
> stuff before our BMessage is ready. :-(

Despite the slow progress (Erik!!!), what's preventing calling 
BMessage::Flatten() yourself and using write=5Fport() manually=3F It should 
be possible to get the target port for the Registrar, right=3F

> > Hehe, sounds very nice overall.
> > I would just like to have a way to keep the notification mechanism
> > available for kernel modules - now if you have an idea to provide
> > this... :-))
> I haven't thought much about that yet. Above is a spontaneous
> brain-storming -- maybe some more concrete ideas will form when 
> thinking a
> bit more about it.

Well, I think it's already the best idea for that problem, but please 
go on, have a better one again ;-P
Of course, the registrar maintaining kernel functions must reside in 
the kernel, and be activated via a syscall. We could place that stuff 
into a kernel module that is loaded on demand of the registrar, and 
registers its service on init.

Adios...
   Axel.