FreeLists / openbeosstorage / [openbeosstorage] Re: Registrar-Based Notification Mechanism

["Axel Dörfler" <axeld@xxxxxxxxxxxxxxxx>]

Ingo Weinhold <bonefish@xxxxxxxxxxxxxxx> wrote:
> since I expect to reach the point where kernel notifications will be
> needed, soon, I think, it's time to present my ideas for a general
> registrar-based notification mechanism for your evaluation.

Nice!

> The general idea is to move as much functionality as possible from 
> the
> kernel to a userland server -- the registrar. The expected positive
> effects are:
> 
> * No (flattened) BMessages in the kernel.
> 
> * Similar performance of the kernel entity causing the notification
> independent of the number of listeners. Maybe even slightly better
> performance for no/one listener.

Just keep in mind that having the node monitor mechanism for the kernel 
itself might also be a very nice idea, and that shouldn't be dropped so 
easily. With a kernel delivering all those nice things, why shouldn't 
it be able to benefit itself from these=3F

> * No more dropping of notification messages, in case the target port 
> is
> full at the moment of the event.

Depends on the implementation - the shared memory buffer could also get 
used up completely, and what would you like to do then=3F Just wait until 
the buffer is free again=3F
The registrar thread collecting the messages should probably run on a 
high priority :-)
Anyway, how does the Registrar know about the target of the message=3F 
The kernel knows this, but the registrar (currently) cannot. The only 
way would be double house-keeping, I think.

> Possible negative effects:
> * Higher `event occurence -> arriving of notification' latency.

Yes, there are more context switchs needed to deliver the message, I 
don't consider this a big problem though, as those notifications aren't 
that time critical, are they=3F

> * More resource usage: thread(s), semaphores and memory

That might be a bigger problem, but nowadays not sooo important, 
although we should prevent doing too many things to let the kernel run 
with a low amount of memory.

> When an event occurs the kernel notifies the registrar, which manages 
> the
> lists of listeners and sends the actual notification messages. The
> kernel->registrar communication is done via shared memory; for the 
> other
> direction syscalls can be used.

If the kernel would send out *all* notifications, that would probably a 
lot more expensive for the case with no listeners, which should be 
considered as the most probable path. Right now, the kernel does 
nothing for those notifications - it makes a hash lookup and that's it.
In that case, it had to copy the notification data into a buffer, and 
the registrar thread had to wake up and check if there are any 
listeners waiting for that info.
Perhaps we should do a little more in the kernel (like maintaining a 
list of vnodes that trigger a notification at all, very similar as it 
is now).
If we had always more than one CPU, that would probably better, though 
:-)

> 1) Subscription/unsubscription for notifications: This is a bit ugly,
> since the functions for that live in libroot and the primary target 
> for
> the request is the registrar. Either the request is passed through a 
> port
> directly to the registrar, or it travels into the kernel and goes the 
> same
> way the to the registrar the notifications take (i.e. through the 
> shared
> memory). Once there, the listener is added to the appropriate
> list/hashtable/...

In that case, why should the subscription functions in the libroot.so 
then=3F And not simply in libbe.so (there is no requirement to have only 
libroot.so make syscalls, although it might be the cleaner design to do 
so :)=3F

> For it should be avoided that the kernel notifies the registrar on an
> event noone is interested in, there still needs to be a structure in 
> the
> kernel, holding the information, which events are listened. The 
> registrar
> needs to tell the kernel whenever this info has changed. This 
> structure
> could be in shared memory too, but, since subscription/unsubscription 
> is
> not time-critical, syscalls may be the better choice. Rare events
> (mounting, appearing/disappearing of device and the like) can, of 
> course,
> always be sent to the registrar.

Ah, you seem to have thought of this already :-)
If we already have a syscall that does the notification, the kernel 
could maintain its list independently. I kinda don't like the situation 
that the kernel is dependend on a userspace application, even if it is 
the registrar :-)
In theory, every application should be able to pick up this service 
provided by the kernel :)

> 2) The kernel-side interface: Once the registrar is running it 
> allocates
> the shared memory and tells the kernel via a syscall. The layout of 
> this
> memory is relatively simple. It has a little header containing 
> general
> information, and the rest is a ring buffer containing notification
> entries. The access is protected by a mutex and there is a semaphore
> counting the entries in the buffer (being released, when a new one is
> added and acquired when one is going to be taken out).
> ...

Looks fine.

[...]
> In case the ring buffer is full, reserve=5Fnotification=5Fentry() 
> allocates
> memory for the entry on the heap and pushes it into a queue. The next 
> call
> to it will first try to empty the queue -- copying its contents into 
> the
> ring buffer -- and proceed as usual, i.e. reserve ring buffer space 
> or
> alloc memory and use the queue respectively.

That is probably not a very good idea, although it's not a big problem 
in the case of BeOS (since the registrar is needed for normal operation 
anyway).
You make the sanity of the kernel depending on the presence of a 
particular application in userland.
Perhaps we should have a libroot function where a team can register to 
provide that service, and once it goes down (i.e. crashes) that 
position is free in the kernel again, and the kernel knows that it 
doesn't have to care about maintaining those notifications anymore 
(therefore, the buffer won't overflow).

Also, with that ring buffer, we could also move the registrar thread(s) 
which provide(s) that service into the kernel, too, saving one context 
switch for a message delivery (but having the (or a) BMessage in the 
kernel again).
Or have special message ports that can't (easily) overflow (because 
basically, that's what you provide there).

> ...
> Of course additionally it must periodically be tried to deliver 
> pending
> messages. And sanity checks need to prevent that for bad targets 
> thousands
> of pending messages are piled.

Sounds very good.

> BTW, there is quite some potential for optimization, e.g. by 
> flattening a
> notification message very early (i.e. directly after setting it up) 
> and
> writing the flattened message to the target ports instead of using 
> the
> high-level API for sending BMessages, which would cause the message 
> to be
> flattened each time it is tried to be sent.

That's something I would do anyway, and not regard as an optimization :
-))

> Mmh, that's it, I think. At least my mind feels a bit emptier now. ;-
> )

Hehe, sounds very nice overall.
I would just like to have a way to keep the notification mechanism 
available for kernel modules - now if you have an idea to provide 
this... :-))

Adios...
   Axel.