[nanomsg] Re: stackable protocols
- From: Drew Crawford <drew@xxxxxxxxxxxxxxxxxx>
- To: nanomsg@xxxxxxxxxxxxx
- Date: Tue, 13 May 2014 00:35:36 -0500
> After all, what does a “session” mean in the context of a bus or star (star
> isn’t in nanomsg, but its an experimental mangos pattern) pattern, or in the
> context of pub/sub?
Well one thing it could mean is something like “sender identifier”. For
example, if every socket had an identifier assigned upon creation, every
message was prepended by this identifier, and the identifier was stripped from
the message and made available via getsocketopt separately.
When applied to REQ/REP, this is the ground floor of a session, minus things
like connect/disconnect notifications, which could very well be shipped
optionally as a stackable layer depending on whether or not you really need
them.
When applied to nn_bus, this is the ground floor of a routing protocol, where
each node on the bus can decide to respond to the message based on its sender
ID. Depending on the application, connect/disconnect notifications may also be
useful in this case as an optional layer, probably via a different
implementation.
> Layering on top of the ‘generic’ framework much less so.
There are a lot of cases where it is useful to layer something on any protocol.
Message CRC32 and verification. Proof of work algorithms. Flags that mark
endianness and code that flips the byte order if the flags are wrong for your
platform. Doing some message-level crypto operation. Embedding system load
statistics into a message so observing parties can throttle their traffic
accordingly. To be clear, I’m not saying any of these are particularly good
candidates for core features. I’m merely saying that nanomsg should lend
itself to developers who want to experiment with this sort of thing.
Experimenting is too hard right now.
> For what its worth, my mangos port has no NN_MAX_SOCKETS. I don’t know the
> point where it will bog down, but I guess the number is quite high.
I wish I was using go… unfortunately there are a lot of other problems for me
to go that way.
On May 13, 2014, at 12:10 AM, Garrett D'Amore <garrett@xxxxxxxxxx> wrote:
> For what its worth, my mangos port has no NN_MAX_SOCKETS. I don’t know the
> point where it will bog down, but I guess the number is quite high.
>
> I don’t think the nanomsg does a very good job of extensibility — it can be
> done, but its not terribly elegant, and the state machine model imposed on
> both protocols and transports makes adding a new one of either of those
> somewhat onerous. That said, for people familiar with the internals, it
> might not be hard.
>
> And FWIW, while I think I did an excellent job of making it easy to add new
> transports, adding new protocols in mangos is still kind of challenging
> because you need to understand some of the undocumented internals.
>
> In the case of your sessions model, a transport might give you what you want
> (you could actually *layer* a transport on top of req/rep in nanomsg — could
> be funny), but I’m not sure that’s right given the connect/reconnect
> handling. (Unless your transport layer had a way to pass session state
> around somehow — its not precisely clear how that would work.)
>
> Layering on top of a single pattern (req/rep) makes sense. Layering on top
> of the ‘generic’ framework much less so. After all, what does a “session”
> mean in the context of a bus or star (star isn’t in nanomsg, but its an
> experimental mangos pattern) pattern, or in the context of pub/sub? I think
> you’re looking there for specific semantics which don’t map well to nanomsg
> “generically”, although I can definitely see a layering approach on top of
> req/rep, and maybe push/pull and pair. (In fact, the pair semantic is
> probably perfect for what I imagine your session layer would look like.)
>
> In other words, your ideas for layering, which sound on the face of it nice,
> break down because of the wildly different semantics of the different
> patterns. To be honest, rather than extending nanomsg, I’d probably just
> create a library that sat on top nanomsg and exposed/utilized just that
> subset of the patterns that made sense for your particular application
> semantics.
>
> Now all that said, having a library mechanism (and protocol extensions) to
> pass “user-defined metadata” might be useful. Might. (I’m imagining an
> extension to the headers.) From nanomsg’s point of view, we’d just treat
> this as a secondary user data area, something that gets passed unmolested by
> nanomsg itself. Handling the semantics of that would be left to the
> application (and there’d be some calls to set/get the data area. This would
> be easy in core nanomsg. I do question how much actual value there is in it,
> though.
>
> --
> Garrett D'Amore
> Sent with Airmail
>
> On May 12, 2014 at 7:17:26 PM, Drew Crawford (drew@xxxxxxxxxxxxxxxxxx) wrote:
>
>> I’m not completely sure, but I may maintain one of the larger nanomsg forks
>> in terms of patch size against master. To put some numbers on it, it is
>> about 2300 lines of changes, which is larger than rep, xrep, req, and xreq
>> combined, and has about 1k unwitting users as part of a proprietary
>> application. I mention this to establish that I have some idea what I’m
>> talking about on the subject of extending nanomsg to new use cases. (Whether
>> or not my extensions are ill-advised is a different question…)
>>
>> Nanomsg has two “official” extension mechanisms—protocols and transports. So
>> the idea is, if you have a new feature you want to implement, you write a
>> new transport or protocol.
>>
>> This is a nice theory but in practice there are problems. One of them is the
>> oldest problem in object-oriented programming: inheritance hell.
>>
>> When you write a protocol (or transport for that matter) you have to pick a
>> particular “lower” level to work from, like a superclass. For example rep
>> chooses xrep, xrep chooses nn_pipe, etc.
>>
>> When you write a new feature however there may not be one single obvious
>> underlying protocol. For example I have raised a couple of threads now about
>> session support, (which is debatable whether or not it is a good idea, but
>> anyway)—in this case there is no “obvious” protocol to piggyback on. For
>> example you could choose rep/req. You could also choose xrep/xreq, but if
>> somebody wants “rep with sessions” then you have to implement a “new”
>> srep/sreq that sits atop the session layer that sits atop xrep. Less
>> obviously, session support is useful for protocols like nn_bus, and maybe
>> xbus, but this is hard to see at first, so you probably made some wrong
>> choices in your design and thus the inheritance tree.
>>
>> Now say you want to add another feature orthogonal to sessions to nanomsg.
>> Now you have protocols like rep + sessions + other_feature, rep + sessions,
>> rep + other_feature, rep, xrep, xrep + sessions + other_feature, etc. The
>> thing totally explodes.
>>
>> This is not very much like real networking infrastructure, which acts in
>> user-stackable layers (like TCP/IP) and also is not very unixy, where you
>> can glue simple tools together to get the job done.
>>
>> Another approach you might try is to implement your feature inside an
>> nn_device (with another name like nn_add_session_device). The advantage of
>> this approach is that the application programmer can bolt together a lot of
>> sockets and devices at runtime to get a socket chain that supports REQ with
>> sessions and other_feature but not third_feature as he desires.
>>
>> The key phrase there is “a lot of sockets”. In my testing today, doing this
>> approach very seriously in production can easily blow past NN_MAX_SOCKETS
>> and can cause performance problems.
>>
>> What I end up doing, is ignoring nanomsg’s extension mechanisms entirely and
>> introducing a new pluggable mechanism that is configured through
>> nn_setsockopt and friends. In this way, arbitrary code (features) can be
>> injected at the application developer’s direction that preprocess the
>> message before any protocol code sees it, into any socket type. (Whether
>> injecting that code into nn_bus specifically makes any sense is a different
>> question.) So you can configure your NN_REQ with both sessions and
>> other_feature through socket options and this has the advantage of A)
>> building protocols dynamically out of smaller pieces and B) not contributing
>> to NN_MAX_SOCKETS and performance headaches.
>>
>> There are some other ways to solve this problem, like delegation patterns,
>> or a more formal responder chain. I don’t mean to suggest that nn_setsockopt
>> is the right approach. I just mean that there are clear deficiencies in the
>> existing pluggable transports that make certain types of nanomsg extensions
>> very hard to implement, that have nothing to do with the features
>> themselves. It’s been said a few times that sessions are hard due to memory
>> management or cleanup, but in practice the hardest part for me has been to
>> figure out *where to put the code*.
>>
>> I want to take the temperature on implementing some kind of dynamically
>> stackable extension mechanism upstream. This problem is a large obstacle to
>> anyone who wants to experimentally implement a new nanomsg feature or an
>> ecosystem that goes beyond just language bindings and into doing new things
>> with the library. I have kind-of-sort-of done something good enough for my
>> purposes but having been through that pain I certainly would not advise
>> anybody else to do wade through the same mess themselves. So I think it is
>> worth exploring whether something sensible can be done to lower this barrier
>> in core.
>>
>> Drew
Other related posts:
