[nanomsg] Re: stackable protocols
- From: Garrett D'Amore <garrett@xxxxxxxxxx>
- To: Drew Crawford <drew@xxxxxxxxxxxxxxxxxx>, nanomsg@xxxxxxxxxxxxx
- Date: Mon, 12 May 2014 22:10:23 -0700
For what its worth, my mangos port has no NN_MAX_SOCKETS. I don’t know the
point where it will bog down, but I guess the number is quite high.
I don’t think the nanomsg does a very good job of extensibility — it can be
done, but its not terribly elegant, and the state machine model imposed on both
protocols and transports makes adding a new one of either of those somewhat
onerous. That said, for people familiar with the internals, it might not be
hard.
And FWIW, while I think I did an excellent job of making it easy to add new
transports, adding new protocols in mangos is still kind of challenging because
you need to understand some of the undocumented internals.
In the case of your sessions model, a transport might give you what you want
(you could actually *layer* a transport on top of req/rep in nanomsg — could be
funny), but I’m not sure that’s right given the connect/reconnect handling.
(Unless your transport layer had a way to pass session state around somehow —
its not precisely clear how that would work.)
Layering on top of a single pattern (req/rep) makes sense. Layering on top of
the ‘generic’ framework much less so. After all, what does a “session” mean in
the context of a bus or star (star isn’t in nanomsg, but its an experimental
mangos pattern) pattern, or in the context of pub/sub? I think you’re looking
there for specific semantics which don’t map well to nanomsg “generically”,
although I can definitely see a layering approach on top of req/rep, and maybe
push/pull and pair. (In fact, the pair semantic is probably perfect for what I
imagine your session layer would look like.)
In other words, your ideas for layering, which sound on the face of it nice,
break down because of the wildly different semantics of the different patterns.
To be honest, rather than extending nanomsg, I’d probably just create a
library that sat on top nanomsg and exposed/utilized just that subset of the
patterns that made sense for your particular application semantics.
Now all that said, having a library mechanism (and protocol extensions) to pass
“user-defined metadata” might be useful. Might. (I’m imagining an extension
to the headers.) From nanomsg’s point of view, we’d just treat this as a
secondary user data area, something that gets passed unmolested by nanomsg
itself. Handling the semantics of that would be left to the application (and
there’d be some calls to set/get the data area. This would be easy in core
nanomsg. I do question how much actual value there is in it, though.
--
Garrett D'Amore
Sent with Airmail
On May 12, 2014 at 7:17:26 PM, Drew Crawford (drew@xxxxxxxxxxxxxxxxxx) wrote:
I’m not completely sure, but I may maintain one of the larger nanomsg forks in
terms of patch size against master. To put some numbers on it, it is about 2300
lines of changes, which is larger than rep, xrep, req, and xreq combined, and
has about 1k unwitting users as part of a proprietary application. I mention
this to establish that I have some idea what I’m talking about on the subject
of extending nanomsg to new use cases. (Whether or not my extensions are
ill-advised is a different question…)
Nanomsg has two “official” extension mechanisms—protocols and transports. So
the idea is, if you have a new feature you want to implement, you write a new
transport or protocol.
This is a nice theory but in practice there are problems. One of them is the
oldest problem in object-oriented programming: inheritance hell.
When you write a protocol (or transport for that matter) you have to pick a
particular “lower” level to work from, like a superclass. For example rep
chooses xrep, xrep chooses nn_pipe, etc.
When you write a new feature however there may not be one single obvious
underlying protocol. For example I have raised a couple of threads now about
session support, (which is debatable whether or not it is a good idea, but
anyway)—in this case there is no “obvious” protocol to piggyback on. For
example you could choose rep/req. You could also choose xrep/xreq, but if
somebody wants “rep with sessions” then you have to implement a “new” srep/sreq
that sits atop the session layer that sits atop xrep. Less obviously, session
support is useful for protocols like nn_bus, and maybe xbus, but this is hard
to see at first, so you probably made some wrong choices in your design and
thus the inheritance tree.
Now say you want to add another feature orthogonal to sessions to nanomsg. Now
you have protocols like rep + sessions + other_feature, rep + sessions, rep +
other_feature, rep, xrep, xrep + sessions + other_feature, etc. The thing
totally explodes.
This is not very much like real networking infrastructure, which acts in
user-stackable layers (like TCP/IP) and also is not very unixy, where you can
glue simple tools together to get the job done.
Another approach you might try is to implement your feature inside an nn_device
(with another name like nn_add_session_device). The advantage of this approach
is that the application programmer can bolt together a lot of sockets and
devices at runtime to get a socket chain that supports REQ with sessions and
other_feature but not third_feature as he desires.
The key phrase there is “a lot of sockets”. In my testing today, doing this
approach very seriously in production can easily blow past NN_MAX_SOCKETS and
can cause performance problems.
What I end up doing, is ignoring nanomsg’s extension mechanisms entirely and
introducing a new pluggable mechanism that is configured through nn_setsockopt
and friends. In this way, arbitrary code (features) can be injected at the
application developer’s direction that preprocess the message before any
protocol code sees it, into any socket type. (Whether injecting that code into
nn_bus specifically makes any sense is a different question.) So you can
configure your NN_REQ with both sessions and other_feature through socket
options and this has the advantage of A) building protocols dynamically out of
smaller pieces and B) not contributing to NN_MAX_SOCKETS and performance
headaches.
There are some other ways to solve this problem, like delegation patterns, or a
more formal responder chain. I don’t mean to suggest that nn_setsockopt is the
right approach. I just mean that there are clear deficiencies in the existing
pluggable transports that make certain types of nanomsg extensions very hard to
implement, that have nothing to do with the features themselves. It’s been said
a few times that sessions are hard due to memory management or cleanup, but in
practice the hardest part for me has been to figure out *where to put the
code*.
I want to take the temperature on implementing some kind of dynamically
stackable extension mechanism upstream. This problem is a large obstacle to
anyone who wants to experimentally implement a new nanomsg feature or an
ecosystem that goes beyond just language bindings and into doing new things
with the library. I have kind-of-sort-of done something good enough for my
purposes but having been through that pain I certainly would not advise anybody
else to do wade through the same mess themselves. So I think it is worth
exploring whether something sensible can be done to lower this barrier in core.
Drew
Other related posts: