[nanomsg] Re: Trying to implement "directory" pattern
- From: Martin Sustrik <sustrik@xxxxxxxxxx>
- To: nanomsg@xxxxxxxxxxxxx
- Date: Sat, 23 Feb 2013 07:19:49 +0100
Hi Paul,
I'm trying to implement the following message pattern, I call it
"directory" (have a better name?).
Directory pattern has two socket types: ASK and ANSWER. They are mostly
analogous to REQ and REP types respectively. The difference is that
ANSWER socket can choose what requests it serves. This works by setting
SUBSCRIBE option on the socket, which sends the subscription to ASK
socket, similarly to how it's done for PUB/SUB sockets in crossroads IO
(with subscription forwarding). ASK socket keeps a trie of prefixes and
for each node of the trie a list of pipes (ANSWER endpoints) is kept to
load-balance between them. So each message matches against trie to find
a subset of pipes that can be used to serve a request.
This looks like merging multiple REQ/REP topologies into a single one.
What's the ultimate goal here? To limit the number of open TCP ports?
Illustration attached. At the picture there is a single node A which
serves requests for "cars" and two nodes (B and C) which serve requests
for "computers"
This way we get a request-reply pattern with stateful routing, while
keeping important properties of other patterns (scalability, uniformity
and interjection)
In the future it's intended to use same infrastructure that will be
implemented for PUB/SUB subscriptions (i.e. plugins for subscription
matching)
I currently have the following questions:
1. Does AF_SP_RAW intended for asynchronous socket handling, similarly
to how X* sockets worked in zeromq? It matters because xask and xanswer
going to operate on subscriptions with messages, and asynchronous
sockets should subscribe by socket option. However, non-raw answer
socket can only process one request at a time, which is unviable for
asynchronous applications.
Yes. That's the case. The intuition is: when you open two raw sockets
and make pass messages between them you should get a fully functional
intermediate device.
In other words, raw sockets are used everywhere (at endpoints as well as
in devices); the "normal" sockets are just a thing layer on top of raw
sockets to provide end-to-end functionality -- as such they are used
only on the edges of the topology.
2. When pipe is added to answer socket, all subscriptions should be
resent. Is there a way to put all the subscriptions into a pipe without
any limits? Otherwise, I need either build a complex state machine to
track which subscriptions are already resent, or bundle all
subscriptions to a single message (inventing some ad-hoc message
format). BTW, similar problem with just adding a subscription, when some
output pipes are full. What crossroads does in this case?
I think you are going to run into problems here.
Sending same data to multiple destinations reliably has the effect of
slow/dead/malevolent consumer being able to block the whole topology.
Subscriptions, being just the data in the end, experience the same problem.
What it means that one hanged up ANSWER application could possibly block
the whole datacenter.
You'll have to think out of the box here. For example, pub/sub solves
the problem by allowing just one upstream pub socket per sub socket.
Other tricks are: Allowing for unreliability. Not sending at all, doing
filtering locally. Etc.
3. I'm going to use NN_RESEND_IVL and NN_SUBSCRIBE options, but they
have same integer value. So I either need to reinvent names, or we need
to set unique numbers for built-in socket options. I think the latter is
preferable, as it also less error prone (can't setsockopt(NN_RESEND_IVL)
for SUB socket)
Ah, there are socket option levels now, same as in POSIX.
So NN_RESEND_IVL is specific to REQ/REP pattern (NN_REQREP option level)
and NN_SUBSCRIBE is specific to pub/sub (NN_PUBSUB option level). You
should define an unique option level for your protocol and define the
option constants as you see fit. Separate option level will guarantee
that you won't clash with other protocols.
4. Martin, could you describe a bit, how chunks work? I need to mark
messages for being either subscriptions or request/reply, as well as
keep a chain of routing data like for req-rep. Any suggestions?
OK. Basically, message is composed of header and body. Each is
represented as "chunk". "chunk" is just a binary buffer.
From the transport you'll get a message with just the body and no
header. What you have to do is to separate the message into header and
body in raw socket's recv() function.
So, you initilaise the header to appropriate size, copy the header data
from the body chunk to the header chunk and trim the header from the
body chunk (trim -- as opposed to copy -- allows to retain zero-copy
guarantees).
Note that inproc transport passes your messages in parsed state (header
and body being separate) so you don't have to do the above for inproc.
5. The POLLOUT will always be signaled. It's useless to check if at
least one pipe is not full, as the subset of pipes can be used for each
message. The end to end resending mechanism will be used to fix the
problem. Is the tradeoff ok?
I think so.
Martin
Other related posts:
