[nanomsg] Re: accessing control IDs

  • From: Drew Crawford <drew@xxxxxxxxxxxxxxxxxx>
  • To: nanomsg@xxxxxxxxxxxxx
  • Date: Wed, 7 May 2014 03:50:58 -0500

> The reason is that REQ/REP is meant for a very specific use case:
> Assembling multiple servers into a CPU sharing topology.

If what you mean is, the computations that you want to do are all “pure” 
“side-effect-free” functions and you don’t particularly care who is allowed to 
make requests then this is a fine system.

However the moment you step beyond those two requirements you are in trouble.  
If your requests wants to write to a file or database (such as HTTP 
POST/PUT/DELETE) now you have problems coming out of your ears.  For example, 
suppose you issue a request that deletes an object in the database.  Somebody 
is listening and now they, too, can replay your packet to delete the object in 
the database, even if you took appropriate message-level precautions.  This is 
a difficult problem to solve, and it is *impossible* to solve without 
introducing state somewhere in the system (whether in the application or the 
transport layer).  This problem is inherent whenever you have side effects.  
Even if you have some distributed filesystem or a distributed database, that is 
partitioned one per worker, you have this problem.  It is inherent in the 
concept of a side effect; it is completely independent of the topology.  It is 
even true of side effects that do not alter state: for example, placing a phone 
call to a human does not in the ordinary sense mutate any state but a replay 
attack that produces a lot of phone calls is very annoying.

If you want to limit access to only certain users (possibly to bail you out of 
the first problem), you also have problems coming out of your ears.  Now in 
theory these can be solved at the transport layer, for example, firewalls that 
only allow trusted IPs to connect.  But such solutions are (when the only 
security measure) completely inappropriate on the open internet, and are even 
inappropriate when working in a datacenter environment like AWS.  You can do it 
right, but it involves VPNs and all kinds of complicated overhead, and it is an 
open question in my mind whether there is even a transport-layer solution that 
exists today that would perform at scale.

I’m not necessarily saying that REQ/REP is wrong on principle, but I am saying 
that there should be much clearer documentation on what it is for and what it 
is not for.  If you violate either of these two principles (and probably others 
I haven’t enumerated) you are in for a world of pain.

> From my point of view, if I wanted to speak to a specific peer, I
> would use TCP. That's what's it was designed for and what it is good at.


I think it is self-evident from my reply to the above, but really what i want 
is a request/reply messaging system that doesn’t shoot you if your REP involves 
saving to a database now and again, and lends itself to some authentication 
scheme other than “configure your firewall”.  I have actually made a 
considerable amount of progress on the second problem in a private fork, but 
was hoping that a solution on the first problem would be more mergeable and 
keep the topics of my fork to a smaller number.

It’s easy to say “you want TCP” but the reality is I want a request/reply 
system on TCP.  Which is, in a very literal sense, what nanomsg is.  It takes 
messages of arbitrary length and moves them along a TCP stream in a reasonable 
way; certainly more reasonable than whatever I would come up with left to my 
own devices.

The thing is that nanomsg exposes request/reply without sessions; and TCP 
exposes sessions without request/reply.  I shouldn’t be forced to choose; this 
should be stackable.  There should be a way to get request/reply messaging and 
sessions at the same time in the same API.

I have come around to your position on implementation, that relying on TCP 
information to derive sessions is not a good approach.  So perhaps the solution 
looks like appending 6 bytes to the message (although zero-copy is a bit of an 
issue).  However I still believe that this is a problem for a networking 
library to solve and not a problem for an application.

> On a tangent, here's my networking equivalent to the CAP theorem.
> Sustrik's theorem: You can't have *P*artition, *S*tate and
> *R*eliability at the same time.


I’m not totally sure what you mean by “state” here.  By the common definition, 
REQ/REP sockets have “state”; as they have, minimally: a reference to a 
transport, a URL for that transport, a table of socket options, a table of 
channel IDs, at least one stored request for later retransmission, etc.  I 
guess you are using “state” in some more specific sense.

On May 7, 2014, at 1:12 AM, Martin Sustrik <sustrik@xxxxxxxxxx> wrote:

> Signed PGP part
> On 07/05/14 05:13, Drew Crawford wrote:
> 
> >> Nanomsg is stateless from the point of view of the app.  Adding
> >> session state would be inappropriate.
> >
> > I have seen this stated several times in this thread, but I have
> > not seen any design justification for this position.  Such a
> > position should be defensible.
> 
> The reason is that REQ/REP is meant for a very specific use case:
> Assembling multiple servers into a CPU sharing topology.
> 
> Processing is load-balanced among the cluster but that only works if
> there's no state on the nodes.
> 
> Same applies to reliability. REQ/REP is reliable because it can resend
> any messages that were lost or not processed in time. Here the
> assumption is that the message will go to a different node the next
> time as the old one may be broken or dead. For this to work there
> can't be any state stored on the processing nodes.
> 
> > I can construct a rationale for why there should be a
> > REQ-REP-*like* protocol that is stateless. That would clearly be
> > useful.  However, in my view it is an error to give it the name
> > REQ/REP, because this is confusingly similar to the ZeroMQ protocol
> > with the same name.  It would be tolerable if nanomsg REQ/REP was
> > merely API-incompatible and wire-incompatible with the protocol
> > from ZeroMQ; that is to be expected. However when the difference is
> > not just in wireformat or argument order but is actually
> > conceptually orthogonal to the way ZeroMQ REQ/REP is used (which is
> > stateful) in a way that is hard to see up front it is time to
> > invent a new name for a new idea.  The connection between the two
> > protocols is so flimsy that sharing a name is actively misleading.
> 
> I've designed ZeroMQ REQ/REP and I can tell you that it was intended
> to be stateless. What's done today with the ROUTER socket is a hack
> similar to, say, if you hacked your TCP endpoint to send unreliable
> UDP-like packets.
> 
> > That aside, you are taking a much stronger position than merely
> > one protocol.  You are saying “nobody should implement a stateful
> > protocol in nanomsg”.  Well, why not?
> 
> Exactly, that's the question! Why yes or why not?
> 
> From my point of view, if I wanted to speak to a specific peer, I
> would use TCP. That's what's it was designed for and what it is good at.
> 
> Both ZeroMQ and nanomsg, on the other hand, were designed to speak to
> a cluster of peers. If you try to emulate TCP on top of that, you'll
> have to fight all the nanomsg features that will get into your way.
> 
> So, why do it at all?
> 
> > Clearly many applications have stateful networking problems.  How
> > should they solve these problems?
> >
> > * Should they each implement state atop nanomsg in their own
> > incompatible way?  This is duplication of effort, and probably
> > leads to buggy code * Should somebody consolidate those stateful
> > implementations into a library that sits atop nanomsg?  Now
> > language bindings have to be written for that library in addition
> > to nanomsg.  This is duplication of effort, and probably leads to
> > buggy code * Should these applications not use nanomsg at all, and
> > instead use ZeroMQ?  But there are many reasons to prefer nanomsg
> > including use of C, BSD socket API, context-free implementation,
> > pluggable transports, routing priorities, zero-copy, etc. etc.
> > This option basically tells applications “because you have stateful
> > networking, you can enjoy none of these benefits” * Should these
> > applications fork nanomsg and hack in a stateful alternative?  This
> > is duplication of effort, and probably leads to buggy code
> >
> >
> > I think what you imagine is that if you make stateful programs
> > difficult, people will rethink their stateful problems into
> > stateless ones.  Sometimes that is true.  But other times, people
> > will rethink their stateful programs and decide being stateful is
> > the right decision for their problem.  There should be some vision
> > to address this case other than “deal with it”.
> >
> > It is one thing to declare stateful problems out of scope for a
> > protocol.  It is a different thing to declare them out of scope for
> > a networking library.  People have stateful problems; and this
> > doesn’t disappear merely because stateless protocols are
> > well-behaved or more scalable.
> >
> > I don’t know exactly what the right solution looks like.  Maybe
> > it’s creating protocols that are deliberately stateful.  Maybe
> > it’s introducing some kind of pluggable session layer.  But I am
> > absolutely certain that throwing up our hands and declaring
> > anything stateful as out of scope would move nanomsg out of the
> > “zeromq done right” search space and turn it into a very
> > specialized tool for very specialized problems.
> 
> I would suggest you just create a new protocol for stateless messaging
> and try to implement that. You'll see what kind of problems you'll run
> into.
> 
> On a tangent, here's my networking equivalent to the CAP theorem.
> Sustrik's theorem: You can't have *P*artition, *S*tate and
> *R*eliability at the same time.
> 
> Note that REQ/REP goes for PR, while PUB/SUB or SURVEY goes for PS.
> 
> Martin
> 
> 

Other related posts: