[nanomsg] Re: accessing control IDs
- From: Drew Crawford <drew@xxxxxxxxxxxxxxxxxx>
- To: "nanomsg@xxxxxxxxxxxxx" <nanomsg@xxxxxxxxxxxxx>
- Date: Tue, 6 May 2014 23:00:49 -0500
How about some kind of API-compatible req/rep clone, say "sreq/srep" (s for
state) that implements session functionality?
Sent from my iPad
> On May 6, 2014, at 10:50 PM, Achille Roussel <achille.roussel@xxxxxxxxx>
> wrote:
>
> I think REQ/REP being stateless is a great thing, it protects the user from
> misusing the tool and solves a single problem. Now it is true that in its
> current state nanomsg is missing some sort of stateful protocol, but if
> you're trying to use REQ/REP when you need to maintain states between
> requests coming from a remote node then you're probably not using the right
> tool to solve your problem.
>
> HTTP is stateless but it is showing its limits for today's web, and is being
> slowly replaced by SPDY which is a stateful protocol... Maybe nanomsg should
> get some sort of hybrid protocol or implement SPDY itself (which is a
> stateful req/rep with some sort of pub/sub on top of it).
>
> Achille Roussel
> +1 415 490 6339
>
>> On May 6, 2014, at 8:13 PM, Drew Crawford <drew@xxxxxxxxxxxxxxxxxx> wrote:
>>
>>> Nanomsg is stateless from the point of view of the app. Adding session
>>> state would be inappropriate.
>>
>> I have seen this stated several times in this thread, but I have not seen
>> any design justification for this position. Such a position should be
>> defensible.
>>
>> I can construct a rationale for why there should be a REQ-REP-*like*
>> protocol that is stateless. That would clearly be useful. However, in my
>> view it is an error to give it the name REQ/REP, because this is confusingly
>> similar to the ZeroMQ protocol with the same name. It would be tolerable if
>> nanomsg REQ/REP was merely API-incompatible and wire-incompatible with the
>> protocol from ZeroMQ; that is to be expected. However when the difference
>> is not just in wireformat or argument order but is actually conceptually
>> orthogonal to the way ZeroMQ REQ/REP is used (which is stateful) in a way
>> that is hard to see up front it is time to invent a new name for a new idea.
>> The connection between the two protocols is so flimsy that sharing a name
>> is actively misleading.
>>
>> That aside, you are taking a much stronger position than merely one
>> protocol. You are saying “nobody should implement a stateful protocol in
>> nanomsg”. Well, why not?
>>
>> Clearly many applications have stateful networking problems. How should
>> they solve these problems?
>>
>> Should they each implement state atop nanomsg in their own incompatible way?
>> This is duplication of effort, and probably leads to buggy code
>> Should somebody consolidate those stateful implementations into a library
>> that sits atop nanomsg? Now language bindings have to be written for that
>> library in addition to nanomsg. This is duplication of effort, and probably
>> leads to buggy code
>> Should these applications not use nanomsg at all, and instead use ZeroMQ?
>> But there are many reasons to prefer nanomsg including use of C, BSD socket
>> API, context-free implementation, pluggable transports, routing priorities,
>> zero-copy, etc. etc. This option basically tells applications “because you
>> have stateful networking, you can enjoy none of these benefits”
>> Should these applications fork nanomsg and hack in a stateful alternative?
>> This is duplication of effort, and probably leads to buggy code
>>
>> I think what you imagine is that if you make stateful programs difficult,
>> people will rethink their stateful problems into stateless ones. Sometimes
>> that is true. But other times, people will rethink their stateful programs
>> and decide being stateful is the right decision for their problem. There
>> should be some vision to address this case other than “deal with it”.
>>
>> It is one thing to declare stateful problems out of scope for a protocol.
>> It is a different thing to declare them out of scope for a networking
>> library. People have stateful problems; and this doesn’t disappear merely
>> because stateless protocols are well-behaved or more scalable.
>>
>> I don’t know exactly what the right solution looks like. Maybe it’s
>> creating protocols that are deliberately stateful. Maybe it’s introducing
>> some kind of pluggable session layer. But I am absolutely certain that
>> throwing up our hands and declaring anything stateful as out of scope would
>> move nanomsg out of the “zeromq done right” search space and turn it into a
>> very specialized tool for very specialized problems.
>>
>>> On May 6, 2014, at 8:23 PM, Garrett D'Amore <garrett@xxxxxxxxxx> wrote:
>>>
>>> Nanomsg is stateless from the point of view of the app. Adding session
>>> state would be inappropriate. The pipe is almost certainly insufficient by
>>> itself and recognize that it is subject to change as a result of connects
>>> or disconnects in the underlying transport.
>>>
>>> The information I would like to make available would allow for actual
>>> strong authentication on those transports that can support it, but
>>> admittedly it's only on receipt and it ties to a specific message so at the
>>> abstract level is still stateless.
>>>
>>> Sent from my iPhone
>>>
>>>> On May 6, 2014, at 4:20 PM, Drew Crawford <drew@xxxxxxxxxxxxxxxxxx> wrote:
>>>>
>>>> I don’t think putting it in the body is the right solution, or at least
>>>> not right for every case.
>>>>
>>>> For one thing it requires allocating storage for this is in the messages.
>>>> It is an interesting question how much storage is required but the naive
>>>> implementation would be a 128-bit GUID. For a 4-byte messages this bloats
>>>> the network traffic significantly. Particularly when there is a simple
>>>> solution with zero overhead—pull the pipe key from nanomsg.
>>>>
>>>> For a another thing, ZeroMQ clearly and unambiguously supports this model
>>>> (via ROUTER). So anyone porting ZeroMQ code is in for a rough time
>>>> implementing their own scheme atop nanomsg on all the codebases to get the
>>>> same behavior they had before. There are of course legitimate reasons to
>>>> be incompatible with ZeroMQ (such as achieving compatibility instead with
>>>> BSD sockets) but I think this case is much more harmful than helpful and
>>>> does not isolate itself to one particular system or codebase.
>>>>
>>>>> let’s think about cases where the client sending requests loses
>>>>> connection, then reconnects with a different address, or connects from
>>>>> multiple endpoints (mobile device, desktop computer, …), if remote
>>>>> endpoint information is provided by nanomsg all these endpoints will
>>>>> appear as different entities, whereas in your application logic they
>>>>> should be considered the same.
>>>>
>>>>
>>>> You should probably not rely on transport-layer guarantees to authenticate
>>>> a user. However *given* transport-layer information, it *becomes possible
>>>> to implement* many authentication schemes.
>>>>
>>>> For example (and this is the problem that motivated this discussion)
>>>> suppose I have some decision oracle which can determine with complete
>>>> certainty whether a particular user intended to send a packet. Then
>>>>
>>>>> if (!oracle_user_sent_message(user,message)) {
>>>>> end_session();
>>>>> }
>>>>
>>>> Now if we know (or can guess) that John sent the message via
>>>> transport-layer information, the solution is straightforward. But trying
>>>> all the possible users is impractical for a slow oracle. So the
>>>> transport-layer information can comprise part of an application-level
>>>> authentication scheme to identify which user the oracle should be asked
>>>> about. Even for sockets that do not have 1:1 fanout to users, the fanout
>>>> may divide the users into enough buckets that asking the oracle about each
>>>> member in the bucket becomes practical.
>>>>
>>>> Now of course we could prepend some session ID to the message rather than
>>>> rely on transport-level data, and bloat the message accordingly. However
>>>> anybody who gets ahold of the session ID could spoof messages with that
>>>> session ID from anywhere on the network. Now these would be rejected by
>>>> our perfect oracle, but not before ending the user’s session, comprising a
>>>> DDoS attack against the legitimate user. Alternatively, relying on the
>>>> TCP information significantly increases the difficulty of the attack,
>>>> requires a TCP MITM technique or some other advanced persistent threat
>>>> capability to execute. This is a major security and reliability advantage
>>>> to relying on TCP data in my situation.
>>>>
>>>> Drew
>>>>
>>>>
>>>>
>>>>
>>>>> On May 6, 2014, at 12:13 PM, Achille Roussel <achille.roussel@xxxxxxxxx>
>>>>> wrote:
>>>>>
>>>>> Could you put the state information in the body of your message instead
>>>>> of attempting to get it from nanomsg. HTTP is also stateless but websites
>>>>> maintain more or less state using cookies, session ids or access tokens…
>>>>> maybe you can implement this at the application logic level.
>>>>>
>>>>> I think it’s a sain design to have your transport protocol separated from
>>>>> your application logic, let’s think about cases where the client sending
>>>>> requests loses connection, then reconnects with a different address, or
>>>>> connects from multiple endpoints (mobile device, desktop computer, …), if
>>>>> remote endpoint information is provided by nanomsg all these endpoints
>>>>> will appear as different entities, whereas in your application logic they
>>>>> should be considered the same.
>>>>>
>>>>>> On May 6, 2014, at 1:23 AM, Drew Crawford <drew@xxxxxxxxxxxxxxxxxx>
>>>>>> wrote:
>>>>>>
>>>>>>> There are many cases that require state full networking
>>>>>>>
>>>>>> I’m in such a case. The open question at this point is how to achieve
>>>>>> it.
>>>>>>
>>>>>>
>>>>>>> On May 5, 2014, at 10:52 PM, Apostolis Xekoukoulotakis
>>>>>>> <xekoukou@xxxxxxxxx> wrote:
>>>>>>>
>>>>>>> Req rep were designed by default to be stateless, that is why finding
>>>>>>> the address of the message has been hidden on purpose.
>>>>>>>
>>>>>>> There are many cases that require state full networking but state full
>>>>>>> is more difficult because it requires that you implement an update
>>>>>>> mechanism on the routing information.
>>>>>>>
>>>>>>>> On May 6, 2014 5:02 AM, "Drew Crawford" <drew@xxxxxxxxxxxxxxxxxx>
>>>>>>>> wrote:
>>>>>>>> I have dug a little deeper into this. it appears that in global.c [1]
>>>>>>>> msg_controllen is never set. I’m not sure if that’s expected.
>>>>>>>>
>>>>>>>> The attached patch sets controllen based on the size of the chunk.
>>>>>>>> Whether right or wrong, this seems to produce the behavior expected by
>>>>>>>> zerotacg and Achille, e.g., control bytes are emitted in the RAW case.
>>>>>>>> The 8 bytes are
>>>>>>>>
>>>>>>>>> d0,4e,c0,00,c1,7f,00,00,
>>>>>>>>
>>>>>>>>
>>>>>>>> Three of which (bytes[3],bytes[4],and bytes[5]) seem to change from
>>>>>>>> run-to-run. This is mildly surprising, because the RFC documents the
>>>>>>>> control ID at being 32 bits, so one would expect four bytes to change
>>>>>>>> from one execution to the next. I’m also unable to account for the
>>>>>>>> presence of the remaining bytes. Something may be wrong with my
>>>>>>>> patch, or with my understanding of the codebase or RFC.
>>>>>>>>
>>>>>>>> This is an interesting line of inquiry, but since a solution along
>>>>>>>> this line has the limitation of requiring me to implement my own
>>>>>>>> end-to-end behaviors on top of a raw socket, I’m wondering if it would
>>>>>>>> be desirable to introduce an API for this purpose
>>>>>>>>
>>>>>>>>> /* Returns an integer that uniquely identifies the immediate sender
>>>>>>>>> of the most-recently-received message. Returns an error if no
>>>>>>>>> messages have ever been received on the socket */
>>>>>>>>
>>>>>>>>> int nn_sender(int socket);
>>>>>>>>
>>>>>>>> Such API could work equally well for raw sockets as full sockets,
>>>>>>>> could be implemented for different socket topologies, and does not
>>>>>>>> introduce an application-layer dependency on parsing the control
>>>>>>>> header format.
>>>>>>>>
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://github.com/nanomsg/nanomsg/blob/master/src/core/global.c#L817
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On May 5, 2014, at 4:38 PM, Drew Crawford <drew@xxxxxxxxxxxxxxxxxx>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I thought about that, however, msg_controllen still returns -1 when
>>>>>>>>> using raw sockets, suggesting there are no control information
>>>>>>>>> available, as the sample below illustrates. Maybe something is wrong
>>>>>>>>> with the code sample?
>>>>>>>>>
>>>>>>>>> Another problem is that use of raw sockets would require me to roll
>>>>>>>>> my own end-to-end behavior which may be undesirable.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> int client = nn_socket(AF_SP,NN_REQ);
>>>>>>>>>> int server = nn_socket(AF_SP_RAW,NN_REP);
>>>>>>>>>> nn_connect(client,"inproc://test");
>>>>>>>>>> nn_bind(server,"inproc://test");
>>>>>>>>>> nn_send(client,"A",1,0);
>>>>>>>>>>
>>>>>>>>>> int rc;
>>>>>>>>>> void *body;
>>>>>>>>>> void *control;
>>>>>>>>>> struct nn_iovec iov;
>>>>>>>>>> struct nn_msghdr hdr;
>>>>>>>>>>
>>>>>>>>>> iov.iov_base = &body;
>>>>>>>>>> iov.iov_len = NN_MSG;
>>>>>>>>>> memset (&hdr, 0, sizeof (hdr));
>>>>>>>>>> hdr.msg_iov = &iov;
>>>>>>>>>> hdr.msg_iovlen = 1;
>>>>>>>>>> hdr.msg_control = &control;
>>>>>>>>>> hdr.msg_controllen = NN_MSG;
>>>>>>>>>> rc = nn_recvmsg (server, &hdr, 0);
>>>>>>>>>> print_array(body,rc,"body”); //contains only A
>>>>>>>>>>
>>>>>>>>>> printf("msg_iovlen %d\n",hdr.msg_iovlen); // 1
>>>>>>>>>> printf("msg_controllen %d\n",hdr.msg_controllen); // -1
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On May 5, 2014, at 4:32 PM, Achille Roussel
>>>>>>>>>> <achille.roussel@xxxxxxxxx> wrote:
>>>>>>>>>>
>>>>>>>>>> You have to use AF_SP_RAW sockets to get access to these info in the
>>>>>>>>>> control header when receiving a message with nn_recvmsg.
>>>>>>>>>>
>>>>>>>>>>> On May 5, 2014, at 2:27 PM, Drew Crawford <drew@xxxxxxxxxxxxxxxxxx>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I have a REP socket. I’m trying to identify the channel (sender or
>>>>>>>>>>> forwarder) on which some message has arrived to the socket. A
>>>>>>>>>>> transport-layer understanding of the sender is not required; any
>>>>>>>>>>> identifying value, such as an integer, is sufficient. Consulting
>>>>>>>>>>> the REQREP spec suggests that the topmost “channel ID”, one of the
>>>>>>>>>>> records in the “backtrace”, is the identifier I’m looking for.
>>>>>>>>>>>
>>>>>>>>>>> Clearly this identifier is not exposed over the nn_recv interface.
>>>>>>>>>>> I had some hopes that it would be accessible in the nn_recvmsg
>>>>>>>>>>> interface, possibly as control information, but it seems not to be
>>>>>>>>>>> the case:
>>>>>>>>>>>
>>>>>>>>>>>> int client = nn_socket(AF_SP,NN_REQ);
>>>>>>>>>>>> int server = nn_socket(AF_SP,NN_REP);
>>>>>>>>>>>> nn_connect(client,"inproc://test");
>>>>>>>>>>>> nn_bind(server,"inproc://test");
>>>>>>>>>>>> nn_send(client,"A",1,0);
>>>>>>>>>>>>
>>>>>>>>>>>> int rc;
>>>>>>>>>>>> void *body;
>>>>>>>>>>>> void *control;
>>>>>>>>>>>> struct nn_iovec iov;
>>>>>>>>>>>> struct nn_msghdr hdr;
>>>>>>>>>>>>
>>>>>>>>>>>> iov.iov_base = &body;
>>>>>>>>>>>> iov.iov_len = NN_MSG;
>>>>>>>>>>>> memset (&hdr, 0, sizeof (hdr));
>>>>>>>>>>>> hdr.msg_iov = &iov;
>>>>>>>>>>>> hdr.msg_iovlen = 1;
>>>>>>>>>>>> hdr.msg_control = &control;
>>>>>>>>>>>> hdr.msg_controllen = NN_MSG;
>>>>>>>>>>>> rc = nn_recvmsg (server, &hdr, 0);
>>>>>>>>>>>> print_array(body,rc,"body”); //contains only A
>>>>>>>>>>>>
>>>>>>>>>>>> printf("msg_iovlen %d\n",hdr.msg_iovlen); //1
>>>>>>>>>>>> printf("msg_controllen %d\n",hdr.msg_controllen); //-1
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have consulted a previous mailing thread on this topic which
>>>>>>>>>>> suggests channel IDs are manipulated in rep.c. Indeed, the
>>>>>>>>>>> information I’m looking for seems to be moved around between
>>>>>>>>>>> nn_sockbase, nn_msg, nn_rep, and similar structures. However I
>>>>>>>>>>> cannot work out a sane way to get those structures from application
>>>>>>>>>>> code.
>>>>>>>>>>>
>>>>>>>>>>> Any suggestions on identifying the sender of a remote message?
>>>>>>>>>>>
>>>>>>>>>>> Drew
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>
Other related posts:
