I’ve been working on a set of enhancements that I think will make it
*vastly* easier to write concurrent REQ applications (basically the idea is
to pull the state machine context for req into a user allocatable thing,
that can be used separately, so you can have many individual synchronous
contexts running against a single *cooked* mode REQ socket. I plan to do
the same thing for survey as well.)
As part of this, I’ve been looking a lot harder at the semantic, and there
are a couple of corner cases that I’m unhappy about; I am proposing to
change the semantics slightly to clear these corner cases up.
a) Multiple pending receive operations against a single socket (or a single
context) — this is largely nonsensical, but in nanomsg today lets you do
this. Only one of them will complete of course, so the others either hang
indefinitely or timeout (if a timeout was set). I’d like to change this
semantic so that if you try to issue a second (or more) receive against a
socket, that the second request returns NNG_ESTATE (EFSM in legacy nanomsg
API). This makes it clear that the user is violating the state machine of
one request to one reply.
b) Cancellation of a request by sending a new request, while a receive
operation was already pending. Today nng and nanomsg just “convert” the
receive, so that it only returns if the id matches the most recent send.
Except of course there is a subtle race condition there, since you can get
a reply with the *old* ID if the receive thread completion was in progress
when you started the new send. This semantic is buggy, and will be
frustrating for users that ever encounter it. It’s also incompatible with
the semantic “a” I proposed above. I’m therefore proposing to change this
so that when you attempt to send on a context (or a socket if not using
contexts), any pending receive would be canceled, and an error for that
receive would go back to the user (NNG_ECANCELED in NNG, probably EFSM for
the legacy API since I don’t have a better way to reflect that — we don’t
have an ECANCELED errno in the legacy API).
None of these apply to *raw* mode sockets; its only for cooked mode. (Raw
mode sockets get no additional semantic processing whatsoever; all REQ
semantics must be implemented by the API user with raw mode sockets, which
is one of the reasons they are so painful to use.)
The fundamental thing that these semantic changes do is strengthen the rule
that a cooked mode socket always matches *one* receive to *one* send, and
puts the burden on the API user to ensure that the ordering of send,
receive is adhered to. Most folks using the cooked mode sockets already do
this by running the operations in a single loop like this:
while (…) {
nng_send();
nng_recv();
}
Folks using cooked mode like this will be unimpacted by the above semantic
changes.
The thing that the new context structures (which I’ll be posting soon) do
is let you have *many* threads (or loops) doing the above, by adding a
context argument (pseudocode):
ctx = ctx_alloc();
while (…) {
nng_ctx_send(ctx, …);
nng_ctx_recv(ctx, …);
}
The actual context send and receive operations will be a bit different,
though, because
I’m intending that they are AIO based. This will make it possible to use
these contexts in places where blocking is inconvenient.
A demo program that shows this, with contexts being used to build a simple
HTTP REST API <-> REQ/REP gateway, will be supplied as well.
I think these contexts will allow us to eliminate about 90+% of the times
that people are using raw mode sockets, and result in a far far friendlier
scalable/concurrent cooked mode usage instead.
Anyway, I wanted to give folks a chance to let me know if the semantic
changes I’m proposing are going to be problematic. I don’t *think* they
will, but I’m not aware of all the corner cases that people code for
everywhere else, hence my inquiry here.
- Garrett
