Isn't the current inproc implementation a shared-memory transport?
Something like this is welcome for ipc for sure, as well as anything
that makes inproc more performant.
As Garret says, the drawback of pipes for inproc is that the kernel
needs to get involved, so out-of-the-box inproc via pipes will have
relatively weak performance, especially in high-load or low-latency use
cases (people who care about this might use kernel bypass tech like
OpenOnload, but that's a small group).
However, I've always suspected that forced multi-threaded architecture
of nanomsg (i.e. worker threads) hurts when people want highest
performance. So perhaps the fear of the kernel is unwarranted.
Can any existing tools be used to easily benchmark the ipc transport
(which is UNIX domain sockets which is mildly less performant than
pipes) within an in-process/intra-thread program and compare it to the
inproc transport doing the same thing?
The upside of using pipes for inproc is that then the ipc transport can
basically use the same transport and get performance wins (versus the
existing implementation). The new ipc would handshake using using UNIX
domain sockets and then "upgrade" to pipes.
-Evan
On 06/02/2015 04:44 AM, Gerry Steele wrote:
inproc would be used where others might use a rungbuffer, as you
suggested and so any requirements should be driven by competing with a
ringbuffer implementation for the inter thread comms.
Might it be more interesting to consider a very low latency shared
memory based transport as common in commercial solutions?
On 01/06/2015, Garrett D'Amore <garrett@xxxxxxxxxx> wrote:
Just so everyone knows it: the inproc transport in libnanomsg is pretty
clearly fraught with problems. At the most basic level, it suffers from a
lack of synchronization control (locks) protecting some of the state shared
between the two endpoints of a connection (in particular the state machine
content is not carefully protected, although both ends participate in
informing each other of new content.)
Since the only conceivably useful case for inproc I can think of involves
different threads on the two different ends of a connection, this makes it
effectively useless until those bugs are fixed.
At least one person has volunteered to work on this part of the code. Ive
tried to fix a few of the issues, but the heart of the issue is going to
require some careful design work with the locking to ensure it works, and
does not create new deadlock scenarios. As I dont use inproc myself, I
cant really say that this has any sense of priority on my part Id like
to fix it, but my own itches tend to get scratched first, and Ive plenty
enough of those.
In the meantime, Id appreciate it if folks can try to avoid using inproc
where possible, and in particular try to refrain from filing new bugs
against inproc. With the lack of thread-safety in it, I suspect *many* of
the strange behaviors folks have reported will go away when inproc is
properly fixed.
Note that none of this applies to mangos. The inproc code in mangos, as
well as the rest of mangos, has been carefully designed to be completely
thread safe, and should not have any issues there. (I cant imagine what
youd use inproc for in a Go program but its there. Inproc in mangos is
not compatible with libnanomsg inproc, in the sense that you cant pass
messages between them.)
(One idea Ive been toying with is to have inproc convert to using a pipe()
based pair of file descriptors. This would let us use the kernel and avoid
the synchronization problems altogether, although it would make inproc
noticeably slower.)
- Garrett
