[nanomsg] Re: Interprocess communication on same host
- From: Alex Elsayed <eternaleye@xxxxxxxxx>
- To: nanomsg@xxxxxxxxxxxxx
- Date: Sun, 07 Sep 2014 11:48:19 -0700
On future versions of Linux there's also the option of using memfds, which
went in for 3.17 - memfds allow you to allocate an in-memory anonymous file,
write to it, and 'seal' it to lock it against changes. The intent is
explicitly that these be used for IPC, via the FD passing mechanism.
In doing testing to find the performance turnover point relative to passing
byte arrays through kdbus (which does exactly two context switches on a one-
way message), the developers found that the value where memfds outperformed
straight copy in 1-to-1 communication was 512KB, and surprisingly was the
same across platforms - x86, ARM, x86_64, etc.
See https://dvdhrm.wordpress.com/tag/memfd/ for more info.
Garrett D'Amore wrote:
> Without some kind of collaboration layer so that parties understand which
> addresses are in use and how it seems kind of useless to me. All the hard
> work still lives in the application. This compares poorly to other
> nanomsg transports.
>
> If you want to map up a bunch of data and use nanomsg to send control data
> only and use shmem for data then just do that. You don't have to do
> anything more than you would if you were going to try to have nanomsg
> handle this for you.
>
> As always, KISS.
>
> A big part of nanomsgs attraction is that it is simple and easy to build
> fault tolerant distributed architectures. Using shared memory flies in
> the face of that.
>
> Sent from my iPhone
>
>> On Sep 6, 2014, at 5:05 AM, Martin Sustrik
>> <sustrik@xxxxxxxxxx> wrote:
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Hi Garrett,
>>
>>> In my experience, people vastly underestimate the performance of
>>> bcopy.
>>>
>>> Unless you are passing around vast amounts of data (why are you
>>> using something like nanomsg in that case, btw?) it simply doesn't
>>> pay off. You lose the intended performance gains in the extra
>>> complexity and locking.
>>>
>>> To make this work well (and this would not be portable outside of
>>> the platform barring unusual measures like RDMA), you'd need a
>>> collaboration layer, a very large shared memory region, and some
>>> kind of ring or consume and produce indices in the buffer.
>>> Probably better to have two separate buffers, one for each
>>> direction, with different MMU settings (cache coherency).
>>>
>>> This also becomes really fragile. A bug in one program can now
>>> take out the other, unless you're very careful to treat the shared
>>> memory region with the same kind of care that you do packet data.
>>> (i.e. don't pass around program state, or pointers, etc.) Don't
>>> assume that the other side won't trash the memory.
>>>
>>> There may be some extreme cases where this complexity is
>>> worthwhile; you *could* use nanomsg to do that. But again, why?
>>> I'd just map the data up, and use POSIX signaling & mutexes to
>>> coordinate access. My guess is that this will be simpler than
>>> trying to coordinate across a simulated network.
>>>
>>> I have a hard time imagining that I'd want to forward data received
>>> from this over some kind of device to other parties in the nanomsg
>>> infrastructure, which is why I don't see much call to make this
>>> work with nanomsg.
>>
>> Agreed with all the above.
>>
>> However, given there is a use case where one process allocates a large
>> chunk of memory (say 1GB) does some work on it, then passes it to
>> another processes et c. I can see no reason why nanomsg should not be
>> able to support that.
>>
>> As already mentioned, most of the infrastructure is already in place:
>> nn_allocmsg() is already allocator-agnostic and so can be used to
>> allocate a message in shmem (say, for IPC message sizes above 1MB):
>>
>> void *p = nn_allocmsg (2000000, NN_IPC);
>>
>> Also, as you may recall, there is a "type" field in IPC protocol which
>> can be used to let the other party know that the message is coming
>> out-of-band, namely in shmem.
>>
>> All that being said, note that I am not proposing to do ring buffers
>> et c. Just allocate very large messages as chunks of shmem and you are
>> done. (Ring buffers are also doable, but hardly worth it IMO.)
>>
>> Ron, if you are interested in this stuff, feel free to implement it.
>> What you have to look at is how nn_allocmsg/nn_freemsg works, add
>> allocation of messages in shmem there, then modify the IPC transport
>> in such a way that it can transport shmem descriptors in addition to
>> the standard IPC bytesteam.
>>
>> Martin
>>
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.11 (GNU/Linux)
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>> iQEcBAEBAgAGBQJUCvhxAAoJENTpVjxCNN9YGe0IAJWeiXA+UgsFJEAexKXKOrO1
>> GubRU9WbMGtmYHo3IgKGFiEna+SUZVPp7QWKPYbsQzeOalKpEU5bx8Sdif69aYVq
>> 0K+zTg5VAdhXkufXukbs9+x9IN45bUqGlTbMt/bsDwgohk17qRzHcyDhl35sZJxu
>> KowYIY8ATyVQA3BkjGGiKLRZ+jA1cdQUwYVqlY9hQMvMsZDHoJxNXdktzgi4UhYg
>> hrcCC6oCqIzs+4T2zYVPAFJyJQCXJIv7gR+cpifLL3lbNug75G651wZt6qaPRoIB
>> VDhRmOIrZnoC8+8mXK3Fmzle+KNZSBeqZBq7EjPIvgXJu1QY5Ku1AWuBuyzIjo4=
>> =+fpz
>> -----END PGP SIGNATURE-----
>>
Other related posts:
