Actually, mangos uses a pooled message cache, and custom allocator, because
otherwise you can have too much stress on the garbage collector in Go. So
I try to avoid it.
As to pure C programs, I rarely feel that this is worthwhile, although a
kmem_cache type situation (allocation from a slab) could be used to some
good effect. This approach is only necessary if your heap implementation
isn’t terribly great. It might be hard to retrofit this approach into
nanomsg (since it doesn’t really have a discrete message structure), but
libnng will easily be able to have customized allocation schemes like this.
And I agree 100% about blocking vs. nonblocking. If you can avoid resource
contention, and your threads implementation does not suck, then blocking
threads are as performant (or nearly so) as non-blocking I/O, and also can
scale quite a bit more to engage with multiple cores, etc. Designing
systems for this requires some care and thought. Which I’m investing in
libnng right now.
(Btw, libnng’s name isn’t set in stone… it may be renamed in the future — I
already have a candidate name in mind.)
- Garrett
On Wed, Dec 14, 2016 at 11:42 PM, Matthew Hall <mhall@xxxxxxxxxxxxxxx>
wrote:
On Wed, Dec 14, 2016 at 04:47:07PM +0000, Benoit Labaere wrote:
There is a per message overhead for several reasons:
- allocation on each send, because the socket takes ownership of the
message when sending.
For a long time I have been wishing for a way to free the messages via
callback handler. Because I usually prefer to create the messages in
special
pooled memory, send them, then mark them free from the pool upon successful
TX. It's basically zero alloc cost but not available in nanomsg or zeromq
yet.
- there is a channel between the user thread and the I/O thread, it is
crossed once by the sent message and once by the 'return code'.
Many people think non-blocking is best. But in my performance studies I
usually find that a very large number (like 4-6 per physical core) of
simple
blocking user threads is faster as it prevents this problem right here, and
the queueing overhead prevents you from managing to totally saturate.
- there is no prefetch of incoming messages.
Yes, you have to try to fit messages into cacheline multiples and proper
alignment then prefetch first 1-3 lines after RX completes before pushing
back
to user code.
Good luck,
Matthew.
