[nanomsg] Re: Getting rid of nn_init() and nn_term()
- From: Martin Sustrik <sustrik@xxxxxxxxxx>
- To: David Nyström <david.c.nystrom@xxxxxxxxx>
- Date: Tue, 05 Feb 2013 23:55:21 +0100
Hi David,
Cacheline bouncing would only become prevalent when the shared
datastructures change, and cache needs to be invalidated. If this
datastructure changes when a socket is created/destroy:ed, this would
mean trouble.
What nanomsg API calls could potentially cause a write to this
datastructure ?
What Chuck referred to was using global defaults for socket options, ie.
set option once globally, rather than setting it separately for each socket.
That shouldn't cause a problem, AFAIU.
In addition to that, global state consists basically of an array which
maps fds to object pointers. Individual nn_* functions then use the
array to access the socket object. Pseudocode:
struct nn_sock *sockets [MAX_SOCKETS];
int nn_send (int s, ...)
{
struct nn_sock *sock = sockets [s];
...
}
As can be seen, pointers to socket objects with similar fds can reside
in a single cache line and thus creation/destruction of a socket can
cause cache line bounce for sockets with similar fds.
This can be avoided either by aligning the pointers to cachelines, or
simply assigning the fds in non-sequential manner (e.g. 0, 16, 32, 48, etc.)
Anyway, currently the prevalent source of latency is thread wake-up, ie.
OS-level eventfd and condition variable objects, not cache bouncing.
Martin
Other related posts:
