Hi David,
Cacheline bouncing would only become prevalent when the shared datastructures change, and cache needs to be invalidated. If this datastructure changes when a socket is created/destroy:ed, this would mean trouble. What nanomsg API calls could potentially cause a write to this datastructure ?
What Chuck referred to was using global defaults for socket options, ie. set option once globally, rather than setting it separately for each socket.
That shouldn't cause a problem, AFAIU.In addition to that, global state consists basically of an array which maps fds to object pointers. Individual nn_* functions then use the array to access the socket object. Pseudocode:
struct nn_sock *sockets [MAX_SOCKETS]; int nn_send (int s, ...) { struct nn_sock *sock = sockets [s]; ... }As can be seen, pointers to socket objects with similar fds can reside in a single cache line and thus creation/destruction of a socket can cause cache line bounce for sockets with similar fds.
This can be avoided either by aligning the pointers to cachelines, or simply assigning the fds in non-sequential manner (e.g. 0, 16, 32, 48, etc.)
Anyway, currently the prevalent source of latency is thread wake-up, ie. OS-level eventfd and condition variable objects, not cache bouncing.
Martin