I think the latter is more likely to be useful and efficient. If you have
very specific memory allocation needs, then I think the API consumer should
provide memory regions. In the Solaris/illumos kernel we have a technique
like this for mblks called “esballoc” —
http://www.unix.com/man-page/opensolaris/9f/esballoc/ — all the high
performance networking NIC card use this.
I’m a little concerned here about how the final API is going to look for
API consumers. I think its really important to have a clean API for users
of the libnanomsg API, and as design constraints use of optimization
features needs to be optional. That is, if an API consumer doesn’t use
your fancy allocation scheme, then it should still work, perhaps less
efficiently.
Further, there are challenges I think because you have to arrange for the
API consumer and the transport to collaborate on the memory allocation
scheme. What I mean is, the transport needs to be able to identify that
the memory it wants to use is already “registered” (I guess this means that
it is mapped for DMA in the system, and registers have been programmed on
the device to identify its location by some kind of numeric ID?), so that
it can do the right thing. I actually have no idea how you’re going to
wind up doing that cleanly — most of my ideas for this wind up looking
pretty ugly. A big part of the problem is that we don’t really reveal the
transport to the API consumer — to do most of what I think you want to do,
you’re going to have to find a solution to break through the abstraction
boundary that libnanomsg provides. (And frankly that abstraction boundary
is a big percentage of the services that libnanomsg is designed to provide
— meaning by doing this you’re sort of negating a sizeable part of the
benefit of libnanomsg — particularly the transport independence, and the
ability to have many transports participating on a single socket.)
I’ve said here and in many other places, I highly recommend implementing a
simple copy scheme in your transport first, and benchmarking that.
Frankly, compared to other operations that libnanomsg does, the copy of
your messaging data is unlikely to have a huge performance impact. The
exception here would be if your application needs to send huge messages
(e.g. >64KB) frequently. (libnanomsg has performance issues that come from
the extra system calls and file descriptors it uses to provide a poll() and
select() compatible notification mechanism — basically it performs a small
write(2) to a notification pipe, which has to be poll()’d, then read().
That’s at least three system calls more per message than we would have if
we just used a simple synchronous pattern using threads. Sadly that
problem cannot be fixed without a major rewrite of libnanomsg, and not
without discarding the poll() and select() compatible semantic. I don’t
see that happening for libnanomsg, ever; its something for a wire
compatible alternate implementation.)
If I were implementing a transport like yours, I’d have preallocated and
premapped buffers available in the transport, and on TX I’d just copy into
one of those; on TX completion I’d recycle that buffer internally. On RX
you receive into your registered memory, and then copy into ordinary
memory. I’d probably keep a pool of allocated RX buffers (normal memory,
not registered with your device) handy to ensure that I could receive
without getting stuck behind malloc.
On a system with DTrace, you might even be able to deep probe to see
whether the bulk of of your time is spent doing the bcopy(), or if that
falls into the noise compared to e.g. the system calls. (I’m not sure what
other introspection tools might be available on Linux or Windows. I’m
mostly an illumos guy, though I use a Mac on the desktop.)
Now that said, if you can show that libnanomsg consumers see a notable
performance difference justifying the complexity (more likely if your needs
are to exchange huge messages, and your transport can move them natively
using zero copy DMA or somesuch), then this becomes much more worthwhile.
- Garrett
On Thu, Apr 14, 2016 at 6:03 AM, Ioannis Charalampidis <
ioannis.charalampidis@xxxxxxx> wrote:
Hi all!
I am going to add an additional feature to the chunks core and I wanted
your feedback in order to chose what would benefit everyone. In my case I
will need to allocate a series of consecutive chunks for optimization
reasons (fewer memory registrations) and since this is not currently
supported I have two solutions planned for implementation :
(1) Shall I introduce a high-level function: nn_chunk_alloc_many( size_t
size, int type, int count, void*** chunks ) that allocates a number of
consecutive chunks using the allocation type specified?
Example:
void * chunks[4];
nn_chunk_alloc_many( 1024, NN_ALLOC_PAGEALIGN, 4, &chunks );
// .. use them as chunks ..
// Free them
nn_chunk_free( chunks[0] );
nn_chunk_free( chunks[1] );
nn_chunk_free( chunks[2] );
nn_chunk_free( chunks[3] );
Pros:
- Very simple and straightforward API from user's PoV
Cons:
- This requires additional reference tracking and custom de-allocator
functions in order to wait for all chunks to be free'd before the actual
memory region is released, but that's easily managed.
- In order to implement the memory registration I will need to know
the buffer base address and overall size (that in case of memory alignment
won't be equal to size * count), therefore introducing a kind of ugly
optional 5h parameter ( struct nn_chunk_meta * meta ) that will be
used to track such information.
- If more fine-grained control is required it's difficult to access
the implementation internals without hacking it (btw, this is something
that I have been fighting with until I decided to actually touch the chunk
code myself).
(2) Or shall I introduce a lower-level function : nn_chunk_init( void *
ptr, size_t ptr_size, nn_chunk_free_fn destructor, void * userptr, void **
chunk ) that initializes a chunk structure to a given buffer? In this
case the user should allocate the consecutive buffer and then call this
function to initialize parts of it as chunks.
Example:
size_t chunk_size = 1024 + nn_chunk_hdrsize();
void * memory = aligned_alloc( sysconf(_SC_PAGESIZE), chunk_size * 4 );
// Create chunks
void * chunks[4];
void * ptr = memory;
for (int i=0; i<4; i++) {
nn_chunk_init( ptr, chunk_size, &free_fn, NULL, &chunks[i] );
ptr = ((uint8_t*)ptr) + chunk_size;
}
// .. use them ..
// Free chunks (calls the given free function, doesn't free anything)
nn_chunk_free( chunks[0] );
nn_chunk_free( chunks[1] );
nn_chunk_free( chunks[2] );
nn_chunk_free( chunks[3] );
// User needs to free memory eventually, or needs to
// implement the high-level logic mentioned before to
// free memory when last chunk is freed
free( memory );
Pros:
- No need to implement the reference tracking, which keeps the chunk
core cleaner
- The custom free function can be used to track
implementation-specific logic (ex. mark buffer as free for re-use)
- No need to track the individual chunks when it's time for clean-up,
just free the allocated memory.
Cons:
- The user needs to do the memory management.
- Very similar API to nn_chunk_alloc_ptr that might introduce
confusion. The difference is that the latter just creates a const
pointer-chunk to the user data, while the former assumes the data given is
a chunk and initializes it as such (writes a chunk header and returns a
pointer to the chunk data).
(3) Or shall I implement both solutions?
Looking forward to your comment/choice!
Cheers,
Ioannis
