[nanomsg] Re: why memcpy data to be sent?
- From: Garrett D'Amore <garrett@xxxxxxxxxx>
- To: "nanomsg@xxxxxxxxxxxxx" <nanomsg@xxxxxxxxxxxxx>
- Date: Tue, 28 Oct 2014 17:57:10 -0700
Bluntly. I think you may be suffering from premature optimization.
Getting to tens of gigabits per second isn't that hard modern hardware.
Profile your app and check to see where it is spending time.
It may be cheaper to throw a little more hardware at the problem and
parallelize than to try extraordinary measures like a user space tcp stack.
Sent from my iPhone
> On Oct 28, 2014, at 5:45 PM, Matthew Hall <mhall@xxxxxxxxxxxxxxx> wrote:
>
>> On Mon, Oct 27, 2014 at 02:13:48PM -0400, Steve Vinoski wrote:
>> Theoretically, yes, but this doesn't help at all when using nanomsg
>> underneath other systems that do their own memory allocation. For my
>> specific case of sending messages, the Erlang VM allocates the memory for
>> the data as part of its per-lightweight-process GC system, and the Erlang
>> VM passes pointers to those blocks to my driver which then passes them
>> along to nanomsg.
>>
>> Maybe I'm missing something, but in looking at the nanomsg code some more I
>> couldn't see any compelling reason for the copy other than the fact that
>> the send functions called after the copy is made don't do the equivalent of
>> writev and instead take just a single block of data.
>>
>> --steve
>
> This is precisely the same sort of issue I was reporting a few weeks ago.
>
> I am working on a security sensor with a user-space network stack to
> eventually scale to 10's of gigabits, and I want to report threat matches via
> nanomsg using input from a non-default memory allocator.
>
> Right now you can't instruct nanomsg to zero-copy from your provided prebuilt
> buffer, and send it out.
>
> So my app ends up working like this:
>
> 1) Find data inside special Huge TLB Pages of network packets.
>
> 2) Create threat report using serialization library (causes gathering copy
> from few spots in packets).
>
> 3) Serialize threat report to buffer in serialization library (need to hack
> it
> eventually to remove this copy).
>
> 4) Nanomsg wants to make a copy to do its send. Instead I'd like to be able
> to
> have it use my buffer for the send, and callback my custom_buffer_free(void*)
> to return it to my pool.
>
> So it's a lot of work to remove all these copies, and past experience has
> shown the hit they cause is painful when you are writing a high-speed app
> like
> mine is going to be.
>
> Matthew.
>
Other related posts:
