[nanomsg] Re: Framing - separation of multiple parts of the message
- From: Jörg Singler <joerg@xxxxxxxxxxxx>
- To: nanomsg@xxxxxxxxxxxxx
- Date: Sat, 23 Nov 2013 10:08:07 +0100
Hi,
isn't there a way which does not require a memcopy when using ipc or
inproc? For my understanding all serializations cause converting a set of
fields into a binary stream, right? How does that fit with ideas of RDMA?
Best, Joerg
2013/11/22 Paul Colomiets <paul@xxxxxxxxxxxxxx>
> Hi Steve,
>
> On Fri, Nov 22, 2013 at 7:28 PM, Steve Atkins <steve@xxxxxxxxxxx> wrote:
> >
> > On Nov 22, 2013, at 9:19 AM, Paul Colomiets <paul@xxxxxxxxxxxxxx> wrote:
> >
> >> Hi Martin,
> >>
> >> On Fri, Nov 22, 2013 at 7:03 PM, Martin Sustrik <sustrik@xxxxxxxxxx>
> wrote:
> >>>
> >>> It would be a nice weekend project to implement 0MQ-style multi-part
> >>> messages on top of nanomsg.
> >>>
> >>
> >> I wouldn't encourage anybody to do the work. For the following reasons:
> >>
> >> 1. Multipart messages are too attractive to use for coarse-grained
> >> protocol. But at the end of the day they suck.
> >>
> >> 2. I've used to think that's multipart are nice, because even simplest
> >> C client can make use of them without complex dependencies. But full
> >> msgpack parser take a few hours to implement (in C), and it's nice,
> >> well known format, that is going to be standardized by IETF
> >> (hopefully). It's even easier to implement subset of msgpack.
> >>
> >> 3. That "project" would need bindings to every language. And there are
> >> already bindings for each and every language for msgpack (or put your
> >> favourite serialization format name here).
> >
> > One place where it'd be useful is where you want to use it for routing
> > information. Cracking open a big protobuf or msgpack payload just to
> find out
> > where it should be sent to is a fair bit of overhead.
> >
>
> The subset of msgpack that parses a string of up to 32 bytes looks like:
>
> if(*c & 0xE0 == 0xA0) {
> memcpy(routing_data, c+1, *c & ~0xE0);
> }
>
> I think it's simple enough to use for routing (And it's not that hard
> if you need more than 32 bytes too). You don't need full parser for
> the purpose.
>
> > The extreme of that is pub/sub - you can't send packed messages over
> > pub/sub and use subscriptions without adding some sort of basic
> > framing (or getting *way* too chummy with the packing format in use).
> >
>
> Yes. But you need different formats based on how do you want to match.
> If you want full match for topic, you might be better served with
> length-prefixed data, like above. But if you want prefix matches to
> work, you need to put a delimiter at the end of the topic (or in some
> cases you might end up without a delimiter at all).
>
> Anyway if we ever need the topics to be matched by nanomsg, e.g. in
> subscription matching plugin, it's easy enough to embed an msgpack
> decoder. E.g. the one in nanoconfig is less than 500 lines of code, it
> parses everything except strings > 4 Gb, and does no memory
> allocations. So should be good enough for the purpose. (And I believe
> we eventually embed nanoconfig anyway :) )
>
> --
> Paul
>
>
Other related posts:
