[nanomsg] Re: towards a more robust socket model

  • From: Martin Sustrik <sustrik@xxxxxxxxxx>
  • To: nanomsg@xxxxxxxxxxxxx
  • Date: Sun, 16 Nov 2014 14:34:52 +0100

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Drew,

First of all, I completely agree. Layered design would improve the
current situation in a big way, and in the end, it's the only sensible
way to handle network protocols anyway.

The only problem, I think, is the implementation. The code is complex
enough even now and if we added more layers there would be a real
danger of it becoming unmanageable and falling apart under its own weight.

One option I've investigated was using coroutines+channels (the Go
way), I've even made a preprocessor that adds coroutine support to C.
Still, using it would mean heavy rewrite of the codebase.

If you have any better idea of what to do, I am all ears. What would
you say should be the first step towards the layered design?

Some minor comments follow:

> I am most familiar with the security/encryption problem because I
> have solved it.  I have worked on this for over a year and my
> solution is currently deployed to around 10k users as part of a
> commercial project. For reasons that will become clear, my solution
> so far only works for REQ/REP, and next year I have a requirement
> to get it working for at least one other protocol family.  So keep
> in mind that at the end of the tunnel for this architecture problem
> is potentially getting commercially-developed security contributed
> back to core.

Good to hear that!

> Meanwhile you want to encrypt and decrypt traffic on a hop-by-hop
> basis, to stop an NSA-level adversary from seeing (ciphertext, but
> identical) packets moving from hop to hop and thereby deducing who
> is talking to who.

Ah, I see, Tor-like onion routing. I was a strong proponent of
end-to-end encryption but this use hints that using both e2e and
hop-by-hop encryption would be a better solution.


> Each component has its output piped to the input of the next
> component, like processes in a Unix shell.  We call the bottom-most
> component which talks to the network a transport, and we call all
> other components, that each talk to the next component in the
> dataflow, a protocol.

Yes. This is where coroutines would fit in. Each protocol can be
modeled using a single coroutine, individual coroutines being
connected using go-like channels.

> This splits WS implementation into 2 protocols, one that can
> handle opcode switching above scalability layer and another that
> handles the bulk of WS below scalability layer.  These protocols
> could be used together, in the case that opcode support is desired,
> or with just the main WS protocol, if opcode support is not
> desired.  Finally, the actual transport (TCP or TLS, which IIRC
> isn’t supported in the current WebSocket “transport”) is moved out
> into a proper transport, where it is pluggable and interchangeable,
> both for WS and also for any ordinary non-WS socket to use.  Under
> this scheme the */actual transport* /really is just a dumb pipe,
> which has been one important philosophical objection to WS opcodes.
> Nor is WS coupled to the */actual transport/*, another
> philosophical objection that has been raised to coupling between 
> protocols and transports.

You are right about TLS, but I think opcodes are not really a problem.
It's only WebSocket developers being silly and introducing a bit of L6
functionality to what is fundamentally an L4 protocol. If we just
ignore the feature and use binary messages, no problem remains.

> This architecture also provides a clear path to more robust
> headers, which I think at present is non-intuitive and leads to
> unexpected situations (like #324).  With this architecture, one
> would simply walk down a socket’s stack and ask each
> protocol/transport how many bytes of overhead for headers it would
> like to reserve [0].  Then nanomsg can do one up-front allocation
> of the correct size for the message.  We could even standardize on
> a struct describing the header format being declared by each
> protocol, [1]  so that for a well-known stack the header can be 
> easily parsed, for debugging or any other purpose.

Not sure whether this would be flexible enough. There are many
protocols out there. Some have variable-length headers (WS), some have
footers as well (SOAP, XMPP). Some require message to be split into
several smaller chunks (PGM) Et c.

> Whereas under the proposed architecture, SP sockets have a natural
> dataflow representation:
> 
> [SP socket] -> [RAW socket] -> [Transport]

Yes, that would be nice.

> Some problems remain.  One problem is how to create these 
> many-protocoled sockets from an API perspective.

Yes, that could make the API a bit ugly, but in the end I think users
want to use pre-defined stacks of protocols rather composing arbitrary
protocols stacks themselves. Which means you just have to think of a
name for each pre-defined stack and you are done.

> Another problem is how to implement ispeer().

AFAICS this is a problem to be solved by each individual protocol in
the stack. There are no cross-layer issues, so we don't have to worry
about it on the architectural level.

Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iQEcBAEBAgAGBQJUaKf8AAoJENTpVjxCNN9Y+z0H/Rr4KTS8CQW+I7nbV3n5hC7e
/Wt62FxYJByWoTKNw5nNzHcPDH3+78smO+Nru6wmmNhOhz6xhM/OvyKM5DetZMeb
oABgaaV3YXSOQKG7Yb+WhIeg6/ZcPK8eX59RYpxLE2INESX2iL0gKi3rzC3+jET6
ylqvwZlFI4YlUiWixA7amOiOy4DFIxbTyt8eqtO9N82mTHwyC1PogBeU3Gg8Sewd
A1z4LaiSsw5eV0iCP1GFkc+wbhdX0ghihW2dBJqqAhwNQ+tVFcv1wgr0iU6HhNMX
bS+x1vxEY8Z+7huZCeLcTmO7HwcPBjaQ3UH1DhuO3PzpZzoaeltfgQ3aTadRUfY=
=t4Qh
-----END PGP SIGNATURE-----

Other related posts: