[nanomsg] Re: Port sharing, tcpmux, web-related roadmap et c.

  • From: Paul Colomiets <paul@xxxxxxxxxxxxxx>
  • To: "nanomsg@xxxxxxxxxxxxx" <nanomsg@xxxxxxxxxxxxx>
  • Date: Fri, 21 Nov 2014 15:42:07 +0200

Hi Martin,

On Fri, Nov 21, 2014 at 12:24 PM, Martin Sustrik <sustrik@xxxxxxxxxx> wrote:
>
>>> You mean expensive in terms of bytes transferred? If so, fair enough.
>>>
>>> However, most arguments against multiple connections don't really stand.
>>> I've written a blog about that in the past: http://250bpm.com/blog:18
>>>
>>
>> Not exactly in terms of bytes transferred. It's ok to open any number
>> of connections between two services, even if they speak websockets.
>> But with browsers:
>>
>> 1. The tend to have limits on number of connections (you can have few
>> tens of services behind a single gateway, and user having 50-100 tabs
>> is not so uncommon).
>
>
> That's ugly. So there's an artificial limit on number of connections that
> forces us to replace time-tested TCP multiplexing with some kind of hack of
> our own for no good reason, no performance improvement or such.
>

Yes. That the reality all web developers live in. There is also
ongoing standard that does that:

http://www.ietf.org/archive/id/draft-ietf-hybi-websocket-multiplexing-11.txt

Propbably there is more recent version somewhere. Or the problem is
going to be solved by HTTP 2.0. I'm not following that trend very
well.

>> 2. Websocket connections are usually pinged to keep them closed fast
>> in case of closing client connection, so traffic is huge
>>
>> 3. Allowing lots of connections per host make server easy DoS attacked
>> (remember NAT'd clients)
>>
>> 4. Some companies have hundreds of employees behind NAT, so can easily
>> exhaust number of ports at client-side
>
>
> I wonder what's the conclusion from 2,3,4? To not use websockets and rather
> use raw TCP services spread over multiple ports?
>

For machine to machine communication, yes. It doesn't matter too much
whether Websocket delimits messages or some other protocol. Except
websocket have mandatory "masking" and crappy handshake which are
weird, but technically doesn't matter.

For browser to server communication the solution is per-message
multiplexing/tunelling.

>> It's not weak. It prevents a javascript on web page
>> http://malicious.attacker/ to connect websocket to http://gmail.com
>> with your credentials.
>> So yes, it's not check against bots. It's check against javascript on
>> malicious web pages.
>
>
> Right, some kind of whitelisting the origins can be added then.
>

Yes. And whitelist may be different for each url, if you allow
multiple different services to attach to different urls at same
host-port.

>>> I am wondering about what the alernative would be. Any ideas?
>>>
>>
>> The gateway, which does access auth checks and some sort of tunelling.
>> It's kinda tcpmux on steroids. But, firstly to have some real
>> authentication and for tunelling, the gateway must read and understand
>> websocket packets itself
>>
>> If the gateway does packet-based routing, it may (I would argue it
>> should) just prefix packet with authentication info and send that
>> packet over persistent nanomsg connection. I.e. you shouldn't have
>> nanomsg connection per websocket connection, but a persistent nanomsg
>> connection between service and gateway.
>>
>> So it may be an nginx module, or standalone http/websockets server
>> like zerogw or mongrel, but not a small and transparent one like
>> tcpmuxd.
>
>
> Ok, so there are 2 concerns, tunnelling and more rigorous initial
> handshaking, right.
>
> I am still not convinced about the former (see above).
>
> As for the latter, can ZeroGW be re-purposed to act as a WebSocket
> multiplexer; i.e. can it, after doing the initial handshake, pass the
> connection to the application, the way that tcpmuxd does?
>

First issue, is that you still can't know which connection message
comes from. And that's insecure in browser. The simplest case to
explain, is if one uses client certificates to authenticate client
connections.

By the way how passing connection to the application is going to work
with wss:// anyway? I don't think there is a TLS implementation that
can serialize state and send it along with SCM_RIGHTS message.


Second, I believe it's architecturally wrong. Let's consider we have a
simple websock-mux M and a service S. On the same box, so connection
hand off with SCM_RIGHTS work (with square brackets denoting box):

Client -> [ M -> S ]

The if service is too slow, to be done on the same machine, we need to
move service S to a separate box(es). We do that by putting
intermediate proxy:

C -> [ M -> P ] -> [ S ]

Now, how the proxy is going to be implemented? It can't handoff
connection to a different machine. It can either create a connection
to S for each client C. So number of connections C -> P between
clients and proxy equals to the number of connections P -> S between
proxy and service. But that the *old*, pre-zeromq way (i.e we could
just put haproxy or nginx there).

The *new* way is to create a single nanomsg socket P -> S between a
proxy and service. But this way we need to rewrite service S when we
start to scale. So why not start with the following scheme, so we can
scale out later:

C -> [ M -> P -> S ]

Then I don't see a reason of not merging M and P. They can't be on
separate boxes anyway, and they tied to a single protocol (i.e. muxer
can't accept websocket where proxy expects raw tcp).

-- 
Paul

Other related posts:

  1. Home
  2. nanomsg
  3. Archive
  4. 11-2014