[nanomsg] Re: WebSocket Transport design considerations
- From: Garrett D'Amore <garrett@xxxxxxxxxx>
- To: "nanomsg@xxxxxxxxxxxxx" <nanomsg@xxxxxxxxxxxxx>
- Date: Thu, 16 Oct 2014 14:36:36 -0700
I think this is a good first start. Longer term we can add real buffering and
tie that to the rest of the protocol. But for now KISS.
Sent from my iPhone
> On Oct 16, 2014, at 2:20 PM, Jack Dunaway <jack@xxxxxxxxxxxxxxxx> wrote:
>
> Reading a byte at a time crossed my mind, but I tossed it as silly! However,
> if there is consensus that this is not out of the question, that's indeed a
> viable workaround.
>
> And now thinking about this idea more seriously... as an optimization, it's
> possible to read 4 bytes at a time, until part of a CRLFCRLF sequence is
> encountered at the end of those 4 bytes, at which point the next read would
> be 3, 2, or 1 byte. There are a few "false positives" along the way, since
> every header line ends in CRLF, but this would roughly cut the number of
> reads by a factor of 4.
>
> To put this into perspective, a "typical" opening handshake is <200 bytes, so
> order of magnitude estimate, we could receive the entire header within ~50
> such iterations in order to establish a connection.
>
> Would any others weigh in on this strategy? Barring better ideas or
> opposition, I think it's a viable path for a first pass.
>
> Best regards,
>
> Jack R. Dunaway | Wirebird Labs LLC
>
>> On Thu, Oct 16, 2014 at 3:01 PM, Garrett D'Amore <garrett@xxxxxxxxxx> wrote:
>> So if you are implementing the handshake you could even dispense with
>> buffering and just do single byte reads. It's not ideal for performance but
>> simple enough and it only impacts connection setup.
>>
>> Sent from my iPhone
>>
>>> On Oct 16, 2014, at 12:35 PM, Jack Dunaway <jack@xxxxxxxxxxxxxxxx> wrote:
>>>
>>> For clarity, this issue is only for the opening handshake. Once the
>>> handshake is complete, the sockets then talk on a (relatively) fixed-width
>>> header binary protocol. It's just the opening handshake with the issue of
>>> non-fixed width HTTP. (for reference RFC 6455
>>> https://tools.ietf.org/html/rfc6455)
>>>
>>> Best regards,
>>>
>>> Jack R. Dunaway | Wirebird Labs LLC
>>>
>>>> On Thu, Oct 16, 2014 at 1:33 PM, Garrett D'Amore <garrett@xxxxxxxxxx>
>>>> wrote:
>>>> This all sounds... strange.
>>>>
>>>> Maybe because you're trying to handle websockets directly, and so wind up
>>>> basically writing an HTTP handler?
>>>>
>>>> It strikes me that once connected, web sockets act like a TCP stream. Or
>>>> should. I have no idea how Windows and CGI express this, but that would
>>>> be the route I'd have gone. The only "tricky" part is getting this hooked
>>>> to a web server for the serving side. (On the client side, I'd hope this
>>>> is basically abstracted away in a library call that looks like of like a
>>>> connect() call with some extra parameters?)
>>>>
>>>> If you're going to write the web sock handler / handoff directly, I'd
>>>> think you wind up basically doing a non-blocking read, parsing the header
>>>> as it comes in (lines at a time), until the HTTP/1.1 header is complete.
>>>> Once you've finished the handshake, you're in normal TCP transport mode, I
>>>> think, right?
>>>>
>>>>
>>>>> On Thu, Oct 16, 2014 at 10:20 AM, Jack Dunaway <jack@xxxxxxxxxxxxxxxx>
>>>>> wrote:
>>>>> Thank you to Paul and Garrett for weighing in. As an update on progress,
>>>>> adding the ws:// protocol to nanomsg has progressed nicely with no
>>>>> external dependencies. I could use some guidance on one remaining
>>>>> question below, and will then be ready for Autobahn testing and pushing
>>>>> to a public repo.
>>>>>
>>>>> The problem: Since WebSockets send an unknown length HTTP syntax opening
>>>>> handshake, the strategy for receiving this handshake is more complicated
>>>>> than the elegant fixed-width handshake headers of other nanomsg
>>>>> transports (see:
>>>>> https://github.com/nanomsg/nanomsg/blob/master/src/transports/utils/streamhdr.c#L180).
>>>>>
>>>>> The complication arises since nn_usock_recv() dispatches an async worker
>>>>> whose completion is determined only once all bytes requested have been
>>>>> received. Whereas, since the WebSocket handshake is a termination
>>>>> sequence delimited protocol, we must scan the input stream for the
>>>>> CRLFCRLF termseq as the final token before determining success.
>>>>>
>>>>> Things I've tried (followed by a tl;dr question at the end):
>>>>>
>>>>> 1) On initializing the handshake, starting a nn_usock_recv() operation
>>>>> requesting an arbitrarily large number of bytes, and immediately starting
>>>>> a backoff timer as well. As the backoff timer times out, the buffer to
>>>>> which the underlying socket worker is writing is parsed. The three
>>>>> outcomes of this parse are 1) valid and accepted opening handshake;
>>>>> commence to send response, 2) valid but unacceptable opening handshake;
>>>>> commence to send a failure message then close the connection, and 3)
>>>>> termseq not yet encountered; try again on the next backoff timer timeout.
>>>>> (An implicit fourth mode exists as well, and that is if all requested
>>>>> bytes are received, this will likely result in a failure of the
>>>>> handshake) The first case is the tricky case -- the nn_usock_recv worker
>>>>> continues attempting to receive the arbitrarily-large number of bytes,
>>>>> whereas it should shut down once the opening handshake has been fully
>>>>> received
>>>>> 2) Modifying the nn_usock_recv() interface
>>>>> (https://github.com/nanomsg/nanomsg/blob/master/src/aio/usock.h#L88) to
>>>>> have options of either synchronous or async receive (analogous to
>>>>> NN_DONTWAIT). Which ties in closely to the third investigation...
>>>>> 3) Introducing nn_usock_recv_raw() to the Windows usock, and also making
>>>>> this a public method on the usock object
>>>>> (https://github.com/nanomsg/nanomsg/blob/master/src/aio/usock_posix.inc#L1037).
>>>>> Porting the POSIX method to the Windows side, I got tangled in a web of
>>>>> learning about WSA and completion ports and overlapped I/O and... and
>>>>> decided to consult the mailing list on where to head.
>>>>> 4) Incrementally increasing the socket timeout as the backoff timer so
>>>>> that each time the usock worker times out is the signal for the parse to
>>>>> occur. However, it appears this is not a valid strategy, since socket
>>>>> options cannot be set during the handshake
>>>>> (https://github.com/nanomsg/nanomsg/blob/master/src/aio/usock_posix.inc#L248),
>>>>> and also because on Windows "socket options SO_RCVTIMEO and SO_SNDTIMEO
>>>>> apply only to blocking sockets"
>>>>> (http://msdn.microsoft.com/en-us/library/windows/desktop/ms741688(v=vs.85).aspx)
>>>>>
>>>>> Things I have investigated but not tried: upgrading the backed worker to
>>>>> complete on a termination sequence rather than on number of bytes
>>>>> received. This is perhaps more performant and more elegant than polling
>>>>> with a backoff timer (or at least, encapsulates this action), yet
>>>>> represents a fundamentally "not so good thing" that could pollute the
>>>>> library with a feature that is only incidentally required for the
>>>>> handshake of one transport. For now, polling with the backoff timer feels
>>>>> a reasonable first goal, and perhaps even the better long-term goal.
>>>>>
>>>>> Conceptually, I feel that #3 above is the most desirable path. In
>>>>> psuedocode, receipt of the opening handshake looks like: 1) Start backoff
>>>>> timer 2) on timeout, call a non-blocking synchronous recv command (no
>>>>> worker), 3) parse, 4) on success, continue handshake; if not enough
>>>>> information has been received increment backoff and goto 1
>>>>>
>>>>> The next best option: 1) Start async recv worker, 2) Start backoff timer,
>>>>> 3) on timeout, parse the buffer to which the async worker is currently
>>>>> writing 4) on success, stop worker and continue handshake; if not enough
>>>>> information has been received, goto 2
>>>>>
>>>>> tl;dr question: Given the current methods available, or suggesting a
>>>>> modification to the usock worker, what is the best way to coordinate
>>>>> polling the recv/parse sequence required here? I would be excited to hear
>>>>> this is a simple problem and I'm missing something obvious!
>>>>>
>>>>> Best regards,
>>>>>
>>>>> Jack R. Dunaway | Wirebird Labs LLC
>>>>>
>>>>>> On Fri, Oct 10, 2014 at 8:02 AM, Garrett D'Amore <garrett@xxxxxxxxxx>
>>>>>> wrote:
>>>>>> There is an rfc for tcp+tls and mangos conforms to that protocol. I had
>>>>>> started enhancing libnanomsg for this as well but got sidetracked and
>>>>>> didn't get far.
>>>>>>
>>>>>> I would be happy to add websocket support to mangos as well. It should
>>>>>> only take a couple of hours. Go makes this very easy. :-)
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> > On Oct 10, 2014, at 4:28 AM, Paul Colomiets <paul@xxxxxxxxxxxxxx>
>>>>>> > wrote:
>>>>>> >
>>>>>> > Hi Jack,
>>>>>> >
>>>>>> >> On Fri, Oct 10, 2014 at 7:38 AM, Jack Dunaway <jack@xxxxxxxxxxxxxxxx>
>>>>>> >> wrote:
>>>>>> >> I have begun investigating development of a WebSocket transport for
>>>>>> >> nanomsg,
>>>>>> >> and I'm curious what design/implementation/brainpower others have put
>>>>>> >> into
>>>>>> >> this already.
>>>>>> >>
>>>>>> >> The first stage "hack" in this investigation has been retrofitting the
>>>>>> >> existing TCP transport
>>>>>> >> (https://github.com/nanomsg/nanomsg/tree/master/src/transports/tcp) to
>>>>>> >> dispatch different handshake and send/recv handlers based on transport
>>>>>> >> scheme.
>>>>>> >>
>>>>>> >> Based on a day's worth of hacking, here are a few early-stage design
>>>>>> >> considerations:
>>>>>> >>
>>>>>> >> 1a) It feels heavy-handed to copy-and-paste the entire TCP transport
>>>>>> >> wholesale, just to change a few key areas where WebSockets differ
>>>>>> >> (e.g., on
>>>>>> >> the surface, only STCP needs non-trivial modification). On the other
>>>>>> >> hand,
>>>>>> >> the IPC transport very closely resembles TCP, so perhaps there's
>>>>>> >> merit in
>>>>>> >> this duplication? That said, does it make more sense to duplicate the
>>>>>> >> TCP
>>>>>> >> protocol as a starting point for the WS protocol, or to extend the
>>>>>> >> existinga
>>>>>> >> TCP?
>>>>>> >
>>>>>> >
>>>>>> > Probably with current codebase you just have to copy. In fact IPC
>>>>>> > (unix sockets), transport is even more similar to TCP, but they don't
>>>>>> > share codebase. I think it's reasonable trade off.
>>>>>> >
>>>>>> >> 2) There exist several WebSocket protocol parsing libraries in C;
>>>>>> >> however,
>>>>>> >> I'm not readily finding a sweet spot of License + Quality + RFC 6455
>>>>>> >> coverage. Ideas?
>>>>>> >
>>>>>> > I think it's easy enough to write your own. You can use autobahn as a
>>>>>> > test suite.
>>>>>> >
>>>>>> >> 3) Since the WebSocket protocol does not have a fixed-width header
>>>>>> >> like the
>>>>>> >> nanomsg TCP wire-level protocol
>>>>>> >> (https://github.com/nanomsg/nanomsg/blob/master/rfc/sp-tcp-mapping-01.txt#L68),
>>>>>> >> this presents a fundamental new challenge buffering and parsing an
>>>>>> >> incoming
>>>>>> >> stream. Thoughts?
>>>>>> >
>>>>>> > I don't think it's really complex stuff.
>>>>>> >
>>>>>> >> 4) For the ws:// protocol, how should a client negotiate scalability
>>>>>> >> protocol? Via parameters on the URI (e.g.,
>>>>>> >> ws://127.0.0.1:9876/?sp=NN_SUB)?
>>>>>> >> And, what shall be the value of Sec-WebSocket-Protocol?
>>>>>> >
>>>>>> > I'd say "Sec-WebSocket-Protocol: x-nanomsg-sub"
>>>>>> >
>>>>>> >> 5) Does it make sense to attack this problem of nanomsg<->WebSockets
>>>>>> >> as a
>>>>>> >> specific application use-case, or as generally desirable transport
>>>>>> >> considered for inclusion into the core nanomsg library?
>>>>>> >
>>>>>> > Don't make a transport if you need websockets for specific application
>>>>>> > usage. Make a "gateway" -- a separate process which does translation,
>>>>>> > using any available http/websocket library.
>>>>>> >
>>>>>> >> 6) wss:// ... ?
>>>>>> >
>>>>>> > Yes, sure. Use libressl :). If you will be doing it, consider also
>>>>>> > looking at tcp+ssl implementation (latter should probably be
>>>>>> > compatible to mangos).
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > Paul
>>>>>> >
>
Other related posts:
