[nanomsg] Re: Pub/Sub behavior
- From: Matthew Hall <mhall@xxxxxxxxxxxxxxx>
- To: nanomsg@xxxxxxxxxxxxx
- Date: Fri, 13 Nov 2015 22:26:03 -0800
On 11/13/15 11:08 AM, Charles McParland wrote:
Using tcp,
I can publish hundreds of messages and
the subscriber - which I've manually slowed down - keeps handing
me the earliest messages in order. So, are these messages getting
cached in the subscriber nanomsg stack ? Can I reach down and
flush old messages and force the subscriber to get "current"?
Hi Chuck,
I also have used nanomsg between C and Java and like the nice simple
APIs. I contributed some important security patches to jnano:
https://github.com/gonzus/jnano/pull/5
I have C code which is doing very high speed processing which can't be
allowed to bog down. For spray-and-pray mode in my code I use this:
rv = nn_send(nn_queue->conn, message, length, NN_DONTWAIT);
You can also do some slightly more sophisticated stuff:
http://nanomsg.org/v0.7/nn_setsockopt.3.html
NN_SNDBUF
NN_RCVBUF
If NN_DONTWAIT is used, packets are tossed out and nn_errno() is EAGAIN.
The buffer limits set with NN_SNDBUF are respected, plus one additional
packet is allowed. The application can apply internal backpressure from
the EAGAIN.
For automatic backpressure (i.e. blocking) just do not specify
NN_DONTWAIT, and the upstream app will slow to allow backpressure.
Another possible option would be reading everything downstream as soon
as it arrives and scheduling it for execution in a Timer Queue with a
deadline. Something like a token-bucket or leaky-bucket algorithm.
Nanomsg and ZeroMQ explicitly and deliberately eschew the more complex
queuing mechanisms because they end up being very counterproductive. In
this paper an expert explains it quite well if you didn't see this already:
http://www.imatix.com/articles:whats-wrong-with-amqp/
I found a lot of it quite obvious from my SW dev career but felt much
better knowing I wasn't the only one thinking it, and learned some
helpful details about how things can go so catastrophically wrong in our
field.
At a high level if producers and consumers have big long lasting
imbalances, the scaling between the tiers is wrong and one side needs to
be shrunk or expanded to balance the load. Trying to do otherwise ends
up causing more problems than it solves.
Matthew.
Other related posts:
