[nanomsg] Re: publisher side filtering
- From: Martin Sustrik <sustrik@xxxxxxxxxx>
- To: nanomsg@xxxxxxxxxxxxx
- Date: Sun, 04 May 2014 07:36:22 +0200
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 03/05/14 23:18, Garrett D'Amore wrote:
> I haven't looked at it. I will but I wanted to think from a clean
> slate first.
I've written about the system implemented in 0MQ here:
http://250bpm.com/pubsub
Crossroads added support for different message matching algorithms
(prefix/regexp/SQL/you name it).
Conceptually, I find it useful to think of forwarding subscriptions to
publishers as of an optimisation:
In the beginning you have the distribution tree, the messages flow
from the publisher to each subscriber. The ultimate subscribers may
choose to drop some of the messages. (That's how nanomsg is
implemented ATM.)
Then, some parts of the tree may implement local optimisation by
subscriber letting the publisher know that it won't need some messages.
I consider this point of view pretty useful, as it allows us to think
of multicast parts of the tree not as some kind of oddity, but rather
fully standard nodes which happen not to implement the optimisation.
Now, here comes the problem:
PUB/SUB allows for multiple publisher/single consumer topologies.
What it means is that the subsription are sent to multiple
destinations. At the same time, the mechanism for passing
subscriptions has to be fully reliable (losing a subscription is not
an option as it would result in faulty business logic).
And as I've tried to explain in many places (in case of need I'll do
it again) reliable transmission to multiple destinations is possibly
only if you accept the fact that a slow/stuck/hung-up destination can
cause the whole tree to slow down or grind to the halt.
Obviously, we cannot accept that in PUB/SUB protocol.
The solution is not obvious though. Some possibilities (all of them
pretty sucky):
1. Don't allow more that one publisher per subscriber.
2. If there's more than one publisher, turn the subscription
forwarding off.
3. Allow for forwarding subscriptions only 1 hop upstream, thus
limiting the possible damage to a single tier of subscribers.
One solutions that is typically mentioned is "kill the connection to
the slow/hung-up producer". The problem with that is the connection
will be re-connected in a short time and then *all* the subscriptions
will be resent, making the congestion situation even worse (you can
think of it as a connection-based equivalent of retransmit storm).
Thoughts?
Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJTZdHWAAoJENTpVjxCNN9YskMIAKLwXkepbpCoTU3ChbvTZ++q
Kp7SvitbTaY8trYTSqU6TUZP0oxf00WkJ5hCsy0U7uQQ8mBs/PRoRyhFy7P9yxJK
NZ/4SiD/jD201S3DTc4LOC1fW4Kl1jh304DOUtypM5wXYnJIRgHVvSEdEdN94YKQ
fPvpPvjvmHBJHEYXJo2soB6QayqqF9m6Ozi/jxxH54NNvZFdckd4seYmQVRWeJ5o
mPI3R6ZbH3Qu4JQzpNlr2XW+nwXVvuRQw5PNcW84pVfQteUUP/OgRjGIEl7ufynP
/8Y5fvsWGoLt73zkjHmNUF9UvN1DFy+jqT/lg8inDv7BcXtjCR4fNi8telHygek=
=bz6T
-----END PGP SIGNATURE-----
Other related posts:
