[nanomsg] Re: Persisted reliable messaging
- From: Jörg Singler <joerg@xxxxxxxxxxxx>
- To: nanomsg@xxxxxxxxxxxxx
- Date: Fri, 22 Nov 2013 12:38:52 +0100
Hi Martin,
thanks. Ok. Your thoughts are highly appreciated and some of them are well
documented as well in the ZMQ guide. I would really like to combine the
lightweightness and speed of nanomsg with certain pluggable subscribers (if
required) which then forward the message to more heavyweight, restartable
queues such as rabbitMQ or various other commercial products which ensure
failover recovery.
For my understanding one could combine very smartly the best out of both
worlds: highspeed on the bus and for those agents which are subscribed for
a subset of messages but need to safely transport the message to a
third-party system one would integrate the failover-save stuff. Or would
you vote more for: don't do that ;-)
Best, Joerg
2013/11/22 Martin Sustrik <sustrik@xxxxxxxxxx>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hi Joerg,
>
> > that nanomsg is a pure networking library is clear for me. I played
> > around with it already. Alright. So let's say you have an internal
> > message bus implemented via the Pub/Sub pattern (because you want
> > to leverage the subscriptions). It is ok that the bus is in terms
> > of crashes somehow atomic (either it works completely or not at
> > all). Some of the subscribers are communicating with external
> > systems and here you want to have more reliability (with
> > persistence). if I get you right: your answer would be not to use
> > nanomsg at all (at least for the communication to the external
> > system), right?
>
> It's really a matter of what you want to achieve. 100% reliability is
> impossible.
>
> If you want to be safe wrt. a box being destroyed (by fire or such)
> you want to have mutliple outward facing boxes, presumably in
> different buildings etc. IIRC RabbitMQ has clustered mode that could
> help you with such setup, but still, you are not safe wrt. split-brain
> syndrome etc.
>
> If all you are interested in is being safe in the face of sender
> process failing, RabbitMQ or a similar solution that writes data to
> the disk, may solve your problem -- unless the process fails again
> once it's restarted :)
>
> Finally, if you want to be safe wrt network failures, TCP handles that
> for you and you can use nanomsg as is.
>
> That being said, you should be aware that with pub/sub there are only
> 2 mutually exclusive options: 1.) System will block/crash because of a
> slow subscriber. 2.) Messages will be dropped when faced with a slow
> consumer.
>
> In other words, you either have a 24/7 support team that checks
> whether the system is blocked and solves the problem when it occurs
> (by killing consumers, adding more disk space or similar), or you
> accept that messages will be lost sometimes.
>
> Martin
>
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJSjy9uAAoJENTpVjxCNN9YKyYIAIxD+W3taDiNc/IR6a5s+0M9
> o5bavmbCoR3AIEm57pMAF18g/mcQSedp18YvjrAbS0eW8tZqOxxKhgYYOfW8Lg8F
> Y7RIKTDOQrAxvjtpkgxzf6bADv4QIjQ4HfGh93B5oGm5RorBCSIF9/yohVBX0+EF
> HrcNzdZnW892zy1CJxSCacgL3c7CEp+Ik1XeeG15XQnB1LeYG8uzz1dVpHpJb3sB
> rPgHSzkPN1YVkd6egc6JKetrKVN+G8okIIBrFV6lMRwfTpbfq7t3zRNPi759fWrr
> tImW8TMxH4tMCF7uzV3UOuYY8cP64EOclOUGEmLEC2xCvJoxRFKsSIKej9gs9aA=
> =0VL4
> -----END PGP SIGNATURE-----
>
>
Other related posts:
