[nanomsg] Re: interop testing and benchmarks
- From: Martin Sustrik <sustrik@xxxxxxxxxx>
- To: nanomsg@xxxxxxxxxxxxx
- Date: Wed, 26 Mar 2014 01:48:29 +0100
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 25/03/14 20:09, Garrett D'Amore wrote:
> As there is now more than a single implementation of nanomsg (er..
> SP), it seems that it would be useful to have some basis for
> comparing implementations. It also also seems that it would be
> useful to have a “conformance” suite of tests, so that new
> implementations can be tested against each other (or at least
> against the “reference”, which in this case I presume to be
> libnanomsg.)
>
> Has anyone given any thought to this? Conformance *could* be
> something as simple as a wrapper around nanocat, but I suspect we
> might want to add a bit more to it, to validate things that are
> easier to check from native C than indirectly via nanocat (think
> especially about edge case validation.) It seems that this is more
> than just a task to write code, but also a documentation effort,
> since one would have to know what the conformance test is going to
> test for. (For example, if a REQ/REQP is used, then there has to
> be agreement about what the conformance test involves. Including
> details like port numbers or IPC paths, message contents, etc.
>
> I’m motivated to make such a test suite and document it, treating
> libnanomsg as the “reference”. But this might be something
> someone else wants to contribute to?
This would be very useful, however, given that nanomsg is a tool for
building distributed -- and thus often non-deterministic -- systems,
validation of conformity is often a pretty hard problem.
> I’m also thinking that it would be useful to have some kind of
> benchmark tests as well. How fast can an implementation respond
> to REQ/REP, bandwidth, etc. This can serve less as a marketing
> tool, and more as a tool to help implementors identify bottlenecks,
> and put effort on fixing them. This could even be handled as part
> of a “conformance” specification. Both sides of a protocol should
> be specified, so that it should be possible to mix and match, and
> measure all four possible combinations of client/server resulting
> from just two different implementations.
With both ZeroMQ and nanomsg we have two simple perf tests:
1. Latency test (doing ping-pong among 2 nodes).
2. Throughput test (tranferring a lot of small messages from one to
another)
These are the two most interesting metrics that can be measured in a
simple way.
Of course, there are zillions of other metrics that may be ineresting,
but it's typically much harder to measure those.
Martin
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJTMiPdAAoJENTpVjxCNN9Y35EH/2BouFuqwuah6YhRdWJFl1yP
f+1MT94B3Gy4th2QjT+OFEdAh7z3fX0sHwP6EC2NhzGJNJZStmsO5reIkue8Nyah
uVIFvPmJIvL7tINj1kjv9BDtuRNixWdIU5SOj0F81cZDxgKXUcD7v1APb8XXUfQR
f+j8WmOwl3OLLu06MExq4Wjx1xXuz8NM2NtBSaIAFVmwQpMofByjDhlPzwZMxLG1
6ipxzBfCE7Y4IxRuHinfOafVovZV8RvgnyI81jEqpLtvAKvHkE04aHbfLi7xJKXw
0AA5UcqbtY5uvtNnrGjt1nb81uSWFe2fzPKN4oSafsySFXWabUUHNAxnbQq5GCQ=
=svSP
-----END PGP SIGNATURE-----
Other related posts:
