[nanomsg] Re: interop testing and benchmarks
- From: 老栋 <yp.fangdong@xxxxxxxxx>
- To: nanomsg@xxxxxxxxxxxxx
- Date: Wed, 26 Mar 2014 13:38:55 +0800
is there anyone want to implement a unit test framwork for nanomsg first?
if no and needed, i am very happy to do it.
BTW, because of the fsm, it seems a little difficult to write unittest for
modules independently.
On Wed, Mar 26, 2014 at 1:13 PM, Garrett D'Amore <garrett@xxxxxxxxxx> wrote:
>
> --
> Garrett D'Amore
> Sent with Airmail
>
> On March 25, 2014 at 5:48:40 PM, Martin Sustrik (sustrik@xxxxxxxxxx)
> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 25/03/14 20:09, Garrett D'Amore wrote:
> > As there is now more than a single implementation of nanomsg (er..
> > SP), it seems that it would be useful to have some basis for
> > comparing implementations. It also also seems that it would be
> > useful to have a "conformance" suite of tests, so that new
> > implementations can be tested against each other (or at least
> > against the "reference", which in this case I presume to be
> > libnanomsg.)
> >
> > Has anyone given any thought to this? Conformance *could* be
> > something as simple as a wrapper around nanocat, but I suspect we
> > might want to add a bit more to it, to validate things that are
> > easier to check from native C than indirectly via nanocat (think
> > especially about edge case validation.) It seems that this is more
> > than just a task to write code, but also a documentation effort,
> > since one would have to know what the conformance test is going to
> > test for. (For example, if a REQ/REQP is used, then there has to
> > be agreement about what the conformance test involves. Including
> > details like port numbers or IPC paths, message contents, etc.
> >
> > I'm motivated to make such a test suite and document it, treating
> > libnanomsg as the "reference". But this might be something
> > someone else wants to contribute to?
>
> This would be very useful, however, given that nanomsg is a tool for
> building distributed -- and thus often non-deterministic -- systems,
> validation of conformity is often a pretty hard problem.
>
>
> Agreed that it may be somewhat difficult. Still, I think we can probably
> make some tests to ensure that protocols work as expected (base protocols),
> and also that certain "expected" failure modes are handled properly. E.g.
> sending a message on a REQ socket should work and be delivered even later,
> when the REP actually connects. Its easy enough to baseline this. (In the
> case of REQ/REP we can also include some testing with devices as part of
> any suite.)
>
>
>
> > I'm also thinking that it would be useful to have some kind of
> > benchmark tests as well. How fast can an implementation respond
> > to REQ/REP, bandwidth, etc. This can serve less as a marketing
> > tool, and more as a tool to help implementors identify bottlenecks,
> > and put effort on fixing them. This could even be handled as part
> > of a "conformance" specification. Both sides of a protocol should
> > be specified, so that it should be possible to mix and match, and
> > measure all four possible combinations of client/server resulting
> > from just two different implementations.
>
> With both ZeroMQ and nanomsg we have two simple perf tests:
>
> 1. Latency test (doing ping-pong among 2 nodes).
> 2. Throughput test (tranferring a lot of small messages from one to
> another)
>
> These are the two most interesting metrics that can be measured in a
> simple way.
>
> Of course, there are zillions of other metrics that may be ineresting,
> but it's typically much harder to measure those.
>
>
> Those are probably a good start. I'll have a look. I'm curious to see
> how my implementation performs. :-)
>
> - Garrett
>
>
>
> Martin
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJTMiPdAAoJENTpVjxCNN9Y35EH/2BouFuqwuah6YhRdWJFl1yP
> f+1MT94B3Gy4th2QjT+OFEdAh7z3fX0sHwP6EC2NhzGJNJZStmsO5reIkue8Nyah
> uVIFvPmJIvL7tINj1kjv9BDtuRNixWdIU5SOj0F81cZDxgKXUcD7v1APb8XXUfQR
> f+j8WmOwl3OLLu06MExq4Wjx1xXuz8NM2NtBSaIAFVmwQpMofByjDhlPzwZMxLG1
> 6ipxzBfCE7Y4IxRuHinfOafVovZV8RvgnyI81jEqpLtvAKvHkE04aHbfLi7xJKXw
> 0AA5UcqbtY5uvtNnrGjt1nb81uSWFe2fzPKN4oSafsySFXWabUUHNAxnbQq5GCQ=
> =svSP
> -----END PGP SIGNATURE-----
>
>
Other related posts:
