-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 25/03/14 20:09, Garrett D'Amore wrote: > As there is now more than a single implementation of nanomsg (er.. > SP), it seems that it would be useful to have some basis for > comparing implementations. It also also seems that it would be > useful to have a “conformance” suite of tests, so that new > implementations can be tested against each other (or at least > against the “reference”, which in this case I presume to be > libnanomsg.) > > Has anyone given any thought to this? Conformance *could* be > something as simple as a wrapper around nanocat, but I suspect we > might want to add a bit more to it, to validate things that are > easier to check from native C than indirectly via nanocat (think > especially about edge case validation.) It seems that this is more > than just a task to write code, but also a documentation effort, > since one would have to know what the conformance test is going to > test for. (For example, if a REQ/REQP is used, then there has to > be agreement about what the conformance test involves. Including > details like port numbers or IPC paths, message contents, etc. > > I’m motivated to make such a test suite and document it, treating > libnanomsg as the “reference”. But this might be something > someone else wants to contribute to? This would be very useful, however, given that nanomsg is a tool for building distributed -- and thus often non-deterministic -- systems, validation of conformity is often a pretty hard problem. > I’m also thinking that it would be useful to have some kind of > benchmark tests as well. How fast can an implementation respond > to REQ/REP, bandwidth, etc. This can serve less as a marketing > tool, and more as a tool to help implementors identify bottlenecks, > and put effort on fixing them. This could even be handled as part > of a “conformance” specification. Both sides of a protocol should > be specified, so that it should be possible to mix and match, and > measure all four possible combinations of client/server resulting > from just two different implementations. With both ZeroMQ and nanomsg we have two simple perf tests: 1. Latency test (doing ping-pong among 2 nodes). 2. Throughput test (tranferring a lot of small messages from one to another) These are the two most interesting metrics that can be measured in a simple way. Of course, there are zillions of other metrics that may be ineresting, but it's typically much harder to measure those. Martin -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTMiPdAAoJENTpVjxCNN9Y35EH/2BouFuqwuah6YhRdWJFl1yP f+1MT94B3Gy4th2QjT+OFEdAh7z3fX0sHwP6EC2NhzGJNJZStmsO5reIkue8Nyah uVIFvPmJIvL7tINj1kjv9BDtuRNixWdIU5SOj0F81cZDxgKXUcD7v1APb8XXUfQR f+j8WmOwl3OLLu06MExq4Wjx1xXuz8NM2NtBSaIAFVmwQpMofByjDhlPzwZMxLG1 6ipxzBfCE7Y4IxRuHinfOafVovZV8RvgnyI81jEqpLtvAKvHkE04aHbfLi7xJKXw 0AA5UcqbtY5uvtNnrGjt1nb81uSWFe2fzPKN4oSafsySFXWabUUHNAxnbQq5GCQ= =svSP -----END PGP SIGNATURE-----