[liblouis-liblouisxml] Re: Why is there an error in this test run?
- From: Bert Frees <bertfrees@xxxxxxxxx>
- To: liblouis-liblouisxml@xxxxxxxxxxxxx
- Date: Fri, 06 Jun 2014 18:38:46 +0200
Hi Keith,
Answers inline.
Keith Creasy writes:
> Mesar.
>
> If you don't mind can we talk about the test suite itself? Really the summary
> doesn't make sense.
>
> Tests run: 13, Failures: 1, Expected failures: 7, Errors: 0
>
> FAIL: run_test_suite.sh
> =============================================
> 1 of 3 tests failed
>
>
> What does this really mean? The only part that really adds up is that 13
> tests were run. It says 1 failed and I guess that much seems to make sense.
> Then it says "expected failures: 7" and I don't understand what that means.
> Does it means that 7 failures were expected or that there were actually 7
> failures that were expected?
I agree that "Expected failures: 7" can be interpreted in two ways. It
means a failure that was expected. You could also say "There were 8
tests of which 7 were expected".
> Then, finally, it reports that 1 of 3 tests failed, which doesn't really add
> up.
>
> IMHO, the summary is pretty useless.
There are two summaries. The first is about the test suite Christian and
I wrote. The second is about all tests. There are three tests in total
and the test suite as a whole is one of them.
I agree that the summary is confusing. You are welcome to improve it.
> Most importantly, no test failures should ever be expected. If we
> perform an operation with known input we should always get known
> output. If we don't get the output we expect then either we have to
> change what we expect the output to be or fix the code to give us the
> output we expect.
I don't agree. An expected failure is also a "passing test". It's just a
special kind of passing test. We check that the result does NOT match
the "correct" output. liblouis also has expected failures. It's a common
thing.
> The goal should always be that all tests pass and no code is committed to the
> main repository until they do.
You are right that ideally for new code tests should be written and that
they should pass before committing. But I added these tests after
discovering bugs, long after the faulty code had been written (written
not by me).
> Is anyone in love with the existing tests?
Well it was only meant as a start. There was no testing framework at all
before and I needed one. It is there to be improved. I'm not
particularly in love with the existing tests, but if you mean you want
to delete the expected failures, I say no, they are very useful. They
remember us of these bugs each time we build the program.
For a description of the software, to download it and links to
project pages go to http://www.abilitiessoft.com
Other related posts:
