[liblouis-liblouisxml] Re: Why is there an error in this test run?
- From: Keith Creasy <kcreasy@xxxxxxx>
- To: "liblouis-liblouisxml@xxxxxxxxxxxxx" <liblouis-liblouisxml@xxxxxxxxxxxxx>
- Date: Fri, 6 Jun 2014 15:58:22 +0000
Mesar.
If you don't mind can we talk about the test suite itself? Really the summary
doesn't make sense.
Tests run: 13, Failures: 1, Expected failures: 7, Errors: 0
FAIL: run_test_suite.sh
=============================================
1 of 3 tests failed
What does this really mean? The only part that really adds up is that 13 tests
were run. It says 1 failed and I guess that much seems to make sense. Then it
says "expected failures: 7" and I don't understand what that means. Does it
means that 7 failures were expected or that there were actually 7 failures that
were expected?
Then, finally, it reports that 1 of 3 tests failed, which doesn't really add up.
IMHO, the summary is pretty useless.
Most importantly, no test failures should ever be expected. If we perform an
operation with known input we should always get known output. If we don't get
the output we expect then either we have to change what we expect the output to
be or fix the code to give us the output we expect. The goal should always be
that all tests pass and no code is committed to the main repository until they
do.
Is anyone in love with the existing tests?
Keith
-----Original Message-----
From: liblouis-liblouisxml-bounce@xxxxxxxxxxxxx
[mailto:liblouis-liblouisxml-bounce@xxxxxxxxxxxxx] On Behalf Of Mesar Hameed
Sent: Friday, June 06, 2014 11:45 AM
To: liblouis-liblouisxml@xxxxxxxxxxxxx
Subject: [liblouis-liblouisxml] Re: Why is there an error in this test run?
Sorry Keith I was not precise.
I ment that the code/configuration should be changed so that the user is
directed to the list/website.
I was not commenting on your particular bug report. :)
thanks,
Mesar
On Fri 06/06/14,15:38, Keith Creasy wrote:
> Well, I agree and strictly speaking I did send it to the list and not to John
> personally. He's just the most knowledgable person I know when it comes to
> this stuff.
>
>
> Thanks for your help with this, all of you.
>
>
> Keith
>
> -----Original Message-----
> From: liblouis-liblouisxml-bounce@xxxxxxxxxxxxx
> [mailto:liblouis-liblouisxml-bounce@xxxxxxxxxxxxx] On Behalf Of Mesar
> Hameed
> Sent: Friday, June 06, 2014 11:27 AM
> To: liblouis-liblouisxml@xxxxxxxxxxxxx
> Subject: [liblouis-liblouisxml] Re: Why is there an error in this test run?
>
> Hi,
> Its probably best if the user is redirected to the mailinglist and to the
> website rather than a particular individual.
>
> thanks,
> Mesar
> On Fri 06/06/14,17:24, Bert Frees wrote:
> > The difference between the output and the expected result appears to
> > be only whitespace. Either the expected result should be adjusted or
> > we should have a more advanced output comparing that ignores
> > whitespace at the end of lines (right now it is just using diff).
> >
> > For the record, I'm not the responsible person either, you knew that
> > :) I'm only trying to help out because I wrote those tests but I
> > have my hands full with liblouis.
> >
> > Bert
> >
> > John J. Boyer writes:
> >
> > > Keith,
> > >
> > > This looks good as far as the output of the test run is concerned.
> > > I don't know why failures are expected. It's something for Bert to
> > > explain. Shouldn't he be the one to receive bug reports? I'm no
> > > longer the appropriate person.
> > >
> > > John
> > >
> > > On Fri, Jun 06, 2014 at 02:57:08PM +0000, Keith Creasy wrote:
> > >> John, I did run the liblouisutdml tests:
> > >>
> > >> Tests run: 13, Failures: 1, Expected failures: 7, Errors: 0
> > >>
> > >> FAIL: run_test_suite.sh
> > >> =============================================
> > >> 1 of 3 tests failed
> > >> Please report to john.boyer@xxxxxxxxxxxxxxxxx
> > >>
> > >>
> > >> Why does it expect 7 failures?
> > >>
> > >>
> > >> Here is the entire output:
> > >>
> > >> Making check in gnulib
> > >> make[1]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/gnulib'
> > >> make check-recursive
> > >> make[2]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/gnulib'
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/gnulib'
> > >> make[3]: Nothing to be done for `check-am'.
> > >> make[3]: Leaving directory `/home/kcreasy/workspace/liblouisutdml/gnulib'
> > >> make[2]: Leaving directory `/home/kcreasy/workspace/liblouisutdml/gnulib'
> > >> make[1]: Leaving directory `/home/kcreasy/workspace/liblouisutdml/gnulib'
> > >> Making check in doc
> > >> make[1]: Entering directory `/home/kcreasy/workspace/liblouisutdml/doc'
> > >> make[1]: Nothing to be done for `check'.
> > >> make[1]: Leaving directory `/home/kcreasy/workspace/liblouisutdml/doc'
> > >> Making check in lbu_files
> > >> make[1]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/lbu_files'
> > >> make[1]: Nothing to be done for `check'.
> > >> make[1]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/lbu_files'
> > >> Making check in liblouisutdml
> > >> make[1]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/liblouisutdml'
> > >> make[1]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/liblouisutdml'
> > >> Making check in tests
> > >> make[1]: Entering directory `/home/kcreasy/workspace/liblouisutdml/tests'
> > >> Making check in test_suite
> > >> make[2]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite'
> > >> Making check in test_01
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_01'
> > >> make[3]: Nothing to be done for `check'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_01'
> > >> Making check in test_02
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_02'
> > >> make[3]: Nothing to be done for `check'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_02'
> > >> Making check in test_03
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_03'
> > >> make[3]: Nothing to be done for `check'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_03'
> > >> Making check in test_04
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_04'
> > >> make[3]: Nothing to be done for `check'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_04'
> > >> Making check in test_05
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_05'
> > >> make[3]: Nothing to be done for `check'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_05'
> > >> Making check in test_06
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_06'
> > >> make[3]: Nothing to be done for `check'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_06'
> > >> Making check in test_07
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_07'
> > >> make[3]: Nothing to be done for `check'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_07'
> > >> Making check in test_08
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_08'
> > >> make[3]: Nothing to be done for `check'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_08'
> > >> Making check in test_09
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_09'
> > >> make[3]: Nothing to be done for `check'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_09'
> > >> Making check in test_10
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_10'
> > >> make[3]: Nothing to be done for `check'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_10'
> > >> Making check in test_11
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_11'
> > >> make[3]: Nothing to be done for `check'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_11'
> > >> Making check in test_12
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_12'
> > >> make[3]: Nothing to be done for `check'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_12'
> > >> Making check in test_13
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_13'
> > >> make[3]: Nothing to be done for `check'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite/test_13'
> > >> make[3]: Entering directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite'
> > >> make[3]: Nothing to be done for `check-am'.
> > >> make[3]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite'
> > >> make[2]: Leaving directory
> > >> `/home/kcreasy/workspace/liblouisutdml/tests/test_suite'
> > >> make[2]: Entering directory `/home/kcreasy/workspace/liblouisutdml/tests'
> > >> make testTranslateString tripleTrans \
> > >> run_test_suite.sh
> > >> make[3]: Entering directory `/home/kcreasy/workspace/liblouisutdml/tests'
> > >> make[3]: `testTranslateString' is up to date.
> > >> make[3]: `tripleTrans' is up to date.
> > >> make[3]: Nothing to be done for `run_test_suite.sh'.
> > >> make[3]: Leaving directory `/home/kcreasy/workspace/liblouisutdml/tests'
> > >> make check-TESTS
> > >> make[3]: Entering directory `/home/kcreasy/workspace/liblouisutdml/tests'
> > >> XFAIL: testTranslateString
> > >> PASS: tripleTrans
> > >> -----------------------------------------------------------------
> > >> --
> > >> -------------
> > >> TESTS
> > >> -----------------------------------------------------------------
> > >> --
> > >> -------------
> > >> Running test_01...
> > >> Testing pagenum, runninghead and footer semantic actions
> > >> \033[1m\033[91mFAIL\033[0m\033[0m Running test_02...
> > >> Testing footer when some text immediately precedes the footer element
> > >> \033[1m\033[93mXFAIL\033[0m\033[0m
> > >> Running test_03...
> > >> footer bug
> > >> \033[1m\033[93mXFAIL\033[0m\033[0m
> > >> Running test_04...
> > >> Testing prehyphenation
> > >> \033[1m\033[92mPASS\033[0m\033[0m Running test_05...
> > >> Testing printPageNumberRange option with printPageNumberAt=bottom
> > >> \033[1m\033[92mPASS\033[0m\033[0m Running test_06...
> > >> Testing printPageNumberRange option with printPageNumberAt=top
> > >> \033[1m\033[93mXFAIL\033[0m\033[0m
> > >> Running test_07...
> > >> Testing orphanControl
> > >> \033[1m\033[92mPASS\033[0m\033[0m Running test_08...
> > >> Testing nested orphanControl. Outer orphanControl assertion should
> > >> not be
> > >> dropped when inner orphanControl is encountered.
> > >> \033[1m\033[93mXFAIL\033[0m\033[0m
> > >> Running test_09...
> > >> Testing hyphenation
> > >> \033[1m\033[92mPASS\033[0m\033[0m Running test_10...
> > >> Testing negative firstLineIndent
> > >> \033[1m\033[93mXFAIL\033[0m\033[0m
> > >> Running test_11...
> > >> Testing braille page numbering table (pageNumberTable)
> > >> \033[1m\033[92mPASS\033[0m\033[0m Running test_12...
> > >> Testing whitespace-only text-node in mixed content.
> > >> \033[1m\033[93mXFAIL\033[0m\033[0m
> > >> Running test_13...
> > >> Testing printPageNumberRange when a page break in print happens right
> > >> before
> > >> a page break in braille (fails when printPageNumberAt=bottom)
> > >> \033[1m\033[93mXFAIL\033[0m\033[0m
> > >>
> > >> Summary:
> > >> --------
> > >> Tests run: 13, Failures: 1, Expected failures: 7, Errors: 0
> > >>
> > >> FAIL: run_test_suite.sh
> > >> =============================================
> > >> 1 of 3 tests failed
> > >> Please report to john.boyer@xxxxxxxxxxxxxxxxx
> > >> =============================================
> > >> make[3]: Leaving directory `/home/kcreasy/workspace/liblouisutdml/tests'
> > >> make[2]: Leaving directory `/home/kcreasy/workspace/liblouisutdml/tests'
> > >> make[1]: Leaving directory `/home/kcreasy/workspace/liblouisutdml/tests'
> > >> Keith
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: liblouis-liblouisxml-bounce@xxxxxxxxxxxxx
> > >> [mailto:liblouis-liblouisxml-bounce@xxxxxxxxxxxxx] On Behalf Of
> > >> John J. Boyer
> > >> Sent: Friday, June 06, 2014 5:30 AM
> > >> To: liblouis-liblouisxml@xxxxxxxxxxxxx
> > >> Subject: [liblouis-liblouisxml] Re: Why is there an error in this test
> > >> run?
> > >>
> > >> Bert,
> > >>
> > >> Thanks. We'll be adding some tests to check UTDML .
> > >>
> > >> John
> > >>
> > >> On Fri, Jun 06, 2014 at 11:13:32AM +0200, Bert Frees wrote:
> > >> > Keith and John,
> > >> >
> > >> > There was a bug in the script indeed when run by itself and not
> > >> > through `make check`. I fixed that bug now, but it doesn't
> > >> > matter really. The script is supposed to be run with make check
> > >> > anyway because make takes care of setting LD_LIBRARY_PATH etc.
> > >> >
> > >> > You may also consider setting the `colors` var in the script to false.
> > >> > It is merely a visual help for myself, but I noticed from your
> > >> > emails that it produces garbage in your shell.
> > >> >
> > >> > Thanks,
> > >> > Bert
> > >> >
> > >> >
> > >> > Bert Frees writes:
> > >> >
> > >> > > This used to work. Haven't used it for a while. Let me check what's
> > >> > > wrong.
> > >> > >
> > >> > > Bert
> > >> > >
> > >> > > Keith Creasy writes:
> > >> > >
> > >> > >> Yes, this is what I found also. Thanks.
> > >> > >>
> > >> > >>
> > >> > >> -----Original Message-----
> > >> > >> From: liblouis-liblouisxml-bounce@xxxxxxxxxxxxx
> > >> > >> [mailto:liblouis-liblouisxml-bounce@xxxxxxxxxxxxx] On Behalf
> > >> > >> Of John J. Boyer
> > >> > >> Sent: Thursday, June 05, 2014 10:56 AM
> > >> > >> To: liblouis-liblouisxml@xxxxxxxxxxxxx
> > >> > >> Subject: [liblouis-liblouisxml] Why is there an error in this test
> > >> > >> run?
> > >> > >>
> > >> > >> Below is a typescript of running the tests in liblouisutdml. There
> > >> > >> seems to be a problem in finding things.
> > >> > >>
> > >> > >> Script started on Thu 05 Jun 2014 09:48:57 AM CDT
> > >> > >> $31:3>./run_test_suite.sh
> > >> > >> ------------------------------------------------------------
> > >> > >> --
> > >> > >> -----
> > >> > >> -------------
> > >> > >> TESTS
> > >> > >> ------------------------------------------------------------
> > >> > >> --
> > >> > >> -----
> > >> > >> -------------
> > >> > >> ./run_test_suite.sh: 49: cd: can't cd to /test_suite/test_* Running
> > >> > >> test_*...
> > >> > >> Can't find configuration file 'styles.cfg'
> > >> > >> Can't open input file input.xml.
> > >> > >>
> > >> > >> [1m[91mERROR[0m[0m
> > >> > >>
> > >> > >> Summary:
> > >> > >> --------
> > >> > >> Tests run: 1, Failures: 0, Expected failures: 0, Errors: 1
> > >> > >>
> > >> > >> $31:3>exit
> > >> > >> exit
> > >> > >>
> > >> > >> Script done on Thu 05 Jun 2014 09:49:48 AM CDT
> > >> > >>
> > >> > >> John
> > >> > For a description of the software, to download it and links to
> > >> > project pages go to http://www.abilitiessoft.com
> > >>
> > >> --
> > >> John J. Boyer; President, Chief Software Developer Abilitiessoft, Inc.
> > >> http://www.abilitiessoft.com
> > >> Madison, Wisconsin USA
> > >> Developing software for people with disabilities
> > >>
> > >> For a description of the software, to download it and links to
> > >> project pages go to http://www.abilitiessoft.com For a
> > >> description of the software, to download it and links to project
> > >> pages go to http://www.abilitiessoft.com
> >
> > For a description of the software, to download it and links to
> > project pages go to http://www.abilitiessoft.com
> For a description of the software, to download it and links to project
> pages go to http://www.abilitiessoft.com
For a description of the software, to download it and links to
project pages go to http://www.abilitiessoft.com
Other related posts:
