Hi Neil,
I'll need some guidance on the format as the current yaml format doesn't
seem suited to MathML/XML/HTML input (at least not in the few examples I
looked at). Currently my code generates a file with input and a file with
expected output, but that can be changed.
Bert,
Thanks for responding. Some questions and answers below.
Regarding the UEB tests: these are very welcome too, but Liblouisutdml
doesn't support UEB math, so the configuration file has yet to be written.
I'm not sure whether it is possible to write a configuration file that
mimics the behavior of the XSLT from APH (that Michael shared on the list a
while back). Worse case, we only add the tests to the Liblouis repository
and use the XSLT to convert MathML to the linear math format, rather than
with Liblouisutdml. As a matter of fact, let's start with that anyway, and
look at Liblouisutdml later?
Let me know where to put them, and I'll add them to that location. If it
is part of liblouis, I'll need some guidance on the format as the current
yaml format doesn't seem suited to MathML/XML/HTML input (at least not in
the few examples I looked at). Currently my code generates a file with
input and a file with expected output, but that can be changed.
Because UEB has multiple valid answers (based on what grade 1 indicators
get used), it is harder to test for correctness
How do you solve this issue in your own (MathCAT) test suite?
In general, I tried to get MathCAT to match the output in the examples I
found in the ICEB and BANA docs. Both offer *guidance *on when to use
grade 1 symbol/word/passage indicators and also capital symbol/word/passage
indicators, although there is at least one example that has different
output for the same expression (i.e., the ICEB example has different
braille than the BANA one), so it isn't possible to completely follow the
guidance exactly. Also, the guidance is very incomplete -- it often boils
down to 'use your best judgement when considering the following things
...'. I had an email discussion with someone from BANA trying to get some
precision for coding the guidance up, but was never able to pin them down
(they'd contradict earlier statements). In the end, there were about 10
examples out of 160 examples where the MathCAT output didn't match the
example. MathCAT internally produces three different versions, and one of
the internal versions did match the expected output and getting someone
else to verify their reasonableness gave me the confidence to mark those
tests as correct. liblouis may pick some other option, and so a failure
would need to be evaluated in UEB as to whether it is truly a failure or
not.
Neil
On Fri, Feb 3, 2023 at 2:47 AM Bert Frees <bertfrees@xxxxxxxxx> wrote:
Awesome, thanks very much Neil!
I don't know when the tests stopped running. Some stuff happened in the
Liblouisutdml repository since your tests were added but I don't follow it
that closely. Either they were disabled on purpose, or we didn't notice
that something broke.
I think it's easiest if we continue the discussion on Github.
It was kind of expected that only a small part of the new tests would
pass. But at least now we have proof that it is bad and, like you say, a
means to improve it.
Regarding the UEB tests: these are very welcome too, but Liblouisutdml
doesn't support UEB math, so the configuration file has yet to be written.
I'm not sure whether it is possible to write a configuration file that
mimics the behavior of the XSLT from APH (that Michael shared on the list a
while back). Worse case, we only add the tests to the Liblouis repository
and use the XSLT to convert MathML to the linear math format, rather than
with Liblouisutdml. As a matter of fact, let's start with that anyway, and
look at Liblouisutdml later?
Because UEB has multiple valid answers (based on what grade 1 indicators
get used), it is harder to test for correctness
How do you solve this issue in your own (MathCAT) test suite?
Thanks,
Bert
Op do 2 feb. 2023 om 02:07 schreef Neil Soiffer <soiffer@xxxxxxxxxxxx>:
With a lot of help from Norbert Markus (thanks!), I was able to get the
MathCAT Nemeth tests into liblouisutdml. I have submitted a PR
<https://github.com/liblouis/liblouisutdml/pull/87>, although it is
failing some checks for reasons I'm unclear about. I did need to make a
couple of modifications to existing files because file2brl reported errors
with them. I'll fix whatever the problems are if someone can explain the
errors
<https://github.com/liblouis/liblouisutdml/actions/runs/4070099837>.
The failure mentions an address sanitizer which usually refers to code
issues, but I didn't change any code. Potentially it is because the MathML
tests fail, but I didn't add the MathCAT tests to the tester, and the older
tests have always failed.
The bad news is that only ~15% of the 169 new tests that I added (which
come from the Nemeth code green book) passed. I submitted an issue
<https://github.com/liblouis/liblouisutdml/issues/88> with the details.
If anyone wants to improve the translation, now they have a means of
knowing whether they are making good progress or not.
I also have MathML to UEB tests. However, I don't know what the
configuration file should be for them. Unfortunately, Norbert doesn't know
either. Can anyone help with that? Because UEB has multiple valid answers
(based on what grade 1 indicators get used), it is harder to test for
correctness, but I can adjust the expected output for whatever failures
show up that are actually valid.
Neil
