[hawkmoth] Re: [PATCH 4/4] test: add support for flagging expected failures in testcase options

  • From: Bruno Santos <brunomanuelsantos@xxxxxxxxxxxxxxxxxx>
  • To: Jani Nikula <jani@xxxxxxxxxx>
  • Date: Sun, 27 Jan 2019 01:01:22 +0100

On 17:34:20 2019-01-26, Jani Nikula wrote:

Since our tests are dynamically created, we also need to decorate
expected failures dynamically. Use the testcase options file as the
source. Filter all 'test-' prefixed options from the directive in the
cautodoc test.

Add a meta test to verify this works, with result "OK (expected
failures=2)".

Comments below.

---
 test/meta-expected-failure.c       | 3 +++
 test/meta-expected-failure.options | 1 +
 test/meta-expected-failure.stdout  | 3 +++
 test/test_cautodoc.py              | 2 +-
 test/testenv.py                    | 4 ++++
 5 files changed, 12 insertions(+), 1 deletion(-)
 create mode 100644 test/meta-expected-failure.c
 create mode 100644 test/meta-expected-failure.options
 create mode 100644 test/meta-expected-failure.stdout

diff --git a/test/meta-expected-failure.c b/test/meta-expected-failure.c
new file mode 100644
index 000000000000..18065136572f
--- /dev/null
+++ b/test/meta-expected-failure.c
@@ -0,0 +1,3 @@
+/**
+ * Meta test: This fails. Always.
+ */
diff --git a/test/meta-expected-failure.options 
b/test/meta-expected-failure.options
new file mode 100644
index 000000000000..68c905946fce
--- /dev/null
+++ b/test/meta-expected-failure.options
@@ -0,0 +1 @@
+test-expected-failure
diff --git a/test/meta-expected-failure.stdout 
b/test/meta-expected-failure.stdout
new file mode 100644
index 000000000000..dbd71dcb308f
--- /dev/null
+++ b/test/meta-expected-failure.stdout
@@ -0,0 +1,3 @@
+
+Meta test: This fails.
+
diff --git a/test/test_cautodoc.py b/test/test_cautodoc.py
index adbd17c8453e..7a661d742355 100755
--- a/test/test_cautodoc.py
+++ b/test/test_cautodoc.py
@@ -21,7 +21,7 @@ def _sphinx_build_output(input_filename, output_filename, 
options,
     with open(os.path.join(app.srcdir, 'index.rst'), 'w') as file:
         fmt = '.. c:autodoc:: {source}\n'
         file.write(fmt.format(source=os.path.basename(input_filename)))
-        for key in options.keys():
+        for key in [k for k in options.keys() if not k.startswith('test-')]:

Maybe, for clarity, define

_TEST_DECORATE_PREFIX = 'test-'

and use that here instead. I think we both need to improve on these
kinds of things by the way.

             fmt = '   :{key}: {value}\n'
             file.write(fmt.format(key=key, value=options[key]))
 
diff --git a/test/testenv.py b/test/testenv.py
index 9ffcc4fbd8c1..a495fc360d44 100644
--- a/test/testenv.py
+++ b/test/testenv.py
@@ -83,4 +83,8 @@ def assign_test_methods(cls, get_output, get_expected):
         options = get_testcase_options(f)
         method = _test_generator(get_output, get_expected, f, **options)
 
+        # expected failure decoration

I personally dislike this sort of comment: it adds nothing to the code.
No other objections.

Regards,
Bruno

+        if options.get('test-expected-failure') is not None:
+            method = unittest.expectedFailure(method)
+
         setattr(cls, _testcase_name(f), method)

Other related posts:

  1. Home
  2. hawkmoth
  3. Archive
  4. 01-2019