Implements detection of unexpected successes and expected failures. Both
classes are represented in the TAP output as 'not ok', unexpected
successes with '# FIXED' metadata and expected failures as '# TODO'.
This brings C++ tests to feature parity with Python-based ones when it
comes to expected failures and unexpected successes.
Unexpected successes were being labeled as failures, which might be
related to [1]. This behaviour is indeed welcome - we want to detect if
tests are unexpectedly succeeding via test pipeline failures, however,
the current implementation of the simpletap did not properly implement
this.
[1] https://bugs.python.org/issue20165
This adds a note to the man page to properly escape filters containing
spaces. It also fixes the unittests to reflect this.
One of the unittests contained an alternative syntax as discussed in
TW-1479 (#1505). It has been extracted into its own unittest and marked
as an expected failure because it is currently not supported.
I verified locally that this avoids skipping all of the
faketime-dependent unittests. The Travis CI logs will have to be
investigated for all of the other docker images on which distros we may
need to apply a similar fix.
This is an attempt at updating all docker configurations by simply
replacing the `python` packages with `python3`. The Travis CI will let
us know if this works.
This change was not included in the previous commit because the sed
command I used included the `/usr/bin/env` path to ensure I don't
replace any occurences of the word `python` which I did not wish to
replace.
This commit updates all tests to enforce the Python3 executable. This is
necessary because the `assertRegex` function we use was renamed to this
name only in Python 3.2 [1]
For reference:
s;/usr/bin/env python;/usr/bin/env python3;g
[1]: https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertRegex
$ task 0000000d-000a-0000-0000-000000000000 export
Cannot subtract strings
$ task 0000000d-0000-0000-0000-000000000000 export
The expression could not be evaluated.
The test was failing because it expected "Created task 1"
but it saw "Created task 0a436fd8-7f06-4168-a66d-bbc940498af9".
Instead, the new code looks at the description of task 1.
The test was failing on my machine on fresh checkout. It failed because
somehow both new-id and new-uuid verbosity were set during the test, so
TW defaulted to printing out uuid, which did not match the assertion.
Test fixed by explicitly setting verbosity to new-id.
