Unexpected successes were being labeled as failures, which might be
related to [1]. This behaviour is indeed welcome - we want to detect if
tests are unexpectedly succeeding via test pipeline failures, however,
the current implementation of the simpletap did not properly implement
this.
[1] https://bugs.python.org/issue20165
This allows tools like `prove` to operate on our test-suite, allowing us
to do things like run recently-failing tests first, randomising test
order, and fancier reporting options.
Changes made:
- TAP output always goes to STDOUT, never to STDERR
- Skipped tests signaled as `ok 1 # skip` as per standard
- Expected failures signaled as `not ok 1 # TODO` as per standard
- `.prove` added to .gitignore
In addition to the state of the test the description now includes the
location of the test, i.e. the filename that contains it.
Useful in central.tasktools.org.
Before they were being treated as "passing" tests.
Since this might cause them to be silently ignored, classifying them
under "skipped" will ensure they won't go unnoticed.
Expected failures != skipped. The former will always be executed.
