kyua report — Generates reports with the results of a test suite run

Description

The kyuareport command parses a results file and generates a user-friendly, plaintext report for user
consumption on the terminal. By default, these reports only display a summary of the execution of the
full test suite to highlight where problems may lie.

The output of kyuareport can be customized to display full details on all executed test cases.
Additionally, the optional arguments to kyuareport are used to select which test programs or test cases
to display. These are filters and are described below in “Test filters”.

Reports generated by kyuareport are notintendedtobemachine-parseable.

The following subcommand options are recognized:

--outputpath
Specifies the path to which the report should be written to. The special values /dev/stdout and
/dev/stderr can be used to specify the standard output and the standard error, respectively.

--results-filepath, -spath
Specifies the results file to operate on. Defaults to ‘LATEST’, which causes kyuareport to
automatically load the latest results file from the current test suite.

The following values are accepted:

‘LATEST’
Requests the load of the latest results file available for the test suite rooted at the current
directory.

Directory
Requests the load of the latest results file available for the test suite rooted at the given
directory.

Test suite name
Requests the load of the latest results file available for the given test suite.

Results identifier
Requests the load of a specific results file.

Explicit file name (aka everything else)
Load the specified results file.

See “Results files” for more details.

--results-filtertypes
Comma-separated list of the test result types to include in the report. The ordering of the values
is respected so that you can determine how you want the list of tests to be shown.

The valid values are: ‘broken’, ‘failed’, ‘passed’, ‘skipped’ and ‘xfail’. If the parameter supplied
to the option is empty, filtering is suppressed and all result types are shown in the report.

The default value for this flag includes all the test results except the passed tests. Showing the
passed tests by default clutters the report with too much information, so only abnormal conditions
are included.

--verbose
Prints a detailed report of the execution. In addition to all the information printed by default,
verbose reports include the runtime context of the test suite run, the metadata of each test case,
and the verbatim output of the test cases.

Resultsfiles
Results files contain, as their name implies, the results of the execution of a test suite. Each test
suite executed by kyua-test(1) generates a new results file, and such results files can be loaded later
on by inspection commands such as kyua-report(1) to analyze their contents.

Results files support identifier-based lookups and also path name lookups. The differences between the
two are described below.

The default naming scheme for the results files provides simple support for identifier-based lookups and
historical recording of test suite runs. Each results file is given an identifier derived from the test
suite that generated it and the time the test suite was run. Kyua can later look up results files by
these fields.

The identifier follows this pattern:

<test_suite>.<YYYYMMDD>-<HHMMSS>-<uuuuuu>

where ‘test_suite’ is the path to the root of the test suite that was run with all slashes replaced by
underscores and ‘YYYYMMDD-HHMMSS-uuuuuu’ is a timestamp with microsecond resolution.

When using the default naming scheme, results files are stored in the ~/.kyua/store/ subdirectory and
each file holds a name of the form:

~/.kyua/store/results.<identifier>.db

Results files are simple SQLite databases with the schema described in the
/usr/share/kyua/store/schema_v?.sql files. For details on the schema, please refer to the heavily
commented SQL file.

Testfilters
A testfilter is a string that is used to match test cases or test programs in a test suite. Filters
have the following form:

test_program_name[:test_case_name]

Where ‘test_program_name’ is the name of a test program or a subdirectory in the test suite, and
‘test_case_name’ is the name of a test case.

Examples

Workflowwithresultsfiles
       If one runs the following command twice in a row:

             kyua test -k /usr/tests/Kyuafile

       the two executions will generate two different files with names like:

             ~/.kyua/store/results.usr_tests.20140731-150500-196784.db~/.kyua/store/results.usr_tests.20140731-151730-997451.db

       Taking advantage of the default naming scheme, the following commands would all generate a report for the
       results of the latest execution of the test suite:

             cd /usr/tests && kyua report
             cd /usr/tests && kyua report --results-file=LATEST
             kyua report --results-file=/usr/tests
             kyua report --results-file=usr_tests
             kyua report --results-file=usr_tests.20140731-151730-997451

       But it is also possible to explicitly load data for older runs or from explicitly-named files:

             kyua report \
                 --results-file=usr_tests.20140731-150500-196784
             kyua report \
                 --results-file=~/.kyua/store/results.usr_tests.20140731-150500-196784.db

Exit Status

       The  kyuareport command returns 0 if no filters were specified or if all filters match one or more test
       cases.  If any filter fails to match any test case, the command returns 1.

       Additional exit codes may be returned as described in kyua(1).

Name

       kyua report — Generates reports with the results of a test suite run

Synopsis

kyuareport    [--outputpath]   [--results-filefile]   [--results-filtertypes]   [--verbose]
            [test_filter1..test_filterN]