1 .. SPDX-License-Identifier: GPL-2.0
3 ============================
4 Tips For Running KUnit Tests
5 ============================
7 Using ``kunit.py run`` ("kunit tool")
8 =====================================
10 Running from any directory
11 --------------------------
13 It can be handy to create a bash function like:
17 function run_kunit() {
18 ( cd "$(git rev-parse --show-toplevel)" && ./tools/testing/kunit/kunit.py run $@ )
22 Early versions of ``kunit.py`` (before 5.6) didn't work unless run from
23 the kernel root, hence the use of a subshell and ``cd``.
25 Running a subset of tests
26 -------------------------
28 ``kunit.py run`` accepts an optional glob argument to filter tests. Currently
29 this only matches against suite names, but this may change in the future.
31 Say that we wanted to run the sysctl tests, we could do so via:
35 $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' > .kunit/.kunitconfig
36 $ ./tools/testing/kunit/kunit.py run 'sysctl*'
38 We're paying the cost of building more tests than we need this way, but it's
39 easier than fiddling with ``.kunitconfig`` files or commenting out
42 However, if we wanted to define a set of tests in a less ad hoc way, the next
45 Defining a set of tests
46 -----------------------
48 ``kunit.py run`` (along with ``build``, and ``config``) supports a
49 ``--kunitconfig`` flag. So if you have a set of tests that you want to run on a
50 regular basis (especially if they have other dependencies), you can create a
51 specific ``.kunitconfig`` for them.
53 E.g. kunit has one for its tests:
57 $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit/.kunitconfig
59 Alternatively, if you're following the convention of naming your
60 file ``.kunitconfig``, you can just pass in the dir, e.g.
64 $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit
67 This is a relatively new feature (5.12+) so we don't have any
68 conventions yet about on what files should be checked in versus just
69 kept around locally. It's up to you and your maintainer to decide if a
70 config is useful enough to submit (and therefore have to maintain).
73 Having ``.kunitconfig`` fragments in a parent and child directory is
74 iffy. There's discussion about adding an "import" statement in these
75 files to make it possible to have a top-level config run tests from all
76 child directories. But that would mean ``.kunitconfig`` files are no
77 longer just simple .config fragments.
79 One alternative would be to have kunit tool recursively combine configs
80 automagically, but tests could theoretically depend on incompatible
81 options, so handling that would be tricky.
83 Generating code coverage reports under UML
84 ------------------------------------------
87 TODO(brendanhiggins@google.com): There are various issues with UML and
88 versions of gcc 7 and up. You're likely to run into missing ``.gcda``
89 files or compile errors. We know one `faulty GCC commit
90 <https://github.com/gcc-mirror/gcc/commit/8c9434c2f9358b8b8bad2c1990edf10a21645f9d>`_
91 but not how we'd go about getting this fixed. The compile errors still
92 need some investigation.
95 TODO(brendanhiggins@google.com): for recent versions of Linux
96 (5.10-5.12, maybe earlier), there's a bug with gcov counters not being
97 flushed in UML. This translates to very low (<1%) reported coverage. This is
98 related to the above issue and can be worked around by replacing the
99 one call to ``uml_abort()`` (it's in ``os_dump_core()``) with a plain
103 This is different from the "normal" way of getting coverage information that is
104 documented in Documentation/dev-tools/gcov.rst.
106 Instead of enabling ``CONFIG_GCOV_KERNEL=y``, we can set these options:
110 CONFIG_DEBUG_KERNEL=y
115 Putting it together into a copy-pastable sequence of commands:
119 # Append coverage options to the current config
120 $ echo -e "CONFIG_DEBUG_KERNEL=y\nCONFIG_DEBUG_INFO=y\nCONFIG_GCOV=y" >> .kunit/.kunitconfig
121 $ ./tools/testing/kunit/kunit.py run
122 # Extract the coverage information from the build dir (.kunit/)
123 $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/
125 # From here on, it's the same process as with CONFIG_GCOV_KERNEL=y
126 # E.g. can generate an HTML report in a tmp dir like so:
127 $ genhtml -o /tmp/coverage_html coverage.info
130 If your installed version of gcc doesn't work, you can tweak the steps:
134 $ ./tools/testing/kunit/kunit.py run --make_options=CC=/usr/bin/gcc-6
135 $ lcov -t "my_kunit_tests" -o coverage.info -c -d .kunit/ --gcov-tool=/usr/bin/gcov-6
138 Running tests manually
139 ======================
141 Running tests without using ``kunit.py run`` is also an important use case.
142 Currently it's your only option if you want to test on architectures other than
145 As running the tests under UML is fairly straightforward (configure and compile
146 the kernel, run the ``./linux`` binary), this section will focus on testing
147 non-UML architectures.
150 Running built-in tests
151 ----------------------
153 When setting tests to ``=y``, the tests will run as part of boot and print
154 results to dmesg in TAP format. So you just need to add your tests to your
155 ``.config``, build and boot your kernel as normal.
157 So if we compiled our kernel with:
162 CONFIG_KUNIT_EXAMPLE_TEST=y
164 Then we'd see output like this in dmesg signaling the test ran and passed:
172 # example_simple_test: initializing
173 ok 1 - example_simple_test
176 Running tests as modules
177 ------------------------
179 Depending on the tests, you can build them as loadable modules.
181 For example, we'd change the config options from before to
186 CONFIG_KUNIT_EXAMPLE_TEST=m
188 Then after booting into our kernel, we can run the test via
192 $ modprobe kunit-example-test
194 This will then cause it to print TAP output to stdout.
197 The ``modprobe`` will *not* have a non-zero exit code if any test
198 failed (as of 5.13). But ``kunit.py parse`` would, see below.
201 You can set ``CONFIG_KUNIT=m`` as well, however, some features will not
202 work and thus some tests might break. Ideally tests would specify they
203 depend on ``KUNIT=y`` in their ``Kconfig``'s, but this is an edge case
204 most test authors won't think about.
205 As of 5.13, the only difference is that ``current->kunit_test`` will
208 Pretty-printing results
209 -----------------------
211 You can use ``kunit.py parse`` to parse dmesg for test output and print out
212 results in the same familiar format that ``kunit.py run`` does.
216 $ ./tools/testing/kunit/kunit.py parse /var/log/dmesg
219 Retrieving per suite results
220 ----------------------------
222 Regardless of how you're running your tests, you can enable
223 ``CONFIG_KUNIT_DEBUGFS`` to expose per-suite TAP-formatted results:
228 CONFIG_KUNIT_EXAMPLE_TEST=m
229 CONFIG_KUNIT_DEBUGFS=y
231 The results for each suite will be exposed under
232 ``/sys/kernel/debug/kunit/<suite>/results``.
233 So using our example config:
237 $ modprobe kunit-example-test > /dev/null
238 $ cat /sys/kernel/debug/kunit/example/results
241 # After removing the module, the corresponding files will go away
242 $ modprobe -r kunit-example-test
243 $ cat /sys/kernel/debug/kunit/example/results
244 /sys/kernel/debug/kunit/example/results: No such file or directory
246 Generating code coverage reports
247 --------------------------------
249 See Documentation/dev-tools/gcov.rst for details on how to do this.
251 The only vaguely KUnit-specific advice here is that you probably want to build
252 your tests as modules. That way you can isolate the coverage from tests from
253 other code executed during boot, e.g.
257 # Reset coverage counters before running the test.
258 $ echo 0 > /sys/kernel/debug/gcov/reset
259 $ modprobe kunit-example-test