1TMT(1) User Commands TMT(1)
2
6 tmt - Test Management Tool
7
9 The tmt tool provides a user-friendly way to work with tests. You can
10 comfortably create new tests, safely and easily run tests across dif‐
11 ferent environments, review test results, debug test code and enable
12 tests in the CI using a consistent and concise config.
13 The python module and command-line tool implement the Metadata Specifi‐
15 cation which allows storing all needed test execution data directly
16 within a git repository. Together with possibility to reference remote
17 repositories it makes it easy to share test coverage across projects
18 and distros.
19 The Flexible Metadata Format fmf is used to store data in both human
21 and machine readable way close to the source code. Thanks to inheri‐
22 tance and elasticity metadata are organized in the structure effi‐
23 ciently, preventing unnecessary duplication.
24
26 There are several metadata levels defined by the specification:
27 Core attributes such as summary or description which are common across
29 all levels are defined by the special L0 metadata.
30 Tests, or L1 metadata, define attributes which are closely related to
32 individual test cases such as test script, framework, directory path
33 where the test should be executed, maximum test duration or packages
34 required to run the test.
35 Plans, also called L2 metadata, are used to group relevant tests and
37 enable them in the CI. They describe how to discover tests for execu‐
38 tion, how to provision the environment, how to prepare it for testing,
39 how to execute tests and report test results.
40 Stories, which implement the L3 metadata, can be used to track imple‐
42 mentation, test and documentation coverage for individual features or
43 requirements. Thanks to this you can track everything in one place, in‐
44 cluding the project implementation progress.
45
47 Command line usage is straightforward:
48 tmt command [options]
50
52 Let's see which tests, plans and stories are available:
53 tmt
55 Initialize the metadata tree in the current directory, optionally with
57 example content based on templates:
58 tmt init
60 tmt init --template base
61 Run all or selected steps for each plan:
63 tmt run
65 tmt run discover
66 tmt run prepare execute
67 List tests, show details, check against the specification:
69 tmt tests ls
71 tmt tests show
72 tmt tests lint
73 Create a new test, import test metadata from other formats:
75 tmt test create
77 tmt test import
78 List plans, show details, check against the specification:
80 tmt plans ls
82 tmt plans show
83 tmt plans lint
84 List stories, check details, show coverage status:
86 tmt stories ls
88 tmt stories show
89 tmt stories coverage
90 Many commands support regular expression filtering and other specific
92 options:
93 tmt stories ls cli
95 tmt stories show create
96 tmt stories coverage --implemented
97 Check help message of individual commands for the full list of avail‐
99 able options.
100
102 Here is the list of the most frequently used commands and options.
103 Run
105 The run command is used to execute test steps. By default all test
106 steps are run. See the L2 Metadata specification for detailed descrip‐
107 tion of individual steps. Here is a brief overview:
108 discover
110 Gather information about test cases to be executed.
111 provision
113 Provision an environment for testing or use localhost.
114 prepare
116 Prepare the environment for testing.
117 execute
119 Run tests using the specified executor.
120 report Provide test results overview and send reports.
122 finish Perform the finishing tasks and clean up provisioned guests.
124 Tests
126 Manage tests (L1 metadata). Check available tests, inspect their meta‐
127 data, gather old metadata from various sources and stored them in the
128 new fmf format.
129 ls List available tests.
131 show Show test details.
133 lint Check tests against the L1 metadata specification.
135 create Create a new test based on given template.
137 import Convert old test metadata into the new fmf format.
139 Plans
141 Manage test plans (L2 metadata). Search for available plans. Explore
142 detailed test step configuration.
143 ls List available plans.
145 show Show plan details.
147 lint Check plans against the L2 metadata specification.
149 Stories
151 Manage user stories. Check available user stories. Explore coverage
152 (test, implementation, documentation).
153 ls List available stories.
155 show Show story details.
157 coverage
159 Show code, test and docs coverage for given stories.
160 export Export selected stories into desired format.
162 Utils
164 Various utility options.
165 --root PATH
167 Path to the metadata tree, current directory used by default.
168 --verbose
170 Print additional information.
171 --debug
173 Turn on debugging output.
174 Check help message of individual commands for the full list of avail‐
176 able options.
177
179 The main tmt package provides the core features with a minimal set of
180 dependencies:
181 sudo dnf install tmt
183 In order to enable additional functionality, such as particular provi‐
185 sion or report plugins, install the respective subpackage:
186 sudo dnf install tmt+test-convert
188 sudo dnf install tmt+provision-container
189 sudo dnf install tmt+provision-virtual
190 If you don't care about disk space and want to have all available fea‐
192 tures right at hand install everything:
193 sudo dnf install tmt+all
195 For CentOS and RHEL, first make sure that you have available the EPEL
197 repository. You might also have to enable additional repositories:
198 sudo dnf config-manager --enable crb # CentOS 9
200 sudo dnf config-manager --enable rhel-CRB # RHEL 9
201 sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
202 sudo dnf install tmt
204 For plugins which cannot work outside of VPN and so live within its
206 walls you need to enable the internal copr repository first. Then you
207 can install either everything or only those you need:
208 sudo dnf install tmt-redhat-all
210 sudo dnf install tmt-redhat-*
211 Impatient to try the fresh features as soon as possible? Install the
213 latest greatest version from the copr repository:
214 sudo dnf copr enable @teemtee/tmt
216 sudo dnf install tmt
217 Not sure, just want to try out how it works? Experiment safely and eas‐
219 ily inside a container:
220 podman run -it --rm quay.io/testing-farm/tmt bash
222 podman run -it --rm quay.io/testing-farm/tmt-all bash
223 When installing using pip you might need to install additional packages
225 on your system:
226 sudo dnf install gcc redhat-rpm-config
228 sudo dnf install {python3,libvirt,krb5,libpq}-devel
229 pip install --user tmt
230 On other distributions than Fedora or RHEL the package names might be
232 different. For example on Ubuntu to install all packages to have provi‐
233 sion plugins working:
234 sudo apt install libkrb5-dev pkg-config libvirt-dev genisoimage qemu-kvm libvirt-daemon-system
236 pip install --user "tmt[provision]"
237 Note: You can omit the --user flag if in a virtual environment.
239
241 The rpm package includes a system wide script which enables the command
242 line completion for bash so no additional config should be needed. If
243 you use a different installation method or prefer another shell, see
244 the instructions below.
245 For Bash, add this to ~/.bashrc:
247 eval "$(_TMT_COMPLETE=source_bash tmt)"
249 For Zsh, add this to ~/.zshrc:
251 eval "$(_TMT_COMPLETE=source_zsh tmt)"
253 For Fish, add this to ~/.config/fish/completions/tmt.fish:
255 eval (env _TMT_COMPLETE=source_fish tmt)
257 Open a new shell to enable completion. Or run the eval command directly
259 in your current shell to enable it temporarily.
260 This is however run every time you start a shell which can cause some
262 delay. To speed it up, write the generated script to a file and then
263 source it from your shell's configuration file. All of this can be
264 achieved using tmt setup completion command. By default, it outputs
265 the completion script to the terminal but it can also add it to your
266 ~/.bashrc or ~/.zshrc using the --install option:
267 tmt setup completion {bash, zsh, fish} --install
269
271 The following exit codes are returned from tmt run. Note that you can
272 use the --quiet option to completely disable output and only check for
273 the exit code.
274 0 At least one test passed, there was no fail, warn or error.
276 1 There was a fail or warn identified, but no error.
278 2 Errors occured during test execution.
280 3 No test results found.
282 4 Tests were executed, and all reported the skip result.
284
286 The list of available environment variables which can be used to adjust
287 the execution.
288 Command Variables
290 The following environment variables can be used to modify behaviour of
291 the tmt command.
292 TMT_DEBUG
294 Enable the desired debug level. Most of the commands support
295 levels from 1 to 3. However, some of the plugins go even deeper
296 when needed.
297 TMT_PLUGINS
299 Path to a directory with additional plugins. Multiple paths sep‐
300 arated with the : character can be provided as well.
301 TMT_CONFIG_DIR
303 Path to an alternative directory with config files. By default
304 ~/.config/tmt is used.
305 TMT_WORKDIR_ROOT
307 Path to root directory containing run workdirs. Defaults to
308 /var/tmp/tmt.
309 NO_COLOR, TMT_NO_COLOR
311 Disable colors in the output, both the actual output and logging
312 messages. Output only plain, non-colored text.
313 Two variables are accepted, one with the usual TMT_ prefix, but
315 tmt accepts also NO_COLOR to support the NO_COLOR effort, see
316 https://no-color.org/ for more information.
317 TMT_FORCE_COLOR
319 Enforce colors in the output, both the actual output and logging
320 messages. Might come handy when tmt's output streams are not
321 terminal-like, yet its output would be displayed by tools with
322 ANSI color support. This is often the case of various CI sys‐
323 tems.
324 Note that TMT_FORCE_COLOR takes priority over NO_COLOR and
326 TMT_NO_COLOR. If user tries both to disable and enable coloriza‐
327 tion, output would be colorized.
328 TMT_SHOW_TRACEBACK
330 By default, when tmt reports an error, the corresponding trace‐
331 back is not printed out. When TMT_SHOW_TRACEBACK is set to any
332 string except 0, traceback would be printed out.
333 TMT_OUTPUT_WIDTH
335 By default, the output width of commands like tmt * show is con‐
336 strained to 79 characters. Set this variable to an integer to
337 change the limit.
338 TMT_GIT_CREDENTIALS_URL_<suffix>, TMT_GIT_CREDENTIALS_VALUE_<suffix>
340 Variable pairs used to provide credentials to clone git reposi‐
341 tories. Suffix identifies the pair and determines the order in
342 which URL regexp is tried.
343 The TMT_GIT_CREDENTIALS_URL_<suffix> contains regexp to search
345 against url to clone. For first successful search the content of
346 the TMT_GIT_CREDENTIALS_VALUE_<suffix> variable is used as the
347 credential value. When it is set to an empty string, unmodified
348 url is used.
349 Example usage:
351 GitLab credentials need to contain nonempty username followed by
353 colon and token value:
354 TMT_GIT_CREDENTIALS_URL_lab='gitlab.com/mysecretproject'
356 TMT_GIT_CREDENTIALS_VALUE_lab='foo:secrettoken'
357 GitHub credentials contain just the token value:
359 TMT_GIT_CREDENTIALS_URL_hub='github.com/teemtee'
361 TMT_GIT_CREDENTIALS_VALUE_hub='secrettoken'
362 Step Variables
364 The following environment variables are provided to the environment
365 during prepare, execute and finish steps:
366 TMT_TREE
368 The full path of the working directory where the metadata tree
369 is copied. This usually contains the whole git repository from
370 which tests have been executed.
371 TMT_PLAN_DATA
373 Path to the common directory used for storing logs and other ar‐
374 tifacts related to the whole plan execution. It is pulled back
375 from the guest and available for inspection after the plan is
376 completed.
377 TMT_PLAN_ENVIRONMENT_FILE
379 Path to the file containing environment variables that should be
380 sourced after prepare and execute steps. These variables will be
381 accessible for all subsequent steps and have lower priority than
382 variables specified by the environment key, environment-file
383 key, or the command line. Variables inside the file have to be
384 in the format of NAME=VALUE and each variable should be on a
385 separate line.
386 Example of the file content:
388 COUNT=1
390 VARIABLE=VALUE
391 ANOTHER_VARIABLE=ANOTHER_VALUE
392 TMT_VERSION
394 The version of tmt.
395 Test Variables
397 The following environment variables are provided to the test during the
398 execution:
399 TMT_TEST_NAME
401 The test name, as a resolved FMF object name starting with /
402 from the root of the hierarchy.
403 TMT_TEST_DATA
405 Path to the directory where test can store logs and other arti‐
406 facts generated during its execution. These will be pulled back
407 from the guest and available for inspection after the test exe‐
408 cution is finished.
409 TMT_TEST_SERIAL_NUMBER
411 The serial number of running test in the whole plan. Each test
412 is assigned its own serial number.
413 TMT_TEST_METADATA
415 Path to a YAML-formatted file with test metadata collected dur‐
416 ing the discover step.
417 TMT_SOURCE_DIR
419 Path to directory with downloaded and extracted sources if the
420 dist-git-source option was used in the discover step.
421 TMT_REBOOT_COUNT
423 During the test execution the tmt-reboot command can be used to
424 request reboot of the guest. This variable contains number of
425 reboots which already happened during the test. Value is set to
426 0 if no reboot occurred.
427 In order to keep backward-compatibility with older tests,
429 rhts-reboot and rstrnt-reboot commands are supported for re‐
430 questing the reboot, variables REBOOTCOUNT and RSTRNT_REBOOT‐
431 COUNT contain number of reboots as well.
432 TMT_TOPOLOGY_BASH, TMT_TOPOLOGY_YAML
434 Paths of files describing existing guests, their roles and the
435 guest on which the test is running. Format of these files is de‐
436 scribed in the Guest Topology Format section of the plan speci‐
437 fication.
438 TMT_TEST_PIDFILE, TMT_TEST_PIDFILE_LOCK
440 Path to a file storing the test process pid and path to its re‐
441 boot-request file, separated by a space. The "LOCK" variable
442 then holds path to a locking file which must be acquired before
443 making any changes to the pid file.
444 TMT_TEST_PIDFILE_ROOT
446 By default, the test pidfile file is stored in /var/tmp direc‐
447 tory. If specified, the directory in this variable would be used
448 instead. The directory permissions should follow the pattern of
449 temporary directory permissions, e.g. chmod 1777, to allow ac‐
450 cess to users with all privilege levels.
451 Plugin Variables
453 Each plugin option can be also specified via environment variable.
454 Variables follow a naming scheme utilizing plugin name, step it belongs
455 to, and the option name:
456 TMT_PLUGIN_${STEP}_${PLUGIN}_${OPTION}
457 All values are upper-cased, with dashes (-) replaced by underscores
459 (_).
460 For example, an execute plugin "tmt" would run with verbosity equal to
462 -vvv:
463 TMT_PLUGIN_EXECUTE_TMT_VERBOSE=3 tmt run ... execute -h tmt ...
465 Command-line takes precedence over environment variables, therefore -v
467 would undo the effect of environment variable, and reduce verbosity to
468 one level only:
469 TMT_PLUGIN_EXECUTE_TMT_VERBOSE=3 tmt run ... execute -h tmt -v ...
471 Environment variables - just like command-line options - take prece‐
473 dence over values stored in files. For example, consider the following
474 discover step:
475 discover:
477 how: fmf
478 url: https://example.org/
479 The following commands would override the URL:
481 tmt run ... discover -h fmf --url https://actual.org/ ...
483 TMT_PLUGIN_DISCOVER_FMF_URL=https://actual.org/ tmt run ...
485 For setting flag-like option, 0 and 1 are the expected value. For exam‐
487 ple, an interactive mode would be enabled in this run:
488 TMT_PLUGIN_EXECUTE_TMT_INTERACTIVE=1 tmt run ... execute -h tmt ...
490 NOTE:
492 The following applies to situations when a plugin is specified on
493 the command line only. Keys of plugins specified in fmf files would
494 not be modified. This is a limit of the current implementation, and
495 will be addressed in the future:
496 # Here the verbosity will not be increased since the plugin is
498 # not mentioned on the command line:
499 $ TMT_PLUGIN_DISCOVER_FMF_VERBOSE=2 tmt run -a
500 # Here the environment variable will take effect:
502 $ TMT_PLUGIN_DISCOVER_FMF_VERBOSE=2 tmt run -a discover -h fmf ...
503
505 Git: https://github.com/teemtee/tmt
506 Docs: http://tmt.readthedocs.io/
508 Stories: https://tmt.readthedocs.io/en/stable/stories.html
510 Issues: https://github.com/teemtee/tmt/issues
512 Releases: https://tmt.readthedocs.io/en/stable/releases.html
514 Copr: https://copr.fedorainfracloud.org/coprs/g/teemtee/tmt/
516 PIP: https://pypi.org/project/tmt/
518 Metadata Specification: https://tmt.readthedocs.io/en/stable/spec.html
520 Flexible Metadata Format: http://fmf.readthedocs.io/
522 Testing Farm: https://docs.testing-farm.io/
524 Packit: https://packit.dev/testing-farm/
526
528 Petr Šplíchal, Miro Hrončok, Alexander Sosedkin, Lukáš Zachar, Petr
529 Menšík, Leoš Pol, Miroslav Vadkerti, Pavel Valena, Jakub Heger, Honza
530 Horák, Rachel Sibley, František Nečas, Michal Ruprich, Martin Kyral,
531 Miloš Prchlík, Tomáš Navrátil, František Lachman, Patrik Kis, Ondrej
532 Mosnáček, Andrea Ficková, Denis Karpelevich, Michal Srb, Jan Ščotka,
533 Artem Zhukov, Vinzenz Feenstra, Inessa Vasilevskaya, Štěpán Němec,
534 Robin Hack, Yulia Kopkova, Ondrej Moriš, Martin Zelený, Karel Šrot,
535 František Zatloukal, Simon Walter, Petr Matyáš, Yariv Rachmani, Pavel
536 Cahyna, Martin Litwora, Brian Grech, Vojtěch Eichler, Philip Daly, Vec‐
537 tor Li, Evgeny Fedin, Guy Inger, Adrián Tomašov, Jan Havlín, Lukáš
538 Kotek, Daniel Diblík, Laura Barcziova, Marián Konček, Marcin Sobczyk,
539 Jiří Jabůrek, Huijing Hei, Tibor Dudlák, Jan Macků, Filip Vágner, Mar‐
540 tin Hoyer, Iveta Česalová, Yi Zhang, Zhaojuan Guo, Natália Bubáková,
541 Michal Josef Špaček, Jiří Popelka, Matej Focko, Yulia Kopkova, Tomáš
542 Bajer, Carlos Rodriguez-Fernandez and James Molet.
543
545 Copyright Red Hat
546 This program is free software; you can redistribute it and/or modify it
548 under the terms of the MIT License.
549 September 2019 TMT(1)