1TMT(1) User Commands TMT(1)
2
3
4
6 tmt - Test Management Tool
7
9 The tmt tool provides a user-friendly way to work with tests. You can
10 comfortably create new tests, safely and easily run tests across dif‐
11 ferent environments, review test results, debug test code and enable
12 tests in the CI using a consistent and concise config.
13
14 The python module and command-line tool implement the Metadata Specifi‐
15 cation which allows storing all needed test execution data directly
16 within a git repository. Together with possibility to reference remote
17 repositories it makes it easy to share test coverage across projects
18 and distros.
19
20 The Flexible Metadata Format fmf is used to store data in both human
21 and machine readable way close to the source code. Thanks to inheri‐
22 tance and elasticity metadata are organized in the structure effi‐
23 ciently, preventing unnecessary duplication.
24
26 There are several metadata levels defined by the specification:
27
28 Core attributes such as summary or description which are common across
29 all levels are defined by the special L0 metadata.
30
31 Tests, or L1 metadata, define attributes which are closely related to
32 individual test cases such as test script, framework, directory path
33 where the test should be executed, maximum test duration or packages
34 required to run the test.
35
36 Plans, also called L2 metadata, are used to group relevant tests and
37 enable them in the CI. They describe how to discover tests for execu‐
38 tion, how to provision the environment, how to prepare it for testing,
39 how to execute tests and report test results.
40
41 Stories, which implement the L3 metadata, can be used to track imple‐
42 mentation, test and documentation coverage for individual features or
43 requirements. Thanks to this you can track everything in one place, in‐
44 cluding the project implementation progress.
45
47 Command line usage is straightforward:
48
49 tmt command [options]
50
52 Let's see which tests, plans and stories are available:
53
54 tmt
55
56 Initialize the metadata tree in the current directory, optionally with
57 example content based on templates:
58
59 tmt init
60 tmt init --template base
61
62 Run all or selected steps for each plan:
63
64 tmt run
65 tmt run discover
66 tmt run prepare execute
67
68 List tests, show details, check against the specification:
69
70 tmt test ls
71 tmt test show
72 tmt test lint
73
74 Create a new test, import test metadata from other formats:
75
76 tmt test create
77 tmt test import
78
79 List plans, show details, check against the specification:
80
81 tmt plan ls
82 tmt plan show
83 tmt plan lint
84
85 List stories, check details, show coverage status:
86
87 tmt story ls
88 tmt story show
89 tmt story coverage
90
91 Many commands support regular expression filtering and other specific
92 options:
93
94 tmt story ls cli
95 tmt story show create
96 tmt story coverage --implemented
97
98 Check help message of individual commands for the full list of avail‐
99 able options.
100
102 Here is the list of the most frequently used commands and options.
103
104 Run
105 The run command is used to execute test steps. By default all test
106 steps are run. See the L2 Metadata specification for detailed descrip‐
107 tion of individual steps. Here is a brief overview:
108
109 discover
110 Gather information about test cases to be executed.
111
112 provision
113 Provision an environment for testing or use localhost.
114
115 prepare
116 Prepare the environment for testing.
117
118 execute
119 Run tests using the specified executor.
120
121 report Provide test results overview and send reports.
122
123 finish Perform the finishing tasks and clean up provisioned guests.
124
125 Test
126 Manage tests (L1 metadata). Check available tests, inspect their meta‐
127 data, gather old metadata from various sources and stored them in the
128 new fmf format.
129
130 ls List available tests.
131
132 show Show test details.
133
134 lint Check tests against the L1 metadata specification.
135
136 create Create a new test based on given template.
137
138 import Convert old test metadata into the new fmf format.
139
140 Plan
141 Manage test plans (L2 metadata). Search for available plans. Explore
142 detailed test step configuration.
143
144 ls List available plans.
145
146 show Show plan details.
147
148 lint Check plans against the L2 metadata specification.
149
150 Story
151 Manage user stories. Check available user stories. Explore coverage
152 (test, implementation, documentation).
153
154 ls List available stories.
155
156 show Show story details.
157
158 coverage
159 Show code, test and docs coverage for given stories.
160
161 export Export selected stories into desired format.
162
163 Utils
164 Various utility options.
165
166 --root PATH
167 Path to the metadata tree, current directory used by default.
168
169 --verbose
170 Print additional information.
171
172 --debug
173 Turn on debugging output.
174
175 Check help message of individual commands for the full list of avail‐
176 able options.
177
179 The main tmt package provides the core features with a minimal set of
180 dependencies:
181
182 sudo dnf install tmt
183
184 In order to enable additional functionality, such as particular provi‐
185 sion or report plugins, install the respective subpackage:
186
187 sudo dnf install tmt-test-convert
188 sudo dnf install tmt-report-html
189 sudo dnf install tmt-provision-container
190 sudo dnf install tmt-provision-virtual
191
192 If you don't care about disk space and want to have all available fea‐
193 tures right at hand install everything:
194
195 sudo dnf install tmt-all
196
197 For CentOS and RHEL, first make sure that you have available the EPEL
198 repository. You might also have to enable additional repositories:
199
200 sudo dnf config-manager --enable powertools # CentOS 8
201 sudo dnf config-manager --enable rhel-CRB # RHEL 8
202 sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
203
204 sudo dnf config-manager --enable crb # CentOS 9
205 sudo dnf config-manager --enable rhel-CRB # RHEL 9
206 sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
207
208 sudo dnf install tmt
209
210 For plugins which cannot work outside of VPN and so live within its
211 walls you need to enable the internal copr repository first. Then you
212 can install either everything or only those you need:
213
214 sudo dnf install tmt-redhat-all
215 sudo dnf install tmt-redhat-*
216
217 Impatient to try the fresh features as soon as possible? Install the
218 latest greatest version from the copr repository:
219
220 sudo dnf copr enable psss/tmt
221 sudo dnf install tmt
222
223 Not sure, just want to try out how it works? Experiment safely and eas‐
224 ily inside a container:
225
226 podman run -it --rm quay.io/testing-farm/tmt bash
227 podman run -it --rm quay.io/testing-farm/tmt-all bash
228
229 When installing using pip you might need to install additional packages
230 on your system:
231
232 sudo dnf install gcc redhat-rpm-config
233 sudo dnf install {python3,libvirt,krb5,libpq}-devel
234 pip install --user tmt
235
236 Note: You can omit the --user flag if in a virtual environment.
237
239 The rpm package includes a system wide script which enables the command
240 line completion for bash so no additional config should be needed. If
241 you use a different installation method or prefer another shell, see
242 the instructions below.
243
244 For Bash, add this to ~/.bashrc:
245
246 eval "$(_TMT_COMPLETE=source_bash tmt)"
247
248 For Zsh, add this to ~/.zshrc:
249
250 eval "$(_TMT_COMPLETE=source_zsh tmt)"
251
252 For Fish, add this to ~/.config/fish/completions/tmt.fish:
253
254 eval (env _TMT_COMPLETE=source_fish tmt)
255
256 Open a new shell to enable completion. Or run the eval command directly
257 in your current shell to enable it temporarily.
258
259 This is however run every time you start a shell which can cause some
260 delay. To speed it up, write the generated script to a file and then
261 source it from your shell's configuration file. All of this can be
262 achieved using tmt setup completion command. By default, it outputs
263 the completion script to the terminal but it can also add it to your
264 ~/.bashrc or ~/.zshrc using the --install option:
265
266 tmt setup completion {bash, zsh, fish} --install
267
269 The following exit codes are returned from tmt run. Note that you can
270 use the --quiet option to completely disable output and only check for
271 the exit code.
272
273 0 At least one test passed, there was no fail, warn or error.
274
275 1 There was a fail or warn identified, but no error.
276
277 2 Errors occured during test execution.
278
279 3 No test results found.
280
282 The following environment variables can be used to modify behaviour of
283 the tmt command.
284
285 TMT_DEBUG
286 Enable the desired debug level. Most of the commands support
287 levels from 1 to 3. However, some of the plugins go even deeper
288 when needed.
289
290 TMT_PLUGINS
291 Path to a directory with additional plugins. Multiple paths sep‐
292 arated with the : character can be provided as well.
293
294 TMT_WORKDIR_ROOT
295 Path to root directory containing run workdirs. Defaults to
296 /var/tmp/tmt.
297
298 NO_COLOR
299 Disable colors in the terminal output. Output only plain,
300 non-colored text. See https://no-color.org/ for more informa‐
301 tion.
302
303 The following environment variables are provided to the test during the
304 execution:
305
306 TMT_TREE
307 The full path of the working directory where the metadata tree
308 is copied. This usually contains the whole git repository from
309 which tests have been executed.
310
311 TMT_TEST_DATA
312 Path to the directory where test can store logs and other arti‐
313 facts generated during its execution. These will be pulled back
314 from the guest and available for inspection after the test exe‐
315 cution is finished.
316
317 TMT_PLAN_DATA
318 Path to the common directory used for storing logs and other ar‐
319 tifacts related to the whole plan execution. It is pulled back
320 from the guest and available for inspection after the plan is
321 completed.
322
323 TMT_SOURCE_DIR
324 Path to directory with downloaded and extracted sources if the
325 dist-git-source option was used in the discover step.
326
327 TMT_REBOOT_COUNT
328 During the test execution the tmt-reboot command can be used to
329 request reboot of the guest. This variable contains number of
330 reboots which already happened during the test. Value is set to
331 0 if no reboot occurred.
332
333 In order to keep backward-compatibility with older tests,
334 rhts-reboot and rstrnt-reboot commands are supported for re‐
335 questing the reboot, variables REBOOTCOUNT and RSTRNT_REBOOT‐
336 COUNT contain number of reboots as well.
337
339 Git: https://github.com/teemtee/tmt
340
341 Docs: http://tmt.readthedocs.io/
342
343 Stories: https://tmt.readthedocs.io/en/stable/stories.html
344
345 Issues: https://github.com/teemtee/tmt/issues
346
347 Releases: https://github.com/teemtee/tmt/releases
348
349 Copr: http://copr.fedoraproject.org/coprs/psss/tmt
350
351 PIP: https://pypi.org/project/tmt/
352
353 Metadata Specification: https://tmt.readthedocs.io/en/stable/spec.html
354
355 Flexible Metadata Format: http://fmf.readthedocs.io/
356
357 Packit & Testing Farm: https://packit.dev/testing-farm/
358
360 Petr Šplíchal, Miro Hrončok, Alexander Sosedkin, Lukáš Zachar, Petr
361 Menšík, Leoš Pol, Miroslav Vadkerti, Pavel Valena, Jakub Heger, Honza
362 Horák, Rachel Sibley, František Nečas, Michal Ruprich, Martin Kyral,
363 Miloš Prchlík, Tomáš Navrátil, František Lachman, Patrik Kis, Ondrej
364 Mosnáček, Andrea Ficková, Denis Karpelevich, Michal Srb, Jan Ščotka,
365 Artem Zhukov, Vinzenz Feenstra, Inessa Vasilevskaya, Štěpán Němec,
366 Robin Hack, Yulia Kopkova, Ondrej Moriš, Martin Zelený, Karel Šrot,
367 František Zatloukal, Simon Walter, Petr Matyáš, Yariv Rachmani, Pavel
368 Cahyna, Martin Litwora, Brian Grech, Vojtěch Eichler, Philip Daly, Vec‐
369 tor Li, Evgeny Fedin, Guy Inger, Adrián Tomašov, Jan Havlín and Lukáš
370 Kotek, Daniel Diblík.
371
373 Copyright Red Hat
374
375 This program is free software; you can redistribute it and/or modify it
376 under the terms of the MIT License.
377
378
379
380
381 September 2019 TMT(1)