1AVOCADO(1) AVOCADO(1)
2
6 avocado - test runner command line tool
7
9 avocado [-h] [-v] [--config [CONFIG_FILE]] [--show [STREAM[:LVL]]] [-s]
10 {config,diff,distro,exec-path,list,multiplex,plugins,run,sys‐
11 info} ...
12
14 Avocado is a modern test framework that is built on the experience ac‐
15 cumulated with autotest (https://autotest.github.io).
16 avocado is also the name of its test runner command line tool, de‐
18 scribed in this man page.
19 For more information about the Avocado project, please check its web‐
21 site: https://avocado-framework.github.io/
22
24 The following list of options are builtin, application level avocado
25 options. Most other options are implemented via plugins and will depend
26 on them being loaded (avocado --help):
27 -h, --help show this help message and exit
29 -v, --version show program's version number and exit
30 --config [CONFIG_FILE]
31 Use custom configuration from a file
32 --enable-paginator Turn the paginator on. Useful when output is too long.
33 -V, --verbose Some commands can produce more information. This
34 option will enable the verbosity when applicable.
35 --show CORE.SHOW List of comma separated builtin logs, or logging
36 streams optionally followed by LEVEL (DEBUG,INFO,...).
37 Builtin streams are: "app": application output;
38 "test": test output; "debug": tracebacks and other
39 debugging info; "early": early logging of other
40 streams, including test (very verbose); "all": all
41 builtin streams; "none": disables regular output
42 (leaving only errors enabled). By default: 'app'
43 Real use of avocado depends on running avocado subcommands. This a typ‐
45 ical list of avocado subcommands:
46 assets Manage assets
48 config Shows avocado config keys
49 diff Shows the difference between 2 jobs.
50 distro Shows detected Linux distribution
51 exec-path Returns path to avocado bash libraries and exits.
52 jobs Manage Avocado jobs
53 list List available tests
54 plugins Displays plugin information
55 replay Runs a new job using a previous job as its
56 configuration
57 run Runs one or more tests (native test, test alias,
58 binary or script)
59 sysinfo Collect system information
60 variants Tool to analyze and visualize test variants and params
61 vmimage Provides VM images acquired from official repositories
62 To get usage instructions for a given subcommand, run it with --help.
64 Example:
65 $ avocado run --help
67 Options for subcommand run (avocado run --help):
69 positional arguments:
71 TEST_REFERENCE List of test references (aliases or paths)
72 optional arguments:
74 -h, --help show this help message and exit
75 -p NAME_VALUE, --test-parameter NAME_VALUE
76 Parameter name and value to pass to all tests. This is
77 only applicable when not using a varianter plugin.
78 This option format must be given in the NAME=VALUE
79 format, and may be given any number of times, or per
80 parameter.
81 --test-runner TEST_RUNNER
82 Selects the runner implementation from one of the
83 installed and active implementations. You can run
84 "avocado plugins" and find the list of valid runners
85 under the "Plugins that run test suites on a job
86 (runners) section. Defaults to "nrunner", which is the
87 new runner. To use the conventional and traditional
88 runner, use "runner".
89 -d, --dry-run Instead of running the test only list them and log
90 their params.
91 --dry-run-no-cleanup Do not automatically clean up temporary directories
92 used by dry-run
93 --force-job-id UNIQUE_JOB_ID
94 Forces the use of a particular job ID. Used internally
95 when interacting with an avocado server. You should
96 not use this option unless you know exactly what
97 you're doing
98 --job-results-dir DIRECTORY
99 Forces to use of an alternate job results directory.
100 --job-category CATEGORY
101 Categorizes this within a directory with the same
102 name, by creating a link to the job result directory
103 --job-timeout SECONDS
104 Set the maximum amount of time (in SECONDS) that tests
105 are allowed to execute. Values <= zero means "no
106 timeout". You can also use suffixes, like: s
107 (seconds), m (minutes), h (hours).
108 --failfast Enable the job interruption on first failed test.
109 --keep-tmp Keep job temporary files (useful for avocado
110 debugging).
111 --ignore-missing-references
112 Force the job execution, even if some of the test
113 references are not resolved to tests. "on" and "off"
114 will be deprecated soon.
115 --disable-sysinfo Enable or disable sysinfo information. Like hardware
116 details, profiles, etc.
117 --execution-order RUN.EXECUTION_ORDER
118 Defines the order of iterating through test suite and
119 test variants
120 --log-test-data-directories
121 Logs the possible data directories for each test. This
122 is helpful when writing new tests and not being sure
123 where to put data files. Look for "Test data
124 directories" in your test log
125 --journal Records test status changes (for use with avocado-
126 journal-replay and avocado-server)
127 --json FILE Enable JSON result format and write it to FILE. Use
128 "-" to redirect to the standard output.
129 --disable-json-job-result
130 Enables default JSON result in the job results
131 directory. File will be named "results.json".
132 --tap FILE Enable TAP result output and write it to FILE. Use "-"
133 to redirect to standard output.
134 --disable-tap-job-result
135 Enables default TAP result in the job results
136 directory. File will be named "results.tap"
137 --tap-include-logs Include test logs as comments in TAP output.
138 -z, --archive Archive (ZIP) files generated by tests
139 output and result format:
141 --store-logging-stream LOGGING_STREAM
142 Store given logging STREAMs in
143 "$JOB_RESULTS_DIR/$STREAM.$LEVEL."
144 --xunit FILE Enable xUnit result format and write it to FILE. Use
145 "-" to redirect to the standard output.
146 --disable-xunit-job-result
147 Enables default xUnit result in the job results
148 directory. File will be named "results.xml".
149 --xunit-job-name XUNIT_JOB_NAME
150 Override the reported job name. By default uses the
151 Avocado job name which is always unique. This is
152 useful for reporting in Jenkins as it only evaluates
153 first-failure from jobs of the same name.
154 --xunit-max-test-log-chars SIZE
155 Limit the attached job log to given number of
156 characters (k/m/g suffix allowed)
157 output check arguments:
159 --output-check-record {none,stdout,stderr,both,combined,all}
160 Record the output produced by each test (from stdout
161 and stderr) into both the current executing result and
162 into reference files. Reference files are used on
163 subsequent runs to determine if the test produced the
164 expected output or not, and the current executing
165 result is used to check against a previously recorded
166 reference file. Valid values: "none" (to explicitly
167 disable all recording) "stdout" (to record standard
168 output *only*), "stderr" (to record standard error
169 *only*), "both" (to record standard output and error
170 in separate files), "combined" (for standard output
171 and error in a single file). "all" is also a valid but
172 deprecated option that is a synonym of "both".
173 --disable-output-check
174 Disables test output (stdout/stderr) check. If this
175 option is given, no output will be checked, even if
176 there are reference files present for the test.
177 loader options:
179 --loaders LOADER_NAME_OR_TEST_TYPE [LOADER_NAME_OR_TEST_TYPE ...]
180 Overrides the priority of the test loaders. You can
181 specify either @loader_name or TEST_TYPE. By default
182 it tries all available loaders according to priority
183 set in settings->plugins.loaders.
184 --external-runner EXTERNAL_RUNNER
185 Path to an specific test runner that allows the use of
186 its own tests. This should be used for running tests
187 that do not conform to Avocado's SIMPLE test interface
188 and can not run standalone. Note: the use of
189 --external-runner overwrites the --loaders to
190 'external_runner'
191 --external-runner-chdir {runner,test}
192 Change directory before executing tests. This option
193 may be necessary because of requirements and/or
194 limitations of the external test runner. If the
195 external runner requires to be run from its own base
196 directory, use 'runner' here. If the external runner
197 runs tests based on files and requires to be run from
198 the directory where those files are located, use
199 'test' here and specify the test directory with the
200 option '--external-runner-testdir'.
201 --external-runner-testdir DIRECTORY
202 Where test files understood by the external test
203 runner are located in the filesystem. Obviously this
204 assumes and only applies to external test runners that
205 run tests from files
206 filtering parameters:
208 -t TAGS, --filter-by-tags TAGS
209 Filter tests based on tags
210 --filter-by-tags-include-empty
211 Include all tests without tags during filtering. This
212 effectively means they will be kept in the test suite
213 found previously to filtering.
214 --filter-by-tags-include-empty-key
215 Include all tests that do not have a matching key in
216 its key:val tags. This effectively means those tests
217 will be kept in the test suite found previously to
218 filtering.
219 JSON serialized based varianter options:
221 --json-variants-load FILE
222 Load the Variants from a JSON serialized file
223 nrunner specific options:
225 --nrunner-shuffle Shuffle the tasks to be executed
226 --nrunner-status-server-listen HOST_PORT
227 URI for listing the status server. Usually a
228 "HOST:PORT" string
229 --nrunner-status-server-uri HOST_PORT
230 URI for connecting to the status server, usually a
231 "HOST:PORT" string. Use this if your status server is
232 in another host, or different port
233 --nrunner-max-parallel-tasks NUMBER_OF_TASKS
234 Number of maximum number tasks running in parallel.
235 You can disable parallel execution by setting this to
236 1. Defaults to the amount of CPUs on this machine.
237 --nrunner-spawner SPAWNER
238 Spawn tasks in a specific spawner. Available spawners:
239 'process' and 'podman'
240 podman spawner specific options:
242 --spawner-podman-bin PODMAN_BIN
243 Path to the podman binary
244 --spawner-podman-image CONTAINER_IMAGE
245 Image name to use when creating the container
246 job replay:
248 --replay JOB_ID Replay a job identified by its (partial) hash id. Use
249 "--replay" latest to replay the latest job.
250 --replay-test-status TEST_STATUS
251 Filter tests to replay by test status.
252 --replay-ignore IGNORE
253 Ignore variants and/or configuration from the source
254 job.
255 --replay-resume Resume an interrupted job
256 wrapper support:
258 --wrapper SCRIPT[:EXECUTABLE]
259 Use a script to wrap executables run by a test. The
260 wrapper is either a path to a script (AKA a global
261 wrapper) or a path to a script followed by colon
262 symbol (:), plus a shell like glob to the target
263 EXECUTABLE. Multiple wrapper options are allowed, but
264 only one global wrapper can be defined.
265 Options for subcommand assets (avocado assets --help):
267 positional arguments:
269 {fetch,register,purge,list}
270 fetch Fetch assets from test source or config file if it's
271 not already in the cache
272 register Register an asset directly to the cacche
273 purge Removes assets cached locally.
274 list List all cached assets.
275 optional arguments:
277 -h, --help show this help message and exit
278 Options for subcommand config (avocado config --help):
280 positional arguments:
282 sub-command
283 reference Show a configuration reference with all registered options
284 optional arguments:
286 -h, --help show this help message and exit
287 --datadir Shows the data directories currently being used by Avocado
288 Options for subcommand diff (avocado diff --help):
290 positional arguments:
292 JOB A job reference, identified by a (partial) unique ID
293 (SHA1) or test results directory.
294 optional arguments:
296 -h, --help show this help message and exit
297 --html FILE Enable HTML output to the FILE where the result should
298 be written.
299 --open-browser Generate and open a HTML report in your preferred
300 browser. If no --html file is provided, create a
301 temporary file.
302 --diff-filter DIFF_FILTER
303 Comma separated filter of diff sections:
304 (no)cmdline,(no)time,(no)variants,(no)results,
305 (no)config,(no)sysinfo (defaults to all enabled).
306 --diff-strip-id Strip the "id" from "id-name;variant" when comparing
307 test results.
308 --create-reports Create temporary files with job reports to be used by
309 other diff tools
310 By default, a textual diff report is generated in the standard output.
312 Options for subcommand distro (avocado distro --help):
314 optional arguments:
316 -h, --help show this help message and exit
317 --distro-def-create Creates a distro definition file based on the path
318 given.
319 --distro-def-name DISTRO_DEF_NAME
320 Distribution short name
321 --distro-def-version DISTRO_DEF_VERSION
322 Distribution major version name
323 --distro-def-release DISTRO_DEF_RELEASE
324 Distribution release version number
325 --distro-def-arch DISTRO_DEF_ARCH
326 Primary architecture that the distro targets
327 --distro-def-path DISTRO.DISTRO_DEF_PATH
328 Top level directory of the distro installation files
329 --distro-def-type {rpm,deb}
330 Distro type (one of: rpm, deb)
331 Options for subcommand exec-path (avocado exec-path --help):
333 optional arguments:
335 -h, --help show this help message and exit
336 Options for subcommand jobs (avocado jobs --help):
338 positional arguments:
340 sub-command
341 list List all known jobs by Avocado
342 show Show details about a specific job. When passing a Job ID,
343 you can use any Job Reference (job_id, "latest", or job
344 results path).
345 get-output-files
346 Download output files generated by tests on
347 AVOCADO_TEST_OUTPUTDIR
348 optional arguments:
350 -h, --help show this help message and exit
351 Options for subcommand list (avocado list --help):
353 positional arguments:
355 references List of test references (aliases or paths). If empty,
356 Avocado will list tests on the configured test source,
357 (see "avocado config --datadir") Also, if there are
358 other test loader plugins active, tests from those
359 plugins might also show up (behavior may vary among
360 plugins)
361 optional arguments:
363 -h, --help show this help message and exit
364 --resolver Uses the Avocado resolver method (part of the nrunner
365 architecture) to detect tests. This is enabled by
366 default and exists only for compatibility purposes,
367 and will be removed soon. To use the legacy (loader)
368 method for finding tests, set the "--loader" option
369 --loader Uses the Avocado legacy (loader) method for finding
370 tests. This option will exist only for a transitional
371 period until the legacy (loader) method is deprecated
372 and removed
373 --write-recipes-to-directory DIRECTORY
374 Writes runnable recipe files to a directory. Valid
375 only when using --resolver.
376 --json JSON_FILE Writes output to a json file.
377 loader options:
379 --loaders LOADER_NAME_OR_TEST_TYPE [LOADER_NAME_OR_TEST_TYPE ...]
380 Overrides the priority of the test loaders. You can
381 specify either @loader_name or TEST_TYPE. By default
382 it tries all available loaders according to priority
383 set in settings->plugins.loaders.
384 --external-runner EXTERNAL_RUNNER
385 Path to an specific test runner that allows the use of
386 its own tests. This should be used for running tests
387 that do not conform to Avocado's SIMPLE test interface
388 and can not run standalone. Note: the use of
389 --external-runner overwrites the --loaders to
390 'external_runner'
391 --external-runner-chdir {runner,test}
392 Change directory before executing tests. This option
393 may be necessary because of requirements and/or
394 limitations of the external test runner. If the
395 external runner requires to be run from its own base
396 directory, use 'runner' here. If the external runner
397 runs tests based on files and requires to be run from
398 the directory where those files are located, use
399 'test' here and specify the test directory with the
400 option '--external-runner-testdir'.
401 --external-runner-testdir DIRECTORY
402 Where test files understood by the external test
403 runner are located in the filesystem. Obviously this
404 assumes and only applies to external test runners that
405 run tests from files
406 filtering parameters:
408 -t TAGS, --filter-by-tags TAGS
409 Filter tests based on tags
410 --filter-by-tags-include-empty
411 Include all tests without tags during filtering. This
412 effectively means they will be kept in the test suite
413 found previously to filtering.
414 --filter-by-tags-include-empty-key
415 Include all tests that do not have a matching key in
416 its key:val tags. This effectively means those tests
417 will be kept in the test suite found previously to
418 filtering.
419 Options for subcommand plugins (avocado plugins --help):
421 optional arguments:
423 -h, --help show this help message and exit
424 Options for subcommand replay (avocado reply --help):
426 positional arguments:
428 SOURCE_JOB_ID Replays a job, identified by: complete or partial Job ID,
429 "latest" for the latest job, the job results path.
430 optional arguments:
432 -h, --help show this help message and exit
433 Options for subcommand sysinfo (avocado sysinfo --help):
435 positional arguments:
437 sysinfodir Directory where Avocado will dump sysinfo data. If one is not
438 given explicitly, it will default to a directory named
439 "sysinfo-" followed by a timestamp in the current working
440 directory.
441 optional arguments:
443 -h, --help show this help message and exit
444 Options for subcommand variants (avocado variants --help):
446 optional arguments:
448 -h, --help show this help message and exit
449 --summary SUMMARY Verbosity of the variants summary. (positive integer -
450 0, 1, ... - or none, brief, normal, verbose, full,
451 max)
452 --variants VARIANTS Verbosity of the list of variants. (positive integer -
453 0, 1, ... - or none, brief, normal, verbose, full,
454 max)
455 -c, --contents [obsoleted by --variants] Shows the node content
456 (variables)
457 --json-variants-dump FILE
458 Dump the Variants to a JSON serialized file
459 environment view options:
461 -d, --debug Use debug implementation to gather more information.
462 tree view options:
464 -t, --tree [obsoleted by --summary] Shows the multiplex tree
465 structure
466 -i, --inherit [obsoleted by --summary] Show the inherited values
467 JSON serialized based varianter options:
469 --json-variants-load FILE
470 Load the Variants from a JSON serialized file
471 Options for subcommand vmimage (avocado vmimage --help):
473 positional arguments:
475 {list,get}
476 list List of all downloaded images
477 get Downloads chosen VMimage if it's not already in the cache
478 optional arguments:
480 -h, --help show this help message and exit
481
483 The most common use of the avocado command line tool is to run a test:
484 $ avocado run examples/tests/sleeptest.py
486 This command will run the sleeptest.py test, as found on the standard
488 test directories. The output should be similar to:
489 JOB ID : <id>
491 JOB LOG : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
492 (1/1) sleeptest.py:SleepTest.test: PASS (1.01 s)
493 RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
494 JOB TIME : 1.11 s
495 The test directories will vary depending on you system and installation
497 method used. Still, it's pretty easy to find that out as shown in the
498 next section.
499
501 When you are developing new tests, frequently you want to look at the
502 straight output of the job log in the stdout, without having to tail
503 the job log. In order to do that, you can use --show=test to the avo‐
504 cado test runner:
505 $ avocado --show=test run examples/tests/sleeptest.py
507 ...
508 PARAMS (key=timeout, path=*, default=None) => None
509 Test metadata:
510 filename: /home/user/avocado/examples/tests/sleeptest.py
511 teststmpdir: /var/tmp/avocado_o98elmi0
512 workdir: /var/tmp/avocado_iyzcj3hn/avocado_job_mwikfsnl/1-examples_tests_sleeptest.py_SleepTest.test
513 START 1-examples/tests/sleeptest.py:SleepTest.test
514 DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
515 PARAMS (key=sleep_length, path=*, default=1) => 1
516 Sleeping for 1.00 seconds
517 DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
518 DATA (filename=stdout.expected) => NOT FOUND (data sources: variant, test, file)
519 DATA (filename=stderr.expected) => NOT FOUND (data sources: variant, test, file)
520 PASS 1-examples/tests/sleeptest.py:SleepTest.test
521 ...
522 Let's say you are debugging a test particularly large, with lots of de‐
524 bug output and you want to reduce this output to only messages with
525 level 'INFO' and higher. You can set job-log-level to info to reduce
526 the amount of output.
527 Edit your ~/.config/avocado/avocado.conf file and add:
529 [job.output]
531 loglevel = INFO
532 Running the same example with this option will give you:
534 $ avocado --show=test run examples/tests/sleeptest.py
536 ...
537 START 1-examples/tests/sleeptest.py:SleepTest.test
538 PASS 1-examples/tests/sleeptest.py:SleepTest.test
539 ...
540 The levels you can choose are the levels available in the python log‐
542 ging system https://docs.python.org/3/library/logging.html#logging-lev‐
543 els, so 'NOTSET', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL', in
544 order of severity.
545 As you can see, the UI output is suppressed and only the job log goes
547 to stdout, making this a useful feature for test development/debugging.
548
550 You may specify --show=none, that means avocado will turn off all run‐
551 ner stdout. Note that --show=none does not affect on disk job logs,
552 those continue to be generated normally.
553
555 You may specify --disable-sysinfo and avocado will not collect profil‐
556 ers, hardware details and other system information, inside the job re‐
557 sult directory.
558
560 The avocado command line tool also has a list command, that lists the
561 known tests in a given path, be it a path to an individual test, or a
562 path to a directory. If no arguments provided, avocado will inspect the
563 contents of the test location being used by avocado (if you are in
564 doubt about which one is that, you may use avocado config --datadir).
565 The output looks like:
566 $ avocado list
568 INSTRUMENTED /usr/share/doc/avocado/tests/abort.py
569 INSTRUMENTED /usr/share/doc/avocado/tests/datadir.py
570 INSTRUMENTED /usr/share/doc/avocado/tests/doublefail.py
571 INSTRUMENTED /usr/share/doc/avocado/tests/doublefree.py
572 INSTRUMENTED /usr/share/doc/avocado/tests/errortest.py
573 INSTRUMENTED /usr/share/doc/avocado/tests/failtest.py
574 INSTRUMENTED /usr/share/doc/avocado/tests/fiotest.py
575 INSTRUMENTED /usr/share/doc/avocado/tests/gdbtest.py
576 INSTRUMENTED /usr/share/doc/avocado/tests/gendata.py
577 INSTRUMENTED /usr/share/doc/avocado/tests/linuxbuild.py
578 INSTRUMENTED /usr/share/doc/avocado/tests/multiplextest.py
579 INSTRUMENTED /usr/share/doc/avocado/tests/passtest.py
580 INSTRUMENTED /usr/share/doc/avocado/tests/skiptest.py
581 INSTRUMENTED /usr/share/doc/avocado/tests/sleeptenmin.py
582 INSTRUMENTED /usr/share/doc/avocado/tests/sleeptest.py
583 INSTRUMENTED /usr/share/doc/avocado/tests/synctest.py
584 INSTRUMENTED /usr/share/doc/avocado/tests/timeouttest.py
585 INSTRUMENTED /usr/share/doc/avocado/tests/warntest.py
586 INSTRUMENTED /usr/share/doc/avocado/tests/whiteboard.py
587 Here, INSTRUMENTED means that the files there are python files with an
589 avocado test class in them, therefore, that they are what we call in‐
590 strumented tests. This means those tests can use all avocado APIs and
591 facilities. Let's try to list a directory with a bunch of executable
592 shell scripts:
593 $ avocado list examples/wrappers/
595 SIMPLE examples/wrappers/bind_cpu0.sh
596 SIMPLE examples/wrappers/dummy.sh
597 SIMPLE examples/wrappers/ltrace.sh
598 SIMPLE examples/wrappers/perf.sh
599 SIMPLE examples/wrappers/rr.sh
600 SIMPLE examples/wrappers/strace.sh
601 SIMPLE examples/wrappers/time.sh
602 SIMPLE examples/wrappers/valgrind.sh
603 Here, SIMPLE means that those files are executables, that avocado will
605 simply execute and return PASS or FAIL depending on their return codes
606 (PASS -> 0, FAIL -> any integer different than 0). You can also provide
607 the --verbose, or -V flag to display files that were detected but are
608 not avocado tests, along with summary information:
609 $ avocado -V list examples/gdb-prerun-scripts/
611 Type Test Tag(s)
612 NOT_A_TEST examples/gdb-prerun-scripts/README
613 NOT_A_TEST examples/gdb-prerun-scripts/pass-sigusr1
614 TEST TYPES SUMMARY
616 ==================
617 not_a_test: 2
618 That summarizes the basic commands you should be using more frequently
620 when you start with avocado. Let's talk now about how avocado stores
621 test results.
622
624 When avocado runs tests, it saves all its results on your system:
625 JOB ID : <id>
627 JOB LOG : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
628 For your convenience, avocado maintains a link to the latest job run
630 (an avocado run command in this context), so you can always use "lat‐
631 est" to browse your test results:
632 $ ls /home/user/avocado/job-results/latest
634 id
635 jobdata
636 job.log
637 results.html
638 results.json
639 results.tap
640 results.xml
641 sysinfo
642 test-results
643 The main log file is job.log, but every test has its own results direc‐
645 tory:
646 $ ls -1 ~/avocado/job-results/latest/test-results/
648 1-sleeptest.py:SleepTest.test
649 Since this is a directory, it should have content similar to:
651 $ ls -1 ~/avocado/job-results/latest/test-results/1-sleeptest.py\:SleepTest.test/
653 data
654 debug.log
655 stderr
656 stdout
657 whiteboard
658
660 Avocado has a powerful tool that enables multiple test scenarios to be
661 run using a single, unmodified test. This mechanism uses a YAML file
662 called the 'multiplex file', that tells avocado how to multiply all
663 possible test scenarios automatically.
664 This is a sample that varies the parameter sleep_length through the
666 scenarios /run/short (sleeps for 0.5 s), /run/medium (sleeps for 1 s),
667 /run/long (sleeps for 5s), /run/longest (sleeps for 10s). The YAML
668 file (multiplex file) that produced the output above is:
669 !mux
671 short:
672 sleep_length: 0.5
673 medium:
674 sleep_length: 1
675 long:
676 sleep_length: 5
677 longest:
678 sleep_length: 10
679 You can execute sleeptest in all variations exposed above with:
681 $ avocado run examples/tests/sleeptest.py -m examples/tests/sleeptest.py.data/sleeptest.yaml
683 And the output should look like:
685 JOB ID : <id>
687 JOB LOG : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
688 (1/4) examples/tests/sleeptest.py:SleepTest.test;run-short-beaf: PASS (0.50 s)
689 (2/4) examples/tests/sleeptest.py:SleepTest.test;run-medium-5595: PASS (1.01 s)
690 (3/4) examples/tests/sleeptest.py:SleepTest.test;run-long-f397: PASS (5.01 s)
691 (4/4) examples/tests/sleeptest.py:SleepTest.test;run-longest-efc4: PASS (10.01 s)
692 RESULTS : PASS 4 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
693 JOB TIME : 16.65 s
694 The test runner supports two kinds of global filters, through the com‐
696 mand line options --mux-filter-only and --mux-filter-out. The mux-fil‐
697 ter-only exclusively includes one or more paths and the mux-filter-out
698 removes one or more paths from being processed.
699 From the previous example, if we are interested to use the variants
701 /run/medium and /run/longest, we do the following command line:
702 $ avocado run examples/tests/sleeptest.py -m examples/tests/sleeptest.py.data/sleeptest.yaml \
704 --mux-filter-only /run/medium /run/longest
705 And if you want to remove /small from the variants created, we do the
707 following:
708 $ avocado run examples/tests/sleeptest.py -m examples/tests/sleeptest.py.data/sleeptest.yaml \
710 --mux-filter-out /run/medium
711 Note that both --mux-filter-only and --mux-filter-out filters can be
713 arranged in the same command line.
714 The multiplexer also supports default paths. The base path is /run/*
716 but it can be overridden by --mux-path, which accepts multiple argu‐
717 ments. What it does: it splits leaves by the provided paths. Each query
718 goes one by one through those sub-trees and first one to hit the match
719 returns the result. It might not solve all problems, but it can help to
720 combine existing YAML files with your ones:
721 qa: # large and complex read-only file, content injected into /qa
723 tests:
724 timeout: 10
725 ...
726 my_variants: !mux # your YAML file injected into /my_variants
727 short:
728 timeout: 1
729 long:
730 timeout: 1000
731 You want to use an existing test which uses params.get('timeout', '*').
733 Then you can use --mux-path '/my_variants/*' '/qa/*' and it'll first
734 look in your variants. If no matches are found, then it would proceed
735 to /qa/*
736 Keep in mind that only slices defined in mux-path are taken into ac‐
738 count for relative paths (the ones starting with *).
739
741 Avocado allows the instrumentation of executables being run by a test
742 in a transparent way. The user specifies a script ("the wrapper") to be
743 used to run the actual program called by the test.
744 If the instrumentation script is implemented correctly, it should not
746 interfere with the test behavior. That is, the wrapper should avoid
747 changing the return status, standard output and standard error messages
748 of the original executable.
749 The user can be specific about which program to wrap (with a shell-like
751 glob), or if that is omitted, a global wrapper that will apply to all
752 programs called by the test.
753 So, for every executable run by the test, the program name will be com‐
755 pared to the pattern to decide whether to wrap it or not. You can have
756 multiples wrappers and patterns defined.
757 Examples:
759 $ avocado run examples/tests/datadir.py --wrapper examples/wrappers/strace.sh
761 Any command created by the test datadir will be wrapped on strace.sh.
763 $ avocado run examples/tests/datadir.py \
765 --wrapper examples/wrappers/ltrace.sh:*make \
766 --wrapper examples/wrappers/perf.sh:*datadir
767 Any command that matches the pattern *make will be wrapper on ltrace.sh
769 and the pattern *datadir will trigger the execution of perf.sh.
770
772 It's quite common to have organically grown test suites in most soft‐
773 ware projects. These usually include a custom built, very specific test
774 runner that knows how to find and run their own tests.
775 Still, running those tests inside Avocado may be a good idea for vari‐
777 ous reasons, including being able to have results in different human
778 and machine readable formats, collecting system information alongside
779 those tests (the Avocado's sysinfo functionality), and more.
780 Avocado makes that possible by means of its "external runner" feature.
782 The most basic way of using it is:
783 $ avocado run --external-runner=/path/to/external_runner foo bar baz
785 In this example, Avocado will report individual test results for tests
787 foo, bar and baz. The actual results will be based on the return code
788 of individual executions of /path/to/external_runner foo, /path/to/ex‐
789 ternal_runner bar and finally /path/to/external_runner baz.
790 As another way to explain an show how this feature works, think of the
792 "external runner" as some kind of interpreter and the individual tests
793 as anything that this interpreter recognizes and is able to execute. A
794 UNIX shell, say /bin/sh could be considered an external runner, and
795 files with shell code could be considered tests:
796 $ echo "exit 0" > /tmp/pass
798 $ echo "exit 1" > /tmp/fail
799 $ avocado run --external-runner=/bin/sh /tmp/pass /tmp/fail
800 JOB ID : <id>
801 JOB LOG : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
802 TESTS : 2
803 (1/2) /tmp/pass: PASS (0.01 s)
804 (2/2) /tmp/fail: FAIL: Exited with status: '1', stdout: '' stderr: '' (0.02 s)
805 RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
806 JOB TIME : 0.11 s
807 This example is pretty obvious, and could be achieved by giving
809 /tmp/pass and /tmp/fail shell "shebangs" (#!/bin/sh), making them exe‐
810 cutable (chmod +x /tmp/pass /tmp/fail), and running them as "SIMPLE"
811 tests.
812 But now consider the following example:
814 $ avocado run --external-runner=/bin/curl http://local-avocado-server:9405/jobs/ \
816 http://remote-avocado-server:9405/jobs/
817 JOB ID : <id>
818 JOB LOG : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
819 TESTS : 2
820 (1/2) http://local-avocado-server:9405/jobs/: PASS (0.02 s)
821 (2/2) http://remote-avocado-server:9405/jobs/: FAIL (3.02 s)
822 RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
823 JOB TIME : 3.14 s
824 This effectively makes /bin/curl an "external test runner", responsible
826 for trying to fetch those URLs, and reporting PASS or FAIL for each of
827 them.
828
830 As a tester, you may want to check if the output of a given application
831 matches an expected output. In order to help with this common use case,
832 we offer the option --output-check-record [mode] to the test runner.
833 If this option is used, it will store the stdout or stderr of the
834 process (or both, if you specified all) being executed to reference
835 files: stdout.expected and stderr.expected.
836 Those files will be recorded in the test data dir. The data dir is in
838 the same directory as the test source file, named
839 [source_file_name.data]. Let's take as an example the test synctest.py.
840 In a fresh checkout of avocado, you can see:
841 examples/tests/synctest.py.data/stderr.expected
843 examples/tests/synctest.py.data/stdout.expected
844 From those 2 files, only stdout.expected is non empty:
846 $ cat examples/tests/synctest.py.data/stdout.expected
848 PAR : waiting
849 PASS : sync interrupted
850 The output files were originally obtained using the test runner and
852 passing the option --output-check-record all to the test runner:
853 $ avocado run --output-check-record all examples/tests/synctest.py
855 JOB ID : <id>
856 JOB LOG : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
857 (1/1) examples/tests/synctest.py:SyncTest.test: PASS (4.00 s)
858 RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
859 JOB TIME : 4.10 s
860 After the reference files are added, the check process is transparent,
862 in the sense that you do not need to provide special flags to the test
863 runner. Now, every time the test is executed, after it is done running,
864 it will check if the outputs are exactly right before considering the
865 test as PASSed. If you want to override the default behavior and skip
866 output check entirely, you may provide the flag --output-check=off to
867 the test runner.
868 The avocado.utils.process APIs have a parameter allow_output_check (de‐
870 faults to all), so that you can select which process outputs will go to
871 the reference files, should you chose to record them. You may choose
872 all, for both stdout and stderr, stdout, for the stdout only, stderr,
873 for only the stderr only, or none, to allow neither of them to be
874 recorded and checked.
875 This process works fine also with simple tests, executables that return
877 0 (PASSed) or != 0 (FAILed). Let's consider our bogus example:
878 $ cat output_record.sh
880 #!/bin/bash
881 echo "Hello, world!"
882 Let's record the output (both stdout and stderr) for this one:
884 $ avocado run output_record.sh --output-check-record all
886 JOB ID : <id>
887 JOB LOG : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
888 (1/1) output_record.sh: PASS (0.01 s)
889 RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
890 JOB HTML : /home/user/avocado/job-results/job-<date>-<shortid>/results.html
891 JOB TIME : 0.11 s
892 After this is done, you'll notice that a the test data directory ap‐
894 peared in the same level of our shell script, containing 2 files:
895 $ ls output_record.sh.data/
897 stderr.expected stdout.expected
898 Let's look what's in each of them:
900 $ cat output_record.sh.data/stdout.expected
902 Hello, world!
903 $ cat output_record.sh.data/stderr.expected
904 $
905 Now, every time this test runs, it'll take into account the expected
907 files that were recorded, no need to do anything else but run the test.
908
910 Avocado has some planned features that depend on knowing the Linux Dis‐
911 tribution being used on the system. The most basic command prints the
912 detected Linux Distribution:
913 $ avocado distro
915 Detected distribution: fedora (x86_64) version 21 release 0
916 Other features are available with the same command when command line
918 options are given, as shown by the --help option.
919 For instance, it possible to create a so-called "Linux Distribution
921 Definition" file, by inspecting an installation tree. The installation
922 tree could be the contents of the official installation ISO or a local
923 network mirror.
924 These files let Avocado pinpoint if a given installed package is part
926 of the original Linux Distribution or something else that was installed
927 from an external repository or even manually. This, in turn, can help
928 detecting regressions in base system packages that affected a given
929 test result.
930 To generate a definition file run:
932 $ avocado distro --distro-def-create --distro-def-name avocadix \
934 --distro-def-version 1 --distro-def-arch x86_64 \
935 --distro-def-type rpm --distro-def-path /mnt/dvd
936 And the output will be something like:
938 Loading distro information from tree... Please wait...
940 Distro information saved to "avocadix-1-x86_64.distro"
941
943 System wide configuration file
944 /etc/avocado/avocado.conf
945 Extra configuration files
947 /etc/avocado/conf.d/
948 User configuration file
950 ~/.config/avocado/avocado.conf
951
953 If you find a bug, please report it over our github page as an issue:
954 https://github.com/avocado-framework/avocado/issues
955
957 Avocado is released under GPLv2 (explicit version)
958 https://gnu.org/licenses/gpl-2.0.html. Even though most of the current
959 code is licensed under a "and any later version" clause, some parts are
960 specifically bound to the version 2 of the license and therefore that's
961 the official license of the prject itself. For more details, please see
962 the LICENSE file in the project source code directory.
963
965 For more information please check Avocado's project website, located at
966 https://avocado-framework.github.io/. There you'll find links to online
967 documentation, source code and community resources.
968
970 Avocado Development Team <avocado-devel@redhat.com>
971 AVOCADO(1)