1AVOCADO(1) AVOCADO(1)
2
6 avocado - test runner command line tool
7
9 avocado [-h] [-v] [--config [CONFIG_FILE]] [--show [STREAM[:LVL]]] [-s]
10 {config,diff,distro,exec-path,list,multiplex,plugins,run,sys‐
11 info} ...
12
14 Avocado is a modern test framework that is built on the experience
15 accumulated with autotest (http://autotest.github.io).
16 avocado is also the name of its test runner command line tool,
18 described in this man page.
19 For more information about the Avocado project, please check its web‐
21 site: http://avocado-framework.github.io/
22
24 The following list of options are builtin, application level avocado
25 options. Most other options are implemented via plugins and will depend
26 on them being loaded (avocado --help):
27 -h, --help show this help message and exit
29 -v, --version show program's version number and exit
30 --config [CONFIG_FILE]
31 Use custom configuration from a file
32 --show [STREAM[:LVL]]
33 List of comma separated builtin logs, or logging
34 streams optionally followed by LEVEL (DEBUG,INFO,...).
35 Builtin streams are: "test": test output; "debug":
36 tracebacks and other debugging info; "app":
37 application output; "early": early logging of other
38 streams, including test (very verbose); "remote":
39 fabric/paramiko debug; "all": all builtin streams;
40 "none": disables regular output (leaving only errors
41 enabled). By default: 'app'
42 -s, --silent disables regular output (leaving only errors enabled)
43 Real use of avocado depends on running avocado subcommands. This a typ‐
45 ical list of avocado subcommands:
46 config Shows avocado config keys
48 diff Shows the difference between 2 jobs.
49 distro Shows detected Linux distribution
50 exec-path Returns path to avocado bash libraries and exits.
51 list List available tests
52 multiplex Tool to analyze and visualize test variants and params
53 plugins Displays plugin information
54 run Runs one or more tests (native test, test alias,
55 binary or script)
56 sysinfo Collect system information
57 To get usage instructions for a given subcommand, run it with --help.
59 Example:
60 $ avocado run --help
62 Options for subcommand run (avocado run --help):
64 positional arguments:
66 TEST_REFERENCE List of test references (aliases or paths)
67 optional arguments:
69 -h, --help show this help message and exit
70 -d, --dry-run Instead of running the test only list them and log
71 their params.
72 --force-job-id UNIQUE_JOB_ID
73 Forces the use of a particular job ID. Used internally
74 when interacting with an avocado server. You should
75 not use this option unless you know exactly what
76 you're doing
77 --job-results-dir DIRECTORY
78 Forces to use of an alternate job results directory.
79 --job-timeout SECONDS
80 Set the maximum amount of time (in SECONDS) that tests
81 are allowed to execute. Values <= zero means "no
82 timeout". You can also use suffixes, like: s
83 (seconds), m (minutes), h (hours).
84 --failfast {on,off} Enable or disable the job interruption on first failed
85 test.
86 --keep-tmp {on,off} Keep job temporary files (useful for avocado
87 debugging). Defaults to off.
88 --sysinfo {on,off} Enable or disable system information (hardware
89 details, profilers, etc.). Current: on
90 --execution-order {tests-per-variant,variants-per-test}
91 How to iterate through test suite and variants
92 output and result format:
94 -s, --silent Silence stdout
95 --show-job-log Display only the job log on stdout. Useful for test
96 debugging purposes. No output will be displayed if you
97 also specify --silent
98 --store-logging-stream [STREAM[:LEVEL] [STREAM[:LEVEL] ...]]
99 Store given logging STREAMs in
100 $JOB_RESULTS_DIR/$STREAM.$LEVEL.
101 --html FILE Enable HTML output to the FILE where the result should
102 be written. The value - (output to stdout) is not
103 supported since not all HTML resources can be embedded
104 into a single file (page resources will be copied to
105 the output file dir)
106 --open-browser Open the generated report on your preferred browser.
107 This works even if --html was not explicitly passed,
108 since an HTML report is always generated on the job
109 results dir. Current: False
110 --html-job-result {on,off}
111 Enables default HTML result in the job results
112 directory. File will be located at
113 "html/results.html".
114 --journal Records test status changes (for use with avocado-
115 journal-replay and avocado-server)
116 --json FILE Enable JSON result format and write it to FILE. Use
117 '-' to redirect to the standard output.
118 --json-job-result {on,off}
119 Enables default JSON result in the job results
120 directory. File will be named "results.json".
121 --tap FILE Enable TAP result output and write it to FILE. Use '-'
122 to redirect to the standard output.
123 --tap-job-result {on,off}
124 Enables default TAP result in the job results
125 directory. File will be named "results.tap".
126 --xunit FILE Enable xUnit result format and write it to FILE. Use
127 '-' to redirect to the standard output.
128 --xunit-job-result {on,off}
129 Enables default xUnit result in the job results
130 directory. File will be named "results.xml".
131 -z, --archive Archive (ZIP) files generated by tests
132 output check arguments:
134 --output-check-record {none,all,stdout,stderr}
135 Record output streams of your tests to reference files
136 (valid options: none (do not record output streams),
137 all (record both stdout and stderr), stdout (record
138 only stderr), stderr (record only stderr). Current:
139 none
140 --output-check {on,off}
141 Enable or disable test output (stdout/stderr) check.
142 If this option is off, no output will be checked, even
143 if there are reference files present for the test.
144 Current: on (output check enabled)
145 loader options:
147 --loaders [LOADERS [LOADERS ...]]
148 Overrides the priority of the test loaders. You can
149 specify either @loader_name or TEST_TYPE. By default
150 it tries all available loaders according to priority
151 set in settings->plugins.loaders.
152 --external-runner EXECUTABLE
153 Path to an specific test runner that allows the use of
154 its own tests. This should be used for running tests
155 that do not conform to Avocado' SIMPLE testinterface
156 and can not run standalone. Note: the use of
157 --external-runner overwrites the --loaders to
158 "external_runner"
159 --external-runner-chdir {runner,test}
160 Change directory before executing tests. This option
161 may be necessary because of requirements and/or
162 limitations of the external test runner. If the
163 external runner requires to be run from its own base
164 directory,use "runner" here. If the external runner
165 runs tests based on files and requires to be run from
166 the directory where those files are located, use
167 "test" here and specify the test directory with the
168 option "--external-runner-testdir". Defaults to "None"
169 --external-runner-testdir DIRECTORY
170 Where test files understood by the external test
171 runner are located in the filesystem. Obviously this
172 assumes and only applies to external test runners that
173 run tests from files
174 filtering parameters:
176 --filter-by-tags TAGS
177 Filter INSTRUMENTED tests based on ":avocado:
178 tags=tag1,tag2" notation in their class docstring
179 --filter-by-tags-include-empty
180 Include all tests without tags during filtering. This
181 effectively means they will be kept in the test suite
182 found previously to filtering.
183 test execution inside docker container:
185 --docker IMAGE Name of the docker image torun tests on.
186 --docker-cmd CMD Override the docker command, eg. 'sudo docker' or
187 other base docker options like hypervisor. Default:
188 'docker'
189 --docker-options OPT Extra options for docker run cmd. (see: man docker-
190 run)
191 --docker-no-cleanup Preserve container after test
192 keep environment variables:
194 --env-keep ENV_KEEP Keep environment variables in remote executions
195 GNU Debugger support:
197 --gdb-run-bin EXECUTABLE[:BREAKPOINT]
198 Run a given executable inside the GNU debugger,
199 pausing at a given breakpoint (defaults to "main")
200 --gdb-prerun-commands EXECUTABLE:COMMANDS
201 After loading an executable in GDB, but before
202 actually running it, execute the GDB commands in the
203 given file. EXECUTABLE is optional, if omitted
204 COMMANDS will apply to all executables
205 --gdb-coredump {on,off}
206 Automatically generate a core dump when the inferior
207 process received a fatal signal such as SIGSEGV or
208 SIGABRT
209 test execution on a remote machine:
211 --remote-hostname REMOTE_HOSTNAME
212 Specify the hostname to login on remote machine
213 --remote-port REMOTE_PORT
214 Specify the port number to login on remote machine.
215 Current: 22
216 --remote-username REMOTE_USERNAME
217 Specify the username to login on remote machine.
218 Current: apahim
219 --remote-password REMOTE_PASSWORD
220 Specify the password to login on remote machine
221 --remote-key-file REMOTE_KEY_FILE
222 Specify an identity file with a private key instead of
223 a password (Example: .pem files from Amazon EC2)
224 --remote-timeout SECONDS
225 Amount of time (in seconds) to wait for a successful
226 connection to the remote machine. Defaults to 60
227 seconds.
228 job replay:
230 --replay REPLAY_JOBID
231 Replay a job identified by its (partial) hash id. Use
232 "--replay latest" to replay the latest job.
233 --replay-test-status REPLAY_TESTSTATUS
234 Filter tests to replay by test status
235 --replay-ignore REPLAY_IGNORE
236 Ignore variants (variants) and/or configuration
237 (config) from the source job
238 resultsdb options:
240 --resultsdb-api RESULTSDB_API
241 Specify the resultsdb API url
242 --resultsdb-logs RESULTSDB_LOGS
243 Specify the URL where the logs are published
244 test execution on a Virtual Machine:
246 --vm-domain VM_DOMAIN
247 Specify Libvirt Domain Name
248 --vm-hypervisor-uri VM_HYPERVISOR_URI
249 Specify hypervisor URI driver connection. Current:
250 qemu:///system
251 --vm-hostname VM_HOSTNAME
252 Specify VM hostname to login. By default Avocado
253 attempts to automatically find the VM IP address.
254 --vm-port VM_PORT Specify the port number to login on VM. Current: 22
255 --vm-username VM_USERNAME
256 Specify the username to login on VM
257 --vm-password VM_PASSWORD
258 Specify the password to login on VM
259 --vm-key-file VM_KEY_FILE
260 Specify an identity file with a private key instead of
261 a password (Example: .pem files from Amazon EC2)
262 --vm-cleanup Restore VM to a previous state, before running tests
263 --vm-timeout SECONDS Amount of time (in seconds) to wait for a successful
264 connection to the virtual machine. Defaults to 120
265 seconds.
266 wrapper support:
268 --wrapper SCRIPT[:EXECUTABLE]
269 Use a script to wrap executables run by a test. The
270 wrapper is either a path to a script (AKA a global
271 wrapper) or a path to a script followed by colon
272 symbol (:), plus a shell like glob to the target
273 EXECUTABLE. Multiple wrapper options are allowed, but
274 only one global wrapper can be defined.
275 yaml to mux options:
277 -m [FILE [FILE ...]], --mux-yaml [FILE [FILE ...]]
278 Location of one or more Avocado multiplex (.yaml)
279 FILE(s) (order dependent)
280 --mux-filter-only [MUX_FILTER_ONLY [MUX_FILTER_ONLY ...]]
281 Filter only path(s) from multiplexing
282 --mux-filter-out [MUX_FILTER_OUT [MUX_FILTER_OUT ...]]
283 Filter out path(s) from multiplexing
284 --mux-path [MUX_PATH [MUX_PATH ...]]
285 List of default paths used to determine path priority
286 when querying for parameters
287 --mux-inject [MUX_INJECT [MUX_INJECT ...]]
288 Inject [path:]key:node values into the final multiplex
289 tree.
290 yaml to mux options [deprecated]:
292 --multiplex [FILE [FILE ...]]
293 DEPRECATED: Location of one or more Avocado multiplex
294 (.yaml) FILE(s) (order dependent)
295 --filter-only [FILTER_ONLY [FILTER_ONLY ...]]
296 DEPRECATED: Filter only path(s) from multiplexing (use
297 --mux-filter-only instead)
298 --filter-out [FILTER_OUT [FILTER_OUT ...]]
299 DEPRECATED: Filter out path(s) from multiplexing (use
300 --mux-filter-out instead)
301 Options for subcommand config (avocado config --help):
303 optional arguments:
305 -h, --help show this help message and exit
306 --datadir Shows the data directories currently being used by
307 avocado
308 --paginator {on,off} Turn the paginator on/off. Current: on
309 Options for subcommand diff (avocado diff --help):
311 positional arguments:
313 <JOB> A job reference, identified by a (partial) unique ID
314 (SHA1) or test results directory.
315 optional arguments:
317 -h, --help show this help message and exit
318 --html FILE Enable HTML output to the FILE where the result should
319 be written.
320 --open-browser Generate and open a HTML report in your preferred
321 browser. If no --html file is provided, create a
322 temporary file.
323 --diff-filter DIFF_FILTER
324 Comma separated filter of diff sections:
325 (no)cmdline,(no)time,(no)variants,(no)results,
326 (no)config,(no)sysinfo (defaults to all enabled).
327 --paginator {on,off} Turn the paginator on/off. Current: on
328 --create-reports Create temporary files with job reports (to be used by
329 other diff tools)
330 Options for subcommand distro (avocado distro --help):
332 optional arguments:
334 -h, --help show this help message and exit
335 --distro-def-create Creates a distro definition file based on the path
336 given
337 --distro-def-name DISTRO_DEF_NAME
338 Distribution short name
339 --distro-def-version DISTRO_DEF_VERSION
340 Distribution major version number
341 ---distro-def-release DISTRO_DEF_RELEASE
342 Distribution release version number
343 --distro-def-arch DISTRO_DEF_ARCH
344 Primary architecture that the distro targets
345 --distro-def-path DISTRO_DEF_PATH
346 Top level directory of the distro installation files
347 --distro-def-type {deb,rpm}
348 Distro type (one of: deb, rpm)
349 Options for subcommand exec-path (avocado exec-path --help):
351 optional arguments:
353 -h, --help show this help message and exit
354 Options for subcommand list (avocado list --help):
356 positional arguments:
358 reference List of test references (aliases or paths). If empty,
359 avocado will list tests on the configured test source,
360 (see 'avocado config --datadir') Also, if there are
361 other test loader plugins active, tests from those
362 plugins might also show up (behavior may vary among
363 plugins)
364 optional arguments:
366 -h, --help show this help message and exit
367 -V, --verbose Whether to show extra information (headers and
368 summary). Current: False
369 --paginator {on,off} Turn the paginator on/off. Current: on
370 loader options:
372 --loaders [LOADERS [LOADERS ...]]
373 Overrides the priority of the test loaders. You can
374 specify either @loader_name or TEST_TYPE. By default
375 it tries all available loaders according to priority
376 set in settings->plugins.loaders.
377 --external-runner EXECUTABLE
378 Path to an specific test runner that allows the use of
379 its own tests. This should be used for running tests
380 that do not conform to Avocado' SIMPLE testinterface
381 and can not run standalone. Note: the use of
382 --external-runner overwrites the --loaders to
383 "external_runner"
384 --external-runner-chdir {runner,test}
385 Change directory before executing tests. This option
386 may be necessary because of requirements and/or
387 limitations of the external test runner. If the
388 external runner requires to be run from its own base
389 directory,use "runner" here. If the external runner
390 runs tests based on files and requires to be run from
391 the directory where those files are located, use
392 "test" here and specify the test directory with the
393 option "--external-runner-testdir". Defaults to "None"
394 --external-runner-testdir DIRECTORY
395 Where test files understood by the external test
396 runner are located in the filesystem. Obviously this
397 assumes and only applies to external test runners that
398 run tests from files
399 filtering parameters:
401 --filter-by-tags TAGS
402 Filter INSTRUMENTED tests based on ":avocado:
403 tags=tag1,tag2" notation in their class docstring
404 --filter-by-tags-include-empty
405 Include all tests without tags during filtering. This
406 effectively means they will be kept in the test suite
407 found previously to filtering.
408 Options for subcommand multiplex (avocado multiplex --help):
410 optional arguments:
412 -h, --help show this help message and exit
413 --summary SUMMARY Verbosity of the variants summary. (positive integer -
414 0, 1, ... - or none, brief, normal, verbose, full,
415 max)
416 --variants VARIANTS Verbosity of the list of variants. (positive integer -
417 0, 1, ... - or none, brief, normal, verbose, full,
418 max)
419 --system-wide Combine the files with the default tree.
420 -c, --contents [obsoleted by --variants] Shows the node content
421 (variables)
422 environment view options:
424 -d, --debug Debug the multiplex tree.
425 tree view options:
427 -t, --tree [obsoleted by --summary] Shows the multiplex tree
428 structure
429 -i, --inherit [obsoleted by --summary] Show the inherited values
430 yaml to mux options:
432 -m [FILE [FILE ...]], --mux-yaml [FILE [FILE ...]]
433 Location of one or more Avocado multiplex (.yaml)
434 FILE(s) (order dependent)
435 --mux-filter-only [MUX_FILTER_ONLY [MUX_FILTER_ONLY ...]]
436 Filter only path(s) from multiplexing
437 --mux-filter-out [MUX_FILTER_OUT [MUX_FILTER_OUT ...]]
438 Filter out path(s) from multiplexing
439 --mux-path [MUX_PATH [MUX_PATH ...]]
440 List of default paths used to determine path priority
441 when querying for parameters
442 --mux-inject [MUX_INJECT [MUX_INJECT ...]]
443 Inject [path:]key:node values into the final multiplex
444 tree.
445 yaml to mux options [deprecated]:
447 --multiplex [FILE [FILE ...]]
448 DEPRECATED: Location of one or more Avocado multiplex
449 (.yaml) FILE(s) (order dependent)
450 --filter-only [FILTER_ONLY [FILTER_ONLY ...]]
451 DEPRECATED: Filter only path(s) from multiplexing (use
452 --mux-filter-only instead)
453 --filter-out [FILTER_OUT [FILTER_OUT ...]]
454 DEPRECATED: Filter out path(s) from multiplexing (use
455 --mux-filter-out instead)
456 Options for subcommand plugins (avocado plugins --help):
458 optional arguments:
460 -h, --help show this help message and exit
461 --paginator {on,off} Turn the paginator on/off. Current: on
462 Options for subcommand sysinfo (avocado sysinfo --help):
464 positional arguments:
466 sysinfodir Dir where to dump sysinfo
467 optional arguments:
469 -h, --help show this help message and exit
470
472 The most common use of the avocado command line tool is to run a test:
473 $ avocado run sleeptest.py
475 This command will run the sleeptest.py test, as found on the standard
477 test directories. The output should be similar to:
478 JOB ID : <id>
480 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
481 (1/1) sleeptest.py:SleepTest.test: PASS (1.01 s)
482 RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
483 JOB TIME : 1.11 s
484 The test directories will vary depending on you system and installation
486 method used. Still, it's pretty easy to find that out as shown in the
487 next section.
488
490 When you are developing new tests, frequently you want to look at the
491 straight output of the job log in the stdout, without having to tail
492 the job log. In order to do that, you can use --show-job-log to the
493 avocado test runner:
494 $ scripts/avocado run examples/tests/sleeptest.py --show-job-log
496 ...
497 PARAMS (key=timeout, path=*, default=None) => None
498 START 1-sleeptest.py:SleepTest.test
499 PARAMS (key=sleep_length, path=*, default=1) => 1
500 Sleeping for 1.00 seconds
501 Not logging /var/log/messages (lack of permissions)
502 PASS 1-sleeptest.py:SleepTest.test
503 ...
504 Let's say you are debugging a test particularly large, with lots of
506 debug output and you want to reduce this output to only messages with
507 level 'INFO' and higher. You can set job-log-level to info to reduce
508 the amount of output.
509 Edit your ~/.config/avocado/avocado.conf file and add:
511 [job.output]
513 loglevel = info
514 Running the same example with this option will give you:
516 $ scripts/avocado run sleeptest --show-job-log
518 ...
519 START 1-sleeptest.py:SleepTest.test
520 PASS 1-sleeptest.py:SleepTest.test
521 ...
522 The levels you can choose are the levels available in the python log‐
524 ging system https://docs.python.org/2/library/logging.html#logging-lev‐
525 els, translated to lowercase strings, so 'notset', 'debug', 'info',
526 'warning', 'error', 'critical', in order of severity.
527 As you can see, the UI output is suppressed and only the job log goes
529 to stdout, making this a useful feature for test development/debugging.
530
532 You may specify --silent, that means avocado will turn off all runner
533 stdout. Even if you specify things like --show-job-log in the CLI,
534 --silent will have precedence and you will not get application stdout.
535 Note that --silent does not affect on disk job logs, those continue to
536 be generated normally.
537
539 You may specify --sysinfo=off and avocado will not collect profilers,
540 hardware details and other system information, inside the job result
541 directory.
542
544 The avocado command line tool also has a list command, that lists the
545 known tests in a given path, be it a path to an individual test, or a
546 path to a directory. If no arguments provided, avocado will inspect the
547 contents of the test location being used by avocado (if you are in
548 doubt about which one is that, you may use avocado config --datadir).
549 The output looks like:
550 $ avocado list
552 INSTRUMENTED /usr/share/avocado/tests/abort.py
553 INSTRUMENTED /usr/share/avocado/tests/datadir.py
554 INSTRUMENTED /usr/share/avocado/tests/doublefail.py
555 INSTRUMENTED /usr/share/avocado/tests/doublefree.py
556 INSTRUMENTED /usr/share/avocado/tests/errortest.py
557 INSTRUMENTED /usr/share/avocado/tests/failtest.py
558 INSTRUMENTED /usr/share/avocado/tests/fiotest.py
559 INSTRUMENTED /usr/share/avocado/tests/gdbtest.py
560 INSTRUMENTED /usr/share/avocado/tests/gendata.py
561 INSTRUMENTED /usr/share/avocado/tests/linuxbuild.py
562 INSTRUMENTED /usr/share/avocado/tests/multiplextest.py
563 INSTRUMENTED /usr/share/avocado/tests/passtest.py
564 INSTRUMENTED /usr/share/avocado/tests/skiptest.py
565 INSTRUMENTED /usr/share/avocado/tests/sleeptenmin.py
566 INSTRUMENTED /usr/share/avocado/tests/sleeptest.py
567 INSTRUMENTED /usr/share/avocado/tests/synctest.py
568 INSTRUMENTED /usr/share/avocado/tests/timeouttest.py
569 INSTRUMENTED /usr/share/avocado/tests/trinity.py
570 INSTRUMENTED /usr/share/avocado/tests/warntest.py
571 INSTRUMENTED /usr/share/avocado/tests/whiteboard.py
572 Here, INSTRUMENTED means that the files there are python files with an
574 avocado test class in them, therefore, that they are what we call
575 instrumented tests. This means those tests can use all avocado APIs and
576 facilities. Let's try to list a directory with a bunch of executable
577 shell scripts:
578 $ avocado list examples/wrappers/
580 SIMPLE examples/wrappers/dummy.sh
581 SIMPLE examples/wrappers/ltrace.sh
582 SIMPLE examples/wrappers/perf.sh
583 SIMPLE examples/wrappers/strace.sh
584 SIMPLE examples/wrappers/time.sh
585 SIMPLE examples/wrappers/valgrind.sh
586 Here, SIMPLE means that those files are executables, that avocado will
588 simply execute and return PASS or FAIL depending on their return codes
589 (PASS -> 0, FAIL -> any integer different than 0). You can also provide
590 the --verbose, or -V flag to display files that were detected but are
591 not avocado tests, along with summary information:
592 $ avocado list examples/gdb-prerun-scripts/ -V
594 Type Test Tag(s)
595 NOT_A_TEST examples/gdb-prerun-scripts/README
596 NOT_A_TEST examples/gdb-prerun-scripts/pass-sigusr1
597 TEST TYPES SUMMARY
599 ==================
600 SIMPLE: 0
601 INSTRUMENTED: 0
602 MISSING: 0
603 NOT_A_TEST: 2
604 That summarizes the basic commands you should be using more frequently
606 when you start with avocado. Let's talk now about how avocado stores
607 test results.
608
610 When avocado runs tests, it saves all its results on your system:
611 JOB ID : <id>
613 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
614 For your convenience, avocado maintains a link to the latest job run
616 (an avocado run command in this context), so you can always use "lat‐
617 est" to browse your test results:
618 $ ls /home/<user>/avocado/job-results/latest
620 id
621 jobdata
622 job.log
623 results.json
624 results.tap
625 results.xml
626 sysinfo
627 test-results
628 The main log file is job.log, but every test has its own results direc‐
630 tory:
631 $ ls -1 ~/avocado/job-results/latest/test-results/
633 1-sleeptest.py:SleepTest.test
634 Since this is a directory, it should have content similar to:
636 $ ls -1 ~/avocado/job-results/latest/test-results/1-sleeptest.py\:SleepTest.test/
638 data
639 debug.log
640 remote.log
641 stderr
642 stdout
643 sysinfo
644 whiteboard
645
647 Avocado has a powerful tool that enables multiple test scenarios to be
648 run using a single, unmodified test. This mechanism uses a YAML file
649 called the 'multiplex file', that tells avocado how to multiply all
650 possible test scenarios automatically.
651 A command by the same name, multiplex, is available on the avocado com‐
653 mand line tool, and enables you to see all the test scenarios that can
654 be run:
655 $ avocado multiplex -m examples/tests/sleeptest.py.data/sleeptest.yaml -c
657 Variants generated:
658 Variant 1: /run/short
660 /run/short:sleep_length => 0.5
661 Variant 2: /run/medium
663 /run/medium:sleep_length => 1
664 Variant 3: /run/long
666 /run/long:sleep_length => 5
667 Variant 4: /run/longest
669 /run/longest:sleep_length => 10
670 This is a sample that varies the parameter sleep_length through the
672 scenarios /run/short (sleeps for 0.5 s), /run/medium (sleeps for 1 s),
673 /run/long (sleeps for 5s), /run/longest (sleeps for 10s). The YAML
674 file (multiplex file) that produced the output above is:
675 !mux
677 short:
678 sleep_length: 0.5
679 medium:
680 sleep_length: 1
681 long:
682 sleep_length: 5
683 longest:
684 sleep_length: 10
685 You can execute sleeptest in all variations exposed above with:
687 $ avocado run sleeptest.py -m examples/tests/sleeptest.py.data/sleeptest.yaml
689 And the output should look like:
691 JOB ID : <id>
693 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
694 (1/4) sleeptest.py:SleepTest.test;1: PASS (0.51 s)
695 (2/4) sleeptest.py:SleepTest.test;2: PASS (1.01 s)
696 (3/4) sleeptest.py:SleepTest.test;3: PASS (5.02 s)
697 (4/4) sleeptest.py:SleepTest.test;4: PASS (10.01 s)
698 RESULTS : PASS 4 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
699 JOB TIME : 16.65 s
700 The multiplex plugin and the test runner supports two kinds of global
702 filters, through the command line options --mux-filter-only and
703 --mux-filter-out. The mux-filter-only exclusively includes one or more
704 paths and the mux-filter-out removes one or more paths from being pro‐
705 cessed.
706 From the previous example, if we are interested to use the variants
708 /run/medium and /run/longest, we do the following command line:
709 $ avocado run sleeptest.py -m examples/tests/sleeptest.py.data/sleeptest.yaml \
711 --mux-filter-only /run/medium /run/longest
712 And if you want to remove /small from the variants created, we do the
714 following:
715 $ avocado run sleeptest.py -m examples/tests/sleeptest.py.data/sleeptest.yaml \
717 --mux-filter-out /run/medium
718 Note that both --mux-filter-only and --mux-filter-out filters can be
720 arranged in the same command line.
721 The multiplexer also supports default paths. The base path is /run/*
723 but it can be overridden by --mux-path, which accepts multiple argu‐
724 ments. What it does: it splits leaves by the provided paths. Each query
725 goes one by one through those sub-trees and first one to hit the match
726 returns the result. It might not solve all problems, but it can help to
727 combine existing YAML files with your ones:
728 qa: # large and complex read-only file, content injected into /qa
730 tests:
731 timeout: 10
732 ...
733 my_variants: !mux # your YAML file injected into /my_variants
734 short:
735 timeout: 1
736 long:
737 timeout: 1000
738 You want to use an existing test which uses params.get('timeout', '*').
740 Then you can use --mux-path '/my_variants/*' '/qa/*' and it'll first
741 look in your variants. If no matches are found, then it would proceed
742 to /qa/*
743 Keep in mind that only slices defined in mux-path are taken into
745 account for relative paths (the ones starting with *).
746
748 One interesting avocado feature is the ability to automatically and
749 transparently run executables that are used on a given test inside the
750 GNU debugger.
751 Suppose you are running a test that uses an external, compiled, image
753 converter. Now suppose you're feeding it with different types of
754 images, including broken image files, and it fails at a given point.
755 You wish you could connect to the debugger at that given source loca‐
756 tion while your test is running. This is how to do just that with avo‐
757 cado:
758 $ avocado run --gdb-run-bin=convert:convert_ppm_to_raw converttest.py
760 The job starts running just as usual, and so does your test:
762 JOB ID : <id>
764 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
765 TESTS : 1
766 (1/1) converttest.py:ConvertTest.test: /
767 The convert executable though, automatically runs inside GDB. Avocado
769 will stop when the given breakpoint is reached:
770 TEST PAUSED because of debugger breakpoint. To DEBUG your application run:
772 /home/<user>/avocado/job-results/job-<date>-<shortid>/test-results/converttest.py/data/convert.gdb.sh
773 NOTE: please use *disconnect* command in gdb before exiting, or else the debugged process will be KILLED
775 From this point, you can run the generated script (convert.gdb.sh) to
777 debug you application.
778 As noted, it is strongly recommended that you disconnect from gdb while
780 your executable is still running. That is, if the executable finished
781 running while you are debugging it, avocado has no way to know about
782 its status.
783 Avocado will automatically send a continue command to the debugger when
785 you disconnect from and exit gdb.
786 If, for some reason you have a custom GDB, or your system does not put
788 GDB on what avocado believes to be the standard location
789 (/usr/bin/gdb), you can override that in the section gdb.paths of your
790 documentation:
791 [gdb.paths]
793 gdb = /usr/bin/gdb
794 gdbserver = /usr/bin/gdbserver
795 So running avocado after setting those will use the appropriate
797 gdb/gdbserver path.
798 If you are debugging a special application and need to setup GDB in
800 custom ways by running GDB commands, you can do that with the
801 --gdb-prerun-commands option:
802 $ avocado run --gdb-run-bin=foo:bar --gdb-prerun-commands=/tmp/disable-signals footest.py
804 In this example, /tmp/disable-signals is a simple text file containing
806 two lines:
807 signal SIGUSR1 pass
809 signal SIGUSR1 nostop
810 Each line is a GDB command, so you can have from simple to very complex
812 debugging environments configured like that.
813
815 Avocado allows the instrumentation of executables being run by a test
816 in a transparent way. The user specifies a script ("the wrapper") to be
817 used to run the actual program called by the test.
818 If the instrumentation script is implemented correctly, it should not
820 interfere with the test behavior. That is, the wrapper should avoid
821 changing the return status, standard output and standard error messages
822 of the original executable.
823 The user can be specific about which program to wrap (with a shell-like
825 glob), or if that is omitted, a global wrapper that will apply to all
826 programs called by the test.
827 So, for every executable run by the test, the program name will be com‐
829 pared to the pattern to decide whether to wrap it or not. You can have
830 multiples wrappers and patterns defined.
831 Examples:
833 $ avocado run datadir.py --wrapper examples/wrappers/strace.sh
835 Any command created by the test datadir will be wrapped on strace.sh.
837 $ avocado run datadir.py --wrapper examples/wrappers/ltrace.sh:*make \
839 --wrapper examples/wrappers/perf.sh:*datadir
840 Any command that matches the pattern *make will be wrapper on ltrace.sh
842 and the pattern *datadir will trigger the execution of perf.sh.
843 Note that it is not possible to use --gdb-run-bin together with --wrap‐
845 per, they are incompatible.
846
848 It's quite common to have organically grown test suites in most soft‐
849 ware projects. These usually include a custom built, very specific test
850 runner that knows how to find and run their own tests.
851 Still, running those tests inside Avocado may be a good idea for vari‐
853 ous reasons, including being able to have results in different human
854 and machine readable formats, collecting system information alongside
855 those tests (the Avocado's sysinfo functionality), and more.
856 Avocado makes that possible by means of its "external runner" feature.
858 The most basic way of using it is:
859 $ avocado run --external-runner=/path/to/external_runner foo bar baz
861 In this example, Avocado will report individual test results for tests
863 foo, bar and baz. The actual results will be based on the return code
864 of individual executions of /path/to/external_runner foo,
865 /path/to/external_runner bar and finally /path/to/external_runner baz.
866 As another way to explain an show how this feature works, think of the
868 "external runner" as some kind of interpreter and the individual tests
869 as anything that this interpreter recognizes and is able to execute. A
870 UNIX shell, say /bin/sh could be considered an external runner, and
871 files with shell code could be considered tests:
872 $ echo "exit 0" > /tmp/pass
874 $ echo "exit 1" > /tmp/fail
875 $ avocado run --external-runner=/bin/sh /tmp/pass /tmp/fail
876 JOB ID : <id>
877 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
878 TESTS : 2
879 (1/2) /tmp/pass: PASS (0.01 s)
880 (2/2) /tmp/fail: FAIL (0.01 s)
881 RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0
882 JOB TIME : 0.11 s
883 This example is pretty obvious, and could be achieved by giving
885 /tmp/pass and /tmp/fail shell "shebangs" (#!/bin/sh), making them exe‐
886 cutable (chmod +x /tmp/pass /tmp/fail), and running them as "SIMPLE"
887 tests.
888 But now consider the following example:
890 $ avocado run --external-runner=/bin/curl http://local-avocado-server:9405/jobs/ \
892 http://remote-avocado-server:9405/jobs/
893 JOB ID : <id>
894 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
895 TESTS : 2
896 (1/2) http://local-avocado-server:9405/jobs/: PASS (0.02 s)
897 (2/2) http://remote-avocado-server:9405/jobs/: FAIL (3.02 s)
898 RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0
899 JOB TIME : 3.14 s
900 This effectively makes /bin/curl an "external test runner", responsible
902 for trying to fetch those URLs, and reporting PASS or FAIL for each of
903 them.
904
906 As a tester, you may want to check if the output of a given application
907 matches an expected output. In order to help with this common use case,
908 we offer the option --output-check-record [mode] to the test runner.
909 If this option is used, it will store the stdout or stderr of the
910 process (or both, if you specified all) being executed to reference
911 files: stdout.expected and stderr.expected.
912 Those files will be recorded in the test data dir. The data dir is in
914 the same directory as the test source file, named
915 [source_file_name.data]. Let's take as an example the test synctest.py.
916 In a fresh checkout of avocado, you can see:
917 examples/tests/synctest.py.data/stderr.expected
919 examples/tests/synctest.py.data/stdout.expected
920 From those 2 files, only stdout.expected is non empty:
922 $ cat examples/tests/synctest.py.data/stdout.expected
924 PAR : waiting
925 PASS : sync interrupted
926 The output files were originally obtained using the test runner and
928 passing the option --output-check-record all to the test runner:
929 $ avocado run --output-check-record all examples/tests/synctest.py
931 JOB ID : <id>
932 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
933 (1/1) examples/tests/synctest.py:SyncTest.test: PASS (4.00 s)
934 RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
935 JOB TIME : 4.10 s
936 After the reference files are added, the check process is transparent,
938 in the sense that you do not need to provide special flags to the test
939 runner. Now, every time the test is executed, after it is done running,
940 it will check if the outputs are exactly right before considering the
941 test as PASSed. If you want to override the default behavior and skip
942 output check entirely, you may provide the flag --output-check=off to
943 the test runner.
944 The avocado.utils.process APIs have a parameter allow_output_check
946 (defaults to all), so that you can select which process outputs will go
947 to the reference files, should you chose to record them. You may choose
948 all, for both stdout and stderr, stdout, for the stdout only, stderr,
949 for only the stderr only, or none, to allow neither of them to be
950 recorded and checked.
951 This process works fine also with simple tests, executables that return
953 0 (PASSed) or != 0 (FAILed). Let's consider our bogus example:
954 $ cat output_record.sh
956 #!/bin/bash
957 echo "Hello, world!"
958 Let's record the output (both stdout and stderr) for this one:
960 $ avocado run output_record.sh --output-check-record all
962 JOB ID : <id>
963 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
964 TESTS : 1
965 (1/1) home/$USER/Code/avocado/output_record.sh: PASS (0.01 s)
966 RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
967 JOB TIME : 0.11 s
968 After this is done, you'll notice that a the test data directory
970 appeared in the same level of our shell script, containing 2 files:
971 $ ls output_record.sh.data/
973 stderr.expected stdout.expected
974 Let's look what's in each of them:
976 $ cat output_record.sh.data/stdout.expected
978 Hello, world!
979 $ cat output_record.sh.data/stderr.expected
980 $
981 Now, every time this test runs, it'll take into account the expected
983 files that were recorded, no need to do anything else but run the test.
984
986 Avocado allows you to execute tests on a remote machine by means of a
987 SSH network connection. The remote machine must be configured to accept
988 remote connections and the Avocado framework have to be installed in
989 both origin and remote machines.
990 When running tests on remote machine, the test sources and its data (if
992 any present) are transferred to the remote target, just before the test
993 execution. After the test execution, all test results are transferred
994 back to the origin machine.
995 Here is how to run the sleeptest example test in a remote machine with
997 IP address 192.168.0.123 (standard port 22), remote user name fedora
998 and remote user password 123456:
999 $ avocado run sleeptest.py --remote-hostname 192.168.0.123 --remote-username fedora --remote-password 123456
1001 The output should look like:
1003 JOB ID : <id>
1005 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
1006 LOGIN : fedora@localhost:22 (TIMEOUT: 60 seconds)
1007 (1/1) sleeptest.py:SleepTest.test: PASS (1.02 s)
1008 RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
1009 JOB TIME : 1.12 s
1010 For more information, please consult the topic Remote Machine Plugin on
1012 Avocado's online documentation.
1013
1015 Avocado has some planned features that depend on knowing the Linux Dis‐
1016 tribution being used on the system. The most basic command prints the
1017 detected Linux Distribution:
1018 $ avocado distro
1020 Detected distribution: fedora (x86_64) version 21 release 0
1021 Other features are available with the same command when command line
1023 options are given, as shown by the --help option.
1024 For instance, it possible to create a so-called "Linux Distribution
1026 Definition" file, by inspecting an installation tree. The installation
1027 tree could be the contents of the official installation ISO or a local
1028 network mirror.
1029 These files let Avocado pinpoint if a given installed package is part
1031 of the original Linux Distribution or something else that was installed
1032 from an external repository or even manually. This, in turn, can help
1033 detecting regressions in base system pacakges that affected a given
1034 test result.
1035 To generate a definition file run:
1037 $ avocado distro --distro-def-create --distro-def-name avocadix \
1039 --distro-def-version 1 --distro-def-arch x86_64 \
1040 --distro-def-type rpm --distro-def-path /mnt/dvd
1041 And the output will be something like:
1043 Loading distro information from tree... Please wait...
1045 Distro information saved to "avocadix-1-x86_64.distro"
1046
1048 System wide configuration file
1049 /etc/avocado/avocado.conf
1050 Extra configuration files
1052 /etc/avocado/conf.d/
1053 User configuration file
1055 ~/.config/avocado/avocado.conf
1056
1058 If you find a bug, please report it over our github page as an issue:
1059 https://github.com/avocado-framework/avocado/issues
1060
1062 Avocado is released under GPLv2 (explicit version)
1063 http://gnu.org/licenses/gpl-2.0.html. Even though most of the current
1064 code is licensed under a "and any later version" clause, some parts are
1065 specifically bound to the version 2 of the license and therefore that's
1066 the official license of the prject itself. For more details, please see
1067 the LICENSE file in the project source code directory.
1068
1070 For more information please check Avocado's project website, located at
1071 http://avocado-framework.github.io/. There you'll find links to online
1072 documentation, source code and community resources.
1073
1075 Avocado Development Team <avocado-devel@redhat.com>
1076 AVOCADO(1)