1AVOCADO(1) AVOCADO(1)
2
6 avocado - test runner command line tool
7
9 avocado [-h] [-v] [--config [CONFIG_FILE]] [--show [STREAM[:LVL]]] [-s]
10 {config,diff,distro,exec-path,list,multiplex,plugins,run,sys‐
11 info} ...
12
14 Avocado is a modern test framework that is built on the experience ac‐
15 cumulated with autotest (http://autotest.github.io).
16 avocado is also the name of its test runner command line tool, de‐
18 scribed in this man page.
19 For more information about the Avocado project, please check its web‐
21 site: http://avocado-framework.github.io/
22
24 The following list of options are builtin, application level avocado
25 options. Most other options are implemented via plugins and will depend
26 on them being loaded (avocado --help):
27 -h, --help show this help message and exit
29 -v, --version show program's version number and exit
30 --config [CONFIG_FILE]
31 Use custom configuration from a file
32 --paginator {on,off} Turn the paginator on/off.
33 --show [STREAM[:LVL]]
34 List of comma separated builtin logs, or logging
35 streams optionally followed by LEVEL (DEBUG,INFO,...).
36 Builtin streams are: "test": test output; "debug":
37 tracebacks and other debugging info; "app":
38 application output; "early": early logging of other
39 streams, including test (very verbose); "all": all
40 builtin streams; "none": disables regular output
41 (leaving only errors enabled). By default: 'app'
42 Real use of avocado depends on running avocado subcommands. This a typ‐
44 ical list of avocado subcommands:
45 config Shows avocado config keys
47 diff Shows the difference between 2 jobs.
48 distro Shows detected Linux distribution
49 exec-path Returns path to avocado bash libraries and exits.
50 list List available tests
51 multiplex Tool to analyze and visualize test variants and params
52 plugins Displays plugin information
53 run Runs one or more tests (native test, test alias,
54 binary or script)
55 sysinfo Collect system information
56 To get usage instructions for a given subcommand, run it with --help.
58 Example:
59 $ avocado run --help
61 Options for subcommand run (avocado run --help):
63 positional arguments:
65 TEST_REFERENCE List of test references (aliases or paths)
66 optional arguments:
68 -h, --help show this help message and exit
69 -d, --dry-run Instead of running the test only list them and log
70 their params.
71 --force-job-id UNIQUE_JOB_ID
72 Forces the use of a particular job ID. Used internally
73 when interacting with an avocado server. You should
74 not use this option unless you know exactly what
75 you're doing
76 --job-results-dir DIRECTORY
77 Forces to use of an alternate job results directory.
78 --job-timeout SECONDS
79 Set the maximum amount of time (in SECONDS) that tests
80 are allowed to execute. Values <= zero means "no
81 timeout". You can also use suffixes, like: s
82 (seconds), m (minutes), h (hours).
83 --failfast Enable the job interruption on first failed test.
84 test.
85 --keep-tmp Keep job temporary files (useful for avocado
86 debugging).
87 --disable-sysinfo Enable or disable sysinfo information. Like hardware
88 details, profiles, etc.
89 --execution-order {tests-per-variant,variants-per-test}
90 How to iterate through test suite and variants
91 output and result format:
93 --store-logging-stream [STREAM[:LEVEL] [STREAM[:LEVEL] ...]]
94 Store given logging STREAMs in
95 $JOB_RESULTS_DIR/$STREAM.$LEVEL.
96 --html FILE Enable HTML output to the FILE where the result should
97 be written. The value - (output to stdout) is not
98 supported since not all HTML resources can be embedded
99 into a single file (page resources will be copied to
100 the output file dir)
101 --open-browser Open the generated report on your preferred browser.
102 This works even if --html was not explicitly passed,
103 since an HTML report is always generated on the job
104 results dir. Current: False
105 --html-job-result {on,off}
106 Enables default HTML result in the job results
107 directory. File will be located at
108 "html/results.html".
109 --journal Records test status changes (for use with avocado-
110 journal-replay and avocado-server)
111 --json FILE Enable JSON result format and write it to FILE. Use
112 '-' to redirect to the standard output.
113 --json-job-result {on,off}
114 Enables default JSON result in the job results
115 directory. File will be named "results.json".
116 --tap FILE Enable TAP result output and write it to FILE. Use '-'
117 to redirect to the standard output.
118 --tap-job-result {on,off}
119 Enables default TAP result in the job results
120 directory. File will be named "results.tap".
121 --tap-include-logs Include test logs as comments in TAP output. Defaults
122 to False
123 --xunit FILE Enable xUnit result format and write it to FILE. Use
124 '-' to redirect to the standard output.
125 --xunit-job-result {on,off}
126 Enables default xUnit result in the job results
127 directory. File will be named "results.xml".
128 --xunit-job-name XUNIT_JOB_NAME
129 Override the reported job name. By default uses the
130 Avocado job name which is always unique. This is
131 useful for reporting in Jenkins as it only evaluates
132 first-failure from jobs of the same name.
133 --xunit-max-test-log-chars SIZE
134 Limit the attached job log to given number of
135 characters (k/m/g suffix allowed)
136 -z, --archive Archive (ZIP) files generated by tests
137 output check arguments:
139 --output-check-record {none,all,stdout,stderr}
140 Record output streams of your tests to reference files
141 (valid options: none (do not record output streams),
142 all (record both stdout and stderr), stdout (record
143 only stderr), stderr (record only stderr). Current:
144 none
145 --disable-output-check
146 Disables test output (stdout/stderr) check. If this
147 option is given, no output will be checked, even if
148 there are reference files present for the test.
149 loader options:
151 --loaders [LOADERS [LOADERS ...]]
152 Overrides the priority of the test loaders. You can
153 specify either @loader_name or TEST_TYPE. By default
154 it tries all available loaders according to priority
155 set in settings->plugins.loaders.
156 --external-runner EXECUTABLE
157 Path to an specific test runner that allows the use of
158 its own tests. This should be used for running tests
159 that do not conform to Avocado' SIMPLE testinterface
160 and can not run standalone. Note: the use of
161 --external-runner overwrites the --loaders to
162 "external_runner"
163 --external-runner-chdir {runner,test}
164 Change directory before executing tests. This option
165 may be necessary because of requirements and/or
166 limitations of the external test runner. If the
167 external runner requires to be run from its own base
168 directory,use "runner" here. If the external runner
169 runs tests based on files and requires to be run from
170 the directory where those files are located, use
171 "test" here and specify the test directory with the
172 option "--external-runner-testdir". Defaults to "None"
173 --external-runner-testdir DIRECTORY
174 Where test files understood by the external test
175 runner are located in the filesystem. Obviously this
176 assumes and only applies to external test runners that
177 run tests from files
178 filtering parameters:
180 --filter-by-tags TAGS
181 Filter tests based on tags
182 --filter-by-tags-include-empty
183 Include all tests without tags during filtering. This
184 effectively means they will be kept in the test suite
185 found previously to filtering.
186 --filter-by-tags-include-empty-key
187 Include all tests that do not have a matching key in
188 its key:val tags. This effectively means those tests
189 will be kept in the test suite found previously to
190 filtering.
191 test execution inside docker container:
193 --docker IMAGE Name of the docker image torun tests on.
194 --docker-cmd CMD Override the docker command, eg. 'sudo docker' or
195 other base docker options like hypervisor. Default:
196 'docker'
197 --docker-options OPT Extra options for docker run cmd. (see: man docker-
198 run)
199 --docker-no-cleanup Preserve container after test
200 keep environment variables:
202 --env-keep ENV_KEEP Keep environment variables in remote executions
203 GNU Debugger support:
205 --gdb-run-bin EXECUTABLE[:BREAKPOINT]
206 Run a given executable inside the GNU debugger,
207 pausing at a given breakpoint (defaults to "main")
208 --gdb-prerun-commands EXECUTABLE:COMMANDS
209 After loading an executable in GDB, but before
210 actually running it, execute the GDB commands in the
211 given file. EXECUTABLE is optional, if omitted
212 COMMANDS will apply to all executables
213 --gdb-coredump {on,off}
214 Automatically generate a core dump when the inferior
215 process received a fatal signal such as SIGSEGV or
216 SIGABRT
217 job replay:
219 --replay REPLAY_JOBID
220 Replay a job identified by its (partial) hash id. Use
221 "--replay latest" to replay the latest job.
222 --replay-test-status REPLAY_TESTSTATUS
223 Filter tests to replay by test status
224 --replay-ignore REPLAY_IGNORE
225 Ignore variants (variants) and/or configuration
226 (config) from the source job
227 resultsdb options:
229 --resultsdb-api RESULTSDB_API
230 Specify the resultsdb API url
231 --resultsdb-logs RESULTSDB_LOGS
232 Specify the URL where the logs are published
233 test execution on a Virtual Machine:
235 --vm-domain VM_DOMAIN
236 Specify Libvirt Domain Name
237 --vm-hypervisor-uri VM_HYPERVISOR_URI
238 Specify hypervisor URI driver connection. Current:
239 qemu:///system
240 --vm-hostname VM_HOSTNAME
241 Specify VM hostname to login. By default Avocado
242 attempts to automatically find the VM IP address.
243 --vm-port VM_PORT Specify the port number to login on VM. Current: 22
244 --vm-username VM_USERNAME
245 Specify the username to login on VM
246 --vm-password VM_PASSWORD
247 Specify the password to login on VM
248 --vm-key-file VM_KEY_FILE
249 Specify an identity file with a private key instead of
250 a password (Example: .pem files from Amazon EC2)
251 --vm-cleanup Restore VM to a previous state, before running tests
252 --vm-timeout SECONDS Amount of time (in seconds) to wait for a successful
253 connection to the virtual machine. Defaults to 120
254 seconds.
255 wrapper support:
257 --wrapper SCRIPT[:EXECUTABLE]
258 Use a script to wrap executables run by a test. The
259 wrapper is either a path to a script (AKA a global
260 wrapper) or a path to a script followed by colon
261 symbol (:), plus a shell like glob to the target
262 EXECUTABLE. Multiple wrapper options are allowed, but
263 only one global wrapper can be defined.
264 yaml to mux options:
266 -m [FILE [FILE ...]], --mux-yaml [FILE [FILE ...]]
267 Location of one or more Avocado multiplex (.yaml)
268 FILE(s) (order dependent)
269 --mux-filter-only [MUX_FILTER_ONLY [MUX_FILTER_ONLY ...]]
270 Filter only path(s) from multiplexing
271 --mux-filter-out [MUX_FILTER_OUT [MUX_FILTER_OUT ...]]
272 Filter out path(s) from multiplexing
273 --mux-path [MUX_PATH [MUX_PATH ...]]
274 List of default paths used to determine path priority
275 when querying for parameters
276 --mux-inject [MUX_INJECT [MUX_INJECT ...]]
277 Inject [path:]key:node values into the final multiplex
278 tree.
279 Options for subcommand config (avocado config --help):
281 optional arguments:
283 -h, --help show this help message and exit
284 --datadir Shows the data directories currently being used by
285 avocado
286 Options for subcommand diff (avocado diff --help):
288 positional arguments:
290 <JOB> A job reference, identified by a (partial) unique ID
291 (SHA1) or test results directory.
292 optional arguments:
294 -h, --help show this help message and exit
295 --html FILE Enable HTML output to the FILE where the result should
296 be written.
297 --open-browser Generate and open a HTML report in your preferred
298 browser. If no --html file is provided, create a
299 temporary file.
300 --diff-filter DIFF_FILTER
301 Comma separated filter of diff sections:
302 (no)cmdline,(no)time,(no)variants,(no)results,
303 (no)config,(no)sysinfo (defaults to all enabled).
304 --create-reports Create temporary files with job reports (to be used by
305 other diff tools)
306 Options for subcommand distro (avocado distro --help):
308 optional arguments:
310 -h, --help show this help message and exit
311 --distro-def-create Creates a distro definition file based on the path
312 given
313 --distro-def-name DISTRO_DEF_NAME
314 Distribution short name
315 --distro-def-version DISTRO_DEF_VERSION
316 Distribution major version number
317 ---distro-def-release DISTRO_DEF_RELEASE
318 Distribution release version number
319 --distro-def-arch DISTRO_DEF_ARCH
320 Primary architecture that the distro targets
321 --distro-def-path DISTRO_DEF_PATH
322 Top level directory of the distro installation files
323 --distro-def-type {deb,rpm}
324 Distro type (one of: deb, rpm)
325 Options for subcommand exec-path (avocado exec-path --help):
327 optional arguments:
329 -h, --help show this help message and exit
330 Options for subcommand list (avocado list --help):
332 positional arguments:
334 reference List of test references (aliases or paths). If empty,
335 avocado will list tests on the configured test source,
336 (see 'avocado config --datadir') Also, if there are
337 other test loader plugins active, tests from those
338 plugins might also show up (behavior may vary among
339 plugins)
340 optional arguments:
342 -h, --help show this help message and exit
343 -V, --verbose Whether to show extra information (headers and
344 summary). Current: False
345 loader options:
347 --loaders [LOADERS [LOADERS ...]]
348 Overrides the priority of the test loaders. You can
349 specify either @loader_name or TEST_TYPE. By default
350 it tries all available loaders according to priority
351 set in settings->plugins.loaders.
352 --external-runner EXECUTABLE
353 Path to an specific test runner that allows the use of
354 its own tests. This should be used for running tests
355 that do not conform to Avocado' SIMPLE testinterface
356 and can not run standalone. Note: the use of
357 --external-runner overwrites the --loaders to
358 "external_runner"
359 --external-runner-chdir {runner,test}
360 Change directory before executing tests. This option
361 may be necessary because of requirements and/or
362 limitations of the external test runner. If the
363 external runner requires to be run from its own base
364 directory,use "runner" here. If the external runner
365 runs tests based on files and requires to be run from
366 the directory where those files are located, use
367 "test" here and specify the test directory with the
368 option "--external-runner-testdir". Defaults to "None"
369 --external-runner-testdir DIRECTORY
370 Where test files understood by the external test
371 runner are located in the filesystem. Obviously this
372 assumes and only applies to external test runners that
373 run tests from files
374 filtering parameters:
376 --filter-by-tags TAGS
377 Filter tests based on tags
378 --filter-by-tags-include-empty
379 Include all tests without tags during filtering. This
380 effectively means they will be kept in the test suite
381 found previously to filtering.
382 --filter-by-tags-include-empty-key
383 Include all tests that do not have a matching key in
384 its key:val tags. This effectively means those tests
385 will be kept in the test suite found previously to
386 filtering.
387 Options for subcommand multiplex (avocado multiplex --help):
389 optional arguments:
391 -h, --help show this help message and exit
392 --summary SUMMARY Verbosity of the variants summary. (positive integer -
393 0, 1, ... - or none, brief, normal, verbose, full,
394 max)
395 --variants VARIANTS Verbosity of the list of variants. (positive integer -
396 0, 1, ... - or none, brief, normal, verbose, full,
397 max)
398 -c, --contents [obsoleted by --variants] Shows the node content
399 (variables)
400 environment view options:
402 -d, --debug Debug the multiplex tree.
403 tree view options:
405 -t, --tree [obsoleted by --summary] Shows the multiplex tree
406 structure
407 -i, --inherit [obsoleted by --summary] Show the inherited values
408 yaml to mux options:
410 -m [FILE [FILE ...]], --mux-yaml [FILE [FILE ...]]
411 Location of one or more Avocado multiplex (.yaml)
412 FILE(s) (order dependent)
413 --mux-filter-only [MUX_FILTER_ONLY [MUX_FILTER_ONLY ...]]
414 Filter only path(s) from multiplexing
415 --mux-filter-out [MUX_FILTER_OUT [MUX_FILTER_OUT ...]]
416 Filter out path(s) from multiplexing
417 --mux-path [MUX_PATH [MUX_PATH ...]]
418 List of default paths used to determine path priority
419 when querying for parameters
420 --mux-inject [MUX_INJECT [MUX_INJECT ...]]
421 Inject [path:]key:node values into the final multiplex
422 tree.
423 Options for subcommand plugins (avocado plugins --help):
425 optional arguments:
427 -h, --help show this help message and exit
428 Options for subcommand sysinfo (avocado sysinfo --help):
430 positional arguments:
432 sysinfodir Dir where to dump sysinfo
433 optional arguments:
435 -h, --help show this help message and exit
436
438 The most common use of the avocado command line tool is to run a test:
439 $ avocado run sleeptest.py
441 This command will run the sleeptest.py test, as found on the standard
443 test directories. The output should be similar to:
444 JOB ID : <id>
446 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
447 (1/1) sleeptest.py:SleepTest.test: PASS (1.01 s)
448 RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
449 JOB TIME : 1.11 s
450 The test directories will vary depending on you system and installation
452 method used. Still, it's pretty easy to find that out as shown in the
453 next section.
454
456 When you are developing new tests, frequently you want to look at the
457 straight output of the job log in the stdout, without having to tail
458 the job log. In order to do that, you can use --show=test to the avo‐
459 cado test runner:
460 $ avocado --show=test run examples/tests/sleeptest.py
462 ...
463 PARAMS (key=timeout, path=*, default=None) => None
464 START 1-sleeptest.py:SleepTest.test
465 PARAMS (key=sleep_length, path=*, default=1) => 1
466 Sleeping for 1.00 seconds
467 Not logging /var/log/messages (lack of permissions)
468 PASS 1-sleeptest.py:SleepTest.test
469 ...
470 Let's say you are debugging a test particularly large, with lots of de‐
472 bug output and you want to reduce this output to only messages with
473 level 'INFO' and higher. You can set job-log-level to info to reduce
474 the amount of output.
475 Edit your ~/.config/avocado/avocado.conf file and add:
477 [job.output]
479 loglevel = info
480 Running the same example with this option will give you:
482 $ avocado --show=test run sleeptest.py
484 ...
485 START 1-sleeptest.py:SleepTest.test
486 PASS 1-sleeptest.py:SleepTest.test
487 ...
488 The levels you can choose are the levels available in the python log‐
490 ging system https://docs.python.org/2/library/logging.html#logging-lev‐
491 els, translated to lowercase strings, so 'notset', 'debug', 'info',
492 'warning', 'error', 'critical', in order of severity.
493 As you can see, the UI output is suppressed and only the job log goes
495 to stdout, making this a useful feature for test development/debugging.
496
498 You may specify --show=none, that means avocado will turn off all run‐
499 ner stdout. Note that --show=none does not affect on disk job logs,
500 those continue to be generated normally.
501
503 You may specify --sysinfo=off and avocado will not collect profilers,
504 hardware details and other system information, inside the job result
505 directory.
506
508 The avocado command line tool also has a list command, that lists the
509 known tests in a given path, be it a path to an individual test, or a
510 path to a directory. If no arguments provided, avocado will inspect the
511 contents of the test location being used by avocado (if you are in
512 doubt about which one is that, you may use avocado config --datadir).
513 The output looks like:
514 $ avocado list
516 INSTRUMENTED /usr/share/doc/avocado/tests/abort.py
517 INSTRUMENTED /usr/share/doc/avocado/tests/datadir.py
518 INSTRUMENTED /usr/share/doc/avocado/tests/doublefail.py
519 INSTRUMENTED /usr/share/doc/avocado/tests/doublefree.py
520 INSTRUMENTED /usr/share/doc/avocado/tests/errortest.py
521 INSTRUMENTED /usr/share/doc/avocado/tests/failtest.py
522 INSTRUMENTED /usr/share/doc/avocado/tests/fiotest.py
523 INSTRUMENTED /usr/share/doc/avocado/tests/gdbtest.py
524 INSTRUMENTED /usr/share/doc/avocado/tests/gendata.py
525 INSTRUMENTED /usr/share/doc/avocado/tests/linuxbuild.py
526 INSTRUMENTED /usr/share/doc/avocado/tests/multiplextest.py
527 INSTRUMENTED /usr/share/doc/avocado/tests/passtest.py
528 INSTRUMENTED /usr/share/doc/avocado/tests/skiptest.py
529 INSTRUMENTED /usr/share/doc/avocado/tests/sleeptenmin.py
530 INSTRUMENTED /usr/share/doc/avocado/tests/sleeptest.py
531 INSTRUMENTED /usr/share/doc/avocado/tests/synctest.py
532 INSTRUMENTED /usr/share/doc/avocado/tests/timeouttest.py
533 INSTRUMENTED /usr/share/doc/avocado/tests/warntest.py
534 INSTRUMENTED /usr/share/doc/avocado/tests/whiteboard.py
535 Here, INSTRUMENTED means that the files there are python files with an
537 avocado test class in them, therefore, that they are what we call in‐
538 strumented tests. This means those tests can use all avocado APIs and
539 facilities. Let's try to list a directory with a bunch of executable
540 shell scripts:
541 $ avocado list examples/wrappers/
543 SIMPLE examples/wrappers/dummy.sh
544 SIMPLE examples/wrappers/ltrace.sh
545 SIMPLE examples/wrappers/perf.sh
546 SIMPLE examples/wrappers/strace.sh
547 SIMPLE examples/wrappers/time.sh
548 SIMPLE examples/wrappers/valgrind.sh
549 Here, SIMPLE means that those files are executables, that avocado will
551 simply execute and return PASS or FAIL depending on their return codes
552 (PASS -> 0, FAIL -> any integer different than 0). You can also provide
553 the --verbose, or -V flag to display files that were detected but are
554 not avocado tests, along with summary information:
555 $ avocado list examples/gdb-prerun-scripts/ -V
557 Type Test Tag(s)
558 NOT_A_TEST examples/gdb-prerun-scripts/README
559 NOT_A_TEST examples/gdb-prerun-scripts/pass-sigusr1
560 TEST TYPES SUMMARY
562 ==================
563 SIMPLE: 0
564 INSTRUMENTED: 0
565 MISSING: 0
566 NOT_A_TEST: 2
567 That summarizes the basic commands you should be using more frequently
569 when you start with avocado. Let's talk now about how avocado stores
570 test results.
571
573 When avocado runs tests, it saves all its results on your system:
574 JOB ID : <id>
576 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
577 For your convenience, avocado maintains a link to the latest job run
579 (an avocado run command in this context), so you can always use "lat‐
580 est" to browse your test results:
581 $ ls /home/<user>/avocado/job-results/latest
583 id
584 jobdata
585 job.log
586 results.json
587 results.tap
588 results.xml
589 sysinfo
590 test-results
591 The main log file is job.log, but every test has its own results direc‐
593 tory:
594 $ ls -1 ~/avocado/job-results/latest/test-results/
596 1-sleeptest.py:SleepTest.test
597 Since this is a directory, it should have content similar to:
599 $ ls -1 ~/avocado/job-results/latest/test-results/1-sleeptest.py\:SleepTest.test/
601 data
602 debug.log
603 stderr
604 stdout
605 sysinfo
606 whiteboard
607
609 Avocado has a powerful tool that enables multiple test scenarios to be
610 run using a single, unmodified test. This mechanism uses a YAML file
611 called the 'multiplex file', that tells avocado how to multiply all
612 possible test scenarios automatically.
613 A command by the same name, multiplex, is available on the avocado com‐
615 mand line tool, and enables you to see all the test scenarios that can
616 be run:
617 $ avocado multiplex -m examples/tests/sleeptest.py.data/sleeptest.yaml -c
619 Variants generated:
620 Variant 1: /run/short
622 /run/short:sleep_length => 0.5
623 Variant 2: /run/medium
625 /run/medium:sleep_length => 1
626 Variant 3: /run/long
628 /run/long:sleep_length => 5
629 Variant 4: /run/longest
631 /run/longest:sleep_length => 10
632 This is a sample that varies the parameter sleep_length through the
634 scenarios /run/short (sleeps for 0.5 s), /run/medium (sleeps for 1 s),
635 /run/long (sleeps for 5s), /run/longest (sleeps for 10s). The YAML
636 file (multiplex file) that produced the output above is:
637 !mux
639 short:
640 sleep_length: 0.5
641 medium:
642 sleep_length: 1
643 long:
644 sleep_length: 5
645 longest:
646 sleep_length: 10
647 You can execute sleeptest in all variations exposed above with:
649 $ avocado run sleeptest.py -m examples/tests/sleeptest.py.data/sleeptest.yaml
651 And the output should look like:
653 JOB ID : <id>
655 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
656 (1/4) sleeptest.py:SleepTest.test;1: PASS (0.51 s)
657 (2/4) sleeptest.py:SleepTest.test;2: PASS (1.01 s)
658 (3/4) sleeptest.py:SleepTest.test;3: PASS (5.02 s)
659 (4/4) sleeptest.py:SleepTest.test;4: PASS (10.01 s)
660 RESULTS : PASS 4 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
661 JOB TIME : 16.65 s
662 The multiplex plugin and the test runner supports two kinds of global
664 filters, through the command line options --mux-filter-only and
665 --mux-filter-out. The mux-filter-only exclusively includes one or more
666 paths and the mux-filter-out removes one or more paths from being pro‐
667 cessed.
668 From the previous example, if we are interested to use the variants
670 /run/medium and /run/longest, we do the following command line:
671 $ avocado run sleeptest.py -m examples/tests/sleeptest.py.data/sleeptest.yaml \
673 --mux-filter-only /run/medium /run/longest
674 And if you want to remove /small from the variants created, we do the
676 following:
677 $ avocado run sleeptest.py -m examples/tests/sleeptest.py.data/sleeptest.yaml \
679 --mux-filter-out /run/medium
680 Note that both --mux-filter-only and --mux-filter-out filters can be
682 arranged in the same command line.
683 The multiplexer also supports default paths. The base path is /run/*
685 but it can be overridden by --mux-path, which accepts multiple argu‐
686 ments. What it does: it splits leaves by the provided paths. Each query
687 goes one by one through those sub-trees and first one to hit the match
688 returns the result. It might not solve all problems, but it can help to
689 combine existing YAML files with your ones:
690 qa: # large and complex read-only file, content injected into /qa
692 tests:
693 timeout: 10
694 ...
695 my_variants: !mux # your YAML file injected into /my_variants
696 short:
697 timeout: 1
698 long:
699 timeout: 1000
700 You want to use an existing test which uses params.get('timeout', '*').
702 Then you can use --mux-path '/my_variants/*' '/qa/*' and it'll first
703 look in your variants. If no matches are found, then it would proceed
704 to /qa/*
705 Keep in mind that only slices defined in mux-path are taken into ac‐
707 count for relative paths (the ones starting with *).
708
710 One interesting avocado feature is the ability to automatically and
711 transparently run executables that are used on a given test inside the
712 GNU debugger.
713 Suppose you are running a test that uses an external, compiled, image
715 converter. Now suppose you're feeding it with different types of im‐
716 ages, including broken image files, and it fails at a given point. You
717 wish you could connect to the debugger at that given source location
718 while your test is running. This is how to do just that with avocado:
719 $ avocado run --gdb-run-bin=convert:convert_ppm_to_raw converttest.py
721 The job starts running just as usual, and so does your test:
723 JOB ID : <id>
725 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
726 TESTS : 1
727 (1/1) converttest.py:ConvertTest.test: /
728 The convert executable though, automatically runs inside GDB. Avocado
730 will stop when the given breakpoint is reached:
731 TEST PAUSED because of debugger breakpoint. To DEBUG your application run:
733 /home/<user>/avocado/job-results/job-<date>-<shortid>/test-results/converttest.py/data/convert.gdb.sh
734 NOTE: please use *disconnect* command in gdb before exiting, or else the debugged process will be KILLED
736 From this point, you can run the generated script (convert.gdb.sh) to
738 debug you application.
739 As noted, it is strongly recommended that you disconnect from gdb while
741 your executable is still running. That is, if the executable finished
742 running while you are debugging it, avocado has no way to know about
743 its status.
744 Avocado will automatically send a continue command to the debugger when
746 you disconnect from and exit gdb.
747 If, for some reason you have a custom GDB, or your system does not put
749 GDB on what avocado believes to be the standard location
750 (/usr/bin/gdb), you can override that in the section gdb.paths of your
751 documentation:
752 [gdb.paths]
754 gdb = /usr/bin/gdb
755 gdbserver = /usr/bin/gdbserver
756 So running avocado after setting those will use the appropriate
758 gdb/gdbserver path.
759 If you are debugging a special application and need to setup GDB in
761 custom ways by running GDB commands, you can do that with the
762 --gdb-prerun-commands option:
763 $ avocado run --gdb-run-bin=foo:bar --gdb-prerun-commands=/tmp/disable-signals footest.py
765 In this example, /tmp/disable-signals is a simple text file containing
767 two lines:
768 signal SIGUSR1 pass
770 signal SIGUSR1 nostop
771 Each line is a GDB command, so you can have from simple to very complex
773 debugging environments configured like that.
774
776 Avocado allows the instrumentation of executables being run by a test
777 in a transparent way. The user specifies a script ("the wrapper") to be
778 used to run the actual program called by the test.
779 If the instrumentation script is implemented correctly, it should not
781 interfere with the test behavior. That is, the wrapper should avoid
782 changing the return status, standard output and standard error messages
783 of the original executable.
784 The user can be specific about which program to wrap (with a shell-like
786 glob), or if that is omitted, a global wrapper that will apply to all
787 programs called by the test.
788 So, for every executable run by the test, the program name will be com‐
790 pared to the pattern to decide whether to wrap it or not. You can have
791 multiples wrappers and patterns defined.
792 Examples:
794 $ avocado run datadir.py --wrapper examples/wrappers/strace.sh
796 Any command created by the test datadir will be wrapped on strace.sh.
798 $ avocado run datadir.py --wrapper examples/wrappers/ltrace.sh:*make \
800 --wrapper examples/wrappers/perf.sh:*datadir
801 Any command that matches the pattern *make will be wrapper on ltrace.sh
803 and the pattern *datadir will trigger the execution of perf.sh.
804 Note that it is not possible to use --gdb-run-bin together with --wrap‐
806 per, they are incompatible.
807
809 It's quite common to have organically grown test suites in most soft‐
810 ware projects. These usually include a custom built, very specific test
811 runner that knows how to find and run their own tests.
812 Still, running those tests inside Avocado may be a good idea for vari‐
814 ous reasons, including being able to have results in different human
815 and machine readable formats, collecting system information alongside
816 those tests (the Avocado's sysinfo functionality), and more.
817 Avocado makes that possible by means of its "external runner" feature.
819 The most basic way of using it is:
820 $ avocado run --external-runner=/path/to/external_runner foo bar baz
822 In this example, Avocado will report individual test results for tests
824 foo, bar and baz. The actual results will be based on the return code
825 of individual executions of /path/to/external_runner foo, /path/to/ex‐
826 ternal_runner bar and finally /path/to/external_runner baz.
827 As another way to explain an show how this feature works, think of the
829 "external runner" as some kind of interpreter and the individual tests
830 as anything that this interpreter recognizes and is able to execute. A
831 UNIX shell, say /bin/sh could be considered an external runner, and
832 files with shell code could be considered tests:
833 $ echo "exit 0" > /tmp/pass
835 $ echo "exit 1" > /tmp/fail
836 $ avocado run --external-runner=/bin/sh /tmp/pass /tmp/fail
837 JOB ID : <id>
838 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
839 TESTS : 2
840 (1/2) /tmp/pass: PASS (0.01 s)
841 (2/2) /tmp/fail: FAIL (0.01 s)
842 RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0
843 JOB TIME : 0.11 s
844 This example is pretty obvious, and could be achieved by giving
846 /tmp/pass and /tmp/fail shell "shebangs" (#!/bin/sh), making them exe‐
847 cutable (chmod +x /tmp/pass /tmp/fail), and running them as "SIMPLE"
848 tests.
849 But now consider the following example:
851 $ avocado run --external-runner=/bin/curl http://local-avocado-server:9405/jobs/ \
853 http://remote-avocado-server:9405/jobs/
854 JOB ID : <id>
855 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
856 TESTS : 2
857 (1/2) http://local-avocado-server:9405/jobs/: PASS (0.02 s)
858 (2/2) http://remote-avocado-server:9405/jobs/: FAIL (3.02 s)
859 RESULTS : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0
860 JOB TIME : 3.14 s
861 This effectively makes /bin/curl an "external test runner", responsible
863 for trying to fetch those URLs, and reporting PASS or FAIL for each of
864 them.
865
867 As a tester, you may want to check if the output of a given application
868 matches an expected output. In order to help with this common use case,
869 we offer the option --output-check-record [mode] to the test runner.
870 If this option is used, it will store the stdout or stderr of the
871 process (or both, if you specified all) being executed to reference
872 files: stdout.expected and stderr.expected.
873 Those files will be recorded in the test data dir. The data dir is in
875 the same directory as the test source file, named
876 [source_file_name.data]. Let's take as an example the test synctest.py.
877 In a fresh checkout of avocado, you can see:
878 examples/tests/synctest.py.data/stderr.expected
880 examples/tests/synctest.py.data/stdout.expected
881 From those 2 files, only stdout.expected is non empty:
883 $ cat examples/tests/synctest.py.data/stdout.expected
885 PAR : waiting
886 PASS : sync interrupted
887 The output files were originally obtained using the test runner and
889 passing the option --output-check-record all to the test runner:
890 $ avocado run --output-check-record all examples/tests/synctest.py
892 JOB ID : <id>
893 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
894 (1/1) examples/tests/synctest.py:SyncTest.test: PASS (4.00 s)
895 RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
896 JOB TIME : 4.10 s
897 After the reference files are added, the check process is transparent,
899 in the sense that you do not need to provide special flags to the test
900 runner. Now, every time the test is executed, after it is done running,
901 it will check if the outputs are exactly right before considering the
902 test as PASSed. If you want to override the default behavior and skip
903 output check entirely, you may provide the flag --output-check=off to
904 the test runner.
905 The avocado.utils.process APIs have a parameter allow_output_check (de‐
907 faults to all), so that you can select which process outputs will go to
908 the reference files, should you chose to record them. You may choose
909 all, for both stdout and stderr, stdout, for the stdout only, stderr,
910 for only the stderr only, or none, to allow neither of them to be
911 recorded and checked.
912 This process works fine also with simple tests, executables that return
914 0 (PASSed) or != 0 (FAILed). Let's consider our bogus example:
915 $ cat output_record.sh
917 #!/bin/bash
918 echo "Hello, world!"
919 Let's record the output (both stdout and stderr) for this one:
921 $ avocado run output_record.sh --output-check-record all
923 JOB ID : <id>
924 JOB LOG : /home/<user>/avocado/job-results/job-<date>-<shortid>/job.log
925 TESTS : 1
926 (1/1) home/$USER/Code/avocado/output_record.sh: PASS (0.01 s)
927 RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
928 JOB TIME : 0.11 s
929 After this is done, you'll notice that a the test data directory ap‐
931 peared in the same level of our shell script, containing 2 files:
932 $ ls output_record.sh.data/
934 stderr.expected stdout.expected
935 Let's look what's in each of them:
937 $ cat output_record.sh.data/stdout.expected
939 Hello, world!
940 $ cat output_record.sh.data/stderr.expected
941 $
942 Now, every time this test runs, it'll take into account the expected
944 files that were recorded, no need to do anything else but run the test.
945
947 Avocado has some planned features that depend on knowing the Linux Dis‐
948 tribution being used on the system. The most basic command prints the
949 detected Linux Distribution:
950 $ avocado distro
952 Detected distribution: fedora (x86_64) version 21 release 0
953 Other features are available with the same command when command line
955 options are given, as shown by the --help option.
956 For instance, it possible to create a so-called "Linux Distribution
958 Definition" file, by inspecting an installation tree. The installation
959 tree could be the contents of the official installation ISO or a local
960 network mirror.
961 These files let Avocado pinpoint if a given installed package is part
963 of the original Linux Distribution or something else that was installed
964 from an external repository or even manually. This, in turn, can help
965 detecting regressions in base system pacakges that affected a given
966 test result.
967 To generate a definition file run:
969 $ avocado distro --distro-def-create --distro-def-name avocadix \
971 --distro-def-version 1 --distro-def-arch x86_64 \
972 --distro-def-type rpm --distro-def-path /mnt/dvd
973 And the output will be something like:
975 Loading distro information from tree... Please wait...
977 Distro information saved to "avocadix-1-x86_64.distro"
978
980 System wide configuration file
981 /etc/avocado/avocado.conf
982 Extra configuration files
984 /etc/avocado/conf.d/
985 User configuration file
987 ~/.config/avocado/avocado.conf
988
990 If you find a bug, please report it over our github page as an issue:
991 https://github.com/avocado-framework/avocado/issues
992
994 Avocado is released under GPLv2 (explicit version) http://gnu.org/li‐
995 censes/gpl-2.0.html. Even though most of the current code is licensed
996 under a "and any later version" clause, some parts are specifically
997 bound to the version 2 of the license and therefore that's the official
998 license of the prject itself. For more details, please see the LICENSE
999 file in the project source code directory.
1000
1002 For more information please check Avocado's project website, located at
1003 http://avocado-framework.github.io/. There you'll find links to online
1004 documentation, source code and community resources.
1005
1007 Avocado Development Team <avocado-devel@redhat.com>
1008 AVOCADO(1)