1AVOCADO(1)                                                          AVOCADO(1)
2
3
4

NAME

6       avocado - test runner command line tool
7

SYNOPSIS

9       avocado [-h] [-v] [--config [CONFIG_FILE]] [--show [STREAM[:LVL]]] [-s]
10              {config,diff,distro,exec-path,list,multiplex,plugins,run,sys‐
11              info} ...
12

DESCRIPTION

14       Avocado is a modern test framework that is built on the experience  ac‐
15       cumulated with autotest (https://autotest.github.io).
16
17       avocado  is  also  the  name  of its test runner command line tool, de‐
18       scribed in this man page.
19
20       For more information about the Avocado project, please check  its  web‐
21       site: https://avocado-framework.github.io/
22

OPTIONS

24       The  following  list  of options are builtin, application level avocado
25       options. Most other options are implemented via plugins and will depend
26       on them being loaded (avocado --help):
27
28          -h, --help            show this help message and exit
29          -v, --version         show program's version number and exit
30          --config [CONFIG_FILE]
31                                Use custom configuration from a file
32          --enable-paginator    Turn the paginator on. Useful when output is too long.
33          -V, --verbose         Some commands can produce more information. This
34                                option will enable the verbosity when applicable.
35          --show CORE.SHOW      List of comma separated builtin logs, or logging
36                                streams optionally followed by LEVEL (DEBUG,INFO,...).
37                                Builtin streams are: "app": application output;
38                                "test": test output; "debug": tracebacks and other
39                                debugging info; "early": early logging of other
40                                streams, including test (very verbose); "all": all
41                                builtin streams; "none": disables regular output
42                                (leaving only errors enabled). By default: 'app'
43
44       Real use of avocado depends on running avocado subcommands. This a typ‐
45       ical list of avocado subcommands:
46
47          assets              Manage assets
48          config              Shows avocado config keys
49          diff                Shows the difference between 2 jobs.
50          distro              Shows detected Linux distribution
51          exec-path           Returns path to avocado bash libraries and exits.
52          jobs                Manage Avocado jobs
53          list                List available tests
54          plugins             Displays plugin information
55          replay              Runs a new job using a previous job as its
56                              configuration
57          run                 Runs one or more tests (native test, test alias,
58                              binary or script)
59          sysinfo             Collect system information
60          variants            Tool to analyze and visualize test variants and params
61          vmimage             Provides VM images acquired from official repositories
62
63       To get usage instructions for a given subcommand, run it  with  --help.
64       Example:
65
66          $ avocado run --help
67
68       Options for subcommand run (avocado run --help):
69
70          positional arguments:
71            TEST_REFERENCE        List of test references (aliases or paths)
72
73          optional arguments:
74            -h, --help            show this help message and exit
75            -p NAME_VALUE, --test-parameter NAME_VALUE
76                                  Parameter name and value to pass to all tests. This is
77                                  only applicable when not using a varianter plugin.
78                                  This option format must be given in the NAME=VALUE
79                                  format, and may be given any number of times, or per
80                                  parameter.
81            --test-runner TEST_RUNNER
82                                  Selects the runner implementation from one of the
83                                  installed and active implementations. You can run
84                                  "avocado plugins" and find the list of valid runners
85                                  under the "Plugins that run test suites on a job
86                                  (runners) section. Defaults to "nrunner", which is the
87                                  new runner. To use the conventional and traditional
88                                  runner, use "runner".
89            -d, --dry-run         Instead of running the test only list them and log
90                                  their params.
91            --dry-run-no-cleanup  Do not automatically clean up temporary directories
92                                  used by dry-run
93            --force-job-id UNIQUE_JOB_ID
94                                  Forces the use of a particular job ID. Used internally
95                                  when interacting with an avocado server. You should
96                                  not use this option unless you know exactly what
97                                  you're doing
98            --job-results-dir DIRECTORY
99                                  Forces to use of an alternate job results directory.
100            --job-category CATEGORY
101                                  Categorizes this within a directory with the same
102                                  name, by creating a link to the job result directory
103            --job-timeout SECONDS
104                                  Set the maximum amount of time (in SECONDS) that tests
105                                  are allowed to execute. Values <= zero means "no
106                                  timeout". You can also use suffixes, like: s
107                                  (seconds), m (minutes), h (hours).
108            --failfast            Enable the job interruption on first failed test.
109            --keep-tmp            Keep job temporary files (useful for avocado
110                                  debugging).
111            --ignore-missing-references
112                                  Force the job execution, even if some of the test
113                                  references are not resolved to tests. "on" and "off"
114                                  will be deprecated soon.
115            --disable-sysinfo     Enable or disable sysinfo information. Like hardware
116                                  details, profiles, etc.
117            --execution-order RUN.EXECUTION_ORDER
118                                  Defines the order of iterating through test suite and
119                                  test variants
120            --log-test-data-directories
121                                  Logs the possible data directories for each test. This
122                                  is helpful when writing new tests and not being sure
123                                  where to put data files. Look for "Test data
124                                  directories" in your test log
125            --journal             Records test status changes (for use with avocado-
126                                  journal-replay and avocado-server)
127            --json FILE           Enable JSON result format and write it to FILE. Use
128                                  "-" to redirect to the standard output.
129            --disable-json-job-result
130                                  Enables default JSON result in the job results
131                                  directory. File will be named "results.json".
132            --tap FILE            Enable TAP result output and write it to FILE. Use "-"
133                                  to redirect to standard output.
134            --disable-tap-job-result
135                                  Enables default TAP result in the job results
136                                  directory. File will be named "results.tap"
137            --tap-include-logs    Include test logs as comments in TAP output.
138            -z, --archive         Archive (ZIP) files generated by tests
139
140          output and result format:
141            --store-logging-stream LOGGING_STREAM
142                                  Store given logging STREAMs in
143                                  "$JOB_RESULTS_DIR/$STREAM.$LEVEL."
144            --xunit FILE          Enable xUnit result format and write it to FILE. Use
145                                  "-" to redirect to the standard output.
146            --disable-xunit-job-result
147                                  Enables default xUnit result in the job results
148                                  directory. File will be named "results.xml".
149            --xunit-job-name XUNIT_JOB_NAME
150                                  Override the reported job name. By default uses the
151                                  Avocado job name which is always unique. This is
152                                  useful for reporting in Jenkins as it only evaluates
153                                  first-failure from jobs of the same name.
154            --xunit-max-test-log-chars SIZE
155                                  Limit the attached job log to given number of
156                                  characters (k/m/g suffix allowed)
157
158          output check arguments:
159            --output-check-record {none,stdout,stderr,both,combined,all}
160                                  Record the output produced by each test (from stdout
161                                  and stderr) into both the current executing result and
162                                  into reference files. Reference files are used on
163                                  subsequent runs to determine if the test produced the
164                                  expected output or not, and the current executing
165                                  result is used to check against a previously recorded
166                                  reference file. Valid values: "none" (to explicitly
167                                  disable all recording) "stdout" (to record standard
168                                  output *only*), "stderr" (to record standard error
169                                  *only*), "both" (to record standard output and error
170                                  in separate files), "combined" (for standard output
171                                  and error in a single file). "all" is also a valid but
172                                  deprecated option that is a synonym of "both".
173            --disable-output-check
174                                  Disables test output (stdout/stderr) check. If this
175                                  option is given, no output will be checked, even if
176                                  there are reference files present for the test.
177
178          loader options:
179            --loaders LOADER_NAME_OR_TEST_TYPE [LOADER_NAME_OR_TEST_TYPE ...]
180                                  Overrides the priority of the test loaders. You can
181                                  specify either @loader_name or TEST_TYPE. By default
182                                  it tries all available loaders according to priority
183                                  set in settings->plugins.loaders.
184            --external-runner EXTERNAL_RUNNER
185                                  Path to an specific test runner that allows the use of
186                                  its own tests. This should be used for running tests
187                                  that do not conform to Avocado's SIMPLE test interface
188                                  and can not run standalone. Note: the use of
189                                  --external-runner overwrites the --loaders to
190                                  'external_runner'
191            --external-runner-chdir {runner,test}
192                                  Change directory before executing tests. This option
193                                  may be necessary because of requirements and/or
194                                  limitations of the external test runner. If the
195                                  external runner requires to be run from its own base
196                                  directory, use 'runner' here. If the external runner
197                                  runs tests based on files and requires to be run from
198                                  the directory where those files are located, use
199                                  'test' here and specify the test directory with the
200                                  option '--external-runner-testdir'.
201            --external-runner-testdir DIRECTORY
202                                  Where test files understood by the external test
203                                  runner are located in the filesystem. Obviously this
204                                  assumes and only applies to external test runners that
205                                  run tests from files
206
207          filtering parameters:
208            -t TAGS, --filter-by-tags TAGS
209                                  Filter tests based on tags
210            --filter-by-tags-include-empty
211                                  Include all tests without tags during filtering. This
212                                  effectively means they will be kept in the test suite
213                                  found previously to filtering.
214            --filter-by-tags-include-empty-key
215                                  Include all tests that do not have a matching key in
216                                  its key:val tags. This effectively means those tests
217                                  will be kept in the test suite found previously to
218                                  filtering.
219
220          JSON serialized based varianter options:
221            --json-variants-load FILE
222                                  Load the Variants from a JSON serialized file
223
224          nrunner specific options:
225            --nrunner-shuffle     Shuffle the tasks to be executed
226            --nrunner-status-server-listen HOST_PORT
227                                  URI for listing the status server. Usually a
228                                  "HOST:PORT" string
229            --nrunner-status-server-uri HOST_PORT
230                                  URI for connecting to the status server, usually a
231                                  "HOST:PORT" string. Use this if your status server is
232                                  in another host, or different port
233            --nrunner-max-parallel-tasks NUMBER_OF_TASKS
234                                  Number of maximum number tasks running in parallel.
235                                  You can disable parallel execution by setting this to
236                                  1. Defaults to the amount of CPUs on this machine.
237            --nrunner-spawner SPAWNER
238                                  Spawn tasks in a specific spawner. Available spawners:
239                                  'process' and 'podman'
240
241          podman spawner specific options:
242            --spawner-podman-bin PODMAN_BIN
243                                  Path to the podman binary
244            --spawner-podman-image CONTAINER_IMAGE
245                                  Image name to use when creating the container
246
247          job replay:
248            --replay JOB_ID       Replay a job identified by its (partial) hash id. Use
249                                  "--replay" latest to replay the latest job.
250            --replay-test-status TEST_STATUS
251                                  Filter tests to replay by test status.
252            --replay-ignore IGNORE
253                                  Ignore variants and/or configuration from the source
254                                  job.
255            --replay-resume       Resume an interrupted job
256
257          wrapper support:
258            --wrapper SCRIPT[:EXECUTABLE]
259                                  Use a script to wrap executables run by a test. The
260                                  wrapper is either a path to a script (AKA a global
261                                  wrapper) or a path to a script followed by colon
262                                  symbol (:), plus a shell like glob to the target
263                                  EXECUTABLE. Multiple wrapper options are allowed, but
264                                  only one global wrapper can be defined.
265
266       Options for subcommand assets (avocado assets --help):
267
268          positional arguments:
269            {fetch,register,purge,list}
270              fetch               Fetch assets from test source or config file if it's
271                                  not already in the cache
272              register            Register an asset directly to the cacche
273              purge               Removes assets cached locally.
274              list                List all cached assets.
275
276          optional arguments:
277            -h, --help            show this help message and exit
278
279       Options for subcommand config (avocado config --help):
280
281          positional arguments:
282            sub-command
283              reference  Show a configuration reference with all registered options
284
285          optional arguments:
286            -h, --help   show this help message and exit
287            --datadir    Shows the data directories currently being used by Avocado
288
289       Options for subcommand diff (avocado diff --help):
290
291          positional arguments:
292            JOB                   A job reference, identified by a (partial) unique ID
293                                  (SHA1) or test results directory.
294
295          optional arguments:
296            -h, --help            show this help message and exit
297            --html FILE           Enable HTML output to the FILE where the result should
298                                  be written.
299            --open-browser        Generate and open a HTML report in your preferred
300                                  browser. If no --html file is provided, create a
301                                  temporary file.
302            --diff-filter DIFF_FILTER
303                                  Comma separated filter of diff sections:
304                                  (no)cmdline,(no)time,(no)variants,(no)results,
305                                  (no)config,(no)sysinfo (defaults to all enabled).
306            --diff-strip-id       Strip the "id" from "id-name;variant" when comparing
307                                  test results.
308            --create-reports      Create temporary files with job reports to be used by
309                                  other diff tools
310
311          By default, a textual diff report is generated in the standard output.
312
313       Options for subcommand distro (avocado distro --help):
314
315          optional arguments:
316            -h, --help            show this help message and exit
317            --distro-def-create   Creates a distro definition file based on the path
318                                  given.
319            --distro-def-name DISTRO_DEF_NAME
320                                  Distribution short name
321            --distro-def-version DISTRO_DEF_VERSION
322                                  Distribution major version name
323            --distro-def-release DISTRO_DEF_RELEASE
324                                  Distribution release version number
325            --distro-def-arch DISTRO_DEF_ARCH
326                                  Primary architecture that the distro targets
327            --distro-def-path DISTRO.DISTRO_DEF_PATH
328                                  Top level directory of the distro installation files
329            --distro-def-type {rpm,deb}
330                                  Distro type (one of: rpm, deb)
331
332       Options for subcommand exec-path (avocado exec-path --help):
333
334          optional arguments:
335            -h, --help  show this help message and exit
336
337       Options for subcommand jobs (avocado jobs --help):
338
339          positional arguments:
340            sub-command
341              list            List all known jobs by Avocado
342              show            Show details about a specific job. When passing a Job ID,
343                              you can use any Job Reference (job_id, "latest", or job
344                              results path).
345              get-output-files
346                              Download output files generated by tests on
347                              AVOCADO_TEST_OUTPUTDIR
348
349          optional arguments:
350            -h, --help        show this help message and exit
351
352       Options for subcommand list (avocado list --help):
353
354          positional arguments:
355            references            List of test references (aliases or paths). If empty,
356                                  Avocado will list tests on the configured test source,
357                                  (see "avocado config --datadir") Also, if there are
358                                  other test loader plugins active, tests from those
359                                  plugins might also show up (behavior may vary among
360                                  plugins)
361
362          optional arguments:
363            -h, --help            show this help message and exit
364            --resolver            Uses the Avocado resolver method (part of the nrunner
365                                  architecture) to detect tests. This is enabled by
366                                  default and exists only for compatibility purposes,
367                                  and will be removed soon. To use the legacy (loader)
368                                  method for finding tests, set the "--loader" option
369            --loader              Uses the Avocado legacy (loader) method for finding
370                                  tests. This option will exist only for a transitional
371                                  period until the legacy (loader) method is deprecated
372                                  and removed
373            --write-recipes-to-directory DIRECTORY
374                                  Writes runnable recipe files to a directory. Valid
375                                  only when using --resolver.
376            --json JSON_FILE      Writes output to a json file.
377
378          loader options:
379            --loaders LOADER_NAME_OR_TEST_TYPE [LOADER_NAME_OR_TEST_TYPE ...]
380                                  Overrides the priority of the test loaders. You can
381                                  specify either @loader_name or TEST_TYPE. By default
382                                  it tries all available loaders according to priority
383                                  set in settings->plugins.loaders.
384            --external-runner EXTERNAL_RUNNER
385                                  Path to an specific test runner that allows the use of
386                                  its own tests. This should be used for running tests
387                                  that do not conform to Avocado's SIMPLE test interface
388                                  and can not run standalone. Note: the use of
389                                  --external-runner overwrites the --loaders to
390                                  'external_runner'
391            --external-runner-chdir {runner,test}
392                                  Change directory before executing tests. This option
393                                  may be necessary because of requirements and/or
394                                  limitations of the external test runner. If the
395                                  external runner requires to be run from its own base
396                                  directory, use 'runner' here. If the external runner
397                                  runs tests based on files and requires to be run from
398                                  the directory where those files are located, use
399                                  'test' here and specify the test directory with the
400                                  option '--external-runner-testdir'.
401            --external-runner-testdir DIRECTORY
402                                  Where test files understood by the external test
403                                  runner are located in the filesystem. Obviously this
404                                  assumes and only applies to external test runners that
405                                  run tests from files
406
407          filtering parameters:
408            -t TAGS, --filter-by-tags TAGS
409                                  Filter tests based on tags
410            --filter-by-tags-include-empty
411                                  Include all tests without tags during filtering. This
412                                  effectively means they will be kept in the test suite
413                                  found previously to filtering.
414            --filter-by-tags-include-empty-key
415                                  Include all tests that do not have a matching key in
416                                  its key:val tags. This effectively means those tests
417                                  will be kept in the test suite found previously to
418                                  filtering.
419
420       Options for subcommand plugins (avocado plugins --help):
421
422          optional arguments:
423            -h, --help            show this help message and exit
424
425       Options for subcommand replay (avocado reply --help):
426
427          positional arguments:
428            SOURCE_JOB_ID  Replays a job, identified by: complete or partial Job ID,
429                           "latest" for the latest job, the job results path.
430
431          optional arguments:
432            -h, --help     show this help message and exit
433
434       Options for subcommand sysinfo (avocado sysinfo --help):
435
436          positional arguments:
437            sysinfodir  Directory where Avocado will dump sysinfo data. If one is not
438                        given explicitly, it will default to a directory named
439                        "sysinfo-" followed by a timestamp in the current working
440                        directory.
441
442          optional arguments:
443            -h, --help  show this help message and exit
444
445       Options for subcommand variants (avocado variants --help):
446
447          optional arguments:
448            -h, --help            show this help message and exit
449            --summary SUMMARY     Verbosity of the variants summary. (positive integer -
450                                  0, 1, ... - or none, brief, normal, verbose, full,
451                                  max)
452            --variants VARIANTS   Verbosity of the list of variants. (positive integer -
453                                  0, 1, ... - or none, brief, normal, verbose, full,
454                                  max)
455            -c, --contents        [obsoleted by --variants] Shows the node content
456                                  (variables)
457            --json-variants-dump FILE
458                                  Dump the Variants to a JSON serialized file
459
460          environment view options:
461            -d, --debug           Use debug implementation to gather more information.
462
463          tree view options:
464            -t, --tree            [obsoleted by --summary] Shows the multiplex tree
465                                  structure
466            -i, --inherit         [obsoleted by --summary] Show the inherited values
467
468          JSON serialized based varianter options:
469            --json-variants-load FILE
470                                  Load the Variants from a JSON serialized file
471
472       Options for subcommand vmimage (avocado vmimage --help):
473
474          positional arguments:
475            {list,get}
476              list      List of all downloaded images
477              get       Downloads chosen VMimage if it's not already in the cache
478
479          optional arguments:
480            -h, --help  show this help message and exit
481

RUNNING A TEST

483       The most common use of the avocado command line tool is to run a test:
484
485          $ avocado run examples/tests/sleeptest.py
486
487       This  command  will run the sleeptest.py test, as found on the standard
488       test directories. The output should be similar to:
489
490          JOB ID    : <id>
491          JOB LOG   : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
492           (1/1) sleeptest.py:SleepTest.test: PASS (1.01 s)
493          RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
494          JOB TIME   : 1.11 s
495
496       The test directories will vary depending on you system and installation
497       method  used.  Still, it's pretty easy to find that out as shown in the
498       next section.
499

DEBUGGING TESTS

501       When you are developing new tests, frequently you want to look  at  the
502       straight  output  of  the job log in the stdout, without having to tail
503       the job log. In order to do that, you can use --show=test to  the  avo‐
504       cado test runner:
505
506          $ avocado --show=test run examples/tests/sleeptest.py
507          ...
508          PARAMS (key=timeout, path=*, default=None) => None
509          Test metadata:
510            filename: /home/user/avocado/examples/tests/sleeptest.py
511            teststmpdir: /var/tmp/avocado_o98elmi0
512            workdir: /var/tmp/avocado_iyzcj3hn/avocado_job_mwikfsnl/1-examples_tests_sleeptest.py_SleepTest.test
513          START 1-examples/tests/sleeptest.py:SleepTest.test
514          DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
515          PARAMS (key=sleep_length, path=*, default=1) => 1
516          Sleeping for 1.00 seconds
517          DATA (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
518          DATA (filename=stdout.expected) => NOT FOUND (data sources: variant, test, file)
519          DATA (filename=stderr.expected) => NOT FOUND (data sources: variant, test, file)
520          PASS 1-examples/tests/sleeptest.py:SleepTest.test
521          ...
522
523       Let's say you are debugging a test particularly large, with lots of de‐
524       bug output and you want to reduce this output  to  only  messages  with
525       level  'INFO'  and  higher. You can set job-log-level to info to reduce
526       the amount of output.
527
528       Edit your ~/.config/avocado/avocado.conf file and add:
529
530          [job.output]
531          loglevel = INFO
532
533       Running the same example with this option will give you:
534
535          $ avocado --show=test run examples/tests/sleeptest.py
536          ...
537          START 1-examples/tests/sleeptest.py:SleepTest.test
538          PASS 1-examples/tests/sleeptest.py:SleepTest.test
539          ...
540
541       The levels you can choose are the levels available in the  python  log‐
542       ging system https://docs.python.org/3/library/logging.html#logging-lev‐
543       els, so 'NOTSET', 'DEBUG', 'INFO', 'WARNING', 'ERROR',  'CRITICAL',  in
544       order of severity.
545
546       As  you  can see, the UI output is suppressed and only the job log goes
547       to stdout, making this a useful feature for test development/debugging.
548

SILENCING RUNNER STDOUT

550       You may specify --show=none, that means avocado will turn off all  run‐
551       ner  stdout.   Note  that --show=none does not affect on disk job logs,
552       those continue to be generated normally.
553

SILENCING SYSINFO REPORT

555       You may specify --disable-sysinfo and avocado will not collect  profil‐
556       ers,  hardware details and other system information, inside the job re‐
557       sult directory.
558

LISTING TESTS

560       The avocado command line tool also has a list command, that  lists  the
561       known  tests  in a given path, be it a path to an individual test, or a
562       path to a directory. If no arguments provided, avocado will inspect the
563       contents  of  the  test  location  being used by avocado (if you are in
564       doubt about which one is that, you may use avocado  config  --datadir).
565       The output looks like:
566
567          $ avocado list
568          INSTRUMENTED /usr/share/doc/avocado/tests/abort.py
569          INSTRUMENTED /usr/share/doc/avocado/tests/datadir.py
570          INSTRUMENTED /usr/share/doc/avocado/tests/doublefail.py
571          INSTRUMENTED /usr/share/doc/avocado/tests/doublefree.py
572          INSTRUMENTED /usr/share/doc/avocado/tests/errortest.py
573          INSTRUMENTED /usr/share/doc/avocado/tests/failtest.py
574          INSTRUMENTED /usr/share/doc/avocado/tests/fiotest.py
575          INSTRUMENTED /usr/share/doc/avocado/tests/gdbtest.py
576          INSTRUMENTED /usr/share/doc/avocado/tests/gendata.py
577          INSTRUMENTED /usr/share/doc/avocado/tests/linuxbuild.py
578          INSTRUMENTED /usr/share/doc/avocado/tests/multiplextest.py
579          INSTRUMENTED /usr/share/doc/avocado/tests/passtest.py
580          INSTRUMENTED /usr/share/doc/avocado/tests/skiptest.py
581          INSTRUMENTED /usr/share/doc/avocado/tests/sleeptenmin.py
582          INSTRUMENTED /usr/share/doc/avocado/tests/sleeptest.py
583          INSTRUMENTED /usr/share/doc/avocado/tests/synctest.py
584          INSTRUMENTED /usr/share/doc/avocado/tests/timeouttest.py
585          INSTRUMENTED /usr/share/doc/avocado/tests/warntest.py
586          INSTRUMENTED /usr/share/doc/avocado/tests/whiteboard.py
587
588       Here,  INSTRUMENTED means that the files there are python files with an
589       avocado test class in them, therefore, that they are what we  call  in‐
590       strumented  tests.  This means those tests can use all avocado APIs and
591       facilities. Let's try to list a directory with a  bunch  of  executable
592       shell scripts:
593
594          $ avocado list examples/wrappers/
595          SIMPLE examples/wrappers/bind_cpu0.sh
596          SIMPLE examples/wrappers/dummy.sh
597          SIMPLE examples/wrappers/ltrace.sh
598          SIMPLE examples/wrappers/perf.sh
599          SIMPLE examples/wrappers/rr.sh
600          SIMPLE examples/wrappers/strace.sh
601          SIMPLE examples/wrappers/time.sh
602          SIMPLE examples/wrappers/valgrind.sh
603
604       Here,  SIMPLE means that those files are executables, that avocado will
605       simply execute and return PASS or FAIL depending on their return  codes
606       (PASS -> 0, FAIL -> any integer different than 0). You can also provide
607       the --verbose, or -V flag to display files that were detected  but  are
608       not avocado tests, along with summary information:
609
610          $ avocado -V list examples/gdb-prerun-scripts/
611          Type       Test                                     Tag(s)
612          NOT_A_TEST examples/gdb-prerun-scripts/README
613          NOT_A_TEST examples/gdb-prerun-scripts/pass-sigusr1
614
615          TEST TYPES SUMMARY
616          ==================
617          not_a_test: 2
618
619       That  summarizes the basic commands you should be using more frequently
620       when you start with avocado. Let's talk now about  how  avocado  stores
621       test results.
622

EXPLORING RESULTS

624       When avocado runs tests, it saves all its results on your system:
625
626          JOB ID    : <id>
627          JOB LOG   : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
628
629       For  your  convenience,  avocado maintains a link to the latest job run
630       (an avocado run command in this context), so you can always  use  "lat‐
631       est" to browse your test results:
632
633          $ ls /home/user/avocado/job-results/latest
634          id
635          jobdata
636          job.log
637          results.html
638          results.json
639          results.tap
640          results.xml
641          sysinfo
642          test-results
643
644       The main log file is job.log, but every test has its own results direc‐
645       tory:
646
647          $ ls -1 ~/avocado/job-results/latest/test-results/
648          1-sleeptest.py:SleepTest.test
649
650       Since this is a directory, it should have content similar to:
651
652          $ ls -1 ~/avocado/job-results/latest/test-results/1-sleeptest.py\:SleepTest.test/
653          data
654          debug.log
655          stderr
656          stdout
657          whiteboard
658

MULTIPLEX FILE

660       Avocado has a powerful tool that enables multiple test scenarios to  be
661       run  using  a  single, unmodified test. This mechanism uses a YAML file
662       called the 'multiplex file', that tells avocado  how  to  multiply  all
663       possible test scenarios automatically.
664
665       This  is  a  sample  that varies the parameter sleep_length through the
666       scenarios /run/short (sleeps for 0.5 s), /run/medium (sleeps for 1  s),
667       /run/long  (sleeps  for  5s),  /run/longest (sleeps for 10s).  The YAML
668       file (multiplex file) that produced the output above is:
669
670          !mux
671          short:
672              sleep_length: 0.5
673          medium:
674              sleep_length: 1
675          long:
676              sleep_length: 5
677          longest:
678              sleep_length: 10
679
680       You can execute sleeptest in all variations exposed above with:
681
682          $ avocado run examples/tests/sleeptest.py -m examples/tests/sleeptest.py.data/sleeptest.yaml
683
684       And the output should look like:
685
686          JOB ID    : <id>
687          JOB LOG   : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
688           (1/4) examples/tests/sleeptest.py:SleepTest.test;run-short-beaf: PASS (0.50 s)
689           (2/4) examples/tests/sleeptest.py:SleepTest.test;run-medium-5595: PASS (1.01 s)
690           (3/4) examples/tests/sleeptest.py:SleepTest.test;run-long-f397: PASS (5.01 s)
691           (4/4) examples/tests/sleeptest.py:SleepTest.test;run-longest-efc4: PASS (10.01 s)
692          RESULTS    : PASS 4 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
693          JOB TIME   : 16.65 s
694
695       The test runner supports two kinds of global filters, through the  com‐
696       mand line options --mux-filter-only and --mux-filter-out.  The mux-fil‐
697       ter-only exclusively includes one or more paths and the  mux-filter-out
698       removes one or more paths from being processed.
699
700       From  the  previous  example,  if we are interested to use the variants
701       /run/medium and /run/longest, we do the following command line:
702
703          $ avocado run examples/tests/sleeptest.py -m examples/tests/sleeptest.py.data/sleeptest.yaml \
704                --mux-filter-only /run/medium /run/longest
705
706       And if you want to remove /small from the variants created, we  do  the
707       following:
708
709          $ avocado run examples/tests/sleeptest.py -m examples/tests/sleeptest.py.data/sleeptest.yaml \
710                --mux-filter-out /run/medium
711
712       Note  that  both  --mux-filter-only and --mux-filter-out filters can be
713       arranged in the same command line.
714
715       The multiplexer also supports default paths. The base  path  is  /run/*
716       but  it  can  be overridden by --mux-path, which accepts multiple argu‐
717       ments. What it does: it splits leaves by the provided paths. Each query
718       goes  one by one through those sub-trees and first one to hit the match
719       returns the result. It might not solve all problems, but it can help to
720       combine existing YAML files with your ones:
721
722          qa: # large and complex read-only file, content injected into /qa
723              tests:
724                  timeout: 10
725              ...
726          my_variants: !mux # your YAML file injected into /my_variants
727              short:
728                  timeout: 1
729              long:
730                  timeout: 1000
731
732       You want to use an existing test which uses params.get('timeout', '*').
733       Then you can use --mux-path '/my_variants/*' '/qa/*'  and  it'll  first
734       look  in  your variants. If no matches are found, then it would proceed
735       to /qa/*
736
737       Keep in mind that only slices defined in mux-path are  taken  into  ac‐
738       count for relative paths (the ones starting with *).
739

WRAP EXECUTABLE RUN BY TESTS

741       Avocado  allows  the instrumentation of executables being run by a test
742       in a transparent way. The user specifies a script ("the wrapper") to be
743       used to run the actual program called by the test.
744
745       If  the  instrumentation script is implemented correctly, it should not
746       interfere with the test behavior. That is,  the  wrapper  should  avoid
747       changing the return status, standard output and standard error messages
748       of the original executable.
749
750       The user can be specific about which program to wrap (with a shell-like
751       glob),  or  if that is omitted, a global wrapper that will apply to all
752       programs called by the test.
753
754       So, for every executable run by the test, the program name will be com‐
755       pared  to the pattern to decide whether to wrap it or not. You can have
756       multiples wrappers and patterns defined.
757
758       Examples:
759
760          $ avocado run examples/tests/datadir.py --wrapper examples/wrappers/strace.sh
761
762       Any command created by the test datadir will be wrapped on strace.sh.
763
764          $ avocado run examples/tests/datadir.py \
765                  --wrapper examples/wrappers/ltrace.sh:*make \
766                  --wrapper examples/wrappers/perf.sh:*datadir
767
768       Any command that matches the pattern *make will be wrapper on ltrace.sh
769       and the pattern *datadir will trigger the execution of perf.sh.
770

RUNNING TESTS WITH AN EXTERNAL RUNNER

772       It's  quite  common to have organically grown test suites in most soft‐
773       ware projects. These usually include a custom built, very specific test
774       runner that knows how to find and run their own tests.
775
776       Still,  running those tests inside Avocado may be a good idea for vari‐
777       ous reasons, including being able to have results  in  different  human
778       and  machine  readable formats, collecting system information alongside
779       those tests (the Avocado's sysinfo functionality), and more.
780
781       Avocado makes that possible by means of its "external runner"  feature.
782       The most basic way of using it is:
783
784          $ avocado run --external-runner=/path/to/external_runner foo bar baz
785
786       In  this example, Avocado will report individual test results for tests
787       foo, bar and baz. The actual results will be based on the  return  code
788       of  individual executions of /path/to/external_runner foo, /path/to/ex‐
789       ternal_runner bar and finally /path/to/external_runner baz.
790
791       As another way to explain an show how this feature works, think of  the
792       "external  runner" as some kind of interpreter and the individual tests
793       as anything that this interpreter recognizes and is able to execute.  A
794       UNIX  shell,  say  /bin/sh  could be considered an external runner, and
795       files with shell code could be considered tests:
796
797          $ echo "exit 0" > /tmp/pass
798          $ echo "exit 1" > /tmp/fail
799          $ avocado run --external-runner=/bin/sh /tmp/pass /tmp/fail
800          JOB ID    : <id>
801          JOB LOG   : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
802          TESTS      : 2
803          (1/2) /tmp/pass: PASS (0.01 s)
804          (2/2) /tmp/fail: FAIL: Exited with status: '1', stdout: '' stderr: '' (0.02 s)
805          RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
806          JOB TIME   : 0.11 s
807
808       This example is  pretty  obvious,  and  could  be  achieved  by  giving
809       /tmp/pass  and /tmp/fail shell "shebangs" (#!/bin/sh), making them exe‐
810       cutable (chmod +x /tmp/pass /tmp/fail), and running  them  as  "SIMPLE"
811       tests.
812
813       But now consider the following example:
814
815          $ avocado run --external-runner=/bin/curl http://local-avocado-server:9405/jobs/ \
816                                                    http://remote-avocado-server:9405/jobs/
817          JOB ID    : <id>
818          JOB LOG   : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
819          TESTS      : 2
820          (1/2) http://local-avocado-server:9405/jobs/: PASS (0.02 s)
821          (2/2) http://remote-avocado-server:9405/jobs/: FAIL (3.02 s)
822          RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
823          JOB TIME   : 3.14 s
824
825       This effectively makes /bin/curl an "external test runner", responsible
826       for trying to fetch those URLs, and reporting PASS or FAIL for each  of
827       them.
828

RECORDING TEST REFERENCE OUTPUT

830       As a tester, you may want to check if the output of a given application
831       matches an expected output. In order to help with this common use case,
832       we  offer  the  option --output-check-record [mode] to the test runner.
833       If this option is used, it will store  the  stdout  or  stderr  of  the
834       process  (or  both,  if  you specified all) being executed to reference
835       files: stdout.expected and stderr.expected.
836
837       Those files will be recorded in the test data dir. The data dir  is  in
838       the    same    directory    as    the    test    source   file,   named
839       [source_file_name.data]. Let's take as an example the test synctest.py.
840       In a fresh checkout of avocado, you can see:
841
842          examples/tests/synctest.py.data/stderr.expected
843          examples/tests/synctest.py.data/stdout.expected
844
845       From those 2 files, only stdout.expected is non empty:
846
847          $ cat examples/tests/synctest.py.data/stdout.expected
848          PAR : waiting
849          PASS : sync interrupted
850
851       The  output  files  were  originally obtained using the test runner and
852       passing the option --output-check-record all to the test runner:
853
854          $ avocado run --output-check-record all examples/tests/synctest.py
855          JOB ID    : <id>
856          JOB LOG   : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
857           (1/1) examples/tests/synctest.py:SyncTest.test: PASS (4.00 s)
858          RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
859          JOB TIME   : 4.10 s
860
861       After the reference files are added, the check process is  transparent,
862       in  the sense that you do not need to provide special flags to the test
863       runner. Now, every time the test is executed, after it is done running,
864       it  will  check if the outputs are exactly right before considering the
865       test as PASSed. If you want to override the default behavior  and  skip
866       output  check  entirely, you may provide the flag --output-check=off to
867       the test runner.
868
869       The avocado.utils.process APIs have a parameter allow_output_check (de‐
870       faults to all), so that you can select which process outputs will go to
871       the reference files, should you chose to record them.  You  may  choose
872       all,  for  both stdout and stderr, stdout, for the stdout only, stderr,
873       for only the stderr only, or none, to  allow  neither  of  them  to  be
874       recorded and checked.
875
876       This process works fine also with simple tests, executables that return
877       0 (PASSed) or != 0 (FAILed). Let's consider our bogus example:
878
879          $ cat output_record.sh
880          #!/bin/bash
881          echo "Hello, world!"
882
883       Let's record the output (both stdout and stderr) for this one:
884
885          $ avocado run output_record.sh --output-check-record all
886          JOB ID    : <id>
887          JOB LOG   : /home/user/avocado/job-results/job-<date>-<shortid>/job.log
888           (1/1) output_record.sh: PASS (0.01 s)
889          RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
890          JOB HTML   : /home/user/avocado/job-results/job-<date>-<shortid>/results.html
891          JOB TIME   : 0.11 s
892
893       After this is done, you'll notice that a the test  data  directory  ap‐
894       peared in the same level of our shell script, containing 2 files:
895
896          $ ls output_record.sh.data/
897          stderr.expected  stdout.expected
898
899       Let's look what's in each of them:
900
901          $ cat output_record.sh.data/stdout.expected
902          Hello, world!
903          $ cat output_record.sh.data/stderr.expected
904          $
905
906       Now,  every  time  this test runs, it'll take into account the expected
907       files that were recorded, no need to do anything else but run the test.
908

LINUX DISTRIBUTION UTILITIES

910       Avocado has some planned features that depend on knowing the Linux Dis‐
911       tribution  being  used on the system. The most basic command prints the
912       detected Linux Distribution:
913
914          $ avocado distro
915          Detected distribution: fedora (x86_64) version 21 release 0
916
917       Other features are available with the same command  when  command  line
918       options are given, as shown by the --help option.
919
920       For  instance,  it  possible  to create a so-called "Linux Distribution
921       Definition" file, by inspecting an installation tree. The  installation
922       tree  could be the contents of the official installation ISO or a local
923       network mirror.
924
925       These files let Avocado pinpoint if a given installed package  is  part
926       of the original Linux Distribution or something else that was installed
927       from an external repository or even manually. This, in turn,  can  help
928       detecting  regressions  in  base  system packages that affected a given
929       test result.
930
931       To generate a definition file run:
932
933          $ avocado distro --distro-def-create --distro-def-name avocadix  \
934                           --distro-def-version 1 --distro-def-arch x86_64 \
935                           --distro-def-type rpm --distro-def-path /mnt/dvd
936
937       And the output will be something like:
938
939          Loading distro information from tree... Please wait...
940          Distro information saved to "avocadix-1-x86_64.distro"
941

FILES

943          System wide configuration file
944              /etc/avocado/avocado.conf
945
946          Extra configuration files
947              /etc/avocado/conf.d/
948
949          User configuration file
950              ~/.config/avocado/avocado.conf
951

BUGS

953       If you find a bug, please report it over our github page as  an  issue:
954       https://github.com/avocado-framework/avocado/issues
955

LICENSE

957       Avocado     is     released     under    GPLv2    (explicit    version)
958       https://gnu.org/licenses/gpl-2.0.html. Even though most of the  current
959       code is licensed under a "and any later version" clause, some parts are
960       specifically bound to the version 2 of the license and therefore that's
961       the official license of the prject itself. For more details, please see
962       the LICENSE file in the project source code directory.
963

MORE INFORMATION

965       For more information please check Avocado's project website, located at
966       https://avocado-framework.github.io/. There you'll find links to online
967       documentation, source code and community resources.
968

AUTHOR

970       Avocado Development Team <avocado-devel@redhat.com>
971
972
973
974
975                                                                    AVOCADO(1)
Impressum