1ZOPE.TESTRUNNER(1)               User Commands              ZOPE.TESTRUNNER(1)
2
3
4

NAME

6       zope.testrunner - Zope testrunner script
7

DESCRIPTION

9       usage: zope-testrunner [-h] [--package PACKAGE] [--module MODULE]
10
11       [--test TEST] [--unit] [--non-unit] [--layer LAYER]
12              [-a  AT_LEVEL] [--all] [--list-tests] [--require-unique] [--ver‐
13              bose] [--quiet] [--progress]  [--no-progress]  [--auto-progress]
14              [--color] [--no-color] [--auto-color] [--subunit] [--subunit-v2]
15              [--slow-test  N]  [-1]   [--show-secondary-failures]   [--ndiff]
16              [--udiff]   [--cdiff]  [--ignore-new-thread  REGEXP]  [--buffer]
17              [--stop-on-error] [--post-mortem] [--gc  GC]  [--gc-option  {DE‐
18              BUG_COLLECTABLE,DEBUG_SAVEALL,DEBUG_STATS,DEBUG_OBJECTS,DE‐
19              BUG_LEAK,DEBUG_UNCOLLECTABLE,DEBUG_INSTANCES}] [--gc-after-test]
20              [--repeat  REPEAT]  [--report-refcounts]  [--coverage  COVERAGE]
21              [--profile {cProfile}]  [--profile-directory  PROF_DIR]  [--path
22              PATH]   [--test-path   TEST_PATH]   [--package-path   ARG   ARG]
23              [--tests-pattern   TESTS_PATTERN]   [--suite-name    SUITE_NAME]
24              [--test-file-pattern    TEST_FILE_PATTERN]   [--ignore_dir   IG‐
25              NORE_DIR] [--shuffle] [--shuffle-seed SHUFFLE_SEED]  [--version]
26              [-j       PROCESSES]       [--keepbytecode]      [--usecompiled]
27              [--exit-with-status] [legacy_module_filter] [legacy_test_filter]
28
29       Discover and run unittest tests
30
31   positional arguments:
32       legacy_module_filter
33              DEPRECATED: Prefer to use --module.
34
35       legacy_test_filter
36              DEPRECATED: Prefer to use --test.
37
38   options:
39       -h, --help
40              show this help message and exit
41
42   Searching and filtering:
43              Options in this group are used to define which tests to run.
44
45       --package PACKAGE, --dir PACKAGE, -s PACKAGE
46              Search the given package's directories for tests.  This  can  be
47              specified  more  than once to run tests in multiple parts of the
48              source tree. For example, if refactoring interfaces,  you  don't
49              want  to  see  the way you have broken setups for tests in other
50              packages.  You *just* want to run the interface tests.  Packages
51              are  supplied  as  dotted  names. For compatibility with the old
52              test runner, forward and backward slashed in package  names  are
53              converted  to dots. (In the special case of packages spread over
54              multiple directories, only directories within  the  test  search
55              path are searched. See the --path option.)
56
57       --module MODULE, -m MODULE
58              Specify a test-module filter as a regular expression.  This is a
59              case-sensitive regular expression, used in  search  (not  match)
60              mode,  to  limit  which test modules are searched for tests. The
61              regular expressions are checked against dotted module names.  In
62              an  extension  of  Python  regexp  notation,  a  leading  "!" is
63              stripped and causes the sense of  the  remaining  regexp  to  be
64              negated  (so  "!bc" matches any string that does not match "bc",
65              and vice versa). The option can be specified multiple  test-mod‐
66              ule  filters.  Test modules matching any of the test filters are
67              searched. If no test-module filter is specified, then  all  test
68              modules are used.
69
70       --test TEST, -t TEST
71              Specify  a  test  filter  as  a  regular  expression.  This is a
72              case-sensitive regular expression, used in  search  (not  match)
73              mode,  to  limit  which tests are run. In an extension of Python
74              regexp notation, a leading "!" is stripped and causes the  sense
75              of  the  remaining  regexp  to  be negated (so "!bc" matches any
76              string that does not match "bc", and vice versa). The option can
77              be  specified  multiple  test filters. Tests matching any of the
78              test filters are included. If no test filter is specified,  then
79              all tests are run.
80
81       --unit, -u
82              Run only unit tests, ignoring any layer options.
83
84       --non-unit, -f
85              Run tests other than unit tests.
86
87       --layer LAYER
88              Specify  a  test  layer to run. The option can be given multiple
89              times to specify more than one layer. If not specified, all lay‐
90              ers  are run. It is common for the running script to provide de‐
91              fault values for this option. Layers are specified  regular  ex‐
92              pressions, used in search mode, for dotted names of objects that
93              define a layer. In an extension of  Python  regexp  notation,  a
94              leading  "!"  is  stripped and causes the sense of the remaining
95              regexp to be negated (so "!bc" matches any string that does  not
96              match  "bc",  and  vice  versa).  The layer named 'zope.testrun‐
97              ner.layer.UnitTests' is reserved for unit tests,  however,  take
98              note of the --unit and non-unit options.
99
100       -a AT_LEVEL, --at-level AT_LEVEL
101              Run  the tests at the given level. Any test at a level at or be‐
102              low this is run, any test at a level  above  this  is  not  run.
103              Level <= 0 runs all tests.
104
105       --all  Run tests at all levels.
106
107       --list-tests
108              List all tests that matched your filters. Do not run any tests.
109
110       --require-unique
111              Require that all test IDs be unique and raise an error if dupli‐
112              cates are encountered.
113
114   Reporting:
115              Reporting options control basic aspects of test-runner output
116
117       --verbose, -v
118              Make output more verbose. Increment the verbosity level.
119
120       --quiet, -q
121              Make the output minimal, overriding any verbosity options.
122
123       --progress, -p
124              Output progress status
125
126       --no-progress
127              Do not output progress status. This is the default, but  can  be
128              used to counter a previous use of --progress or -p.
129
130       --auto-progress
131              Output progress status, but only when stdout is a terminal.
132
133       --color, -c
134              Colorize the output.
135
136       --no-color, -C
137              Do not colorize the output. This is the default, but can be used
138              to counter a previous use of --color or -c.
139
140       --auto-color
141              Colorize the output, but only when stdout is a terminal.
142
143       --subunit
144              Use subunit v1 output. Will not be colorized.
145
146       --subunit-v2
147              Use subunit v2 output. Will not be colorized.
148
149       --slow-test N
150              With -c and -vvv, highlight tests that take longer than  N  sec‐
151              onds (default: 10).
152
153       -1, --hide-secondary-failures
154              Report  only the first failure in a doctest. (Examples after the
155              failure are still executed, in case they do any cleanup.)
156
157       --show-secondary-failures
158              Report all failures in a doctest. This is the default,  but  can
159              be used to counter a default use of -1 or --hide-secondary-fail‐
160              ures.
161
162       --ndiff
163              When there is a doctest failure, show it as  a  diff  using  the
164              ndiff.py utility.
165
166       --udiff
167              When there is a doctest failure, show it as a unified diff.
168
169       --cdiff
170              When there is a doctest failure, show it as a context diff.
171
172       --ignore-new-thread REGEXP
173              If  a thread with this name is left behind, don't report this at
174              the end. This is a case-sensitive regular  expression,  used  in
175              match  mode. This option can be used multiple times. If a thread
176              name matches any of them, it will be ignored.
177
178       --buffer
179              Buffer the standard output and  standard  error  streams  during
180              each  test.  Output  during  a passing test is discarded. Output
181              during failing or erroring tests is echoed. This option  is  en‐
182              abled  by  default  if  --subunit  or --subunit-v2 is in use, to
183              avoid corrupting the subunit stream.
184
185   Analysis:
186              Analysis options provide tools for analysing test output.
187
188       --stop-on-error, --stop, -x
189              Stop running tests after first test failure or error.
190
191       --post-mortem, --pdb, -D
192              Enable post-mortem debugging of test failures
193
194       --gc GC, -g GC
195              Set the garbage collector generation threshold. This can be used
196              to  stress  memory and gc correctness. Some crashes are only re‐
197              producible when the threshold is set to  1  (aggressive  garbage
198              collection).  Do  "--gc  0"  to disable garbage collection alto‐
199              gether. The --gc option can be used up to 3 times to specify  up
200              to 3 of the 3 Python gc_threshold settings.
201
202       --gc-option      {DEBUG_COLLECTABLE,DEBUG_SAVEALL,DEBUG_STATS,DEBUG_OB‐
203       JECTS,DEBUG_LEAK,DEBUG_UNCOLLECTABLE,DEBUG_INSTANCES},  -G  {DEBUG_COL‐
204       LECTABLE,DEBUG_SAVEALL,DEBUG_STATS,DEBUG_OBJECTS,DEBUG_LEAK,DEBUG_UN‐
205       COLLECTABLE,DEBUG_INSTANCES}
206              Set a Python gc-module debug flag. This option can be used  more
207              than once to set multiple flags.
208
209       --gc-after-test
210              After  each  test, call 'gc.collect' and record the return value
211              *rv*; when *rv* is non-zero, output '!'  on  verbosity  level  1
212              and '[*rv*]' on higher verbosity levels. On verbosity level 4 or
213              higher output detailed cycle information.
214
215       --repeat REPEAT, -N REPEAT
216              Repeat the tests the given number of times. This option is  used
217              to  make  sure  that  tests leave their environment in the state
218              they found it and, with the --report-refcounts  option  to  look
219              for memory leaks.
220
221       --report-refcounts, -r
222              After each run of the tests, output a report summarizing changes
223              in refcounts by object type.  This  option  that  requires  that
224              Python was built with the --with-pydebug option to configure.
225
226       --coverage COVERAGE
227              Perform  code-coverage analysis, saving trace data to the direc‐
228              tory with the given name. A code coverage summary is printed  to
229              standard out.
230
231       --profile {cProfile}
232              Run  the  tests  under  cProfiler  and display the top 50 stats,
233              sorted by cumulative time and number of calls.
234
235       --profile-directory PROF_DIR
236              Directory  for  temporary  profiler  files.  All   files   named
237              tests_profile.*.prof  in  this directory will be removed. If you
238              intend to run multiple instances of the test runner in parallel,
239              be sure to tell them to use different directories, so they won't
240              step on each other's toes.
241
242   Setup:
243              Setup options are normally supplied by  the  testrunner  script,
244              although they can be overridden by users.
245
246       --path PATH
247              Specify a path to be added to Python's search path.  This option
248              can be used multiple times to specify multiple search paths. The
249              path  is  usually  specified  by  the test-runner script itself,
250              rather than by users of the script, although it can be  overrid‐
251              den  by  users.   Only tests found in the path will be run. This
252              option also specifies directories to be searched for tests.  See
253              the search_directory.
254
255       --test-path TEST_PATH
256              Specify  a  path  to be searched for tests, but not added to the
257              Python search path. This option can be used  multiple  times  to
258              specify  multiple search paths. The path is usually specified by
259              the test-runner script itself,  rather  than  by  users  of  the
260              script, although it can be overridden by users. Only tests found
261              in the path will be run.
262
263       --package-path ARG ARG
264              Specify a path to be searched for tests, but not  added  to  the
265              Python  search  path.  Also specify a package for files found in
266              this path. This is  used  to  deal  with  directories  that  are
267              stitched  into  packages  that  are  not  otherwise searched for
268              tests. This option takes 2 arguments specifying the path and the
269              package.  This option can be used multiple times to specify mul‐
270              tiple search  paths.  The  path  is  usually  specified  by  the
271              test-runner  script  itself, rather than by users of the script,
272              although it can be overridden by users. Only tests found in  the
273              path will be run.
274
275       --tests-pattern TESTS_PATTERN
276              The test runner looks for modules containing tests. It uses this
277              pattern to identify these modules. The  modules  may  be  either
278              packages or python files. If a test module is a package, it uses
279              the value given by  the  test-file-pattern  to  identify  python
280              files within the package containing tests.
281
282       --suite-name SUITE_NAME
283              Specify the name of the object in each test_module that contains
284              the module's test suite.
285
286       --test-file-pattern TEST_FILE_PATTERN
287              Specify a pattern for identifying python files  within  a  tests
288              package. See the documentation for the --tests-pattern option.
289
290       --ignore_dir IGNORE_DIR
291              Specifies  the  name  of  a directory to ignore when looking for
292              tests.
293
294       --shuffle
295              Shuffles the order in which tests are ran.
296
297       --shuffle-seed SHUFFLE_SEED
298              Value used to initialize the tests shuffler. Specify a value  to
299              create repeatable random ordered tests.
300
301   Other:
302              Other options
303
304       --version
305              Print the version of the testrunner, and exit.
306
307       -j PROCESSES
308              Use  up  to given number of parallel processes to execute tests.
309              May decrease test run time substantially. Defaults to 1.
310
311       --keepbytecode, -k
312              Normally, the test runner scans the test paths and the test  di‐
313              rectories looking for and deleting pyc or pyo files without cor‐
314              responding py files. This is to prevent spurious  test  failures
315              due  to  finding compiled modules where source modules have been
316              deleted. This scan can be time consuming. Using this option dis‐
317              ables  this  scan.  If  you know you haven't removed any modules
318              since last running the tests, can make  the  test  run  go  much
319              faster.
320
321       --usecompiled
322              Normally,  a  package must contain an __init__.py file, and only
323              .py files can contain test code. When this option is  specified,
324              compiled Python files (.pyc and .pyo) can be used instead: a di‐
325              rectory containing __init__.pyc or __init__.pyo is also  consid‐
326              ered  to  be a package, and if file XYZ.py contains tests but is
327              absent while XYZ.pyc or XYZ.pyo exists then the  compiled  files
328              will  be  used.  This  is necessary when running tests against a
329              tree where the .py files have been removed after compilation  to
330              .pyc/.pyo. Use of this option implies --keepbytecode.
331
332       --exit-with-status
333              DEPRECATED: The test runner will always exit with a status.
334
335
336
337zope.testrunner version 6.1       August 2023               ZOPE.TESTRUNNER(1)
Impressum