1ZOPE.TESTRUNNER(1) User Commands ZOPE.TESTRUNNER(1)
2
6 zope.testrunner - Zope testrunner script
7
9 usage: zope-testrunner [-h] [--package PACKAGE] [--module MODULE]
10 [--test TEST] [--unit] [--non-unit] [--layer LAYER]
12 [-a AT_LEVEL] [--all] [--list-tests] [--require-unique] [--ver‐
13 bose] [--quiet] [--progress] [--no-progress] [--auto-progress]
14 [--color] [--no-color] [--auto-color] [--subunit] [--subunit-v2]
15 [--slow-test N] [-1] [--show-secondary-failures] [--ndiff]
16 [--udiff] [--cdiff] [--ignore-new-thread REGEXP] [--buffer]
17 [--stop-on-error] [--post-mortem] [--gc GC] [--gc-option
18 {DEBUG_COL‐
19 LECTABLE,DEBUG_INSTANCES,DEBUG_OBJECTS,DEBUG_STATS,DEBUG_LEAK,DEBUG_UNCOL‐
20 LECTABLE,DEBUG_SAVEALL}] [--repeat REPEAT] [--report-refcounts]
21 [--coverage COVERAGE] [--profile {cProfile}] [--profile-direc‐
22 tory PROF_DIR] [--path PATH] [--test-path TEST_PATH] [--pack‐
23 age-path PACKAGE_PATH PACKAGE_PATH] [--tests-pattern TESTS_PAT‐
24 TERN] [--suite-name SUITE_NAME] [--test-file-pattern
25 TEST_FILE_PATTERN] [--ignore_dir IGNORE_DIR] [--shuffle]
26 [--shuffle-seed SHUFFLE_SEED] [--version] [-j PROCESSES]
27 [--keepbytecode] [--usecompiled] [--exit-with-status]
28 [legacy_module_filter] [legacy_test_filter]
29 Discover and run unittest tests
31 positional arguments:
33 legacy_module_filter
34 DEPRECATED: Prefer to use --module.
35 legacy_test_filter
37 DEPRECATED: Prefer to use --test.
38 optional arguments:
40 -h, --help
41 show this help message and exit
42 Searching and filtering:
44 Options in this group are used to define which tests to run.
45 --package PACKAGE, --dir PACKAGE, -s PACKAGE
47 Search the given package's directories for tests. This can be
48 specified more than once to run tests in multiple parts of the
49 source tree. For example, if refactoring interfaces, you don't
50 want to see the way you have broken setups for tests in other
51 packages. You *just* want to run the interface tests. Packages
52 are supplied as dotted names. For compatibility with the old
53 test runner, forward and backward slashed in package names are
54 converted to dots. (In the special case of packages spread over
55 multiple directories, only directories within the test search
56 path are searched. See the --path option.)
57 --module MODULE, -m MODULE
59 Specify a test-module filter as a regular expression. This is a
60 case-sensitive regular expression, used in search (not match)
61 mode, to limit which test modules are searched for tests. The
62 regular expressions are checked against dotted module names. In
63 an extension of Python regexp notation, a leading "!" is
64 stripped and causes the sense of the remaining regexp to be
65 negated (so "!bc" matches any string that does not match "bc",
66 and vice versa). The option can be specified multiple test-mod‐
67 ule filters. Test modules matching any of the test filters are
68 searched. If no test-module filter is specified, then all test
69 modules are used.
70 --test TEST, -t TEST
72 Specify a test filter as a regular expression. This is a
73 case-sensitive regular expression, used in search (not match)
74 mode, to limit which tests are run. In an extension of Python
75 regexp notation, a leading "!" is stripped and causes the sense
76 of the remaining regexp to be negated (so "!bc" matches any
77 string that does not match "bc", and vice versa). The option can
78 be specified multiple test filters. Tests matching any of the
79 test filters are included. If no test filter is specified, then
80 all tests are run.
81 --unit, -u
83 Run only unit tests, ignoring any layer options.
84 --non-unit, -f
86 Run tests other than unit tests.
87 --layer LAYER
89 Specify a test layer to run. The option can be given multiple
90 times to specify more than one layer. If not specified, all lay‐
91 ers are run. It is common for the running script to provide
92 default values for this option. Layers are specified regular
93 expressions, used in search mode, for dotted names of objects
94 that define a layer. In an extension of Python regexp notation,
95 a leading "!" is stripped and causes the sense of the remaining
96 regexp to be negated (so "!bc" matches any string that does not
97 match "bc", and vice versa). The layer named 'zope.testrun‐
98 ner.layer.UnitTests' is reserved for unit tests, however, take
99 note of the --unit and non-unit options.
100 -a AT_LEVEL, --at-level AT_LEVEL
102 Run the tests at the given level. Any test at a level at or
103 below this is run, any test at a level above this is not run.
104 Level 0 runs all tests.
105 --all Run tests at all levels.
107 --list-tests
109 List all tests that matched your filters. Do not run any tests.
110 --require-unique
112 Require that all test IDs be unique and raise an error if dupli‐
113 cates are encountered.
114 Reporting:
116 Reporting options control basic aspects of test-runner output
117 --verbose, -v
119 Make output more verbose. Increment the verbosity level.
120 --quiet, -q
122 Make the output minimal, overriding any verbosity options.
123 --progress, -p
125 Output progress status
126 --no-progress
128 Do not output progress status. This is the default, but can be
129 used to counter a previous use of --progress or -p.
130 --auto-progress
132 Output progress status, but only when stdout is a terminal.
133 --color, -c
135 Colorize the output.
136 --no-color, -C
138 Do not colorize the output. This is the default, but can be used
139 to counter a previous use of --color or -c.
140 --auto-color
142 Colorize the output, but only when stdout is a terminal.
143 --subunit
145 Use subunit v1 output. Will not be colorized.
146 --subunit-v2
148 Use subunit v2 output. Will not be colorized.
149 --slow-test N
151 With -c and -vvv, highlight tests that take longer than N sec‐
152 onds (default: 10).
153 -1, --hide-secondary-failures
155 Report only the first failure in a doctest. (Examples after the
156 failure are still executed, in case they do any cleanup.)
157 --show-secondary-failures
159 Report all failures in a doctest. This is the default, but can
160 be used to counter a default use of -1 or --hide-secondary-fail‐
161 ures.
162 --ndiff
164 When there is a doctest failure, show it as a diff using the
165 ndiff.py utility.
166 --udiff
168 When there is a doctest failure, show it as a unified diff.
169 --cdiff
171 When there is a doctest failure, show it as a context diff.
172 --ignore-new-thread REGEXP
174 If a thread with this name is left behind, don't report this at
175 the end. This is a case-sensitive regular expression, used in
176 match mode. This option can be used multiple times. If a thread
177 name matches any of them, it will be ignored.
178 --buffer
180 Buffer the standard output and standard error streams during
181 each test. Output during a passing test is discarded. Output
182 during failing or erroring tests is echoed. This option is
183 enabled by default if --subunit or --subunit-v2 is in use, to
184 avoid corrupting the subunit stream.
185 Analysis:
187 Analysis options provide tools for analysing test output.
188 --stop-on-error, --stop, -x
190 Stop running tests after first test failure or error.
191 --post-mortem, --pdb, -D
193 Enable post-mortem debugging of test failures
194 --gc GC, -g GC
196 Set the garbage collector generation threshold. This can be used
197 to stress memory and gc correctness. Some crashes are only
198 reproducible when the threshold is set to 1 (aggressive garbage
199 collection). Do "--gc 0" to disable garbage collection alto‐
200 gether. The --gc option can be used up to 3 times to specify up
201 to 3 of the 3 Python gc_threshold settings.
202 --gc-option {DEBUG_COL‐
204 LECTABLE,DEBUG_INSTANCES,DEBUG_OBJECTS,DEBUG_STATS,DEBUG_LEAK,DEBUG_UNCOL‐
205 LECTABLE,DEBUG_SAVEALL}, -G {DEBUG_COL‐
206 LECTABLE,DEBUG_INSTANCES,DEBUG_OBJECTS,DEBUG_STATS,DEBUG_LEAK,DEBUG_UNCOL‐
207 LECTABLE,DEBUG_SAVEALL}
208 Set a Python gc-module debug flag. This option can be used more
209 than once to set multiple flags.
210 --repeat REPEAT, -N REPEAT
212 Repeat the tests the given number of times. This option is used
213 to make sure that tests leave their environment in the state
214 they found it and, with the --report-refcounts option to look
215 for memory leaks.
216 --report-refcounts, -r
218 After each run of the tests, output a report summarizing changes
219 in refcounts by object type. This option that requires that
220 Python was built with the --with-pydebug option to configure.
221 --coverage COVERAGE
223 Perform code-coverage analysis, saving trace data to the direc‐
224 tory with the given name. A code coverage summary is printed to
225 standard out.
226 --profile {cProfile}
228 Run the tests under cProfiler and display the top 50 stats,
229 sorted by cumulative time and number of calls.
230 --profile-directory PROF_DIR
232 Directory for temporary profiler files. All files named
233 tests_profile.*.prof in this directory will be removed. If you
234 intend to run multiple instances of the test runner in parallel,
235 be sure to tell them to use different directories, so they won't
236 step on each other's toes.
237 Setup:
239 Setup options are normally supplied by the testrunner script,
240 although they can be overridden by users.
241 --path PATH
243 Specify a path to be added to Python's search path. This option
244 can be used multiple times to specify multiple search paths. The
245 path is usually specified by the test-runner script itself,
246 rather than by users of the script, although it can be overrid‐
247 den by users. Only tests found in the path will be run. This
248 option also specifies directories to be searched for tests. See
249 the search_directory.
250 --test-path TEST_PATH
252 Specify a path to be searched for tests, but not added to the
253 Python search path. This option can be used multiple times to
254 specify multiple search paths. The path is usually specified by
255 the test-runner script itself, rather than by users of the
256 script, although it can be overridden by users. Only tests found
257 in the path will be run.
258 --package-path PACKAGE_PATH PACKAGE_PATH
260 Specify a path to be searched for tests, but not added to the
261 Python search path. Also specify a package for files found in
262 this path. This is used to deal with directories that are
263 stitched into packages that are not otherwise searched for
264 tests. This option takes 2 arguments. The first is a path name.
265 The second is the package name. This option can be used multiple
266 times to specify multiple search paths. The path is usually
267 specified by the test-runner script itself, rather than by users
268 of the script, although it can be overridden by users. Only
269 tests found in the path will be run.
270 --tests-pattern TESTS_PATTERN
272 The test runner looks for modules containing tests. It uses this
273 pattern to identify these modules. The modules may be either
274 packages or python files. If a test module is a package, it uses
275 the value given by the test-file-pattern to identify python
276 files within the package containing tests.
277 --suite-name SUITE_NAME
279 Specify the name of the object in each test_module that contains
280 the module's test suite.
281 --test-file-pattern TEST_FILE_PATTERN
283 Specify a pattern for identifying python files within a tests
284 package. See the documentation for the --tests-pattern option.
285 --ignore_dir IGNORE_DIR
287 Specifies the name of a directory to ignore when looking for
288 tests.
289 --shuffle
291 Shuffles the order in which tests are ran.
292 --shuffle-seed SHUFFLE_SEED
294 Value used to initialize the tests shuffler. Specify a value to
295 create repeatable random ordered tests.
296 Other:
298 Other options
299 --version
301 Print the version of the testrunner, and exit.
302 -j PROCESSES
304 Use up to given number of parallel processes to execute tests.
305 May decrease test run time substantially. Defaults to 1.
306 --keepbytecode, -k
308 Normally, the test runner scans the test paths and the test
309 directories looking for and deleting pyc or pyo files without
310 corresponding py files. This is to prevent spurious test fail‐
311 ures due to finding compiled modules where source modules have
312 been deleted. This scan can be time consuming. Using this option
313 disables this scan. If you know you haven't removed any modules
314 since last running the tests, can make the test run go much
315 faster.
316 --usecompiled
318 Normally, a package must contain an __init__.py file, and only
319 .py files can contain test code. When this option is specified,
320 compiled Python files (.pyc and .pyo) can be used instead: a
321 directory containing __init__.pyc or __init__.pyo is also con‐
322 sidered to be a package, and if file XYZ.py contains tests but
323 is absent while XYZ.pyc or XYZ.pyo exists then the compiled
324 files will be used. This is necessary when running tests against
325 a tree where the .py files have been removed after compilation
326 to .pyc/.pyo. Use of this option implies --keepbytecode.
327 --exit-with-status
329 DEPRECATED: The test runner will always exit with a status.
330zope.testrunner version 5.1 March 2020 ZOPE.TESTRUNNER(1)