1ZOPE.TESTRUNNER(1) User Commands ZOPE.TESTRUNNER(1)
2
6 zope.testrunner - Zope testrunner script
7
9 usage: zope-testrunner [-h] [--package PACKAGE] [--module MODULE]
10 [--test TEST] [--unit] [--non-unit] [--layer LAYER]
12 [-a AT_LEVEL] [--all] [--list-tests] [--require-unique] [--ver‐
13 bose] [--quiet] [--progress] [--no-progress] [--auto-progress]
14 [--color] [--no-color] [--auto-color] [--subunit] [--subunit-v2]
15 [--slow-test N] [-1] [--show-secondary-failures] [--ndiff]
16 [--udiff] [--cdiff] [--ignore-new-thread REGEXP] [--buffer]
17 [--stop-on-error] [--post-mortem] [--gc GC] [--gc-option {DE‐
18 BUG_STATS,DEBUG_LEAK,DEBUG_OBJECTS,DEBUG_COLLECTABLE,DE‐
19 BUG_SAVEALL,DEBUG_UNCOLLECTABLE,DEBUG_INSTANCES}] [--gc-af‐
20 ter-test] [--repeat REPEAT] [--report-refcounts] [--coverage
21 COVERAGE] [--profile {cProfile}] [--profile-directory PROF_DIR]
22 [--path PATH] [--test-path TEST_PATH] [--package-path ARG ARG]
23 [--tests-pattern TESTS_PATTERN] [--suite-name SUITE_NAME]
24 [--test-file-pattern TEST_FILE_PATTERN] [--ignore_dir IG‐
25 NORE_DIR] [--shuffle] [--shuffle-seed SHUFFLE_SEED] [--version]
26 [-j PROCESSES] [--keepbytecode] [--usecompiled]
27 [--exit-with-status] [legacy_module_filter] [legacy_test_filter]
28 Discover and run unittest tests
30 positional arguments:
32 legacy_module_filter
33 DEPRECATED: Prefer to use --module.
34 legacy_test_filter
36 DEPRECATED: Prefer to use --test.
37 options:
39 -h, --help
40 show this help message and exit
41 Searching and filtering:
43 Options in this group are used to define which tests to run.
44 --package PACKAGE, --dir PACKAGE, -s PACKAGE
46 Search the given package's directories for tests. This can be
47 specified more than once to run tests in multiple parts of the
48 source tree. For example, if refactoring interfaces, you don't
49 want to see the way you have broken setups for tests in other
50 packages. You *just* want to run the interface tests. Packages
51 are supplied as dotted names. For compatibility with the old
52 test runner, forward and backward slashed in package names are
53 converted to dots. (In the special case of packages spread over
54 multiple directories, only directories within the test search
55 path are searched. See the --path option.)
56 --module MODULE, -m MODULE
58 Specify a test-module filter as a regular expression. This is a
59 case-sensitive regular expression, used in search (not match)
60 mode, to limit which test modules are searched for tests. The
61 regular expressions are checked against dotted module names. In
62 an extension of Python regexp notation, a leading "!" is
63 stripped and causes the sense of the remaining regexp to be
64 negated (so "!bc" matches any string that does not match "bc",
65 and vice versa). The option can be specified multiple test-mod‐
66 ule filters. Test modules matching any of the test filters are
67 searched. If no test-module filter is specified, then all test
68 modules are used.
69 --test TEST, -t TEST
71 Specify a test filter as a regular expression. This is a
72 case-sensitive regular expression, used in search (not match)
73 mode, to limit which tests are run. In an extension of Python
74 regexp notation, a leading "!" is stripped and causes the sense
75 of the remaining regexp to be negated (so "!bc" matches any
76 string that does not match "bc", and vice versa). The option can
77 be specified multiple test filters. Tests matching any of the
78 test filters are included. If no test filter is specified, then
79 all tests are run.
80 --unit, -u
82 Run only unit tests, ignoring any layer options.
83 --non-unit, -f
85 Run tests other than unit tests.
86 --layer LAYER
88 Specify a test layer to run. The option can be given multiple
89 times to specify more than one layer. If not specified, all lay‐
90 ers are run. It is common for the running script to provide de‐
91 fault values for this option. Layers are specified regular ex‐
92 pressions, used in search mode, for dotted names of objects that
93 define a layer. In an extension of Python regexp notation, a
94 leading "!" is stripped and causes the sense of the remaining
95 regexp to be negated (so "!bc" matches any string that does not
96 match "bc", and vice versa). The layer named 'zope.testrun‐
97 ner.layer.UnitTests' is reserved for unit tests, however, take
98 note of the --unit and non-unit options.
99 -a AT_LEVEL, --at-level AT_LEVEL
101 Run the tests at the given level. Any test at a level at or be‐
102 low this is run, any test at a level above this is not run.
103 Level 0 runs all tests.
104 --all Run tests at all levels.
106 --list-tests
108 List all tests that matched your filters. Do not run any tests.
109 --require-unique
111 Require that all test IDs be unique and raise an error if dupli‐
112 cates are encountered.
113 Reporting:
115 Reporting options control basic aspects of test-runner output
116 --verbose, -v
118 Make output more verbose. Increment the verbosity level.
119 --quiet, -q
121 Make the output minimal, overriding any verbosity options.
122 --progress, -p
124 Output progress status
125 --no-progress
127 Do not output progress status. This is the default, but can be
128 used to counter a previous use of --progress or -p.
129 --auto-progress
131 Output progress status, but only when stdout is a terminal.
132 --color, -c
134 Colorize the output.
135 --no-color, -C
137 Do not colorize the output. This is the default, but can be used
138 to counter a previous use of --color or -c.
139 --auto-color
141 Colorize the output, but only when stdout is a terminal.
142 --subunit
144 Use subunit v1 output. Will not be colorized.
145 --subunit-v2
147 Use subunit v2 output. Will not be colorized.
148 --slow-test N
150 With -c and -vvv, highlight tests that take longer than N sec‐
151 onds (default: 10).
152 -1, --hide-secondary-failures
154 Report only the first failure in a doctest. (Examples after the
155 failure are still executed, in case they do any cleanup.)
156 --show-secondary-failures
158 Report all failures in a doctest. This is the default, but can
159 be used to counter a default use of -1 or --hide-secondary-fail‐
160 ures.
161 --ndiff
163 When there is a doctest failure, show it as a diff using the
164 ndiff.py utility.
165 --udiff
167 When there is a doctest failure, show it as a unified diff.
168 --cdiff
170 When there is a doctest failure, show it as a context diff.
171 --ignore-new-thread REGEXP
173 If a thread with this name is left behind, don't report this at
174 the end. This is a case-sensitive regular expression, used in
175 match mode. This option can be used multiple times. If a thread
176 name matches any of them, it will be ignored.
177 --buffer
179 Buffer the standard output and standard error streams during
180 each test. Output during a passing test is discarded. Output
181 during failing or erroring tests is echoed. This option is en‐
182 abled by default if --subunit or --subunit-v2 is in use, to
183 avoid corrupting the subunit stream.
184 Analysis:
186 Analysis options provide tools for analysing test output.
187 --stop-on-error, --stop, -x
189 Stop running tests after first test failure or error.
190 --post-mortem, --pdb, -D
192 Enable post-mortem debugging of test failures
193 --gc GC, -g GC
195 Set the garbage collector generation threshold. This can be used
196 to stress memory and gc correctness. Some crashes are only re‐
197 producible when the threshold is set to 1 (aggressive garbage
198 collection). Do "--gc 0" to disable garbage collection alto‐
199 gether. The --gc option can be used up to 3 times to specify up
200 to 3 of the 3 Python gc_threshold settings.
201 --gc-option {DEBUG_STATS,DEBUG_LEAK,DEBUG_OBJECTS,DEBUG_COLLECTABLE,DE‐
203 BUG_SAVEALL,DEBUG_UNCOLLECTABLE,DEBUG_INSTANCES}, -G {DEBUG_STATS,DE‐
204 BUG_LEAK,DEBUG_OBJECTS,DEBUG_COLLECTABLE,DEBUG_SAVEALL,DEBUG_UNCOL‐
205 LECTABLE,DEBUG_INSTANCES}
206 Set a Python gc-module debug flag. This option can be used more
207 than once to set multiple flags.
208 --gc-after-test
210 After each test, call 'gc.collect' and record the return value
211 *rv*; when *rv* is non-zero, output '!' on verbosity level 1
212 and '[*rv*]' on higher verbosity levels. On verbosity level 4 or
213 higher output detailed cycle information.
214 --repeat REPEAT, -N REPEAT
216 Repeat the tests the given number of times. This option is used
217 to make sure that tests leave their environment in the state
218 they found it and, with the --report-refcounts option to look
219 for memory leaks.
220 --report-refcounts, -r
222 After each run of the tests, output a report summarizing changes
223 in refcounts by object type. This option that requires that
224 Python was built with the --with-pydebug option to configure.
225 --coverage COVERAGE
227 Perform code-coverage analysis, saving trace data to the direc‐
228 tory with the given name. A code coverage summary is printed to
229 standard out.
230 --profile {cProfile}
232 Run the tests under cProfiler and display the top 50 stats,
233 sorted by cumulative time and number of calls.
234 --profile-directory PROF_DIR
236 Directory for temporary profiler files. All files named
237 tests_profile.*.prof in this directory will be removed. If you
238 intend to run multiple instances of the test runner in parallel,
239 be sure to tell them to use different directories, so they won't
240 step on each other's toes.
241 Setup:
243 Setup options are normally supplied by the testrunner script,
244 although they can be overridden by users.
245 --path PATH
247 Specify a path to be added to Python's search path. This option
248 can be used multiple times to specify multiple search paths. The
249 path is usually specified by the test-runner script itself,
250 rather than by users of the script, although it can be overrid‐
251 den by users. Only tests found in the path will be run. This
252 option also specifies directories to be searched for tests. See
253 the search_directory.
254 --test-path TEST_PATH
256 Specify a path to be searched for tests, but not added to the
257 Python search path. This option can be used multiple times to
258 specify multiple search paths. The path is usually specified by
259 the test-runner script itself, rather than by users of the
260 script, although it can be overridden by users. Only tests found
261 in the path will be run.
262 --package-path ARG ARG
264 Specify a path to be searched for tests, but not added to the
265 Python search path. Also specify a package for files found in
266 this path. This is used to deal with directories that are
267 stitched into packages that are not otherwise searched for
268 tests. This option takes 2 arguments specifying the path and the
269 package. This option can be used multiple times to specify mul‐
270 tiple search paths. The path is usually specified by the
271 test-runner script itself, rather than by users of the script,
272 although it can be overridden by users. Only tests found in the
273 path will be run.
274 --tests-pattern TESTS_PATTERN
276 The test runner looks for modules containing tests. It uses this
277 pattern to identify these modules. The modules may be either
278 packages or python files. If a test module is a package, it uses
279 the value given by the test-file-pattern to identify python
280 files within the package containing tests.
281 --suite-name SUITE_NAME
283 Specify the name of the object in each test_module that contains
284 the module's test suite.
285 --test-file-pattern TEST_FILE_PATTERN
287 Specify a pattern for identifying python files within a tests
288 package. See the documentation for the --tests-pattern option.
289 --ignore_dir IGNORE_DIR
291 Specifies the name of a directory to ignore when looking for
292 tests.
293 --shuffle
295 Shuffles the order in which tests are ran.
296 --shuffle-seed SHUFFLE_SEED
298 Value used to initialize the tests shuffler. Specify a value to
299 create repeatable random ordered tests.
300 Other:
302 Other options
303 --version
305 Print the version of the testrunner, and exit.
306 -j PROCESSES
308 Use up to given number of parallel processes to execute tests.
309 May decrease test run time substantially. Defaults to 1.
310 --keepbytecode, -k
312 Normally, the test runner scans the test paths and the test di‐
313 rectories looking for and deleting pyc or pyo files without cor‐
314 responding py files. This is to prevent spurious test failures
315 due to finding compiled modules where source modules have been
316 deleted. This scan can be time consuming. Using this option dis‐
317 ables this scan. If you know you haven't removed any modules
318 since last running the tests, can make the test run go much
319 faster.
320 --usecompiled
322 Normally, a package must contain an __init__.py file, and only
323 .py files can contain test code. When this option is specified,
324 compiled Python files (.pyc and .pyo) can be used instead: a di‐
325 rectory containing __init__.pyc or __init__.pyo is also consid‐
326 ered to be a package, and if file XYZ.py contains tests but is
327 absent while XYZ.pyc or XYZ.pyo exists then the compiled files
328 will be used. This is necessary when running tests against a
329 tree where the .py files have been removed after compilation to
330 .pyc/.pyo. Use of this option implies --keepbytecode.
331 --exit-with-status
333 DEPRECATED: The test runner will always exit with a status.
334zope.testrunner version 5.5 June 2022 ZOPE.TESTRUNNER(1)