1GABBI(1) Gabbi GABBI(1)
2
6 gabbi - Gabbi Documentation
7 Gabbi tests are expressed in YAML as a series of HTTP requests with
9 their expected response:
10 tests:
12 - name: retrieve root
13 GET: /
14 status: 200
15 This will trigger a GET request to / on the configured Target Host. The
17 test will pass if the response's status code is 200.
18
20 The top-level tests category contains an ordered sequence of test dec‐
21 larations, each describing the expected response to a given request:
22 Metadata
24 ┌───────────────┬─────────────────────┬───────────────────┐
25 │Key │ Description │ Notes │
26 ├───────────────┼─────────────────────┼───────────────────┤
27 │name │ The test's name. │ required │
28 │ │ Must be unique │ │
29 │ │ within a file. │ │
30 ├───────────────┼─────────────────────┼───────────────────┤
31 │desc │ An arbitrary string │ │
32 │ │ describing the │ │
33 │ │ test. │ │
34 ├───────────────┼─────────────────────┼───────────────────┤
35 │verbose │ If True or all │ defaults to False │
36 │ │ (synonymous), │ │
37 │ │ prints a represen‐ │ │
38 │ │ tation of the cur‐ │ │
39 │ │ rent request and │ │
40 │ │ response to stdout, │ │
41 │ │ including both │ │
42 │ │ headers and body. │ │
43 │ │ If set to headers │ │
44 │ │ or body, only the │ │
45 │ │ corresponding part │ │
46 │ │ of the request and │ │
47 │ │ response will be │ │
48 │ │ printed. If the │ │
49 │ │ output is a TTY, │ │
50 │ │ colors will be │ │
51 │ │ used. If the body │ │
52 │ │ content-type is │ │
53 │ │ JSON it will be │ │
54 │ │ formatted for im‐ │ │
55 │ │ proved readability. │ │
56 │ │ See VerboseHttp for │ │
57 │ │ details. │ │
58 ├───────────────┼─────────────────────┼───────────────────┤
59 │skip │ A string message │ defaults to False │
60 │ │ which if set will │ │
61 │ │ cause the test to │ │
62 │ │ be skipped with the │ │
63 │ │ provided message. │ │
64 └───────────────┴─────────────────────┴───────────────────┘
65 │xfail │ Determines whether │ defaults to False │
68 │ │ to expect this test │ │
69 │ │ to fail. Note that │ │
70 │ │ the test will be │ │
71 │ │ run anyway. │ │
72 ├───────────────┼─────────────────────┼───────────────────┤
73 │use_prior_test │ Determines if this │ defaults to True │
74 │ │ test will be run in │ │
75 │ │ sequence (after) │ │
76 │ │ the test prior to │ │
77 │ │ it in the list of │ │
78 │ │ tests within a │ │
79 │ │ file. To be con‐ │ │
80 │ │ crete, when this is │ │
81 │ │ True the test is │ │
82 │ │ dependent on the │ │
83 │ │ prior test and if │ │
84 │ │ that prior has not │ │
85 │ │ yet run, it wil be │ │
86 │ │ run, even if only │ │
87 │ │ the current test │ │
88 │ │ has been selected. │ │
89 │ │ Set this to False │ │
90 │ │ to allow selecting │ │
91 │ │ a test without de‐ │ │
92 │ │ pendencies. │ │
93 ├───────────────┼─────────────────────┼───────────────────┤
94 │cert_validate │ States whether the │ defaults to True │
95 │ │ underlying HTTP │ │
96 │ │ client should at‐ │ │
97 │ │ tempt to validate │ │
98 │ │ SSL certificates. │ │
99 │ │ In many test envi‐ │ │
100 │ │ ronment certifi‐ │ │
101 │ │ cates will be │ │
102 │ │ self-signed so │ │
103 │ │ changing this may │ │
104 │ │ be requried. It can │ │
105 │ │ also be changed │ │
106 │ │ when Loading and │ │
107 │ │ Running Tests or │ │
108 │ │ using gabbi-run. │ │
109 ├───────────────┼─────────────────────┼───────────────────┤
110 │disable_re‐ │ If True, means that │ defaults to False │
111 │sponse_handler │ the response body │ │
112 │ │ will not be pro‐ │ │
113 │ │ cessed to Python │ │
114 │ │ data. This can be │ │
115 │ │ necessary if a re‐ │ │
116 │ │ sponse claims a │ │
117 │ │ content-type but │ │
118 │ │ the body is not ac‐ │ │
119 │ │ tually that type │ │
120 │ │ but it is still │ │
121 │ │ necessary to run │ │
122 │ │ tests against the │ │
123 │ │ response. In that │ │
124 │ │ situation, if dis‐ │ │
125 │ │ able_response_han‐ │ │
126 │ │ dler is False the │ │
127 │ │ test will be │ │
128 │ │ treated as a fail‐ │ │
129 │ │ ure. │ │
130 └───────────────┴─────────────────────┴───────────────────┘
131 NOTE:
133 When tests are generated dynamically, the TestCase name will include
134 the respective test's name, lowercased with spaces transformed to _.
135 In at least some test runners this will allow you to select and fil‐
136 ter on test name.
137 Request Parameters
139 ┌────────────────────┬───────────────────────┬─────────────────────┐
140 │Key │ Description │ Notes │
141 ├────────────────────┼───────────────────────┼─────────────────────┤
142 │any uppercase │ Any such key is │ │
143 │string │ considered an HTTP │ │
144 │ │ method, with the │ │
145 │ │ corresponding value │ │
146 │ │ expressing the URL. │ │
147 │ │ │ │
148 │ │ This is a shortcut │ │
149 │ │ combining method │ │
150 │ │ and url into a sin‐ │ │
151 │ │ gle statement: │ │
152 │ │ │ │
153 │ │ GET: /index │ │
154 │ │ │ │
155 │ │ corresponds │ │
156 │ │ to: │ │
157 │ │ │ │
158 │ │ method: GET │ │
159 │ │ url: /index │ │
160 ├────────────────────┼───────────────────────┼─────────────────────┤
161 │method │ The HTTP request │ defaults to GET │
162 │ │ method. │ │
163 ├────────────────────┼───────────────────────┼─────────────────────┤
164 │url │ The URL to request. │ Either this or the │
165 │ │ This can either be a │ shortcut above is │
166 │ │ full path (e.g. "/in‐ │ required │
167 │ │ dex") or a fully │ │
168 │ │ qualified URL (i.e. │ │
169 │ │ including host and │ │
170 │ │ scheme, e.g. "‐ │ │
171 │ │ http://exam‐ │ │
172 │ │ ple.org/index") — see │ │
173 │ │ Target Host for de‐ │ │
174 │ │ tails. │ │
175 ├────────────────────┼───────────────────────┼─────────────────────┤
176 │request_headers │ A dictionary of │ │
177 │ │ key-value pairs rep‐ │ │
178 │ │ resenting request │ │
179 │ │ header names and val‐ │ │
180 │ │ ues. These will be │ │
181 │ │ added to the con‐ │ │
182 │ │ structed request. │ │
183 ├────────────────────┼───────────────────────┼─────────────────────┤
184 │query_parameters │ A dictionary of query │ │
185 │ │ parameters that will │ │
186 │ │ be added to the url │ │
187 │ │ as query string. If │ │
188 │ │ that URL already con‐ │ │
189 │ │ tains a set of query │ │
190 │ │ parameters, those wil │ │
191 │ │ be extended. See │ │
192 │ │ Example Tests for a │ │
193 │ │ demonstration of how │ │
194 │ │ the data is struc‐ │ │
195 │ │ tured. │ │
196 └────────────────────┴───────────────────────┴─────────────────────┘
197 │data │ A representation to │ │
200 │ │ pass as the body of a │ │
201 │ │ request. Note that │ │
202 │ │ content-type in re‐ │ │
203 │ │ quest_headers should │ │
204 │ │ also be set — see │ │
205 │ │ Data for details. │ │
206 ├────────────────────┼───────────────────────┼─────────────────────┤
207 │redirects │ If True, redirects │ defaults to False │
208 │ │ will automatically be │ │
209 │ │ followed. │ │
210 ├────────────────────┼───────────────────────┼─────────────────────┤
211 │ssl │ Determines whether │ defaults to False │
212 │ │ the request uses SSL │ │
213 │ │ (i.e. HTTPS). Note │ │
214 │ │ that the url's scheme │ │
215 │ │ takes precedence if │ │
216 │ │ present — see Target │ │
217 │ │ Host for details. │ │
218 └────────────────────┴───────────────────────┴─────────────────────┘
219 Response Expectations
221 ┌────────────────────┬─────────────────────┬─────────────────┐
222 │Key │ Description │ Notes │
223 ├────────────────────┼─────────────────────┼─────────────────┤
224 │status │ The expected re‐ │ defaults to 200 │
225 │ │ sponse status code. │ │
226 │ │ Multiple acceptable │ │
227 │ │ response codes may │ │
228 │ │ be provided, sepa‐ │ │
229 │ │ rated by || (e.g. │ │
230 │ │ 302 || 301 — note, │ │
231 │ │ however, that this │ │
232 │ │ indicates ambigu‐ │ │
233 │ │ ity, which is gen‐ │ │
234 │ │ erally undesir‐ │ │
235 │ │ able). │ │
236 ├────────────────────┼─────────────────────┼─────────────────┤
237 │response_headers │ A dictionary of │ │
238 │ │ key-value pairs │ │
239 │ │ representing ex‐ │ │
240 │ │ pected response │ │
241 │ │ header names and │ │
242 │ │ values. If a │ │
243 │ │ header's value is │ │
244 │ │ wrapped in /.../, │ │
245 │ │ it will be treated │ │
246 │ │ as a regular ex‐ │ │
247 │ │ pression to search │ │
248 │ │ for in the response │ │
249 │ │ header. │ │
250 ├────────────────────┼─────────────────────┼─────────────────┤
251 │response_forbid‐ │ A list of headers │ │
252 │den_headers │ which must not be │ │
253 │ │ present. │ │
254 ├────────────────────┼─────────────────────┼─────────────────┤
255 │response_strings │ A list of string │ │
256 │ │ fragments expected │ │
257 │ │ to be present in │ │
258 │ │ the response body. │ │
259 └────────────────────┴─────────────────────┴─────────────────┘
260 │response_json_paths │ A dictionary of │ │
266 │ │ JSONPath rules │ │
267 │ │ paired with ex‐ │ │
268 │ │ pected matches. Us‐ │ │
269 │ │ ing this rule re‐ │ │
270 │ │ quires that the │ │
271 │ │ content being sent │ │
272 │ │ from the server is │ │
273 │ │ JSON (i.e. a con‐ │ │
274 │ │ tent type of appli‐ │ │
275 │ │ cation/json or con‐ │ │
276 │ │ taining +json) │ │
277 │ │ │ │
278 │ │ If the value is │ │
279 │ │ wrapped in /.../ │ │
280 │ │ the result of the │ │
281 │ │ JSONPath query will │ │
282 │ │ be searched for the │ │
283 │ │ value as a regular │ │
284 │ │ expression. │ │
285 ├────────────────────┼─────────────────────┼─────────────────┤
286 │poll │ A dictionary of two │ │
287 │ │ keys: │ │
288 │ │ │ │
289 │ │ • count: An │ │
290 │ │ integer │ │
291 │ │ stating │ │
292 │ │ the number │ │
293 │ │ of times │ │
294 │ │ to attempt │ │
295 │ │ this test │ │
296 │ │ before │ │
297 │ │ giving up. │ │
298 │ │ │ │
299 │ │ • delay: A │ │
300 │ │ floating │ │
301 │ │ point num‐ │ │
302 │ │ ber of │ │
303 │ │ seconds to │ │
304 │ │ delay be‐ │ │
305 │ │ tween at‐ │ │
306 │ │ tempts. │ │
307 │ │ │ │
308 │ │ This makes │ │
309 │ │ it possible │ │
310 │ │ to poll for │ │
311 │ │ a resource │ │
312 │ │ created via │ │
313 │ │ an asynchro‐ │ │
314 │ │ nous re‐ │ │
315 │ │ quest. Use │ │
316 │ │ with cau‐ │ │
317 │ │ tion. │ │
318 └────────────────────┴─────────────────────┴─────────────────┘
319 Note that many of these items allow substitutions.
321 Default values for a file's tests may be provided via the top-level de‐
323 faults category. These take precedence over the global defaults (ex‐
324 plained below).
325 For examples see the gabbi tests, Example Tests and the gabbi-demo tu‐
327 torial.
328
330 The top-level fixtures category contains a sequence of named Fixtures.
331
333 response_* keys are examples of Response Handlers. Custom handlers may
334 be created by test authors for specific use cases. See Content Handlers
335 for more information.
336
338 There are a number of magical variables that can be used to make refer‐
339 ence to the state of a current test, the one just prior or any test
340 prior to the current one. The variables are replaced with real values
341 during test processing.
342 Global
344 • $ENVIRON['<environment variable>']: The name of an environment vari‐
345 able. Its value will replace the magical variable. If the string
346 value of the environment variable is "True" or "False" then the re‐
347 sulting value will be the corresponding boolean, not a string.
348 Current Test
350 • $SCHEME: The current scheme/protocol (usually http or https).
351 • $NETLOC: The host and potentially port of the request.
353 Immediately Prior Test
355 • $COOKIE: All the cookies set by any Set-Cookie headers in the prior
356 response, including only the cookie key and value pairs and no meta‐
357 data (e.g. expires or domain).
358 • $URL: The URL defined in the prior request, after substitutions have
360 been made. For backwards compatibility with earlier releases
361 $LAST_URL may also be used, but if $HISTORY (see below) is being
362 used, $URL must be used.
363 • $LOCATION: The location header returned in the prior response.
365 • $HEADERS['<header>']: The value of any header from the prior re‐
367 sponse.
368 • $RESPONSE['<json path>']: A JSONPath query into the prior response.
370 See JSONPath for more on formatting.
371 Any Previous Test
373 • $HISTORY['<test name>'].<magical variable expression>: Any variable
374 which refers to a prior test may be used in an expression that refers
375 to any earlier test in the same file by identifying the target test
376 by its name in a $HISTORY dictionary. For example, to refer to a
377 value in a JSON object in the response of a test named post json:
378 $HISTORY['post json'].$RESPONSE['$.key']
380 This is a very powerful feature that could lead to test that are dif‐
382 ficult for humans to read. Take care to optimize for the maintainers
383 that will come after you, not yourself.
384 Casting
386 For $ENVIRON and $RESPONSE it is possible to attempt to cast the value
387 to another type: int, float, str, or bool. If the cast fails an excep‐
388 tion will be raised and the test will fail.
389 This functionality only works when the magical variable is the whole
391 value of a YAML entry. If the variable is intermixed with other data,
392 an exception will be raised and the test will fail.
393 The format for a cast is to append a : and the cast type after the type
395 of the magical variable. For example:
396 $RESPONSE:int['$.some_string_value']
398 WARNING:
400 Prior to the introduction of this feature, $ENVIRON would already do
401 some automatic casting of numbers to ints and floats and the strings
402 True and False to booleans. This continues to be the case, but only
403 if no cast is provided.
404 NOTE:
406 Where a single-quote character, ', is shown in the variables above
407 you may also use a double-quote character, ", but in any given ex‐
408 pression the same character must be used at both ends.
409 All of these variables may be used in all of the following fields:
411 • skip
413 • url
415 • query_parameters
417 • data
419 • request_headers (in both the key and value)
421 • response_strings
423 • response_json_paths (in both the key and value, see json path substi‐
425 tution for more info)
426 • response_headers (in both the key and value)
428 • response_forbidden_headers
430 • count and delay fields of poll
432 With these variables it ought to be possible to traverse an API without
434 any explicit statements about the URLs being used. If you need a re‐
435 placement on a field that is not currently supported please raise an
436 issue or provide a patch.
437 As all of these features needed to be tested in the development of
439 gabbi itself, the gabbi tests are a good source of examples on how to
440 use the functionality. See also Example Tests for a collection of exam‐
441 ples and the gabbi-demo tutorial.
442
444 The data key has some special handing to allow for a bit more flexibil‐
445 ity when doing a POST or PUT:
446 • If the value is not a string (that is, it is a sequence or structure)
448 it is treated as a data structure that will be turned into a string
449 by the dumps method on the relevant content handler. For example if
450 the content-type of the body is application/json the data structure
451 will be turned into a JSON string.
452 • If the value is a string that begins with <@ then the rest of the
454 string is treated as a filepath to be loaded. The path is relative to
455 the test directory and may not traverse up into parent directories.
456 • If the value is an undecorated string, that's the value.
458 NOTE:
460 When reading from a file care should be taken to ensure that a rea‐
461 sonable content-type is set for the data as this will control if any
462 encoding is done of the resulting string value. If it is text, json,
463 xml or javascript it will be encoded to UTF-8.
464 To run gabbi tests with a test harness they must be generated in some
466 fashion and then run. This is accomplished by a test loader. Initially
467 gabbi only supported those test harnesses that supported the load_tests
468 protocol in UnitTest. It now possible to also build and run tests with
469 pytest with some limitations described below.
470 NOTE:
472 It is also possible to run gabbi tests from the command line. See
473 YAML Runner.
474 NOTE:
476 By default gabbi will load YAML files using the safe_load function.
477 This means only basic YAML types are allowed in the file. For most
478 use cases this is fine. If you need custom types (for example, to
479 match NaN) it is possible to set the safe_yaml parameter of
480 build_tests() to False. If custom types are used, please keep in
481 mind that this can limit the portability of the YAML files to other
482 contexts.
483 WARNING:
485 If test are being run with a runner that supports concurrency (such
486 as testrepository) it is critical that the test runner is informed
487 of how to group the tests into their respective suites. The usual
488 way to do this is to use a regular expression that groups based on
489 the name of the yaml files. For example, when using testrepository
490 the .testr.conf file needs an entry similar to the following:
491 group_regex=gabbi\.suitemaker\.(test_[^_]+_[^_]+)
493
495 To run the tests with a load_tests style loader a test file containing
496 a load_tests method is required. That will look a bit like:
497 """A sample test module."""
499 # For pathname munging
501 import os
502 # The module that build_tests comes from.
504 from gabbi import driver
505 # We need access to the WSGI application that hosts our service
507 from myapp import wsgiapp
508 # We're using fixtures in the YAML files, we need to know where to
511 # load them from.
512 from myapp.test import fixtures
513 # By convention the YAML files are put in a directory named
515 # "gabbits" that is in the same directory as the Python test file.
516 TESTS_DIR = 'gabbits'
517 def load_tests(loader, tests, pattern):
520 """Provide a TestSuite to the discovery process."""
521 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
522 # Pass "require_ssl=True" as an argument to force all tests
523 # to use SSL in requests.
524 return driver.build_tests(test_dir, loader,
525 intercept=wsgiapp.app,
526 fixture_module=fixtures)
527 For details on the arguments available when building tests see
530 build_tests().
531 Once the test loader has been created, it needs to be run. There are
533 many options. Which is appropriate depends very much on your environ‐
534 ment. Here are some examples using unittest or testtools that require
535 minimal knowledge to get started.
536 By file:
538 python -m testtools.run -v test/test_loader.py
540 By module:
542 python -m testttols.run -v test.test_loader
544 python -m unittest -v test.test_loader
546 Using test discovery to locate all tests in a directory tree:
548 python -m testtools.run discover
550 python -m unittest discover test
552 See the source distribution and the tutorial repo for more advanced op‐
554 tions, including using testrepository and subunit.
555
557 Since pytest does not support the load_tests system, a different way of
558 generating tests is required. Two techniques are supported.
559 The original method (described below) used yield statements to generate
561 tests which pytest would collect. This style of tests is deprecated as
562 of pytest>=3.0 so a new style using pytest fixtures has been developed.
563 WARNING:
565 The pytest loader now requires that test_loader_name be set when
566 gabbi.driver.py_test_generator() is called.
567 pytest >= 3.0
569 In the newer technique, a test file is created that uses the
570 pytest_generate_tests hook. Special care must be taken to always import
571 the test_pytest method which is the base test that the pytest hook
572 parametrizes to generate the tests from the YAML files. Without the
573 method, the hook will not be called and no tests generated.
574 Here is a simple example file:
576 """A sample pytest module for pytest >= 3.0."""
578 # For pathname munging
580 import os
581 # The module that py_test_generator comes from.
583 from gabbi import driver
584 # We need test_pytest so that pytest test collection works properly.
586 # Without this, the pytest_generate_tests method below will not be
587 # called.
588 from gabbi.driver import test_pytest # noqa
589 # We need access to the WSGI application that hosts our service
591 from myapp import wsgiapp
592 # We're using fixtures in the YAML files, we need to know where to
594 # load them from.
595 from myapp.test import fixtures
596 # By convention the YAML files are put in a directory named
598 # "gabbits" that is in the same directory as the Python test file.
599 TESTS_DIR = 'gabbits'
600 def pytest_generate_tests(metafunc):
603 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
604 driver.py_test_generator(
605 test_dir, intercept=wsgiapp.app,
606 test_loader_name=__name__,
607 fixture_module=fixtures, metafunc=metafunc)
608 This can then be run with the usual pytest commands. For example:
611 py.test -svx pytest3.0-example.py
613 pytest < 3.0
615 When using the older technique, test file must be created that calls
616 py_test_generator() and yields the generated tests. That will look a
617 bit like this:
618 """A sample pytest module."""
620 # For pathname munging
622 import os
623 # The module that build_tests comes from.
625 from gabbi import driver
626 # We need access to the WSGI application that hosts our service
628 from myapp import wsgiapp
629 # We're using fixtures in the YAML files, we need to know where to
631 # load them from.
632 from myapp.test import fixtures
633 # By convention the YAML files are put in a directory named
635 # "gabbits" that is in the same directory as the Python test file.
636 TESTS_DIR = 'gabbits'
637 def test_gabbits():
640 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
641 # Pass "require_ssl=True" as an argument to force all tests
642 # to use SSL in requests.
643 test_generator = driver.py_test_generator(
644 test_loader_name=__name__,
645 test_dir, intercept=wsgiapp.app,
646 fixture_module=fixtures)
647 for test in test_generator:
649 yield test
650 This can then be run with the usual pytest commands. For example:
653 py.test -svx pytest-example.py
655 The older technique will continue to work with all versions of
657 pytest<4.0 but >=3.0 will produce warnings. If you want to use the
658 older technique but not see the warnings add --disable-pytest-warnings
659 parameter to the invocation of py.test.
660 What follows is a commented example of some tests in a single file
662 demonstrating many of the Test Format features. See Loading and Running
663 Tests for the Python needed to integrate with a testing harness.
664 # Fixtures can be used to set any necessary configuration, such as a
667 # persistence layer, and establish sample data. They operate per
668 # file. They are context managers, each one wrapping the next in the
669 # sequence.
670 fixtures:
672 - ConfigFixture
673 - SampleDataFixture
674 # There is an included fixture named "SkipAllFixture" which can be
676 # used to declare that all the tests in the given file are to be
677 # skipped.
678 # Each test file can specify a set of defaults that will be used for
680 # every request. This is useful for always specifying a particular
681 # header or always requiring SSL. These values will be used on every
682 # test in the file unless overriden. Lists and dicts are merged one
683 # level deep, except for "data" which is copied verbatim whether it
684 # is a string, list or dict (it can be all three).
685 defaults:
687 ssl: True
688 request_headers:
689 x-my-token: zoom
690 # The tests themselves are a list under a "tests" key. It's useful
692 # to use plenty of whitespace to help readability.
693 tests:
695 # Each request *must* have a name which is unique to the file. When it
697 # becomes a TestCase the name will be lowercased and spaces will
698 # become "_". Use that generated name when limiting test runs.
699 - name: a test for root
701 desc: Some explanatory text that could be used by other tooling
702 # The URL can either be relative to a host specified elsewhere or
704 # be a fully qualified "http" or "https" URL. *You* are responsible
705 # for url-encoding the URL.
706 url: /
708 method: GET
709 # If no status or method are provided they default to "200" and
711 # "GET".
712 # Instead of explicitly stating "url" and "method" you can join
714 # those two keys into one key representing the method. The method
715 # *must* be uppercase.
716 - name: another test for root
718 desc: Same test as above but with GET key
719 GET: /
720 # A single test can override settings in defaults (set above).
722 - name: root without ssl redirects
724 ssl: False
725 GET: /
726 status: 302
727 # When evaluating response headers it is possible to use a regular
729 # expression to not have to test the whole value. Regular expressions match
730 # anywhere in the output, not just at the beginning.
731 response_headers:
733 location: /^https/
734 # By default redirects will not be followed. This can be changed.
736 - name: follow root without ssl redirect
738 ssl: False
739 redirects: True
740 GET: /
741 status: 200 # This is the response code after the redirect.
742 # URLs can express query parameters in two ways: either in the url
744 # value directly, or as query_parameters. If both are used then
745 # query_parameters are appended. In this example the resulting URL
746 # will be equivalient to
747 # /foo?section=news&article=1&article=2&date=yesterday
748 # but not necessarily in that order.
749 - name: create a url with parameters
751 GET: /foo?section=news
752 query_parameters:
753 article:
754 - 1
755 - 2
756 date: yesterday
757 # Request headers can be used to declare media-type choices and
759 # experiment with authorization handling (amongst other things).
760 # Response headers allow evaluating headers in the response. These
761 # two together form the core value of gabbi.
762 - name: test accept
764 GET: /resource
765 request_headers:
766 accept: application/json
767 response_headers:
768 content-type: /application/json/
769 # If a header must not be present in a response at all that can be
771 # expressed in a test as follows.
772 - name: test forbidden headers
774 GET: /resource
775 response_forbidden_headers:
776 - x-special-header
777 # All of the above requests have defaulted to a "GET" method. When
779 # using "POST", "PUT" or "PATCH", the "data" key provides the
780 # request body.
781 - name: post some text
783 POST: /text_repo
784 request_headers:
785 content-type: text/plain
786 data: "I'm storing this"
787 status: 201
788 # If the data is not a string, it will be transformed into JSON.
790 # You must supply an appropriate content-type request header.
791 - name: post some json
793 POST: /json_repo
794 request_headers:
795 content-type: application/json
796 data:
797 name: smith
798 abode: castle
799 status: 201
800 # If the data is a string prepended with "<@" the value will be
802 # treated as the name of a file in the same directory as the YAML
803 # file. Again, you must supply an appropriate content-type. If the
804 # content-type is one of several "text-like" types, the content will
805 # be assumed to be UTF-8 encoded.
806 - name: post an image
808 POST: /image_repo
809 request_headers:
810 content-type: image/png
811 data: <@kittens.png
812 # A single request can be marked to be skipped.
814 - name: patch an image
816 skip: patching images not yet implemented
817 PATCH: /image_repo/12d96fb8-e78c-11e4-8c03-685b35afa334
818 # Or a single request can be marked that it is expected to fail.
820 - name: check allow headers
822 desc: the framework doesn't do allow yet
823 xfail: True
824 PUT: /post_only_url
825 status: 405
826 response_headers:
827 allow: POST
828 # The body of a response can be evaluated with response handlers.
830 # The most simple checks for a set of strings anywhere in the
831 # response. Note that the strings are members of a list.
832 - name: check for css file
834 GET: /blog/posts/12
835 response_strings:
836 - normalize.css
837 # For JSON responses, JSONPath rules can be used.
839 - name: post some json get back json
841 POST: /json_repo
842 request_headers:
843 content-type: application/json
844 data:
845 name: smith
846 abode: castle
847 status: 201
848 response_json_paths:
849 $.name: smith
850 $.abode: castle
851 # Requests run in sequence. One test can make reference to the test
853 # immediately prior using some special variables.
854 # "$LOCATION" contains the "location" header in the previous
855 # response.
856 # "$HEADERS" is a pseudo dictionary containing all the headers of
857 # the previous response.
858 # "$ENVIRON" is a pseudo dictionary providing access to the current
859 # environment.
860 # "$RESPONSE" provides access to the JSON in the prior response, via
861 # JSONPath. See http://jsonpath-rw.readthedocs.io/ for
862 # jsonpath-rw formatting.
863 # $SCHEME and $NETLOC provide access to the current protocol and
864 # location (host and port).
865 - name: get the thing we just posted
867 GET: $LOCATION
868 request_headers:
869 x-magic-exchange: $HEADERS['x-magic-exchange']
870 x-token: $ENVIRON['OS_TOKEN']
871 response_json_paths:
872 $.name: $RESPONSE['$.name']
873 $.abode: $RESPONSE['$.abode']
874 response_headers:
875 content-location: /$SCHEME://$NETLOC/
876 # For APIs where resource creation is asynchronous it can be
878 # necessary to poll for the resulting resource. First we create the
879 # resource in one test. The next test uses the "poll" key to loop
880 # with a delay for a set number of times.
881 - name: create asynch
883 POST: /async_creator
884 request_headers:
885 content-type: application/json
886 data:
887 name: jones
888 abode: bungalow
889 status: 202
890 - name: poll for created resource
892 GET: $LOCATION
893 poll:
894 count: 10 # try up to ten times
895 delay: .5 # wait .5 seconds between each try
896 response_json_paths:
897 $.name: $RESPONSE['$.name']
898 $.abode: $RESPONSE['$.abode']
899 Gabbi supports JSONPath both for validating JSON response bodies and
902 within substitutions.
903 JSONPath expressions are provided by jsonpath_rw, with jsonpath_rw_ext
905 custom extensions to address common requirements:
906 1. Sorting via sorted and [/property].
908 2. Filtering via [?property = value].
910 3. Returning the respective length via len.
912 (These apply both to arrays and key-value pairs.)
914 Here is a JSONPath example demonstrating some of these features. Given
916 JSON data as follows:
917 {
919 "pets": [
920 {"type": "cat", "sound": "meow"},
921 {"type": "dog", "sound": "woof"}
922 ]
923 }
924 If the ordering of the list in pets is predictable and reliable it is
926 relatively straightforward to test values:
927 response_json_paths:
929 # length of list is two
930 $.pets.`len`: 2
931 # sound of second item in list is woof
932 $.pets[1].sound: woof
933 If the ordering is not predictable additional effort is required:
935 response_json_paths:
937 # sort by type
938 $.pets[/type][0].sound: meow
939 # sort by type, reversed
940 $.pets[\type][0].sound: woof
941 # all the sounds
942 $.pets[/type]..sound: ['meow', 'woof']
943 # filter by type = dog
944 $.pets[?type = "dog"].sound: woof
945 If it is necessary to validate the entire JSON response use a JSONPath
947 of $:
948 response_json_paths:
950 $:
951 pets:
952 - type: cat
953 sound: meow
954 - type: dog
955 sound: woof
956 This is not a technique that should be used frequently as it can lead
958 to difficult to read tests and it also indicates that your gabbi tests
959 are being used to test your serializers and data models, not just your
960 API interactions.
961 It is also possible to read raw JSON from disk for either all or some
963 of a JSON response:
964 response_json_paths:
966 $: @<data.json
967 or:
969 response_json_paths:
971 $.pets: <@pets.json
972 $.pets[0]: <@cat.json
973 Examples like this can be found in one of gabbi's own tests.
975 If it is desired to load YAML files like the JSON ones above, two
977 things must be done:
978 1. The YAMLDiskLoadingJSONHandler custom content handler must be passed
980 to the driver through the content_handlers argument. See Extensions
981 on how to do this.
982 2. The YAML files to load must be placed in a subdirectory to prevent
984 the test runner from consuming them as test files to run:
985 response_json_paths:
987 $: @<subdir/values.yaml
988 When reading from disk you can apply the same JSONPath by adding a ':'
990 to the end of your file name. This allows you to store multiple API re‐
991 sponses into a single file to reduce file management when constructing
992 your tests.
993 Given JSON data as follows:
995 {
997 "values": [{
998 "pets": [{
999 "type": "cat",
1000 "sound": "meow"
1001 }, {
1002 "type": "dog",
1003 "sound": "woof"
1004 }]
1005 }, {
1006 "people": [{
1007 "name": "chris",
1008 "id": 1
1009 }, {
1010 "name": "justin",
1011 "id": 2
1012 }]
1013 }]
1014 }
1015 You can write your tests like the following:
1017 response_json_paths:
1019 $.pets: <@pets.json
1020 $.pets[?type = "cat"].sound: <@values.json:$.values[0].pets[?type = "cat"].sound
1021 Although placing more than one API response into a single JSON file may
1023 seem convenient, keep in mind there is a tradeoff in readability that
1024 should not be overlooked before implementing this technique.
1025 Examples like this can be found in one of gabbi's yaml-from-disk tests.
1027 There are more JSONPath examples in Example Tests and in the
1029 jsonpath_rw and jsonpath_rw_ext documentation.
1030
1032 Substitutions can be made in both the left (query) and right (expected)
1033 hand sides of the json path expression. When subtitutions are used in
1034 the query, care must be taken to ensure proper quoting of the resulting
1035 value. For example if there is a uuid (with hyphens) at $RE‐
1036 SPONSE['$.id'] then this expression may fail:
1037 $.nested.structure.$RESPONSE['$.id'].name: foobar
1039 as it will evaluate to something like:
1041 $.nested.structure.ADC8AAFC-D564-40D1-9724-7680D3C010C2.name: foobar
1043 which may be treated as an arithemtic expression by the json path
1045 parser. The test author should write:
1046 $.nested.structure["$RESPONSE['$.id']"].name: foobar
1048 to quote the result of the substitution.
1050 The target host is the host on which the API to be tested can be found.
1052 Gabbi intends to preserve the flow and semantics of HTTP interactions
1053 as much as possible, and every HTTP request needs to be directed at a
1054 host of some form. Gabbi provides three ways to control this:
1055 • Using wsgi-intercept to provide a fake socket and WSGI environment on
1057 an arbitrary host and port attached to a WSGI application (see
1058 intercept examples).
1059 • Using fully qualified url values in the YAML defined tests (see full
1061 examples).
1062 • Using a host and (optionally) port defined at test build time (see
1064 live examples).
1065 The intercept and live methods are mutually exclusive per test builder,
1067 but either kind of test can freely intermix fully qualified URLs into
1068 the sequence of tests in a YAML file.
1069 For test driven development and local tests the intercept style of
1071 testing lowers test requirements (no web server required) and is fast.
1072 Interception is performed as part of Fixtures processing as the most
1073 deeply nested fixture. This allows any configuration or database setup
1074 to be performed prior to the WSGI application being created.
1075 For the implementation of the above see build_tests().
1077 Each suite of tests is represented by a single YAML file, and may op‐
1079 tionally use one or more fixtures to provide the necessary environment
1080 required by the tests in that file.
1081 Fixtures are implemented as nested context managers. Subclasses of
1083 GabbiFixture must implement start_fixture and stop_fixture methods for
1084 creating and destroying, respectively, any resources managed by the
1085 fixture. While the subclass may choose to implement __init__ it is im‐
1086 portant that no exceptions are thrown in that method, otherwise the
1087 stack of context managers will fail in unexpected ways. Instead ini‐
1088 tialization of real resources should happen in start_fixture.
1089 At this time there is no mechanism for the individual tests to have any
1091 direct awareness of the fixtures. The fixtures exist, conceptually, on
1092 the server side of the API being tested.
1093 Fixtures may do whatever is required by the testing environment, how‐
1095 ever there are two common scenarios:
1096 • Establishing (and then resetting when a test suite has finished) any
1098 baseline configuration settings and persistence systems required for
1099 the tests.
1100 • Creating sample data for use by the tests.
1102 If a fixture raises unittest.case.SkipTest during start_fixture all the
1104 tests in the current file will be skipped. This makes it possible to
1105 skip the tests if some optional configuration (such as a particular
1106 type of database) is not available.
1107 If an exception is raised while a fixture is being used, information
1109 about the exception will be stored on the fixture so that the stop_fix‐
1110 ture method can decide if the exception should change how the fixture
1111 should clean up. The exception information can be found on exc_type,
1112 exc_value and traceback method attributes.
1113 If an exception is raised when a fixture is started (in start_fixture)
1115 the first test in the suite using the fixture will be marked with an
1116 error using the traceback from the exception and all the tests in the
1117 suite will be skipped. This ensures that fixture failure is adequately
1118 captured and reported by test runners.
1119 In some contexts (for example CI environments with a large number of
1121 tests being run in a broadly concurrent environment where output is
1122 logged to a single file) it can be important to capture and consolidate
1123 stray output that is produced during the tests and display it associ‐
1124 ated with an individual test. This can help debugging and avoids unus‐
1125 able output that is the result of multiple streams being interleaved.
1126 Inner fixtures have been added to support this. These are fixtures more
1128 in line with the tradtional unittest concept of fixtures: a class on
1129 which setUp and cleanUp is automatically called.
1130 build_tests() accepts a named parameter arguments of inner_fixtures.
1132 The value of that argument may be an ordered list of fixtures.Fixture
1133 classes that will be called when each individual test is set up.
1134 An example fixture that could be useful is the FakeLogger.
1136 NOTE:
1138 At this time inner_fixtures are not supported when using the pytest
1139 loader.
1140 Content handlers are responsible for preparing request data and evalu‐
1142 ating response data based on the content-type of the request and re‐
1143 sponse. A content handler operates as follows:
1144 • Structured YAML data provided via the data attribute is converted to
1146 a string or bytes sequence and used as request body.
1147 • The response body (a string or sequence of bytes) is transformed into
1149 a content-type dependent structure and stored in an internal attri‐
1150 bute named response_data that is:
1151 • used when evaluating the response body
1153 • used in $RESPONSE[] substitutions
1155 By default, gabbi provides content handlers for JSON. In that content
1157 handler the data test key is converted from structured YAML into a JSON
1158 string. Response bodies are converted from a JSON string into a data
1159 structure in response_data that is used when evaluating re‐
1160 sponse_json_paths entries in a test or doing JSONPath-based $RESPONSE[]
1161 substitutions.
1162 A YAMLDiskLoadingJSONHandler has been added to extend the JSON handler.
1164 It works the same way as the JSON handler except for when evaluating
1165 the response_json_paths handle, data that is read from disk can be ei‐
1166 ther in JSON or YAML format. The YAMLDiskLoadingJSONHandler is not en‐
1167 abled by default and must be added as shown in the Extensions section
1168 in order to be used in the tests.
1169 Further content handlers can be added as extensions. Test authors may
1171 need these extensions for their own suites, or enterprising developers
1172 may wish to create and distribute extensions for others to use.
1173 NOTE:
1175 One extension that is likely to be useful is a content handler that
1176 turns data into url-encoded form data suitable for POST and turns an
1177 HTML response into a DOM object.
1178
1180 Content handlers are an evolution of the response handler concept in
1181 earlier versions gabbi. To preserve backwards compatibility with exist‐
1182 ing response handlers, old style response handlers are still allowed,
1183 but new handlers should implement the content handler interface (de‐
1184 scribed below).
1185 Registering additional custom handlers is done by passing a subclass of
1187 ContentHandler to build_tests():
1188 driver.build_tests(test_dir, loader, host=None,
1190 intercept=simple_wsgi.SimpleWsgi,
1191 content_handlers=[MyContentHandler])
1192 If pytest is being used:
1194 driver.py_test_generator(test_dir, intercept=simple_wsgi.SimpleWsgi,
1196 test_loader_name=__name__,
1197 content_handlers=[MyContenHandler])
1198 Gabbi provides an additional custom handler named YAMLDiskLoadingJSON‐
1200 Handler. This can be used for loading JSON and YAML files from disk
1201 when evaluating the response_json_paths handle.
1202 WARNING:
1204 YAMLDiskLoadingJSONHandler shares the same content-type as the de‐
1205 fault JSONHandler. When there are multiple handlers listed that ac‐
1206 cept the same content-type, the one that is earliest in the list
1207 will be used.
1208 With gabbi-run, custom handlers can be loaded via the --response-han‐
1210 dler option -- see load_response_handlers() for details.
1211 NOTE:
1213 The use of the --response-handler argument is done to preserve back‐
1214 wards compatibility and avoid excessive arguments. Both types of
1215 handler may be passed to the argument.
1216 Implementation Details
1218 Creating a content handler requires subclassing ContentHandler and im‐
1219 plementing several methods. These methods are described below, but in‐
1220 specting JSONHandler will be instructive in highlighting required argu‐
1221 ments and techniques.
1222 To provide a response_<something> response-body evaluator a subclass
1224 must define:
1225 • test_key_suffix: This, along with the prefix response_, forms the key
1227 used in the test structure. It is a class level string.
1228 • test_key_value: The key's default value, either an empty list ([]) or
1230 empty dict ({}). It is a class level value.
1231 • action: An instance method which tests the expected values against
1233 the HTTP response - it is invoked for each entry, with the parameters
1234 depending on the default value. The arguments to action are (in or‐
1235 der):
1236 • self: The current instance.
1238 • test: The currently active HTTPTestCase
1240 • item: The current entry if test_key_value is a list, otherwise the
1242 key half of the key/value pair at this entry.
1243 • value: None if test_key_value is a list, otherwise the value half
1245 of the key/value pair at this entry.
1246 To translate request or response bodies to or from structured data a
1248 subclass must define an accepts method. This should return True if this
1249 class is willing to translate the provided content-type. During request
1250 processing it is given the value of the content-type header that will
1251 be sent in the request. During response processing it is given the
1252 value of the content-type header of the response. This makes it possi‐
1253 ble to handle different request and response bodies in the same han‐
1254 dler, if desired. For example a handler might accept applica‐
1255 tion/x-www-form-urlencoded and text/html.
1256 If accepts is defined two additional static methods should be defined:
1258 • dumps: Turn structured Python data from the data key in a test into a
1260 string or byte stream. The optional test param allows you to access
1261 the current test case which may help with manipulations for custom
1262 content handlers, e.g. multipart/form-data needs to add a boundary to
1263 the Content-Type header in order to mark the appropriate sections of
1264 the body.
1265 • loads: Turn a string or byte stream in a response into a Python data
1267 structure. Gabbi will put this data on the response_data attribute on
1268 the test, where it can be used in the evaluations described above (in
1269 the action method) or in $RESPONSE handling. An example usage here
1270 would be to turn HTML into a DOM.
1271 • load_data_file: Load data from disk into a Python data structure.
1273 Gabbi will call this method when response_<something> contains an
1274 item where the right hand side value starts with <@. The test param
1275 allows you to access the current test case and provides a
1276 load_data_file method which should be used because it verifies the
1277 data is loaded within the test diectory and returns the file source
1278 as a string. The load_data_file method was introduced to re-use the
1279 JSONHandler in order to support loading YAML files from disk through
1280 the implementation of an additional custom handler, see
1281 YAMLDiskLoadingJSONHandler for details.
1282 Finally if a replacer class method is defined, then when a $RESPONSE
1284 substitution is encountered, replacer will be passed the response_data
1285 of the prior test and the argument within the $RESPONSE.
1286 Please see the JSONHandler source for additional detail.
1288 If there is a running web service that needs to be tested and creating
1290 a test loader with build_tests() is either inconvenient or overkill it
1291 is possible to run YAML test files directly from the command line with
1292 the console-script gabbi-run. It accepts YAML on stdin or as multiple
1293 file arguments, and generates and runs tests and outputs a summary of
1294 the results.
1295 The provided YAML may not use custom Fixtures but otherwise uses the
1297 default Test Format. Target Host information is either expressed di‐
1298 rectly in the YAML file or provided on the command line:
1299 gabbi-run [host[:port]] < /my/test.yaml
1301 or:
1303 gabbi-run http://host:port < /my/test.yaml
1305 To test with one or more files the following command syntax may be
1307 used:
1308 gabbi-run http://host:port -- /my/test.yaml /my/other.yaml
1310 NOTE:
1312 The filename arguments must come after a -- and all other arguments
1313 (host, port, prefix, failfast) must come before the --.
1314 NOTE:
1316 If files are provided, test output will use names including the name
1317 of the file. If any single file includes an error, the name of the
1318 file will be included in a summary of failed files at the end of the
1319 test report.
1320 To facilitate using the same tests against the same application mounted
1322 in different locations in a WSGI server, a prefix may be provided as a
1323 second argument:
1324 gabbi-run host[:port] [prefix] < /my/test.yaml
1326 or in the target URL:
1328 gabbi-run http://host:port/prefix < /my/test.yaml
1330 The value of prefix will be prepended to the path portion of URLs that
1332 are not fully qualified.
1333 Anywhere host is used, if it is a raw IPV6 address it should be wrapped
1335 in [ and ].
1336 If https is used in the target, then the tests in the provided YAML
1338 will default to ssl: True.
1339 Use -k or --insecure to not validate certificates when making https
1341 connections.
1342 If a -x or --failfast argument is provided then gabbi-run will exit af‐
1344 ter the first test failure.
1345 Use -v or --verbose with a value of all, headers or body to turn on
1347 verbosity for all tests being run.
1348 Use -q or --quiet to silence test runner output.
1350 Use -r or --response-handler to load a custom response or content han‐
1352 dler for use with tests.
1353 Use -l to load response handlers relative to the current working direc‐
1355 tory.
1356 For example, to load a handler named HTMLHandler from the handlers.html
1358 module relative to the current directory:
1359 gabbi-run -l -r handlers.html:HTMLHandler http://example.com <
1360 my.yaml
1361 These are informal release notes for gabbi since version 1.0.0, high‐
1363 lighting major features and changes. For more detail see the commit
1364 logs on GitHub.
1365
1367 • Handle SNI when host request header is any case.
1368
1370 • Clear the PoolManager after each test to avoid running out of socket
1371 file descriptors.
1372
1374 • Add -l argument to gabbi-run to allow loading content and response
1375 handlers relative to the local directory.
1376
1378 • Fix naming of tests when using pytest. This changes means that set‐
1379 ting test_loader_name is now required when using pytest.
1380
1382 • Support SNI in https requests when host request header is set.
1383
1385 • Publish a docker container that runs gabbi-run with each tagged re‐
1386 lease.
1387
1389 • For the $ENVIRON and $RESPONSE substitutions it is now possible to
1390 cast the value to a type of int, float, str, or bool.
1391 • The JSONHandler is now more strict about how it detects that a body
1393 content is JSON, avoiding some errors where the content-type header
1394 suggests JSON but the content cannot be decoded as such.
1395 • Better error message when content cannot be decoded.
1397 • Addition of the disable_response_handler test setting for those cases
1399 when the test author has no control over the content-type header and
1400 it is wrong.
1401
1403 • Allow substitutions in skip for more flexible skipping, notablly with
1404 $ENVIRON.
1405 • Better test name output when running with unittest and verbose.
1407
1409 • Remove support for Python 3.5. Ensure testing support for 3.8, 3.9,
1410 and pypy3.
1411 • Adapt to new behavior in urllib.parse.
1413 • Correct fixture start when using pytest.
1415
1417 • If no content-type is provided with a response and verbose is on for
1418 responses, display the response body as if it were text.
1419
1421 • Properly declare that gabbi 2.x is Python 3 only.
1422
1424 • Correct management of response handler default fields.
1425
1427 • Drop support for Python 2. If you need Python 2 support, use an older
1428 version.
1429 • Stop using testtools and fixtures. These two modules present several
1431 difficulties and their maintenance situation suggests those difficul‐
1432 ties will not be resolved. Since Python 2 support is being removed,
1433 the need for the modules can be removed as well without losing func‐
1434 tionality. "Inner fixtures" that use the fixtures.Fixture interface
1435 should continue to work.
1436
1438 • Add support for not validating certificates in https requests. Con‐
1439 trolled by the cert_validate attribute in individual tests and
1440 build_tests() and the -k or --insecure argument to gabbi-run.
1441
1443 • Support pytest 5.0.0 in Python >=3.5. For earlier versions of Python,
1444 pytest<5.0.0 will be used; the pytest project is dropping support for
1445 older versions of Python.
1446
1448 • Use pytest<5.0.0 until gabbi has solutions for the changes in 5.0.0.
1449
1451 • A -q argument is added to gabbi-run to suppress output from the test
1452 runner.
1453
1455 • Adjust loading of YAML to be ready for new release of PyYAML.
1456
1458 • Provide the YAMLDiskLoadingJSONHandler class that allows test result
1459 data for response_json_path checks to be loaded from YAML-on-disk.
1460
1462 • Use JSONPath to select a portion of data-on-disk in re‐
1463 sponse_json_path checks.
1464 • Restrict PyYAML to <4.0.
1466
1468 • Allow listing of tests with no host configured. When host is an empty
1469 string, tests can be listed (for discovery), but will be skipped on
1470 run.
1471
1473 • JSON $RESPONSE substitutions in the data field may be complex types
1474 (lists and dicts), not solely strings.
1475
1477 • When the HTTP response begins with a bad status line, have BadSta‐
1478 tusLine be raised from urllib3.
1479
1481 • Allow substitutions in the key portion of request and response head‐
1482 ers, not just the value.
1483
1485 • Remove support for Python 3.3.
1486 • Make handling of fixture-level skips in pytest actually work.
1488
1490 • Add safe_yaml parameter to build_tests().
1491
1493 • use_prior_test is added to test Metadata.
1494 • Extensive cleanups in regular expression handling when constructing
1496 tests from YAML.
1497
1499 JSONPath handling gets two improvements:
1500 • The value side of a response_json_paths entry can be loaded from a
1502 file using the <@file.json syntax also used in Data.
1503 • The key side of a response_json_paths entry can use substitutions.
1505 This was already true for the value side.
1506
1508 Substitutions in $RESPONSE handling now preserve numeric types instead
1509 of casting to a string. This is useful when servers are expecting
1510 strong types and tests want to send response data back to the server.
1511
1513 count and delay test keys allow substitutions.
1514 gabbi.driver.build_tests() accepts a verbose parameter to set test
1515 verbosity for an entire session.
1516
1518 Better failure reporting when using gabbi-run with multiple files. Test
1519 names are based on the files and a summary of failed files is provided
1520 at the end of the report.
1521
1523 Effectively capture a failure in a fixture and report the traceback.
1524 Without this some test runners swallow the error and discovering prob‐
1525 lems when developing fixtures can be quite challenging.
1526
1528 Thanks to Samuel Fekete, tests can use the $HISTORY dictionary to refer
1529 to any prior test in the same file, not just the one immediately prior,
1530 when doing substitutions.
1531
1533 Filenames used to read data into tests using the <@ syntax may now use
1534 pathnames relative to the YAML file. See Data.
1535 gabbi-run gains a --verbose parameter to force all tests run in a ses‐
1537 sion to run with verbose set.
1538 When using pytest to load tests, a new mechanism is available which
1540 avoids warnings produced in when using a version of pytest greater than
1541 3.0.
1542
1544 When verbosely displaying request and response bodies that are JSON,
1545 pretty print for improved readability.
1546
1548 Allow gabbi-run to accept multiple filenames as command line arguments
1549 instead of reading tests from stdin.
1550
1552 Switch from response handlers to Content Handlers to allow more flexi‐
1553 ble processing of both response _and_ request bodies.
1554 Add inner fixtures for per test fixtures, useful for output capturing.
1556
1558 Allow the test_loader_name arg to gabbi.driver.build_tests() to over‐
1559 ride the prefix of the pretty printed name of generated tests.
1560
1562 String values in JSONPath matches may be wrapped in /.../` to be
1563 treated as regular expressions.
1564
1566 Better documentation of how to run gabbi in a concurrent environment.
1567 Improved handling of pytest fixtures and test counts.
1568
1570 Add url to gabbi.driver.build_tests() to use instead of host, port and
1571 prefix.
1572
1574 Add require_ssl to gabbi.driver.build_tests() to force use of SSL.
1575
1577 Add $COOKIE substitution.
1578
1580 Correctly support IPV6 hosts.
1581
1583 Add $LAST_URL substitution.
1584
1586 Introduce support for loading and running tests with pytest.
1587
1589 Use urllib3 instead of httplib2 for driving HTTP requests.
1590
1592 Add sorting and filtering to JSONPath handling.
1593
1595 Add the response_forbidden_headers to response expectations.
1596
1598 Instead of:
1599 tests:
1601 - name: a simple get
1602 url: /some/path
1603 method: get
1604 1.7.0 also makes it possible to:
1606 tests:
1608 - name: a simple get
1609 GET: /some/path
1610 Any upper case key is treated as a method.
1612
1614 Enhanced flexibility and colorization when setting tests to be verbose.
1615
1617 Adds the query_parameters key to request parameters.
1618
1620 The start of improvements and extensions to JSONPath handling. In this
1621 case the addition of the len function.
1622
1624 Vastly improved output and behavior in gabbi-run.
1625
1627 Version 1 was the first release with a commitment to a stable Test For‐
1628 mat. Since then new fields have been added but have not been taken
1629 away.
1630 The following people have contributed code to gabbi. Thanks to them.
1632 Thanks also to all the people who have made gabbi better by reporting
1633 issues and their successes and failures with using gabbi.
1634 • Chris Dent
1636 • FND
1638 • Mehdi Abaakouk
1640 • Tom Viner
1642 • Jason Myers
1644 • Josh Leeb-du Toit
1646 • Duc Truong
1648 • Zane Bitter
1650 • Ryan Spencer
1652 • Kim Raymoure
1654 • Travis Truman
1656 • Samuel Fekete
1658 • Michael McCune
1660 • Imran Hayder
1662 • Julien Danjou
1664 • Trevor McCasland
1666 • Danek Duvall
1668 • Marc Abramowitz
1670 • Scott Wallace
1672 NOTE:
1674 This section provides a collection of questions with answers that
1675 don't otherwise fit in the rest of the documentation. If something
1676 is missing, please create an issue.
1677 As this document grows it will gain a more refined structure.
1679
1681 Is gabbi only for testing Python-based APIs?
1682 No, you can use gabbi-run to test an HTTP service built in any program‐
1683 ming language.
1684 How do I run just one test?
1686 Each YAML file contains a sequence of tests, each test within each file
1687 has a name. That name is translated to the name of the test by replac‐
1688 ing spaces with an _.
1689 When running tests that are generated dynamically, filtering based on
1691 the test name prior to the test being collected will not work in some
1692 test runners. Test runners that use a --load-list functionality can be
1693 convinced to filter after discovery.
1694 pytest does this directly with the -k keyword flag.
1696 When using testrepository with tox as used in gabbi's own tests it is
1698 possible to pass a filter in the tox command:
1699 tox -epy27 -- get_the_widget
1701 When using testtools.run and similar test runners it's a bit more com‐
1703 plicated. It is necessary to provide the full name of the test as a
1704 list to --load-list:
1705 python -m testtools.run --load-list \
1707 <(echo package.tests.test_api.yamlfile_get_the_widge.test_request)
1708 How do I run just one test, without running prior tests in a sequence?
1710 By default, when you select a single test to run, all tests prior to
1711 that one in a file will be run as well: the file is treated as as se‐
1712 quence of dependent tests. If you do not want this you can adjust the
1713 use_prior_test test metadata in one of three ways:
1714 • Set it in the YAML file for the one test you are concerned with.
1716 • Set the defaults for all tests in that file.
1718 • set use_prior_test to false when calling build_tests()
1720 Be aware that doing this breaks a fundamental assumption that gabbi
1722 makes about how tests work. Any substitutions will fail.
1723
1725 Can I have variables in my YAML file?
1726 Gabbi provides the $ENVIRON substitution which can operate a bit like
1727 variables that are set elsewhere and then used in the tests defined by
1728 the YAML.
1729 If you find it necessary to have variables within a single YAML file
1731 you take advantage of YAML alias nodes list this:
1732 vars:
1734 - &uuid_1 5613AABF-BAED-4BBA-887A-252B2D3543F8
1735 tests:
1737 - name: send a uuid to a post
1738 POST: /resource
1739 request_headers:
1740 content-type: application/json
1741 data:
1742 uuid: *uuid_1
1743 You can alias all sorts of nodes, not just single items. Be aware that
1745 the replacement of an alias node happens while the YAML is being
1746 loaded, before gabbi does any processing.
1747 How many tests should be put in one YAML file?
1749 For the sake of readability it is best to keep each YAML file rela‐
1750 tively short. Since each YAML file represents a sequence of requests,
1751 it usually makes sense to create a new file when a test is not depen‐
1752 dent on any before it.
1753 It's tempting to put all the tests for any resource or URL in the same
1755 file, but this eventually leads to files that are too long and are thus
1756 difficult to read.
1757
1759 A single HTTP request represented as a subclass of unittest.TestCase
1760 The test case encapsulates the request headers and body and expected
1762 response headers and body. When the test is run an HTTP request is made
1763 using urllib3. Assertions are made against the response.
1764 class gabbi.case.HTTPTestCase(methodName='runTest')
1766 Bases: TestCase
1767 Encapsulate a single HTTP request as a TestCase.
1769 If the test is a member of a sequence of requests, ensure that
1771 prior tests are run.
1772 To keep the test harness happy we need to make sure the setUp
1774 and tearDown are only run once.
1775 assert_in_or_print_output(expected, iterable)
1777 Assert the iterable contains expected or print some out‐
1778 put.
1779 If the output is long, it is limited by either
1781 GABBI_MAX_CHARS_OUTPUT in the environment or the
1782 MAX_CHARS_OUTPUT constant.
1783 base_test = {'cert_validate': True, 'data': '', 'desc': '',
1785 'disable_response_handler': False, 'method': 'GET', 'name': '',
1786 'poll': {}, 'query_parameters': {}, 'redirects': False, 're‐
1787 quest_headers': {}, 'skip': '', 'ssl': False, 'status': '200',
1788 'url': '', 'use_prior_test': True, 'verbose': False, 'xfail':
1789 False}
1790 get_content_handler(content_type)
1792 Determine the content handler for this media type.
1793 load_data_file(filename)
1795 Read a file from the current test directory.
1796 replace_template(message, escape_regex=False)
1798 Replace magic strings in message.
1799 run(result=None)
1801 Store the current result handler on this test.
1802 setUp()
1804 Hook method for setting up the test fixture before exer‐
1805 cising it.
1806 shortDescription()
1808 Returns a one-line description of the test, or None if no
1809 description has been provided.
1810 The default implementation of this method returns the
1812 first line of the specified test method's docstring.
1813 tearDown()
1815 Hook method for deconstructing the test fixture after
1816 testing it.
1817 test_request()
1819 Run this request if it has not yet run.
1820 If there is a prior test in the sequence, run it first.
1822 gabbi.case.potentialFailure(func)
1824 Decorate a test method that is expected to fail if 'xfail' is
1825 true.
1826
1830 Manage fixtures for gabbi at the test suite level.
1831 class gabbi.fixture.GabbiFixture
1833 Bases: object
1834 A context manager that operates as a fixture.
1836 Subclasses must implement start_fixture and stop_fixture, each
1838 of which contain the logic for stopping and starting whatever
1839 the fixture is. What a fixture is is left as an exercise for the
1840 implementor.
1841 These context managers will be nested so any actual work needs
1843 to happen in start_fixture and stop_fixture and not in __init__.
1844 Otherwise exception handling will not work properly.
1845 start_fixture()
1847 Implement the actual workings of starting the fixture
1848 here.
1849 stop_fixture()
1851 Implement the actual workings of stopping the fixture
1852 here.
1853 exception gabbi.fixture.GabbiFixtureError
1855 Bases: Exception
1856 Generic exception for GabbiFixture.
1858 class gabbi.fixture.SkipAllFixture
1860 Bases: GabbiFixture
1861 A fixture that skips all the tests in the current suite.
1863 start_fixture()
1865 Implement the actual workings of starting the fixture
1866 here.
1867 gabbi.fixture.nest(fixtures)
1869 Nest a series of fixtures.
1870 This is duplicated from nested in the stdlib, which has been
1872 deprecated because of issues with how exceptions are difficult
1873 to handle during __init__. Gabbi needs to nest an unknown number
1874 of fixtures dynamically, so the with syntax that replaces nested
1875 will not work.
1876
1878 Package for response and content handlers that process the body of a
1879 response in various ways.
1880 handlers.base Module
1882 Base classes for response and content handlers.
1883 class gabbi.handlers.base.ContentHandler
1885 Bases: ResponseHandler
1886 A subclass of ResponseHandlers that adds content handling.
1888 static accepts(content_type)
1890 Return True if this handler can handler this type.
1891 static dumps(data, pretty=False, test=None)
1893 Return structured data as a string.
1894 If pretty is true, prettify.
1896 static load_data_file(test, file_path)
1898 Return the string content of the file specified by the
1899 file_path.
1900 static loads(data)
1902 Create structured (Python) data from a stream.
1903 If there is a failure decoding then the handler should
1905 repackage the error as a gabbi.exception.GabbiDataLoadEr‐
1906 ror.
1907 classmethod replacer(response_data, path)
1909 Return the string that is replacing RESPONSE.
1910 class gabbi.handlers.base.ResponseHandler
1912 Bases: object
1913 Add functionality for making assertions about an HTTP response.
1915 A subclass may implement two methods: action and preprocess.
1917 preprocess takes one argument, the TestCase. It is called ex‐
1919 actly once for each test before looping across the assertions.
1920 It is used, rarely, to copy the test.output into a useful form
1921 (such as a parsed DOM).
1922 action takes two or three arguments. If test_key_value is a list
1924 action is called with the test case and a single list item. If
1925 test_key_value is a dict then action is called with the test
1926 case and a key and value pair.
1927 action(test, item, value=None)
1929 Test an individual entry for this response handler.
1930 If the entry is a key value pair the key is in item and
1932 the value in value. Otherwise the entry is considered a
1933 single item from a list.
1934 preprocess(test)
1936 Do any pre-single-test preprocessing.
1937 test_key_suffix = ''
1939 test_key_value = []
1941 handlers.core Module
1943 Core response handlers.
1944 class gabbi.handlers.core.ForbiddenHeadersResponseHandler
1946 Bases: ResponseHandler
1947 Test that listed headers are not in the response.
1949 action(test, forbidden, value=None)
1951 Test an individual entry for this response handler.
1952 If the entry is a key value pair the key is in item and
1954 the value in value. Otherwise the entry is considered a
1955 single item from a list.
1956 test_key_suffix = 'forbidden_headers'
1958 test_key_value = []
1960 class gabbi.handlers.core.HeadersResponseHandler
1962 Bases: ResponseHandler
1963 Compare expected headers with actual headers.
1965 If a header value is wrapped in / it is treated as a raw regular
1967 expression.
1968 Headers values are always treated as strings.
1970 action(test, header, value=None)
1972 Test an individual entry for this response handler.
1973 If the entry is a key value pair the key is in item and
1975 the value in value. Otherwise the entry is considered a
1976 single item from a list.
1977 test_key_suffix = 'headers'
1979 test_key_value = {}
1981 class gabbi.handlers.core.StringResponseHandler
1983 Bases: ResponseHandler
1984 Test for matching strings in the the response body.
1986 action(test, expected, value=None)
1988 Test an individual entry for this response handler.
1989 If the entry is a key value pair the key is in item and
1991 the value in value. Otherwise the entry is considered a
1992 single item from a list.
1993 test_key_suffix = 'strings'
1995 test_key_value = []
1997 handlers.jsonhandler Module
1999 JSON-related content handling.
2000 class gabbi.handlers.jsonhandler.JSONHandler
2002 Bases: ContentHandler
2003 A ContentHandler for JSON
2005 • Structured test data is turned into JSON when request con‐
2007 tent-type is JSON.
2008 • Response bodies that are JSON strings are made into Python
2010 data on the test response_data attribute when the response
2011 content-type is JSON.
2012 • A response_json_paths response handler is added.
2014 • JSONPaths in $RESPONSE substitutions are supported.
2016 static accepts(content_type)
2018 Return True if this handler can handler this type.
2019 action(test, path, value=None)
2021 Test json_paths against json data.
2022 static dumps(data, pretty=False, test=None)
2024 Return structured data as a string.
2025 If pretty is true, prettify.
2027 static extract_json_path_value(data, path)
2029 Extract the value at JSON Path path from the data.
2030 The input data is a Python datastructure, not a JSON
2032 string.
2033 static load_data_file(test, file_path)
2035 Return the string content of the file specified by the
2036 file_path.
2037 static loads(data)
2039 Create structured (Python) data from a stream.
2040 If there is a failure decoding then the handler should
2042 repackage the error as a gabbi.exception.GabbiDataLoadEr‐
2043 ror.
2044 classmethod replacer(response_data, match)
2046 Return the string that is replacing RESPONSE.
2047 test_key_suffix = 'json_paths'
2049 test_key_value = {}
2051 handlers.yaml_disk_loading_jsonhandler Module
2053 JSON-related content handling with YAML data disk loading.
2054 class gabbi.handlers.yaml_disk_loading_jsonhandler.YAMLDiskLoadingJSON‐
2056 Handler
2057 Bases: JSONHandler
2058 A ContentHandler for JSON responses that loads YAML from disk
2060 • Structured test data is turned into JSON when request con‐
2062 tent-type is JSON.
2063 • Response bodies that are JSON strings are made into Python
2065 data on the test response_data attribute when the response
2066 content-type is JSON.
2067 • A response_json_paths response handler is added. Data read
2069 from disk during this handle will be loaded with the
2070 yaml.safe_load method to support both JSON and YAML data
2071 sources from disk.
2072 • JSONPaths in $RESPONSE substitutions are supported.
2074 static load_data_file(test, file_path)
2076 Return the string content of the file specified by the
2077 file_path.
2078
2080 A TestSuite for containing gabbi tests.
2081 This suite has two features: the contained tests are ordered and there
2083 are suite-level fixtures that operate as context managers.
2084 class gabbi.suite.GabbiSuite(tests=())
2086 Bases: TestSuite
2087 A TestSuite with fixtures.
2089 The suite wraps the tests with a set of nested context managers
2091 that operate as fixtures.
2092 If a fixture raises unittest.case.SkipTest during setup, all the
2094 tests in this suite will be skipped.
2095 run(result, debug=False)
2097 Override TestSuite run to start suite-level fixtures.
2098 To avoid exception confusion, use a null Fixture when
2100 there are no fixtures.
2101 start(result, tests=None)
2103 Start fixtures when using pytest.
2104 stop() Stop fixtures when using pytest.
2106 gabbi.suite.noop(*args)
2108 A noop method used to disable collected tests.
2109
2112 TestRunner and TestResult for gabbi-run.
2113 class gabbi.reporter.ConciseTestResult(stream, descriptions, verbosity)
2115 Bases: TextTestResult
2116 A TextTestResult with simple but useful output.
2118 If the output is a tty or GABBI_FORCE_COLOR is set in the envi‐
2120 ronment, output will be colorized.
2121 addError(test, err)
2123 Called when an error has occurred. 'err' is a tuple of
2124 values as returned by sys.exc_info().
2125 addExpectedFailure(test, err)
2127 Called when an expected failure/error occurred.
2128 addFailure(test, err)
2130 Called when an error has occurred. 'err' is a tuple of
2131 values as returned by sys.exc_info().
2132 addSkip(test, reason)
2134 Called when a test is skipped.
2135 addSuccess(test)
2137 Called when a test has completed successfully
2138 addUnexpectedSuccess(test)
2140 Called when a test was expected to fail, but succeed.
2141 getDescription(test)
2143 printErrorList(flavor, errors)
2145 startTest(test)
2147 Called when the given test is about to be run
2148 class gabbi.reporter.ConciseTestRunner(stream=None, descriptions=True,
2150 verbosity=1, failfast=False, buffer=False, resultclass=None, warn‐
2151 ings=None, *, tb_locals=False)
2152 Bases: TextTestRunner
2153 A TextTestRunner that uses ConciseTestResult for reporting re‐
2155 sults.
2156 resultclass
2158 alias of ConciseTestResult
2159 class gabbi.reporter.PyTestResult(stream=None, descriptions=None, ver‐
2161 bosity=None)
2162 Bases: TestResult
2163 Wrap a test result to allow it to work with pytest.
2165 The main behaviors here are:
2167 • to turn what had been exceptions back into exceptions
2169 • use pytest's skip and xfail methods
2171 addError(test, err)
2173 Called when an error has occurred. 'err' is a tuple of
2174 values as returned by sys.exc_info().
2175 addExpectedFailure(test, err)
2177 Called when an expected failure/error occurred.
2178 addFailure(test, err)
2180 Called when an error has occurred. 'err' is a tuple of
2181 values as returned by sys.exc_info().
2182 addSkip(test, reason)
2184 Called when a test is skipped.
2185
2187 Utility functions grab bag.
2188 gabbi.utils.create_url(base_url, host, port=None, prefix='', ssl=False)
2190 Given pieces of a path-based url, return a fully qualified url.
2191 gabbi.utils.decode_response_content(header_dict, content)
2193 Decode content to a proper string.
2194 gabbi.utils.extract_content_type(header_dict, default='application/bi‐
2196 nary')
2197 Extract parsed content-type from headers.
2198 gabbi.utils.get_colorizer(stream)
2200 Return a function to colorize a string.
2201 Only if stream is a tty .
2203 gabbi.utils.host_info_from_target(target, prefix=None)
2205 Turn url or host:port and target into test destination.
2206 gabbi.utils.load_yaml(handle=None, yaml_file=None, safe=True)
2208 Read and parse any YAML file or filehandle.
2209 Let exceptions flow where they may.
2211 If no file or handle is provided, read from STDIN.
2213 gabbi.utils.not_binary(content_type)
2215 Decide if something is content we'd like to treat as a string.
2216 gabbi.utils.parse_content_type(content_type, default_charset='utf-8')
2218 Parse content type value for media type and charset.
2219
2221 Gabbi specific exceptions.
2222 exception gabbi.exception.GabbiDataLoadError
2224 Bases: ValueError
2225 An exception to alert when data streams cannot be loaded.
2227 exception gabbi.exception.GabbiFormatError
2229 Bases: ValueError
2230 An exception to encapsulate poorly formed test data.
2232 exception gabbi.exception.GabbiSyntaxWarning
2234 Bases: SyntaxWarning
2235 A warning about syntax that is not desirable.
2237
2240 Keep one single global jsonpath parser.
2241 gabbi.json_parser.parse(path)
2243 Parse a JSONPath expression use the global parser.
2244 Gabbi is a tool for running HTTP tests where requests and responses are
2246 expressed as declarations in YAML files:
2247 tests:
2249 - name: retrieve items
2250 GET: /items
2251 See the rest of these docs for more details on the many features and
2253 formats for setting request headers and bodies and evaluating re‐
2254 sponses.
2255 Tests can be run from the command line with gabbi-run or programmati‐
2257 cally using either py.test or unittest-style test runners. See
2258 installation instructions below.
2259 The name is derived from "gabby": excessively talkative. In a test en‐
2261 vironment having visibility of what a test is actually doing is a good
2262 thing. This is especially true when the goal of a test is to test the
2263 HTTP, not the testing infrastructure. Gabbi tries to put the HTTP in‐
2264 teraction in the foreground of testing.
2265 If you want to get straight to creating tests look at Example Tests,
2267 the test files in the source distribution and Test Format. A gabbi-demo
2268 repository provides a tutorial of using gabbi to build an API, via the
2269 commit history of the repo.
2270
2272 Gabbi works to bridge the gap between human readable YAML files (see
2273 Test Format for details) that represent HTTP requests and expected re‐
2274 sponses and the rather complex world of automated testing.
2275 Each YAML file represents an ordered list of HTTP requests along with
2277 the expected responses. This allows a single file to represent a
2278 process in the API being tested. For example:
2279 • Create a resource.
2281 • Retrieve a resource.
2283 • Delete a resource.
2285 • Retrieve a resource again to confirm it is gone.
2287 At the same time it is still possible to ask gabbi to run just one re‐
2289 quest. If it is in a sequence of tests, those tests prior to it in the
2290 YAML file will be run (in order). In any single process any test will
2291 only be run once. Concurrency is handled such that one file runs in one
2292 process.
2293 These features mean that it is possible to create tests that are useful
2295 for both humans (as tools for learning, improving and developing APIs)
2296 and automated CI systems.
2297 Significant flexibility and power is available in the Test Format to
2299 make it relatively straightforward to test existing complex APIs. This
2300 extended functionality includes the use of JSONPath to query response
2301 bodies and templating of test data to allow access to the prior HTTP
2302 response in the current request. For APIs which do not use JSON addi‐
2303 tional Content Handlers can be created.
2304 Care should be taken when using this functionality when you are creat‐
2306 ing a new API. If your API is so complex that it needs complex test
2307 files then you may wish to take that as a sign that your API itself too
2308 complex. One goal of gabbi is to encourage transparent and comprehensi‐
2309 ble APIs.
2310 Though gabbi is written in Python and under the covers uses unittest
2312 data structures and processes, there is no requirement that the Target
2313 Host be a Python-based service. Anything talking HTTP can be tested. A
2314 YAML Runner makes it possible to simply create YAML files and point
2315 them at a running server.
2316
2318 As a Python package, gabbi is typically installed via pip:
2319 pip install gabbi
2321 You might want to create a virtual environment; an isolated context for
2323 Python packages, keeping gabbi cleany separated from the rest of your
2324 system.
2325 Python 3 comes with a built-in tool to create virtual environments:
2327 python3 -m venv venv
2329 . venv/bin/activate
2330 pip install gabbi
2332 This way we can later use deactivate and safely remove the venv direc‐
2334 tory, thus erasing any trace of gabbi from the system.
2335 If you prefer to not install gabbi, or perhaps want to use it in a dy‐
2337 namic fashion in a CI setting, there is an official container image
2338 hosted at docker hub as cdent/gabbi. It allows running gabbi-run with
2339 any arguments you might need, providing tests on STDIN of via a mounted
2340 volume.
2341
2343 Chris Dent
2344 Jan 20, 2023 GABBI(1)