1GABBI(1) Gabbi GABBI(1)
2
6 gabbi - Gabbi Documentation
7 Gabbi tests are expressed in YAML as a series of HTTP requests with
9 their expected response:
10 tests:
12 - name: retrieve root
13 GET: /
14 status: 200
15 This will trigger a GET request to / on the configured host. The test
17 will pass if the response's status code is 200.
18
20 The top-level tests category contains an ordered sequence of test dec‐
21 larations, each describing the expected response to a given request:
22 Metadata
24 ┌───────────────┬─────────────────────┬───────────────────┐
25 │Key │ Description │ Notes │
26 ├───────────────┼─────────────────────┼───────────────────┤
27 │name │ The test's name. │ required │
28 │ │ Must be unique │ │
29 │ │ within a file. │ │
30 ├───────────────┼─────────────────────┼───────────────────┤
31 │desc │ An arbitrary string │ │
32 │ │ describing the │ │
33 │ │ test. │ │
34 ├───────────────┼─────────────────────┼───────────────────┤
35 │verbose │ If True or all │ defaults to False │
36 │ │ (synonymous), │ │
37 │ │ prints a represen‐ │ │
38 │ │ tation of the cur‐ │ │
39 │ │ rent request and │ │
40 │ │ response to stdout, │ │
41 │ │ including both │ │
42 │ │ headers and body. │ │
43 │ │ If set to headers │ │
44 │ │ or body, only the │ │
45 │ │ corresponding part │ │
46 │ │ of the request and │ │
47 │ │ response will be │ │
48 │ │ printed. If the │ │
49 │ │ output is a TTY, │ │
50 │ │ colors will be │ │
51 │ │ used. If the body │ │
52 │ │ content-type is │ │
53 │ │ JSON it will be │ │
54 │ │ formatted for im‐ │ │
55 │ │ proved readability. │ │
56 │ │ See VerboseHttp for │ │
57 │ │ details. │ │
58 ├───────────────┼─────────────────────┼───────────────────┤
59 │skip │ A string message │ defaults to False │
60 │ │ which if set will │ │
61 │ │ cause the test to │ │
62 │ │ be skipped with the │ │
63 │ │ provided message. │ │
64 └───────────────┴─────────────────────┴───────────────────┘
65 │xfail │ Determines whether │ defaults to False │
68 │ │ to expect this test │ │
69 │ │ to fail. Note that │ │
70 │ │ the test will be │ │
71 │ │ run anyway. │ │
72 ├───────────────┼─────────────────────┼───────────────────┤
73 │use_prior_test │ Determines if this │ defaults to True │
74 │ │ test will be run in │ │
75 │ │ sequence (after) │ │
76 │ │ the test prior to │ │
77 │ │ it in the list of │ │
78 │ │ tests within a │ │
79 │ │ file. To be con‐ │ │
80 │ │ crete, when this is │ │
81 │ │ True the test is │ │
82 │ │ dependent on the │ │
83 │ │ prior test and if │ │
84 │ │ that prior has not │ │
85 │ │ yet run, it wil be │ │
86 │ │ run, even if only │ │
87 │ │ the current test │ │
88 │ │ has been selected. │ │
89 │ │ Set this to False │ │
90 │ │ to allow selecting │ │
91 │ │ a test without de‐ │ │
92 │ │ pendencies. │ │
93 ├───────────────┼─────────────────────┼───────────────────┤
94 │cert_validate │ States whether the │ defaults to True │
95 │ │ underlying HTTP │ │
96 │ │ client should at‐ │ │
97 │ │ tempt to validate │ │
98 │ │ SSL certificates. │ │
99 │ │ In many test envi‐ │ │
100 │ │ ronment certifi‐ │ │
101 │ │ cates will be │ │
102 │ │ self-signed so │ │
103 │ │ changing this may │ │
104 │ │ be requried. It can │ │
105 │ │ also be changed │ │
106 │ │ when loader or us‐ │ │
107 │ │ ing gabbi-run. │ │
108 └───────────────┴─────────────────────┴───────────────────┘
109 NOTE:
111 When tests are generated dynamically, the TestCase name will include
112 the respective test's name, lowercased with spaces transformed to _.
113 In at least some test runners this will allow you to select and fil‐
114 ter on test name.
115 Request Parameters
117 ┌────────────────────┬───────────────────────┬─────────────────────┐
118 │Key │ Description │ Notes │
119 └────────────────────┴───────────────────────┴─────────────────────┘
120 │any uppercase │ Any such key is │ │
134 │string │ considered an HTTP │ │
135 │ │ method, with the │ │
136 │ │ corresponding value │ │
137 │ │ expressing the URL. │ │
138 │ │ │ │
139 │ │ This is a shortcut │ │
140 │ │ combining method │ │
141 │ │ and url into a sin‐ │ │
142 │ │ gle statement: │ │
143 │ │ │ │
144 │ │ GET: /index │ │
145 │ │ │ │
146 │ │ corresponds │ │
147 │ │ to: │ │
148 │ │ │ │
149 │ │ method: GET │ │
150 │ │ url: /index │ │
151 ├────────────────────┼───────────────────────┼─────────────────────┤
152 │method │ The HTTP request │ defaults to GET │
153 │ │ method. │ │
154 ├────────────────────┼───────────────────────┼─────────────────────┤
155 │url │ The URL to request. │ Either this or the │
156 │ │ This can either be a │ shortcut above is │
157 │ │ full path (e.g. "/in‐ │ required │
158 │ │ dex") or a fully │ │
159 │ │ qualified URL (i.e. │ │
160 │ │ including host and │ │
161 │ │ scheme, e.g. "‐ │ │
162 │ │ http://exam‐ │ │
163 │ │ ple.org/index") — see │ │
164 │ │ host for details. │ │
165 ├────────────────────┼───────────────────────┼─────────────────────┤
166 │request_headers │ A dictionary of │ │
167 │ │ key-value pairs rep‐ │ │
168 │ │ resenting request │ │
169 │ │ header names and val‐ │ │
170 │ │ ues. These will be │ │
171 │ │ added to the con‐ │ │
172 │ │ structed request. │ │
173 ├────────────────────┼───────────────────────┼─────────────────────┤
174 │query_parameters │ A dictionary of query │ │
175 │ │ parameters that will │ │
176 │ │ be added to the url │ │
177 │ │ as query string. If │ │
178 │ │ that URL already con‐ │ │
179 │ │ tains a set of query │ │
180 │ │ parameters, those wil │ │
181 │ │ be extended. See ex‐ │ │
182 │ │ ample for a demon‐ │ │
183 │ │ stration of how the │ │
184 │ │ data is structured. │ │
185 ├────────────────────┼───────────────────────┼─────────────────────┤
186 │data │ A representation to │ │
187 │ │ pass as the body of a │ │
188 │ │ request. Note that │ │
189 │ │ content-type in re‐ │ │
190 │ │ quest_headers should │ │
191 │ │ also be set — see │ │
192 │ │ Data for details. │ │
193 ├────────────────────┼───────────────────────┼─────────────────────┤
194 │redirects │ If True, redirects │ defaults to False │
195 │ │ will automatically be │ │
196 │ │ followed. │ │
197 └────────────────────┴───────────────────────┴─────────────────────┘
198 │ssl │ Determines whether │ defaults to False │
200 │ │ the request uses SSL │ │
201 │ │ (i.e. HTTPS). Note │ │
202 │ │ that the url's scheme │ │
203 │ │ takes precedence if │ │
204 │ │ present — see host │ │
205 │ │ for details. │ │
206 └────────────────────┴───────────────────────┴─────────────────────┘
207 Response Expectations
209 ┌────────────────────┬─────────────────────┬─────────────────┐
210 │Key │ Description │ Notes │
211 ├────────────────────┼─────────────────────┼─────────────────┤
212 │status │ The expected re‐ │ defaults to 200 │
213 │ │ sponse status code. │ │
214 │ │ Multiple acceptable │ │
215 │ │ response codes may │ │
216 │ │ be provided, sepa‐ │ │
217 │ │ rated by || (e.g. │ │
218 │ │ 302 || 301 — note, │ │
219 │ │ however, that this │ │
220 │ │ indicates ambigu‐ │ │
221 │ │ ity, which is gen‐ │ │
222 │ │ erally undesir‐ │ │
223 │ │ able). │ │
224 ├────────────────────┼─────────────────────┼─────────────────┤
225 │response_headers │ A dictionary of │ │
226 │ │ key-value pairs │ │
227 │ │ representing ex‐ │ │
228 │ │ pected response │ │
229 │ │ header names and │ │
230 │ │ values. If a │ │
231 │ │ header's value is │ │
232 │ │ wrapped in /.../, │ │
233 │ │ it will be treated │ │
234 │ │ as a regular ex‐ │ │
235 │ │ pression to search │ │
236 │ │ for in the response │ │
237 │ │ header. │ │
238 ├────────────────────┼─────────────────────┼─────────────────┤
239 │response_forbid‐ │ A list of headers │ │
240 │den_headers │ which must not be │ │
241 │ │ present. │ │
242 ├────────────────────┼─────────────────────┼─────────────────┤
243 │response_strings │ A list of string │ │
244 │ │ fragments expected │ │
245 │ │ to be present in │ │
246 │ │ the response body. │ │
247 └────────────────────┴─────────────────────┴─────────────────┘
248 │response_json_paths │ A dictionary of │ │
266 │ │ JSONPath rules │ │
267 │ │ paired with ex‐ │ │
268 │ │ pected matches. Us‐ │ │
269 │ │ ing this rule re‐ │ │
270 │ │ quires that the │ │
271 │ │ content being sent │ │
272 │ │ from the server is │ │
273 │ │ JSON (i.e. a con‐ │ │
274 │ │ tent type of appli‐ │ │
275 │ │ cation/json or con‐ │ │
276 │ │ taining +json) │ │
277 │ │ │ │
278 │ │ If the value is │ │
279 │ │ wrapped in /.../ │ │
280 │ │ the result of the │ │
281 │ │ JSONPath query will │ │
282 │ │ be searched for the │ │
283 │ │ value as a regular │ │
284 │ │ expression. │ │
285 ├────────────────────┼─────────────────────┼─────────────────┤
286 │poll │ A dictionary of two │ │
287 │ │ keys: │ │
288 │ │ │ │
289 │ │ • count: An │ │
290 │ │ integer │ │
291 │ │ stating │ │
292 │ │ the number │ │
293 │ │ of times │ │
294 │ │ to attempt │ │
295 │ │ this test │ │
296 │ │ before │ │
297 │ │ giving up. │ │
298 │ │ │ │
299 │ │ • delay: A │ │
300 │ │ floating │ │
301 │ │ point num‐ │ │
302 │ │ ber of │ │
303 │ │ seconds to │ │
304 │ │ delay be‐ │ │
305 │ │ tween at‐ │ │
306 │ │ tempts. │ │
307 │ │ │ │
308 │ │ This makes │ │
309 │ │ it possible │ │
310 │ │ to poll for │ │
311 │ │ a resource │ │
312 │ │ created via │ │
313 │ │ an asynchro‐ │ │
314 │ │ nous re‐ │ │
315 │ │ quest. Use │ │
316 │ │ with cau‐ │ │
317 │ │ tion. │ │
318 └────────────────────┴─────────────────────┴─────────────────┘
319 Note that many of these items allow substitutions.
321 Default values for a file's tests may be provided via the top-level de‐
323 faults category. These take precedence over the global defaults (ex‐
324 plained below).
325 For examples see the gabbi tests, example and the gabbi-demo tutorial.
327
329 The top-level fixtures category contains a sequence of named fixtures.
330
332 response_* keys are examples of Response Handlers. Custom handlers may
333 be created by test authors for specific use cases. See handlers for
334 more information.
335
337 There are a number of magical variables that can be used to make refer‐
338 ence to the state of a current test, the one just prior or any test
339 prior to the current one. The variables are replaced with real values
340 during test processing.
341 Global
343 • $ENVIRON['<environment variable>']: The name of an environment vari‐
344 able. Its value will replace the magical variable. If the string
345 value of the environment variable is "True" or "False" then the re‐
346 sulting value will be the corresponding boolean, not a string.
347 Current Test
349 • $SCHEME: The current scheme/protocol (usually http or https).
350 • $NETLOC: The host and potentially port of the request.
352 Immediately Prior Test
354 • $COOKIE: All the cookies set by any Set-Cookie headers in the prior
355 response, including only the cookie key and value pairs and no meta‐
356 data (e.g. expires or domain).
357 • $URL: The URL defined in the prior request, after substitutions have
359 been made. For backwards compatibility with earlier releases
360 $LAST_URL may also be used, but if $HISTORY (see below) is being
361 used, $URL must be used.
362 • $LOCATION: The location header returned in the prior response.
364 • $HEADERS['<header>']: The value of any header from the prior re‐
366 sponse.
367 • $RESPONSE['<json path>']: A JSONPath query into the prior response.
369 See jsonpath for more on formatting.
370 Any Previous Test
372 • $HISTORY['<test name>'].<magical variable expression>: Any variable
373 which refers to a prior test may be used in an expression that refers
374 to any earlier test in the same file by identifying the target test
375 by its name in a $HISTORY dictionary. For example, to refer to a
376 value in a JSON object in the response of a test named post json:
377 $HISTORY['post json'].$RESPONSE['$.key']
379 This is a very powerful feature that could lead to test that are dif‐
381 ficult for humans to read. Take care to optimize for the maintainers
382 that will come after you, not yourself.
383 NOTE:
385 Where a single-quote character, ', is shown in the variables above
386 you may also use a double-quote character, ", but in any given ex‐
387 pression the same character must be used at both ends.
388 All of these variables may be used in all of the following fields:
390 • url
392 • query_parameters
394 • data
396 • request_headers (in both the key and value)
398 • response_strings
400 • response_json_paths (in both the key and value, see json path substi‐
402 tution for more info)
403 • response_headers (in both the key and value)
405 • response_forbidden_headers
407 • count and delay fields of poll
409 With these variables it ought to be possible to traverse an API without
411 any explicit statements about the URLs being used. If you need a re‐
412 placement on a field that is not currently supported please raise an
413 issue or provide a patch.
414 As all of these features needed to be tested in the development of
416 gabbi itself, the gabbi tests are a good source of examples on how to
417 use the functionality. See also example for a collection of examples
418 and the gabbi-demo tutorial.
419
421 The data key has some special handing to allow for a bit more flexibil‐
422 ity when doing a POST or PUT:
423 • If the value is not a string (that is, it is a sequence or structure)
425 it is treated as a data structure that will be turned into a string
426 by the dumps method on the relevant content handler. For example if
427 the content-type of the body is application/json the data structure
428 will be turned into a JSON string.
429 • If the value is a string that begins with <@ then the rest of the
431 string is treated as a filepath to be loaded. The path is relative to
432 the test directory and may not traverse up into parent directories.
433 • If the value is an undecorated string, that's the value.
435 NOTE:
437 When reading from a file care should be taken to ensure that a rea‐
438 sonable content-type is set for the data as this will control if any
439 encoding is done of the resulting string value. If it is text, json,
440 xml or javascript it will be encoded to UTF-8.
441 To run gabbi tests with a test harness they must be generated in some
443 fashion and then run. This is accomplished by a test loader. Initially
444 gabbi only supported those test harnesses that supported the load_tests
445 protocol in UnitTest. It now possible to also build and run tests with
446 pytest with some limitations described below.
447 NOTE:
449 It is also possible to run gabbi tests from the command line. See
450 runner.
451 NOTE:
453 By default gabbi will load YAML files using the safe_load function.
454 This means only basic YAML types are allowed in the file. For most
455 use cases this is fine. If you need custom types (for example, to
456 match NaN) it is possible to set the safe_yaml parameter of
457 build_tests() to False. If custom types are used, please keep in
458 mind that this can limit the portability of the YAML files to other
459 contexts.
460 WARNING:
462 If test are being run with a runner that supports concurrency (such
463 as testrepository) it is critical that the test runner is informed
464 of how to group the tests into their respective suites. The usual
465 way to do this is to use a regular expression that groups based on
466 the name of the yaml files. For example, when using testrepository
467 the .testr.conf file needs an entry similar to the following:
468 group_regex=gabbi\.suitemaker\.(test_[^_]+_[^_]+)
470
472 To run the tests with a load_tests style loader a test file containing
473 a load_tests method is required. That will look a bit like:
474 """A sample test module."""
476 # For pathname munging
478 import os
479 # The module that build_tests comes from.
481 from gabbi import driver
482 # We need access to the WSGI application that hosts our service
484 from myapp import wsgiapp
485 # We're using fixtures in the YAML files, we need to know where to
488 # load them from.
489 from myapp.test import fixtures
490 # By convention the YAML files are put in a directory named
492 # "gabbits" that is in the same directory as the Python test file.
493 TESTS_DIR = 'gabbits'
494 def load_tests(loader, tests, pattern):
497 """Provide a TestSuite to the discovery process."""
498 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
499 # Pass "require_ssl=True" as an argument to force all tests
500 # to use SSL in requests.
501 return driver.build_tests(test_dir, loader,
502 intercept=wsgiapp.app,
503 fixture_module=fixtures)
504 For details on the arguments available when building tests see
507 build_tests().
508 Once the test loader has been created, it needs to be run. There are
510 many options. Which is appropriate depends very much on your environ‐
511 ment. Here are some examples using unittest or testtools that require
512 minimal knowledge to get started.
513 By file:
515 python -m testtools.run -v test/test_loader.py
517 By module:
519 python -m testttols.run -v test.test_loader
521 python -m unittest -v test.test_loader
523 Using test discovery to locate all tests in a directory tree:
525 python -m testtools.run discover
527 python -m unittest discover test
529 See the source distribution and the tutorial repo for more advanced op‐
531 tions, including using testrepository and subunit.
532
534 Since pytest does not support the load_tests system, a different way of
535 generating tests is required. Two techniques are supported.
536 The original method (described below) used yield statements to generate
538 tests which pytest would collect. This style of tests is deprecated as
539 of pytest>=3.0 so a new style using pytest fixtures has been developed.
540 pytest >= 3.0
542 In the newer technique, a test file is created that uses the
543 pytest_generate_tests hook. Special care must be taken to always import
544 the test_pytest method which is the base test that the pytest hook
545 parametrizes to generate the tests from the YAML files. Without the
546 method, the hook will not be called and no tests generated.
547 Here is a simple example file:
549 """A sample pytest module for pytest >= 3.0."""
551 # For pathname munging
553 import os
554 # The module that py_test_generator comes from.
556 from gabbi import driver
557 # We need test_pytest so that pytest test collection works properly.
559 # Without this, the pytest_generate_tests method below will not be
560 # called.
561 from gabbi.driver import test_pytest # noqa
562 # We need access to the WSGI application that hosts our service
564 from myapp import wsgiapp
565 # We're using fixtures in the YAML files, we need to know where to
567 # load them from.
568 from myapp.test import fixtures
569 # By convention the YAML files are put in a directory named
571 # "gabbits" that is in the same directory as the Python test file.
572 TESTS_DIR = 'gabbits'
573 def pytest_generate_tests(metafunc):
576 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
577 driver.py_test_generator(
578 test_dir, intercept=wsgiapp.app,
579 fixture_module=fixtures, metafunc=metafunc)
580 This can then be run with the usual pytest commands. For example:
583 py.test -svx pytest3.0-example.py
585 pytest < 3.0
587 When using the older technique, test file must be created that calls
588 py_test_generator() and yields the generated tests. That will look a
589 bit like this:
590 """A sample pytest module."""
592 # For pathname munging
594 import os
595 # The module that build_tests comes from.
597 from gabbi import driver
598 # We need access to the WSGI application that hosts our service
600 from myapp import wsgiapp
601 # We're using fixtures in the YAML files, we need to know where to
603 # load them from.
604 from myapp.test import fixtures
605 # By convention the YAML files are put in a directory named
607 # "gabbits" that is in the same directory as the Python test file.
608 TESTS_DIR = 'gabbits'
609 def test_gabbits():
612 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
613 # Pass "require_ssl=True" as an argument to force all tests
614 # to use SSL in requests.
615 test_generator = driver.py_test_generator(
616 test_dir, intercept=wsgiapp.app,
617 fixture_module=fixtures)
618 for test in test_generator:
620 yield test
621 This can then be run with the usual pytest commands. For example:
624 py.test -svx pytest-example.py
626 The older technique will continue to work with all versions of
628 pytest<4.0 but >=3.0 will produce warnings. If you want to use the
629 older technique but not see the warnings add --disable-pytest-warnings
630 parameter to the invocation of py.test.
631 What follows is a commented example of some tests in a single file
633 demonstrating many of the format features. See loader for the Python
634 needed to integrate with a testing harness.
635 # Fixtures can be used to set any necessary configuration, such as a
638 # persistence layer, and establish sample data. They operate per
639 # file. They are context managers, each one wrapping the next in the
640 # sequence.
641 fixtures:
643 - ConfigFixture
644 - SampleDataFixture
645 # There is an included fixture named "SkipAllFixture" which can be
647 # used to declare that all the tests in the given file are to be
648 # skipped.
649 # Each test file can specify a set of defaults that will be used for
651 # every request. This is useful for always specifying a particular
652 # header or always requiring SSL. These values will be used on every
653 # test in the file unless overriden. Lists and dicts are merged one
654 # level deep, except for "data" which is copied verbatim whether it
655 # is a string, list or dict (it can be all three).
656 defaults:
658 ssl: True
659 request_headers:
660 x-my-token: zoom
661 # The tests themselves are a list under a "tests" key. It's useful
663 # to use plenty of whitespace to help readability.
664 tests:
666 # Each request *must* have a name which is unique to the file. When it
668 # becomes a TestCase the name will be lowercased and spaces will
669 # become "_". Use that generated name when limiting test runs.
670 - name: a test for root
672 desc: Some explanatory text that could be used by other tooling
673 # The URL can either be relative to a host specified elsewhere or
675 # be a fully qualified "http" or "https" URL. *You* are responsible
676 # for url-encoding the URL.
677 url: /
679 method: GET
680 # If no status or method are provided they default to "200" and
682 # "GET".
683 # Instead of explicitly stating "url" and "method" you can join
685 # those two keys into one key representing the method. The method
686 # *must* be uppercase.
687 - name: another test for root
689 desc: Same test as above but with GET key
690 GET: /
691 # A single test can override settings in defaults (set above).
693 - name: root without ssl redirects
695 ssl: False
696 GET: /
697 status: 302
698 # When evaluating response headers it is possible to use a regular
700 # expression to not have to test the whole value. Regular expressions match
701 # anywhere in the output, not just at the beginning.
702 response_headers:
704 location: /^https/
705 # By default redirects will not be followed. This can be changed.
707 - name: follow root without ssl redirect
709 ssl: False
710 redirects: True
711 GET: /
712 status: 200 # This is the response code after the redirect.
713 # URLs can express query parameters in two ways: either in the url
715 # value directly, or as query_parameters. If both are used then
716 # query_parameters are appended. In this example the resulting URL
717 # will be equivalient to
718 # /foo?section=news&article=1&article=2&date=yesterday
719 # but not necessarily in that order.
720 - name: create a url with parameters
722 GET: /foo?section=news
723 query_parameters:
724 article:
725 - 1
726 - 2
727 date: yesterday
728 # Request headers can be used to declare media-type choices and
730 # experiment with authorization handling (amongst other things).
731 # Response headers allow evaluating headers in the response. These
732 # two together form the core value of gabbi.
733 - name: test accept
735 GET: /resource
736 request_headers:
737 accept: application/json
738 response_headers:
739 content-type: /application/json/
740 # If a header must not be present in a response at all that can be
742 # expressed in a test as follows.
743 - name: test forbidden headers
745 GET: /resource
746 response_forbidden_headers:
747 - x-special-header
748 # All of the above requests have defaulted to a "GET" method. When
750 # using "POST", "PUT" or "PATCH", the "data" key provides the
751 # request body.
752 - name: post some text
754 POST: /text_repo
755 request_headers:
756 content-type: text/plain
757 data: "I'm storing this"
758 status: 201
759 # If the data is not a string, it will be transformed into JSON.
761 # You must supply an appropriate content-type request header.
762 - name: post some json
764 POST: /json_repo
765 request_headers:
766 content-type: application/json
767 data:
768 name: smith
769 abode: castle
770 status: 201
771 # If the data is a string prepended with "<@" the value will be
773 # treated as the name of a file in the same directory as the YAML
774 # file. Again, you must supply an appropriate content-type. If the
775 # content-type is one of several "text-like" types, the content will
776 # be assumed to be UTF-8 encoded.
777 - name: post an image
779 POST: /image_repo
780 request_headers:
781 content-type: image/png
782 data: <@kittens.png
783 # A single request can be marked to be skipped.
785 - name: patch an image
787 skip: patching images not yet implemented
788 PATCH: /image_repo/12d96fb8-e78c-11e4-8c03-685b35afa334
789 # Or a single request can be marked that it is expected to fail.
791 - name: check allow headers
793 desc: the framework doesn't do allow yet
794 xfail: True
795 PUT: /post_only_url
796 status: 405
797 response_headers:
798 allow: POST
799 # The body of a response can be evaluated with response handlers.
801 # The most simple checks for a set of strings anywhere in the
802 # response. Note that the strings are members of a list.
803 - name: check for css file
805 GET: /blog/posts/12
806 response_strings:
807 - normalize.css
808 # For JSON responses, JSONPath rules can be used.
810 - name: post some json get back json
812 POST: /json_repo
813 request_headers:
814 content-type: application/json
815 data:
816 name: smith
817 abode: castle
818 status: 201
819 response_json_paths:
820 $.name: smith
821 $.abode: castle
822 # Requests run in sequence. One test can make reference to the test
824 # immediately prior using some special variables.
825 # "$LOCATION" contains the "location" header in the previous
826 # response.
827 # "$HEADERS" is a pseudo dictionary containing all the headers of
828 # the previous response.
829 # "$ENVIRON" is a pseudo dictionary providing access to the current
830 # environment.
831 # "$RESPONSE" provides access to the JSON in the prior response, via
832 # JSONPath. See http://jsonpath-rw.readthedocs.io/ for
833 # jsonpath-rw formatting.
834 # $SCHEME and $NETLOC provide access to the current protocol and
835 # location (host and port).
836 - name: get the thing we just posted
838 GET: $LOCATION
839 request_headers:
840 x-magic-exchange: $HEADERS['x-magic-exchange']
841 x-token: $ENVIRON['OS_TOKEN']
842 response_json_paths:
843 $.name: $RESPONSE['$.name']
844 $.abode: $RESPONSE['$.abode']
845 response_headers:
846 content-location: /$SCHEME://$NETLOC/
847 # For APIs where resource creation is asynchronous it can be
849 # necessary to poll for the resulting resource. First we create the
850 # resource in one test. The next test uses the "poll" key to loop
851 # with a delay for a set number of times.
852 - name: create asynch
854 POST: /async_creator
855 request_headers:
856 content-type: application/json
857 data:
858 name: jones
859 abode: bungalow
860 status: 202
861 - name: poll for created resource
863 GET: $LOCATION
864 poll:
865 count: 10 # try up to ten times
866 delay: .5 # wait .5 seconds between each try
867 response_json_paths:
868 $.name: $RESPONSE['$.name']
869 $.abode: $RESPONSE['$.abode']
870 Gabbi supports JSONPath both for validating JSON response bodies and
873 within substitutions.
874 JSONPath expressions are provided by jsonpath_rw, with jsonpath_rw_ext
876 custom extensions to address common requirements:
877 1. Sorting via sorted and [/property].
879 2. Filtering via [?property = value].
881 3. Returning the respective length via len.
883 (These apply both to arrays and key-value pairs.)
885 Here is a JSONPath example demonstrating some of these features. Given
887 JSON data as follows:
888 {
890 "pets": [
891 {"type": "cat", "sound": "meow"},
892 {"type": "dog", "sound": "woof"}
893 ]
894 }
895 If the ordering of the list in pets is predictable and reliable it is
897 relatively straightforward to test values:
898 response_json_paths:
900 # length of list is two
901 $.pets.`len`: 2
902 # sound of second item in list is woof
903 $.pets[1].sound: woof
904 If the ordering is not predictable additional effort is required:
906 response_json_paths:
908 # sort by type
909 $.pets[/type][0].sound: meow
910 # sort by type, reversed
911 $.pets[\type][0].sound: woof
912 # all the sounds
913 $.pets[/type]..sound: ['meow', 'woof']
914 # filter by type = dog
915 $.pets[?type = "dog"].sound: woof
916 If it is necessary to validate the entire JSON response use a JSONPath
918 of $:
919 response_json_paths:
921 $:
922 pets:
923 - type: cat
924 sound: meow
925 - type: dog
926 sound: woof
927 This is not a technique that should be used frequently as it can lead
929 to difficult to read tests and it also indicates that your gabbi tests
930 are being used to test your serializers and data models, not just your
931 API interactions.
932 It is also possible to read raw JSON from disk for either all or some
934 of a JSON response:
935 response_json_paths:
937 $: @<data.json
938 or:
940 response_json_paths:
942 $.pets: <@pets.json
943 $.pets[0]: <@cat.json
944 Examples like this can be found in one of gabbi's own tests.
946 If it is desired to load YAML files like the JSON ones above, two
948 things must be done:
949 1. The YAMLDiskLoadingJSONHandler custom content handler must be passed
951 to the driver through the content_handlers argument. See Extensions
952 on how to do this.
953 2. The YAML files to load must be placed in a subdirectory to prevent
955 the test runner from consuming them as test files to run:
956 response_json_paths:
958 $: @<subdir/values.yaml
959 When reading from disk you can apply the same JSONPath by adding a ':'
961 to the end of your file name. This allows you to store multiple API re‐
962 sponses into a single file to reduce file management when constructing
963 your tests.
964 Given JSON data as follows:
966 {
968 "values": [{
969 "pets": [{
970 "type": "cat",
971 "sound": "meow"
972 }, {
973 "type": "dog",
974 "sound": "woof"
975 }]
976 }, {
977 "people": [{
978 "name": "chris",
979 "id": 1
980 }, {
981 "name": "justin",
982 "id": 2
983 }]
984 }]
985 }
986 You can write your tests like the following:
988 response_json_paths:
990 $.pets: <@pets.json
991 $.pets[?type = "cat"].sound: <@values.json:$.values[0].pets[?type = "cat"].sound
992 Although placing more than one API response into a single JSON file may
994 seem convenient, keep in mind there is a tradeoff in readability that
995 should not be overlooked before implementing this technique.
996 Examples like this can be found in one of gabbi's yaml-from-disk tests.
998 There are more JSONPath examples in example and in the jsonpath_rw and
1000 jsonpath_rw_ext documentation.
1001
1003 Substitutions can be made in both the left (query) and right (expected)
1004 hand sides of the json path expression. When subtitutions are used in
1005 the query, care must be taken to ensure proper quoting of the resulting
1006 value. For example if there is a uuid (with hyphens) at $RE‐
1007 SPONSE['$.id'] then this expression may fail:
1008 $.nested.structure.$RESPONSE['$.id'].name: foobar
1010 as it will evaluate to something like:
1012 $.nested.structure.ADC8AAFC-D564-40D1-9724-7680D3C010C2.name: foobar
1014 which may be treated as an arithemtic expression by the json path
1016 parser. The test author should write:
1017 $.nested.structure["$RESPONSE['$.id']"].name: foobar
1019 to quote the result of the substitution.
1021 The target host is the host on which the API to be tested can be found.
1023 Gabbi intends to preserve the flow and semantics of HTTP interactions
1024 as much as possible, and every HTTP request needs to be directed at a
1025 host of some form. Gabbi provides three ways to control this:
1026 • Using wsgi-intercept to provide a fake socket and WSGI environment on
1028 an arbitrary host and port attached to a WSGI application (see
1029 intercept examples).
1030 • Using fully qualified url values in the YAML defined tests (see full
1032 examples).
1033 • Using a host and (optionally) port defined at test build time (see
1035 live examples).
1036 The intercept and live methods are mutually exclusive per test builder,
1038 but either kind of test can freely intermix fully qualified URLs into
1039 the sequence of tests in a YAML file.
1040 For test driven development and local tests the intercept style of
1042 testing lowers test requirements (no web server required) and is fast.
1043 Interception is performed as part of fixtures processing as the most
1044 deeply nested fixture. This allows any configuration or database setup
1045 to be performed prior to the WSGI application being created.
1046 For the implementation of the above see build_tests().
1048 Each suite of tests is represented by a single YAML file, and may op‐
1050 tionally use one or more fixtures to provide the necessary environment
1051 required by the tests in that file.
1052 Fixtures are implemented as nested context managers. Subclasses of
1054 GabbiFixture must implement start_fixture and stop_fixture methods for
1055 creating and destroying, respectively, any resources managed by the
1056 fixture. While the subclass may choose to implement __init__ it is im‐
1057 portant that no exceptions are thrown in that method, otherwise the
1058 stack of context managers will fail in unexpected ways. Instead ini‐
1059 tialization of real resources should happen in start_fixture.
1060 At this time there is no mechanism for the individual tests to have any
1062 direct awareness of the fixtures. The fixtures exist, conceptually, on
1063 the server side of the API being tested.
1064 Fixtures may do whatever is required by the testing environment, how‐
1066 ever there are two common scenarios:
1067 • Establishing (and then resetting when a test suite has finished) any
1069 baseline configuration settings and persistence systems required for
1070 the tests.
1071 • Creating sample data for use by the tests.
1073 If a fixture raises unittest.case.SkipTest during start_fixture all the
1075 tests in the current file will be skipped. This makes it possible to
1076 skip the tests if some optional configuration (such as a particular
1077 type of database) is not available.
1078 If an exception is raised while a fixture is being used, information
1080 about the exception will be stored on the fixture so that the stop_fix‐
1081 ture method can decide if the exception should change how the fixture
1082 should clean up. The exception information can be found on exc_type,
1083 exc_value and traceback method attributes.
1084 If an exception is raised when a fixture is started (in start_fixture)
1086 the first test in the suite using the fixture will be marked with an
1087 error using the traceback from the exception and all the tests in the
1088 suite will be skipped. This ensures that fixture failure is adequately
1089 captured and reported by test runners.
1090 In some contexts (for example CI environments with a large number of
1092 tests being run in a broadly concurrent environment where output is
1093 logged to a single file) it can be important to capture and consolidate
1094 stray output that is produced during the tests and display it associ‐
1095 ated with an individual test. This can help debugging and avoids unus‐
1096 able output that is the result of multiple streams being interleaved.
1097 Inner fixtures have been added to support this. These are fixtures more
1099 in line with the tradtional unittest concept of fixtures: a class on
1100 which setUp and cleanUp is automatically called.
1101 build_tests() accepts a named parameter arguments of inner_fixtures.
1103 The value of that argument may be an ordered list of fixtures.Fixture
1104 classes that will be called when each individual test is set up.
1105 An example fixture that could be useful is the FakeLogger.
1107 NOTE:
1109 At this time inner_fixtures are not supported when using the pytest
1110 loader.
1111 Content handlers are responsible for preparing request data and evalu‐
1113 ating response data based on the content-type of the request and re‐
1114 sponse. A content handler operates as follows:
1115 • Structured YAML data provided via the data attribute is converted to
1117 a string or bytes sequence and used as request body.
1118 • The response body (a string or sequence of bytes) is transformed into
1120 a content-type dependent structure and stored in an internal attri‐
1121 bute named response_data that is:
1122 • used when evaluating the response body
1124 • used in $RESPONSE[] substitutions
1126 By default, gabbi provides content handlers for JSON. In that content
1128 handler the data test key is converted from structured YAML into a JSON
1129 string. Response bodies are converted from a JSON string into a data
1130 structure in response_data that is used when evaluating re‐
1131 sponse_json_paths entries in a test or doing JSONPath-based $RESPONSE[]
1132 substitutions.
1133 A YAMLDiskLoadingJSONHandler has been added to extend the JSON handler.
1135 It works the same way as the JSON handler except for when evaluating
1136 the response_json_paths handle, data that is read from disk can be ei‐
1137 ther in JSON or YAML format. The YAMLDiskLoadingJSONHandler is not en‐
1138 abled by default and must be added as shown in the Extensions section
1139 in order to be used in the tests.
1140 Further content handlers can be added as extensions. Test authors may
1142 need these extensions for their own suites, or enterprising developers
1143 may wish to create and distribute extensions for others to use.
1144 NOTE:
1146 One extension that is likely to be useful is a content handler that
1147 turns data into url-encoded form data suitable for POST and turns an
1148 HTML response into a DOM object.
1149
1151 Content handlers are an evolution of the response handler concept in
1152 earlier versions gabbi. To preserve backwards compatibility with exist‐
1153 ing response handlers, old style response handlers are still allowed,
1154 but new handlers should implement the content handler interface (de‐
1155 scribed below).
1156 Registering additional custom handlers is done by passing a subclass of
1158 ContentHandler to build_tests():
1159 driver.build_tests(test_dir, loader, host=None,
1161 intercept=simple_wsgi.SimpleWsgi,
1162 content_handlers=[MyContentHandler])
1163 If pytest is being used:
1165 driver.py_test_generator(test_dir, intercept=simple_wsgi.SimpleWsgi,
1167 content_handlers=[MyContenHandler])
1168 Gabbi provides an additional custom handler named YAMLDiskLoadingJSON‐
1170 Handler. This can be used for loading JSON and YAML files from disk
1171 when evaluating the response_json_paths handle.
1172 WARNING:
1174 YAMLDiskLoadingJSONHandler shares the same content-type as the de‐
1175 fault JSONHandler. When there are multiple handlers listed that ac‐
1176 cept the same content-type, the one that is earliest in the list
1177 will be used.
1178 With gabbi-run, custom handlers can be loaded via the --response-han‐
1180 dler option -- see load_response_handlers() for details.
1181 NOTE:
1183 The use of the --response-handler argument is done to preserve back‐
1184 wards compatibility and avoid excessive arguments. Both types of
1185 handler may be passed to the argument.
1186 Implementation Details
1188 Creating a content handler requires subclassing ContentHandler and im‐
1189 plementing several methods. These methods are described below, but in‐
1190 specting JSONHandler will be instructive in highlighting required argu‐
1191 ments and techniques.
1192 To provide a response_<something> response-body evaluator a subclass
1194 must define:
1195 • test_key_suffix: This, along with the prefix response_, forms the key
1197 used in the test structure. It is a class level string.
1198 • test_key_value: The key's default value, either an empty list ([]) or
1200 empty dict ({}). It is a class level value.
1201 • action: An instance method which tests the expected values against
1203 the HTTP response - it is invoked for each entry, with the parameters
1204 depending on the default value. The arguments to action are (in or‐
1205 der):
1206 • self: The current instance.
1208 • test: The currently active HTTPTestCase
1210 • item: The current entry if test_key_value is a list, otherwise the
1212 key half of the key/value pair at this entry.
1213 • value: None if test_key_value is a list, otherwise the value half
1215 of the key/value pair at this entry.
1216 To translate request or response bodies to or from structured data a
1218 subclass must define an accepts method. This should return True if this
1219 class is willing to translate the provided content-type. During request
1220 processing it is given the value of the content-type header that will
1221 be sent in the request. During response processing it is given the
1222 value of the content-type header of the response. This makes it possi‐
1223 ble to handle different request and response bodies in the same han‐
1224 dler, if desired. For example a handler might accept applica‐
1225 tion/x-www-form-urlencoded and text/html.
1226 If accepts is defined two additional static methods should be defined:
1228 • dumps: Turn structured Python data from the data key in a test into a
1230 string or byte stream. The optional test param allows you to access
1231 the current test case which may help with manipulations for custom
1232 content handlers, e.g. multipart/form-data needs to add a boundary to
1233 the Content-Type header in order to mark the appropriate sections of
1234 the body.
1235 • loads: Turn a string or byte stream in a response into a Python data
1237 structure. Gabbi will put this data on the response_data attribute on
1238 the test, where it can be used in the evaluations described above (in
1239 the action method) or in $RESPONSE handling. An example usage here
1240 would be to turn HTML into a DOM.
1241 • load_data_file: Load data from disk into a Python data structure.
1243 Gabbi will call this method when response_<something> contains an
1244 item where the right hand side value starts with <@. The test param
1245 allows you to access the current test case and provides a
1246 load_data_file method which should be used because it verifies the
1247 data is loaded within the test diectory and returns the file source
1248 as a string. The load_data_file method was introduced to re-use the
1249 JSONHandler in order to support loading YAML files from disk through
1250 the implementation of an additional custom handler, see YAMLDiskLoad‐
1251 ingJSONHandler for details.
1252 Finally if a replacer class method is defined, then when a $RESPONSE
1254 substitution is encountered, replacer will be passed the response_data
1255 of the prior test and the argument within the $RESPONSE.
1256 Please see the JSONHandler source for additional detail.
1258 If there is a running web service that needs to be tested and creating
1260 a test loader with build_tests() is either inconvenient or overkill it
1261 is possible to run YAML test files directly from the command line with
1262 the console-script gabbi-run. It accepts YAML on stdin or as multiple
1263 file arguments, and generates and runs tests and outputs a summary of
1264 the results.
1265 The provided YAML may not use custom fixtures but otherwise uses the
1267 default format. host information is either expressed directly in the
1268 YAML file or provided on the command line:
1269 gabbi-run [host[:port]] < /my/test.yaml
1271 or:
1273 gabbi-run http://host:port < /my/test.yaml
1275 To test with one or more files the following command syntax may be
1277 used:
1278 gabbi-run http://host:port -- /my/test.yaml /my/other.yaml
1280 NOTE:
1282 The filename arguments must come after a -- and all other arguments
1283 (host, port, prefix, failfast) must come before the --.
1284 NOTE:
1286 If files are provided, test output will use names including the name
1287 of the file. If any single file includes an error, the name of the
1288 file will be included in a summary of failed files at the end of the
1289 test report.
1290 To facilitate using the same tests against the same application mounted
1292 in different locations in a WSGI server, a prefix may be provided as a
1293 second argument:
1294 gabbi-run host[:port] [prefix] < /my/test.yaml
1296 or in the target URL:
1298 gabbi-run http://host:port/prefix < /my/test.yaml
1300 The value of prefix will be prepended to the path portion of URLs that
1302 are not fully qualified.
1303 Anywhere host is used, if it is a raw IPV6 address it should be wrapped
1305 in [ and ].
1306 If https is used in the target, then the tests in the provided YAML
1308 will default to ssl: True.
1309 Use -k or --insecure to not validate certificates when making https
1311 connections.
1312 If a -x or --failfast argument is provided then gabbi-run will exit af‐
1314 ter the first test failure.
1315 Use -v or --verbose with a value of all, headers or body to turn on
1317 verbosity for all tests being run.
1318 Use -q or --quiet to silence test runner output.
1320 These are informal release notes for gabbi since version 1.0.0, high‐
1322 lighting major features and changes. For more detail see the commit
1323 logs on GitHub.
1324
1326 • If no content-type is provided with a response and verbose is on for
1327 responses, display the response body as if it were text.
1328
1330 • Properly declare that gabbi 2.x is Python 3 only.
1331
1333 • Correct management of response handler default fields.
1334
1336 • Drop support for Python 2. If you need Python 2 support, use an older
1337 version.
1338 • Stop using testtools and fixtures. These two modules present several
1340 difficulties and their maintenance situation suggests those difficul‐
1341 ties will not be resolved. Since Python 2 support is being removed,
1342 the need for the modules can be removed as well without losing func‐
1343 tionality. "Inner fixtures" that use the fixtures.Fixture interface
1344 should continue to work.
1345
1347 • Add support for not validating certificates in https requests. Con‐
1348 trolled by the cert_validate attribute in individual tests and
1349 build_tests() and the -k or --insecure argument to gabbi-run.
1350
1352 • Support pytest 5.0.0 in Python >=3.5. For earlier versions of Python,
1353 pytest<5.0.0 will be used; the pytest project is dropping support for
1354 older versions of Python.
1355
1357 • Use pytest<5.0.0 until gabbi has solutions for the changes in 5.0.0.
1358
1360 • A -q argument is added to gabbi-run to suppress output from the test
1361 runner.
1362
1364 • Adjust loading of YAML to be ready for new release of PyYAML.
1365
1367 • Provide the YAMLDiskLoadingJSONHandler class that allows test result
1368 data for response_json_path checks to be loaded from YAML-on-disk.
1369
1371 • Use jsonpath to select a portion of data-on-disk in re‐
1372 sponse_json_path checks.
1373 • Restrict PyYAML to <4.0.
1375
1377 • Allow listing of tests with no host configured. When host is an empty
1378 string, tests can be listed (for discovery), but will be skipped on
1379 run.
1380
1382 • JSON $RESPONSE substitutions in the data field may be complex types
1383 (lists and dicts), not solely strings.
1384
1386 • When the HTTP response begins with a bad status line, have BadSta‐
1387 tusLine be raised from urllib3.
1388
1390 • Allow substitutions in the key portion of request and response head‐
1391 ers, not just the value.
1392
1394 • Remove support for Python 3.3.
1395 • Make handling of fixture-level skips in pytest actually work.
1397
1399 • Add safe_yaml parameter to build_tests().
1400
1402 • use_prior_test is added to test metadata.
1403 • Extensive cleanups in regular expression handling when constructing
1405 tests from YAML.
1406
1408 jsonpath handling gets two improvements:
1409 • The value side of a response_json_paths entry can be loaded from a
1411 file using the <@file.json syntax also used in data.
1412 • The key side of a response_json_paths entry can use substitutions.
1414 This was already true for the value side.
1415
1417 Substitutions in $RESPONSE handling now preserve numeric types instead
1418 of casting to a string. This is useful when servers are expecting
1419 strong types and tests want to send response data back to the server.
1420
1422 count and delay test keys allow substitutions.
1423 gabbi.driver.build_tests() accepts a verbose parameter to set test ver‐
1424 bosity for an entire session.
1425
1427 Better failure reporting when using gabbi-run with multiple files. Test
1428 names are based on the files and a summary of failed files is provided
1429 at the end of the report.
1430
1432 Effectively capture a failure in a fixture and report the traceback.
1433 Without this some test runners swallow the error and discovering prob‐
1434 lems when developing fixtures can be quite challenging.
1435
1437 Thanks to Samuel Fekete, tests can use the $HISTORY dictionary to refer
1438 to any prior test in the same file, not just the one immediately prior,
1439 when doing substitutions.
1440
1442 Filenames used to read data into tests using the <@ syntax may now use
1443 pathnames relative to the YAML file. See data.
1444 gabbi-run gains a --verbose parameter to force all tests run in a ses‐
1446 sion to run with verbose set.
1447 When using pytest to load tests, a new mechanism is available which
1449 avoids warnings produced in when using a version of pytest greater than
1450 3.0.
1451
1453 When verbosely displaying request and response bodies that are JSON,
1454 pretty print for improved readability.
1455
1457 Allow gabbi-run to accept multiple filenames as command line arguments
1458 instead of reading tests from stdin.
1459
1461 Switch from response handlers to handlers to allow more flexible pro‐
1462 cessing of both response _and_ request bodies.
1463 Add inner fixtures for per test fixtures, useful for output capturing.
1465
1467 Allow the test_loader_name arg to gabbi.driver.build_tests() to over‐
1468 ride the prefix of the pretty printed name of generated tests.
1469
1471 String values in JSONPath matches may be wrapped in /.../` to be
1472 treated as regular expressions.
1473
1475 Better documentation of how to run gabbi in a concurrent environment.
1476 Improved handling of pytest fixtures and test counts.
1477
1479 Add url to gabbi.driver.build_tests() to use instead of host, port and
1480 prefix.
1481
1483 Add require_ssl to gabbi.driver.build_tests() to force use of SSL.
1484
1486 Add $COOKIE substitution.
1487
1489 Correctly support IPV6 hosts.
1490
1492 Add $LAST_URL substitution.
1493
1495 Introduce support for loading and running tests with pytest.
1496
1498 Use urllib3 instead of httplib2 for driving HTTP requests.
1499
1501 Add sorting and filtering to jsonpath handling.
1502
1504 Add the response_forbidden_headers to response expectations.
1505
1507 Instead of:
1508 tests:
1510 - name: a simple get
1511 url: /some/path
1512 method: get
1513 1.7.0 also makes it possible to:
1515 tests:
1517 - name: a simple get
1518 GET: /some/path
1519 Any upper case key is treated as a method.
1521
1523 Enhanced flexibility and colorization when setting tests to be verbose.
1524
1526 Adds the query_parameters key to request parameters.
1527
1529 The start of improvements and extensions to jsonpath handling. In this
1530 case the addition of the len function.
1531
1533 Vastly improved output and behavior in gabbi-run.
1534
1536 Version 1 was the first release with a commitment to a stable format.
1537 Since then new fields have been added but have not been taken away.
1538 The following people have contributed code to gabbi. Thanks to them.
1540 Thanks also to all the people who have made gabbi better by reporting
1541 issues and their successes and failures with using gabbi.
1542 • Chris Dent
1544 • FND
1546 • Mehdi Abaakouk
1548 • Tom Viner
1550 • Jason Myers
1552 • Josh Leeb-du Toit
1554 • Duc Truong
1556 • Zane Bitter
1558 • Ryan Spencer
1560 • Kim Raymoure
1562 • Travis Truman
1564 • Samuel Fekete
1566 • Michael McCune
1568 • Imran Hayder
1570 • Julien Danjou
1572 • Trevor McCasland
1574 • Danek Duvall
1576 • Marc Abramowitz
1578 NOTE:
1580 This section provides a collection of questions with answers that
1581 don't otherwise fit in the rest of the documentation. If something
1582 is missing, please create an issue.
1583 As this document grows it will gain a more refined structure.
1585
1587 Is gabbi only for testing Python-based APIs?
1588 No, you can use gabbi-run to test an HTTP service built in any program‐
1589 ming language.
1590 How do I run just one test?
1592 Each YAML file contains a sequence of tests, each test within each file
1593 has a name. That name is translated to the name of the test by replac‐
1594 ing spaces with an _.
1595 When running tests that are generated dynamically, filtering based on
1597 the test name prior to the test being collected will not work in some
1598 test runners. Test runners that use a --load-list functionality can be
1599 convinced to filter after discovery.
1600 pytest does this directly with the -k keyword flag.
1602 When using testrepository with tox as used in gabbi's own tests it is
1604 possible to pass a filter in the tox command:
1605 tox -epy27 -- get_the_widget
1607 When using testtools.run and similar test runners it's a bit more com‐
1609 plicated. It is necessary to provide the full name of the test as a
1610 list to --load-list:
1611 python -m testtools.run --load-list \
1613 <(echo package.tests.test_api.yamlfile_get_the_widge.test_request)
1614 How do I run just one test, without running prior tests in a sequence?
1616 By default, when you select a single test to run, all tests prior to
1617 that one in a file will be run as well: the file is treated as as se‐
1618 quence of dependent tests. If you do not want this you can adjust the
1619 use_prior_test test metadata in one of three ways:
1620 • Set it in the YAML file for the one test you are concerned with.
1622 • Set the defaults for all tests in that file.
1624 • set use_prior_test to false when calling build_tests()
1626 Be aware that doing this breaks a fundamental assumption that gabbi
1628 makes about how tests work. Any substitutions will fail.
1629
1631 Can I have variables in my YAML file?
1632 Gabbi provides the $ENVIRON substitution which can operate a bit like
1633 variables that are set elsewhere and then used in the tests defined by
1634 the YAML.
1635 If you find it necessary to have variables within a single YAML file
1637 you take advantage of YAML alias nodes list this:
1638 vars:
1640 - &uuid_1 5613AABF-BAED-4BBA-887A-252B2D3543F8
1641 tests:
1643 - name: send a uuid to a post
1644 POST: /resource
1645 request_headers:
1646 content-type: application/json
1647 data:
1648 uuid: *uuid_1
1649 You can alias all sorts of nodes, not just single items. Be aware that
1651 the replacement of an alias node happens while the YAML is being
1652 loaded, before gabbi does any processing.
1653 How many tests should be put in one YAML file?
1655 For the sake of readability it is best to keep each YAML file rela‐
1656 tively short. Since each YAML file represents a sequence of requests,
1657 it usually makes sense to create a new file when a test is not depen‐
1658 dent on any before it.
1659 It's tempting to put all the tests for any resource or URL in the same
1661 file, but this eventually leads to files that are too long and are thus
1662 difficult to read.
1663
1665 A single HTTP request represented as a subclass of unittest.TestCase
1666 The test case encapsulates the request headers and body and expected
1668 response headers and body. When the test is run an HTTP request is made
1669 using urllib3. Assertions are made against the response.
1670 class gabbi.case.HTTPTestCase(methodName='runTest')
1672 Bases: unittest.case.TestCase
1673 Encapsulate a single HTTP request as a TestCase.
1675 If the test is a member of a sequence of requests, ensure that
1677 prior tests are run.
1678 To keep the test harness happy we need to make sure the setUp
1680 and tearDown are only run once.
1681 assert_in_or_print_output(expected, iterable)
1683 Assert the iterable contains expected or print some out‐
1684 put.
1685 If the output is long, it is limited by either
1687 GABBI_MAX_CHARS_OUTPUT in the environment or the
1688 MAX_CHARS_OUTPUT constant.
1689 base_test = {'cert_validate': True, 'data': '', 'desc': '',
1691 'method': 'GET', 'name': '', 'poll': {}, 'query_parameters': {},
1692 'redirects': False, 'request_headers': {}, 'skip': '', 'ssl':
1693 False, 'status': '200', 'url': '', 'use_prior_test': True, 'ver‐
1694 bose': False, 'xfail': False}
1695 get_content_handler(content_type)
1697 Determine the content handler for this media type.
1698 load_data_file(filename)
1700 Read a file from the current test directory.
1701 replace_template(message, escape_regex=False)
1703 Replace magic strings in message.
1704 run(result=None)
1706 Store the current result handler on this test.
1707 setUp()
1709 Hook method for setting up the test fixture before exer‐
1710 cising it.
1711 tearDown()
1713 Hook method for deconstructing the test fixture after
1714 testing it.
1715 test_request()
1717 Run this request if it has not yet run.
1718 If there is a prior test in the sequence, run it first.
1720 gabbi.case.potentialFailure(func)
1722 Decorate a test method that is expected to fail if 'xfail' is
1723 true.
1724
1728 Manage fixtures for gabbi at the test suite level.
1729 class gabbi.fixture.GabbiFixture
1731 Bases: object
1732 A context manager that operates as a fixture.
1734 Subclasses must implement start_fixture and stop_fixture, each
1736 of which contain the logic for stopping and starting whatever
1737 the fixture is. What a fixture is is left as an exercise for the
1738 implementor.
1739 These context managers will be nested so any actual work needs
1741 to happen in start_fixture and stop_fixture and not in __init__.
1742 Otherwise exception handling will not work properly.
1743 start_fixture()
1745 Implement the actual workings of starting the fixture
1746 here.
1747 stop_fixture()
1749 Implement the actual workings of stopping the fixture
1750 here.
1751 exception gabbi.fixture.GabbiFixtureError
1753 Bases: Exception
1754 Generic exception for GabbiFixture.
1756 class gabbi.fixture.SkipAllFixture
1758 Bases: gabbi.fixture.GabbiFixture
1759 A fixture that skips all the tests in the current suite.
1761 start_fixture()
1763 Implement the actual workings of starting the fixture
1764 here.
1765 gabbi.fixture.nest(fixtures)
1767 Nest a series of fixtures.
1768 This is duplicated from nested in the stdlib, which has been
1770 deprecated because of issues with how exceptions are difficult
1771 to handle during __init__. Gabbi needs to nest an unknown number
1772 of fixtures dynamically, so the with syntax that replaces nested
1773 will not work.
1774
1776 Package for response and content handlers that process the body of a
1777 response in various ways.
1778 handlers.base Module
1780 Base classes for response and content handlers.
1781 class gabbi.handlers.base.ContentHandler
1783 Bases: gabbi.handlers.base.ResponseHandler
1784 A subclass of ResponseHandlers that adds content handling.
1786 static accepts(content_type)
1788 Return True if this handler can handler this type.
1789 static dumps(data, pretty=False, test=None)
1791 Return structured data as a string.
1792 If pretty is true, prettify.
1794 static load_data_file(test, file_path)
1796 Return the string content of the file specified by the
1797 file_path.
1798 static loads(data)
1800 Create structured (Python) data from a stream.
1801 classmethod replacer(response_data, path)
1803 Return the string that is replacing RESPONSE.
1804 class gabbi.handlers.base.ResponseHandler
1806 Bases: object
1807 Add functionality for making assertions about an HTTP response.
1809 A subclass may implement two methods: action and preprocess.
1811 preprocess takes one argument, the TestCase. It is called ex‐
1813 actly once for each test before looping across the assertions.
1814 It is used, rarely, to copy the test.output into a useful form
1815 (such as a parsed DOM).
1816 action takes two or three arguments. If test_key_value is a list
1818 action is called with the test case and a single list item. If
1819 test_key_value is a dict then action is called with the test
1820 case and a key and value pair.
1821 action(test, item, value=None)
1823 Test an individual entry for this response handler.
1824 If the entry is a key value pair the key is in item and
1826 the value in value. Otherwise the entry is considered a
1827 single item from a list.
1828 preprocess(test)
1830 Do any pre-single-test preprocessing.
1831 test_key_suffix = ''
1833 test_key_value = []
1835 handlers.core Module
1837 Core response handlers.
1838 class gabbi.handlers.core.ForbiddenHeadersResponseHandler
1840 Bases: gabbi.handlers.base.ResponseHandler
1841 Test that listed headers are not in the response.
1843 action(test, forbidden, value=None)
1845 Test an individual entry for this response handler.
1846 If the entry is a key value pair the key is in item and
1848 the value in value. Otherwise the entry is considered a
1849 single item from a list.
1850 test_key_suffix = 'forbidden_headers'
1852 test_key_value = []
1854 class gabbi.handlers.core.HeadersResponseHandler
1856 Bases: gabbi.handlers.base.ResponseHandler
1857 Compare expected headers with actual headers.
1859 If a header value is wrapped in / it is treated as a raw regular
1861 expression.
1862 Headers values are always treated as strings.
1864 action(test, header, value=None)
1866 Test an individual entry for this response handler.
1867 If the entry is a key value pair the key is in item and
1869 the value in value. Otherwise the entry is considered a
1870 single item from a list.
1871 test_key_suffix = 'headers'
1873 test_key_value = {}
1875 class gabbi.handlers.core.StringResponseHandler
1877 Bases: gabbi.handlers.base.ResponseHandler
1878 Test for matching strings in the the response body.
1880 action(test, expected, value=None)
1882 Test an individual entry for this response handler.
1883 If the entry is a key value pair the key is in item and
1885 the value in value. Otherwise the entry is considered a
1886 single item from a list.
1887 test_key_suffix = 'strings'
1889 test_key_value = []
1891 handlers.jsonhandler Module
1893 JSON-related content handling.
1894 class gabbi.handlers.jsonhandler.JSONHandler
1896 Bases: gabbi.handlers.base.ContentHandler
1897 A ContentHandler for JSON
1899 • Structured test data is turned into JSON when request con‐
1901 tent-type is JSON.
1902 • Response bodies that are JSON strings are made into Python
1904 data on the test response_data attribute when the response
1905 content-type is JSON.
1906 • A response_json_paths response handler is added.
1908 • JSONPaths in $RESPONSE substitutions are supported.
1910 static accepts(content_type)
1912 Return True if this handler can handler this type.
1913 action(test, path, value=None)
1915 Test json_paths against json data.
1916 static dumps(data, pretty=False, test=None)
1918 Return structured data as a string.
1919 If pretty is true, prettify.
1921 static extract_json_path_value(data, path)
1923 Extract the value at JSON Path path from the data.
1924 The input data is a Python datastructure, not a JSON
1926 string.
1927 static load_data_file(test, file_path)
1929 Return the string content of the file specified by the
1930 file_path.
1931 static loads(data)
1933 Create structured (Python) data from a stream.
1934 classmethod replacer(response_data, match)
1936 Return the string that is replacing RESPONSE.
1937 test_key_suffix = 'json_paths'
1939 test_key_value = {}
1941 handlers.yaml_disk_loading_jsonhandler Module
1943 JSON-related content handling with YAML data disk loading.
1944 class gabbi.handlers.yaml_disk_loading_jsonhandler.YAMLDiskLoadingJSON‐
1946 Handler
1947 Bases: gabbi.handlers.jsonhandler.JSONHandler
1948 A ContentHandler for JSON responses that loads YAML from disk
1950 • Structured test data is turned into JSON when request con‐
1952 tent-type is JSON.
1953 • Response bodies that are JSON strings are made into Python
1955 data on the test response_data attribute when the response
1956 content-type is JSON.
1957 • A response_json_paths response handler is added. Data read
1959 from disk during this handle will be loaded with the
1960 yaml.safe_load method to support both JSON and YAML data
1961 sources from disk.
1962 • JSONPaths in $RESPONSE substitutions are supported.
1964 static load_data_file(test, file_path)
1966 Return the string content of the file specified by the
1967 file_path.
1968
1970 A TestSuite for containing gabbi tests.
1971 This suite has two features: the contained tests are ordered and there
1973 are suite-level fixtures that operate as context managers.
1974 class gabbi.suite.GabbiSuite(tests=())
1976 Bases: unittest.suite.TestSuite
1977 A TestSuite with fixtures.
1979 The suite wraps the tests with a set of nested context managers
1981 that operate as fixtures.
1982 If a fixture raises unittest.case.SkipTest during setup, all the
1984 tests in this suite will be skipped.
1985 run(result, debug=False)
1987 Override TestSuite run to start suite-level fixtures.
1988 To avoid exception confusion, use a null Fixture when
1990 there are no fixtures.
1991 start(result, tests)
1993 Start fixtures when using pytest.
1994 stop() Stop fixtures when using pytest.
1996 gabbi.suite.noop(*args)
1998 A noop method used to disable collected tests.
1999
2002 TestRunner and TestResult for gabbi-run.
2003 class gabbi.reporter.ConciseTestResult(stream, descriptions, verbosity)
2005 Bases: unittest.runner.TextTestResult
2006 A TextTestResult with simple but useful output.
2008 If the output is a tty or GABBI_FORCE_COLOR is set in the envi‐
2010 ronment, output will be colorized.
2011 addError(test, err)
2013 Called when an error has occurred. 'err' is a tuple of
2014 values as returned by sys.exc_info().
2015 addExpectedFailure(test, err)
2017 Called when an expected failure/error occurred.
2018 addFailure(test, err)
2020 Called when an error has occurred. 'err' is a tuple of
2021 values as returned by sys.exc_info().
2022 addSkip(test, reason)
2024 Called when a test is skipped.
2025 addSuccess(test)
2027 Called when a test has completed successfully
2028 addUnexpectedSuccess(test)
2030 Called when a test was expected to fail, but succeed.
2031 getDescription(test)
2033 printErrorList(flavor, errors)
2035 startTest(test)
2037 Called when the given test is about to be run
2038 class gabbi.reporter.ConciseTestRunner(stream=None, descriptions=True,
2040 verbosity=1, failfast=False, buffer=False, resultclass=None, warn‐
2041 ings=None, *, tb_locals=False)
2042 Bases: unittest.runner.TextTestRunner
2043 A TextTestRunner that uses ConciseTestResult for reporting re‐
2045 sults.
2046 resultclass
2048 alias of gabbi.reporter.ConciseTestResult
2049 class gabbi.reporter.PyTestResult(stream=None, descriptions=None, ver‐
2051 bosity=None)
2052 Bases: unittest.result.TestResult
2053 Wrap a test result to allow it to work with pytest.
2055 The main behaviors here are:
2057 • to turn what had been exceptions back into exceptions
2059 • use pytest's skip and xfail methods
2061 addError(test, err)
2063 Called when an error has occurred. 'err' is a tuple of
2064 values as returned by sys.exc_info().
2065 addExpectedFailure(test, err)
2067 Called when an expected failure/error occurred.
2068 addFailure(test, err)
2070 Called when an error has occurred. 'err' is a tuple of
2071 values as returned by sys.exc_info().
2072 addSkip(test, reason)
2074 Called when a test is skipped.
2075
2077 Utility functions grab bag.
2078 gabbi.utils.create_url(base_url, host, port=None, prefix='', ssl=False)
2080 Given pieces of a path-based url, return a fully qualified url.
2081 gabbi.utils.decode_response_content(header_dict, content)
2083 Decode content to a proper string.
2084 gabbi.utils.extract_content_type(header_dict, default='application/bi‐
2086 nary')
2087 Extract parsed content-type from headers.
2088 gabbi.utils.get_colorizer(stream)
2090 Return a function to colorize a string.
2091 Only if stream is a tty .
2093 gabbi.utils.host_info_from_target(target, prefix=None)
2095 Turn url or host:port and target into test destination.
2096 gabbi.utils.load_yaml(handle=None, yaml_file=None, safe=True)
2098 Read and parse any YAML file or filehandle.
2099 Let exceptions flow where they may.
2101 If no file or handle is provided, read from STDIN.
2103 gabbi.utils.not_binary(content_type)
2105 Decide if something is content we'd like to treat as a string.
2106 gabbi.utils.parse_content_type(content_type, default_charset='utf-8')
2108 Parse content type value for media type and charset.
2109
2111 Gabbi specific exceptions.
2112 exception gabbi.exception.GabbiFormatError
2114 Bases: ValueError
2115 An exception to encapsulate poorly formed test data.
2117 exception gabbi.exception.GabbiSyntaxWarning
2119 Bases: SyntaxWarning
2120 A warning about syntax that is not desirable.
2122
2125 Keep one single global jsonpath parser.
2126 gabbi.json_parser.parse(path)
2128 Parse a JSONPath expression use the global parser.
2129 Gabbi is a tool for running HTTP tests where requests and responses are
2131 expressed as declarations in YAML files:
2132 tests:
2134 - name: retrieve items
2135 GET: /items
2136 See the rest of these docs for more details on the many features and
2138 formats for setting request headers and bodies and evaluating re‐
2139 sponses.
2140 Tests can be run from the command line with gabbi-run or programmati‐
2142 cally using either py.test or unittest-style test runners. See
2143 installation instructions below.
2144 The name is derived from "gabby": excessively talkative. In a test en‐
2146 vironment having visibility of what a test is actually doing is a good
2147 thing. This is especially true when the goal of a test is to test the
2148 HTTP, not the testing infrastructure. Gabbi tries to put the HTTP in‐
2149 teraction in the foreground of testing.
2150 If you want to get straight to creating tests look at example, the test
2152 files in the source distribution and format. A gabbi-demo repository
2153 provides a tutorial of using gabbi to build an API, via the commit his‐
2154 tory of the repo.
2155
2157 Gabbi works to bridge the gap between human readable YAML files (see
2158 format for details) that represent HTTP requests and expected responses
2159 and the rather complex world of automated testing.
2160 Each YAML file represents an ordered list of HTTP requests along with
2162 the expected responses. This allows a single file to represent a
2163 process in the API being tested. For example:
2164 • Create a resource.
2166 • Retrieve a resource.
2168 • Delete a resource.
2170 • Retrieve a resource again to confirm it is gone.
2172 At the same time it is still possible to ask gabbi to run just one re‐
2174 quest. If it is in a sequence of tests, those tests prior to it in the
2175 YAML file will be run (in order). In any single process any test will
2176 only be run once. Concurrency is handled such that one file runs in one
2177 process.
2178 These features mean that it is possible to create tests that are useful
2180 for both humans (as tools for learning, improving and developing APIs)
2181 and automated CI systems.
2182 Significant flexibility and power is available in the format to make it
2184 relatively straightforward to test existing complex APIs. This ex‐
2185 tended functionality includes the use of JSONPath to query response
2186 bodies and templating of test data to allow access to the prior HTTP
2187 response in the current request. For APIs which do not use JSON addi‐
2188 tional handlers can be created.
2189 Care should be taken when using this functionality when you are creat‐
2191 ing a new API. If your API is so complex that it needs complex test
2192 files then you may wish to take that as a sign that your API itself too
2193 complex. One goal of gabbi is to encourage transparent and comprehensi‐
2194 ble APIs.
2195 Though gabbi is written in Python and under the covers uses unittest
2197 data structures and processes, there is no requirement that the host be
2198 a Python-based service. Anything talking HTTP can be tested. A runner
2199 makes it possible to simply create YAML files and point them at a run‐
2200 ning server.
2201
2203 As a Python package, gabbi is typically installed via pip:
2204 pip install gabbi
2206 You might want to create a virtual environment; an isolated context for
2208 Python packages, keeping gabbi cleany separated from the rest of your
2209 system.
2210 Python 3 comes with a built-in tool to create virtual environments:
2212 python3 -m venv venv
2214 . venv/bin/activate
2215 pip install gabbi
2217 This way we can later use deactivate and safely remove the venv direc‐
2219 tory, thus erasing any trace of gabbi from the system.
2220
2222 Chris Dent
2223 Jan 27, 2021 GABBI(1)