1GABBI(1) Gabbi GABBI(1)
2
6 gabbi - Gabbi Documentation
7 Gabbi tests are expressed in YAML as a series of HTTP requests with
9 their expected response:
10 tests:
12 - name: retrieve root
13 GET: /
14 status: 200
15 This will trigger a GET request to / on the configured host. The test
17 will pass if the response's status code is 200.
18
20 The top-level tests category contains an ordered sequence of test dec‐
21 larations, each describing the expected response to a given request:
22 Metadata
24 ┌───────────────┬─────────────────────┬───────────────────┐
25 │Key │ Description │ Notes │
26 ├───────────────┼─────────────────────┼───────────────────┤
27 │name │ The test's name. │ required │
28 │ │ Must be unique │ │
29 │ │ within a file. │ │
30 ├───────────────┼─────────────────────┼───────────────────┤
31 │desc │ An arbitrary string │ │
32 │ │ describing the │ │
33 │ │ test. │ │
34 ├───────────────┼─────────────────────┼───────────────────┤
35 │verbose │ If True or all │ defaults to False │
36 │ │ (synonymous), │ │
37 │ │ prints a represen‐ │ │
38 │ │ tation of the cur‐ │ │
39 │ │ rent request and │ │
40 │ │ response to stdout, │ │
41 │ │ including both │ │
42 │ │ headers and body. │ │
43 │ │ If set to headers │ │
44 │ │ or body, only the │ │
45 │ │ corresponding part │ │
46 │ │ of the request and │ │
47 │ │ response will be │ │
48 │ │ printed. If the │ │
49 │ │ output is a TTY, │ │
50 │ │ colors will be │ │
51 │ │ used. If the body │ │
52 │ │ content-type is │ │
53 │ │ JSON it will be │ │
54 │ │ formatted for │ │
55 │ │ improved readabil‐ │ │
56 │ │ ity. See Verbose‐ │ │
57 │ │ Http for details. │ │
58 ├───────────────┼─────────────────────┼───────────────────┤
59 │skip │ A string message │ defaults to False │
60 │ │ which if set will │ │
61 │ │ cause the test to │ │
62 │ │ be skipped with the │ │
63 │ │ provided message. │ │
64 └───────────────┴─────────────────────┴───────────────────┘
65 │xfail │ Determines whether │ defaults to False │
68 │ │ to expect this test │ │
69 │ │ to fail. Note that │ │
70 │ │ the test will be │ │
71 │ │ run anyway. │ │
72 ├───────────────┼─────────────────────┼───────────────────┤
73 │use_prior_test │ Determines if this │ defaults to True │
74 │ │ test will be run in │ │
75 │ │ sequence (after) │ │
76 │ │ the test prior to │ │
77 │ │ it in the list of │ │
78 │ │ tests within a │ │
79 │ │ file. To be con‐ │ │
80 │ │ crete, when this is │ │
81 │ │ True the test is │ │
82 │ │ dependent on the │ │
83 │ │ prior test and if │ │
84 │ │ that prior has not │ │
85 │ │ yet run, it wil be │ │
86 │ │ run, even if only │ │
87 │ │ the current test │ │
88 │ │ has been selected. │ │
89 │ │ Set this to False │ │
90 │ │ to allow selecting │ │
91 │ │ a test without │ │
92 │ │ dependencies. │ │
93 └───────────────┴─────────────────────┴───────────────────┘
94 NOTE:
96 When tests are generated dynamically, the TestCase name will include
97 the respective test's name, lowercased with spaces transformed to _.
98 In at least some test runners this will allow you to select and fil‐
99 ter on test name.
100 Request Parameters
102 ┌────────────────────┬───────────────────────┬─────────────────────┐
103 │Key │ Description │ Notes │
104 ├────────────────────┼───────────────────────┼─────────────────────┤
105 │any uppercase │ Any such key is │ │
106 │string │ considered an HTTP │ │
107 │ │ method, with the │ │
108 │ │ corresponding value │ │
109 │ │ expressing the URL. │ │
110 │ │ │ │
111 │ │ This is a shortcut │ │
112 │ │ combining method │ │
113 │ │ and url into a sin‐ │ │
114 │ │ gle statement: │ │
115 │ │ │ │
116 │ │ GET: /index │ │
117 │ │ │ │
118 │ │ corresponds │ │
119 │ │ to: │ │
120 │ │ │ │
121 │ │ method: GET │ │
122 │ │ url: /index │ │
123 ├────────────────────┼───────────────────────┼─────────────────────┤
124 │method │ The HTTP request │ defaults to GET │
125 │ │ method. │ │
126 └────────────────────┴───────────────────────┴─────────────────────┘
127 │url │ The URL to request. │ Either this or the │
134 │ │ This can either be a │ shortcut above is │
135 │ │ full path (e.g. │ required │
136 │ │ "/index") or a fully │ │
137 │ │ qualified URL (i.e. │ │
138 │ │ including host and │ │
139 │ │ scheme, e.g. "‐ │ │
140 │ │ http://exam‐ │ │
141 │ │ ple.org/index") — see │ │
142 │ │ host for details. │ │
143 ├────────────────────┼───────────────────────┼─────────────────────┤
144 │request_headers │ A dictionary of │ │
145 │ │ key-value pairs rep‐ │ │
146 │ │ resenting request │ │
147 │ │ header names and val‐ │ │
148 │ │ ues. These will be │ │
149 │ │ added to the con‐ │ │
150 │ │ structed request. │ │
151 ├────────────────────┼───────────────────────┼─────────────────────┤
152 │query_parameters │ A dictionary of query │ │
153 │ │ parameters that will │ │
154 │ │ be added to the url │ │
155 │ │ as query string. If │ │
156 │ │ that URL already con‐ │ │
157 │ │ tains a set of query │ │
158 │ │ parameters, those wil │ │
159 │ │ be extended. See │ │
160 │ │ example for a demon‐ │ │
161 │ │ stration of how the │ │
162 │ │ data is structured. │ │
163 ├────────────────────┼───────────────────────┼─────────────────────┤
164 │data │ A representation to │ │
165 │ │ pass as the body of a │ │
166 │ │ request. Note that │ │
167 │ │ content-type in │ │
168 │ │ request_headers │ │
169 │ │ should also be set — │ │
170 │ │ see Data for details. │ │
171 ├────────────────────┼───────────────────────┼─────────────────────┤
172 │redirects │ If True, redirects │ defaults to False │
173 │ │ will automatically be │ │
174 │ │ followed. │ │
175 ├────────────────────┼───────────────────────┼─────────────────────┤
176 │ssl │ Determines whether │ defaults to False │
177 │ │ the request uses SSL │ │
178 │ │ (i.e. HTTPS). Note │ │
179 │ │ that the url's scheme │ │
180 │ │ takes precedence if │ │
181 │ │ present — see host │ │
182 │ │ for details. │ │
183 └────────────────────┴───────────────────────┴─────────────────────┘
184 Response Expectations
186 ┌────────────────────┬─────────────────────┬─────────────────┐
187 │Key │ Description │ Notes │
188 └────────────────────┴─────────────────────┴─────────────────┘
189 │status │ The expected │ defaults to 200 │
200 │ │ response status │ │
201 │ │ code. Multiple │ │
202 │ │ acceptable response │ │
203 │ │ codes may be pro‐ │ │
204 │ │ vided, separated by │ │
205 │ │ || (e.g. 302 || 301 │ │
206 │ │ — note, however, │ │
207 │ │ that this indicates │ │
208 │ │ ambiguity, which is │ │
209 │ │ generally undesir‐ │ │
210 │ │ able). │ │
211 ├────────────────────┼─────────────────────┼─────────────────┤
212 │response_headers │ A dictionary of │ │
213 │ │ key-value pairs │ │
214 │ │ representing │ │
215 │ │ expected response │ │
216 │ │ header names and │ │
217 │ │ values. If a │ │
218 │ │ header's value is │ │
219 │ │ wrapped in /.../, │ │
220 │ │ it will be treated │ │
221 │ │ as a regular │ │
222 │ │ expression to │ │
223 │ │ search for in the │ │
224 │ │ response header. │ │
225 ├────────────────────┼─────────────────────┼─────────────────┤
226 │response_forbid‐ │ A list of headers │ │
227 │den_headers │ which must not be │ │
228 │ │ present. │ │
229 ├────────────────────┼─────────────────────┼─────────────────┤
230 │response_strings │ A list of string │ │
231 │ │ fragments expected │ │
232 │ │ to be present in │ │
233 │ │ the response body. │ │
234 ├────────────────────┼─────────────────────┼─────────────────┤
235 │response_json_paths │ A dictionary of │ │
236 │ │ JSONPath rules │ │
237 │ │ paired with │ │
238 │ │ expected matches. │ │
239 │ │ Using this rule │ │
240 │ │ requires that the │ │
241 │ │ content being sent │ │
242 │ │ from the server is │ │
243 │ │ JSON (i.e. a con‐ │ │
244 │ │ tent type of appli‐ │ │
245 │ │ cation/json or con‐ │ │
246 │ │ taining +json) │ │
247 │ │ │ │
248 │ │ If the value is │ │
249 │ │ wrapped in /.../ │ │
250 │ │ the result of the │ │
251 │ │ JSONPath query will │ │
252 │ │ be searched for the │ │
253 │ │ value as a regular │ │
254 │ │ expression. │ │
255 └────────────────────┴─────────────────────┴─────────────────┘
256 │poll │ A dictionary of two │ │
266 │ │ keys: │ │
267 │ │ │ │
268 │ │ · count: An │ │
269 │ │ integer │ │
270 │ │ stating │ │
271 │ │ the number │ │
272 │ │ of times │ │
273 │ │ to attempt │ │
274 │ │ this test │ │
275 │ │ before │ │
276 │ │ giving up. │ │
277 │ │ │ │
278 │ │ · delay: A │ │
279 │ │ floating │ │
280 │ │ point num‐ │ │
281 │ │ ber of │ │
282 │ │ seconds to │ │
283 │ │ delay │ │
284 │ │ between │ │
285 │ │ attempts. │ │
286 │ │ │ │
287 │ │ This makes │ │
288 │ │ it possible │ │
289 │ │ to poll for │ │
290 │ │ a resource │ │
291 │ │ created via │ │
292 │ │ an asynchro‐ │ │
293 │ │ nous │ │
294 │ │ request. Use │ │
295 │ │ with cau‐ │ │
296 │ │ tion. │ │
297 └────────────────────┴─────────────────────┴─────────────────┘
298 Note that many of these items allow substitutions.
300 Default values for a file's tests may be provided via the top-level
302 defaults category. These take precedence over the global defaults
303 (explained below).
304 For examples see the gabbi tests, example and the gabbi-demo tutorial.
306
308 The top-level fixtures category contains a sequence of named fixtures.
309
311 response_* keys are examples of Response Handlers. Custom handlers may
312 be created by test authors for specific use cases. See handlers for
313 more information.
314
316 There are a number of magical variables that can be used to make refer‐
317 ence to the state of a current test, the one just prior or any test
318 prior to the current one. The variables are replaced with real values
319 during test processing.
320 Global
322 · $ENVIRON['<environment variable>']: The name of an environment vari‐
323 able. Its value will replace the magical variable. If the string
324 value of the environment variable is "True" or "False" then the
325 resulting value will be the corresponding boolean, not a string.
326 Current Test
328 · $SCHEME: The current scheme/protocol (usually http or https).
329 · $NETLOC: The host and potentially port of the request.
331 Immediately Prior Test
333 · $COOKIE: All the cookies set by any Set-Cookie headers in the prior
334 response, including only the cookie key and value pairs and no meta‐
335 data (e.g. expires or domain).
336 · $URL: The URL defined in the prior request, after substitutions have
338 been made. For backwards compatibility with earlier releases
339 $LAST_URL may also be used, but if $HISTORY (see below) is being
340 used, $URL must be used.
341 · $LOCATION: The location header returned in the prior response.
343 · $HEADERS['<header>']: The value of any header from the prior
345 response.
346 · $RESPONSE['<json path>']: A JSONPath query into the prior response.
348 See jsonpath for more on formatting.
349 Any Previous Test
351 · $HISTORY['<test name>'].<magical variable expression>: Any variable
352 which refers to a prior test may be used in an expression that refers
353 to any earlier test in the same file by identifying the target test
354 by its name in a $HISTORY dictionary. For example, to refer to a
355 value in a JSON object in the response of a test named post json:
356 $HISTORY['post json'].$RESPONSE['$.key']
358 This is a very powerful feature that could lead to test that are dif‐
360 ficult for humans to read. Take care to optimize for the maintainers
361 that will come after you, not yourself.
362 NOTE:
364 Where a single-quote character, ', is shown in the variables above
365 you may also use a double-quote character, ", but in any given
366 expression the same character must be used at both ends.
367 All of these variables may be used in all of the following fields:
369 · url
371 · query_parameters
373 · data
375 · request_headers (in both the key and value)
377 · response_strings
379 · response_json_paths (in both the key and value, see json path substi‐
381 tution for more info)
382 · response_headers (in both the key and value)
384 · response_forbidden_headers
386 · count and delay fields of poll
388 With these variables it ought to be possible to traverse an API without
390 any explicit statements about the URLs being used. If you need a
391 replacement on a field that is not currently supported please raise an
392 issue or provide a patch.
393 As all of these features needed to be tested in the development of
395 gabbi itself, the gabbi tests are a good source of examples on how to
396 use the functionality. See also example for a collection of examples
397 and the gabbi-demo tutorial.
398
400 The data key has some special handing to allow for a bit more flexibil‐
401 ity when doing a POST or PUT:
402 · If the value is not a string (that is, it is a sequence or structure)
404 it is treated as a data structure that will be turned into a string
405 by the dumps method on the relevant content handler. For example if
406 the content-type of the body is application/json the data structure
407 will be turned into a JSON string.
408 · If the value is a string that begins with <@ then the rest of the
410 string is treated as a filepath to be loaded. The path is relative to
411 the test directory and may not traverse up into parent directories.
412 · If the value is an undecorated string, that's the value.
414 NOTE:
416 When reading from a file care should be taken to ensure that a rea‐
417 sonable content-type is set for the data as this will control if any
418 encoding is done of the resulting string value. If it is text, json,
419 xml or javascript it will be encoded to UTF-8.
420 To run gabbi tests with a test harness they must be generated in some
422 fashion and then run. This is accomplished by a test loader. Initially
423 gabbi only supported those test harnesses that supported the load_tests
424 protocol in UnitTest. It now possible to also build and run tests with
425 pytest with some limitations described below.
426 NOTE:
428 It is also possible to run gabbi tests from the command line. See
429 runner.
430 NOTE:
432 By default gabbi will load YAML files using the safe_load function.
433 This means only basic YAML types are allowed in the file. For most
434 use cases this is fine. If you need custom types (for example, to
435 match NaN) it is possible to set the safe_yaml parameter of
436 build_tests() to False. If custom types are used, please keep in
437 mind that this can limit the portability of the YAML files to other
438 contexts.
439 WARNING:
441 If test are being run with a runner that supports concurrency (such
442 as testrepository) it is critical that the test runner is informed
443 of how to group the tests into their respective suites. The usual
444 way to do this is to use a regular expression that groups based on
445 the name of the yaml files. For example, when using testrepository
446 the .testr.conf file needs an entry similar to the following:
447 group_regex=gabbi\.suitemaker\.(test_[^_]+_[^_]+)
449
451 To run the tests with a load_tests style loader a test file containing
452 a load_tests method is required. That will look a bit like:
453 """A sample test module."""
455 # For pathname munging
457 import os
458 # The module that build_tests comes from.
460 from gabbi import driver
461 # We need access to the WSGI application that hosts our service
463 from myapp import wsgiapp
464 # We're using fixtures in the YAML files, we need to know where to
467 # load them from.
468 from myapp.test import fixtures
469 # By convention the YAML files are put in a directory named
471 # "gabbits" that is in the same directory as the Python test file.
472 TESTS_DIR = 'gabbits'
473 def load_tests(loader, tests, pattern):
476 """Provide a TestSuite to the discovery process."""
477 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
478 # Pass "require_ssl=True" as an argument to force all tests
479 # to use SSL in requests.
480 return driver.build_tests(test_dir, loader,
481 intercept=wsgiapp.app,
482 fixture_module=fixtures)
483 For details on the arguments available when building tests see
486 build_tests().
487 Once the test loader has been created, it needs to be run. There are
489 many options. Which is appropriate depends very much on your environ‐
490 ment. Here are some examples using unittest or testtools that require
491 minimal knowledge to get started.
492 By file:
494 python -m testtools.run -v test/test_loader.py
496 By module:
498 python -m testttols.run -v test.test_loader
500 python -m unittest -v test.test_loader
502 Using test discovery to locate all tests in a directory tree:
504 python -m testtools.run discover
506 python -m unittest discover test
508 See the source distribution and the tutorial repo for more advanced
510 options, including using testrepository and subunit.
511
513 Since pytest does not support the load_tests system, a different way of
514 generating tests is required. Two techniques are supported.
515 The original method (described below) used yield statements to generate
517 tests which pytest would collect. This style of tests is deprecated as
518 of pytest>=3.0 so a new style using pytest fixtures has been developed.
519 pytest >= 3.0
521 In the newer technique, a test file is created that uses the
522 pytest_generate_tests hook. Special care must be taken to always import
523 the test_pytest method which is the base test that the pytest hook
524 parametrizes to generate the tests from the YAML files. Without the
525 method, the hook will not be called and no tests generated.
526 Here is a simple example file:
528 """A sample pytest module for pytest >= 3.0."""
530 # For pathname munging
532 import os
533 # The module that py_test_generator comes from.
535 from gabbi import driver
536 # We need test_pytest so that pytest test collection works properly.
538 # Without this, the pytest_generate_tests method below will not be
539 # called.
540 from gabbi.driver import test_pytest # noqa
541 # We need access to the WSGI application that hosts our service
543 from myapp import wsgiapp
544 # We're using fixtures in the YAML files, we need to know where to
546 # load them from.
547 from myapp.test import fixtures
548 # By convention the YAML files are put in a directory named
550 # "gabbits" that is in the same directory as the Python test file.
551 TESTS_DIR = 'gabbits'
552 def pytest_generate_tests(metafunc):
555 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
556 driver.py_test_generator(
557 test_dir, intercept=wsgiapp.app,
558 fixture_module=fixtures, metafunc=metafunc)
559 This can then be run with the usual pytest commands. For example:
562 py.test -svx pytest3.0-example.py
564 pytest < 3.0
566 When using the older technique, test file must be created that calls
567 py_test_generator() and yields the generated tests. That will look a
568 bit like this:
569 """A sample pytest module."""
571 # For pathname munging
573 import os
574 # The module that build_tests comes from.
576 from gabbi import driver
577 # We need access to the WSGI application that hosts our service
579 from myapp import wsgiapp
580 # We're using fixtures in the YAML files, we need to know where to
582 # load them from.
583 from myapp.test import fixtures
584 # By convention the YAML files are put in a directory named
586 # "gabbits" that is in the same directory as the Python test file.
587 TESTS_DIR = 'gabbits'
588 def test_gabbits():
591 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
592 # Pass "require_ssl=True" as an argument to force all tests
593 # to use SSL in requests.
594 test_generator = driver.py_test_generator(
595 test_dir, intercept=wsgiapp.app,
596 fixture_module=fixtures)
597 for test in test_generator:
599 yield test
600 This can then be run with the usual pytest commands. For example:
603 py.test -svx pytest-example.py
605 The older technique will continue to work with all versions of
607 pytest<4.0 but >=3.0 will produce warnings. If you want to use the
608 older technique but not see the warnings add --disable-pytest-warnings
609 parameter to the invocation of py.test.
610 What follows is a commented example of some tests in a single file
612 demonstrating many of the format features. See loader for the Python
613 needed to integrate with a testing harness.
614 # Fixtures can be used to set any necessary configuration, such as a
617 # persistence layer, and establish sample data. They operate per
618 # file. They are context managers, each one wrapping the next in the
619 # sequence.
620 fixtures:
622 - ConfigFixture
623 - SampleDataFixture
624 # There is an included fixture named "SkipAllFixture" which can be
626 # used to declare that all the tests in the given file are to be
627 # skipped.
628 # Each test file can specify a set of defaults that will be used for
630 # every request. This is useful for always specifying a particular
631 # header or always requiring SSL. These values will be used on every
632 # test in the file unless overriden. Lists and dicts are merged one
633 # level deep, except for "data" which is copied verbatim whether it
634 # is a string, list or dict (it can be all three).
635 defaults:
637 ssl: True
638 request_headers:
639 x-my-token: zoom
640 # The tests themselves are a list under a "tests" key. It's useful
642 # to use plenty of whitespace to help readability.
643 tests:
645 # Each request *must* have a name which is unique to the file. When it
647 # becomes a TestCase the name will be lowercased and spaces will
648 # become "_". Use that generated name when limiting test runs.
649 - name: a test for root
651 desc: Some explanatory text that could be used by other tooling
652 # The URL can either be relative to a host specified elsewhere or
654 # be a fully qualified "http" or "https" URL. *You* are responsible
655 # for url-encoding the URL.
656 url: /
658 method: GET
659 # If no status or method are provided they default to "200" and
661 # "GET".
662 # Instead of explicitly stating "url" and "method" you can join
664 # those two keys into one key representing the method. The method
665 # *must* be uppercase.
666 - name: another test for root
668 desc: Same test as above but with GET key
669 GET: /
670 # A single test can override settings in defaults (set above).
672 - name: root without ssl redirects
674 ssl: False
675 GET: /
676 status: 302
677 # When evaluating response headers it is possible to use a regular
679 # expression to not have to test the whole value. Regular expressions match
680 # anywhere in the output, not just at the beginning.
681 response_headers:
683 location: /^https/
684 # By default redirects will not be followed. This can be changed.
686 - name: follow root without ssl redirect
688 ssl: False
689 redirects: True
690 GET: /
691 status: 200 # This is the response code after the redirect.
692 # URLs can express query parameters in two ways: either in the url
694 # value directly, or as query_parameters. If both are used then
695 # query_parameters are appended. In this example the resulting URL
696 # will be equivalient to
697 # /foo?section=news&article=1&article=2&date=yesterday
698 # but not necessarily in that order.
699 - name: create a url with parameters
701 GET: /foo?section=news
702 query_parameters:
703 article:
704 - 1
705 - 2
706 date: yesterday
707 # Request headers can be used to declare media-type choices and
709 # experiment with authorization handling (amongst other things).
710 # Response headers allow evaluating headers in the response. These
711 # two together form the core value of gabbi.
712 - name: test accept
714 GET: /resource
715 request_headers:
716 accept: application/json
717 response_headers:
718 content-type: /application/json/
719 # If a header must not be present in a response at all that can be
721 # expressed in a test as follows.
722 - name: test forbidden headers
724 GET: /resource
725 response_forbidden_headers:
726 - x-special-header
727 # All of the above requests have defaulted to a "GET" method. When
729 # using "POST", "PUT" or "PATCH", the "data" key provides the
730 # request body.
731 - name: post some text
733 POST: /text_repo
734 request_headers:
735 content-type: text/plain
736 data: "I'm storing this"
737 status: 201
738 # If the data is not a string, it will be transformed into JSON.
740 # You must supply an appropriate content-type request header.
741 - name: post some json
743 POST: /json_repo
744 request_headers:
745 content-type: application/json
746 data:
747 name: smith
748 abode: castle
749 status: 201
750 # If the data is a string prepended with "<@" the value will be
752 # treated as the name of a file in the same directory as the YAML
753 # file. Again, you must supply an appropriate content-type. If the
754 # content-type is one of several "text-like" types, the content will
755 # be assumed to be UTF-8 encoded.
756 - name: post an image
758 POST: /image_repo
759 request_headers:
760 content-type: image/png
761 data: <@kittens.png
762 # A single request can be marked to be skipped.
764 - name: patch an image
766 skip: patching images not yet implemented
767 PATCH: /image_repo/12d96fb8-e78c-11e4-8c03-685b35afa334
768 # Or a single request can be marked that it is expected to fail.
770 - name: check allow headers
772 desc: the framework doesn't do allow yet
773 xfail: True
774 PUT: /post_only_url
775 status: 405
776 response_headers:
777 allow: POST
778 # The body of a response can be evaluated with response handlers.
780 # The most simple checks for a set of strings anywhere in the
781 # response. Note that the strings are members of a list.
782 - name: check for css file
784 GET: /blog/posts/12
785 response_strings:
786 - normalize.css
787 # For JSON responses, JSONPath rules can be used.
789 - name: post some json get back json
791 POST: /json_repo
792 request_headers:
793 content-type: application/json
794 data:
795 name: smith
796 abode: castle
797 status: 201
798 response_json_paths:
799 $.name: smith
800 $.abode: castle
801 # Requests run in sequence. One test can make reference to the test
803 # immediately prior using some special variables.
804 # "$LOCATION" contains the "location" header in the previous
805 # response.
806 # "$HEADERS" is a pseudo dictionary containing all the headers of
807 # the previous response.
808 # "$ENVIRON" is a pseudo dictionary providing access to the current
809 # environment.
810 # "$RESPONSE" provides access to the JSON in the prior response, via
811 # JSONPath. See http://jsonpath-rw.readthedocs.io/ for
812 # jsonpath-rw formatting.
813 # $SCHEME and $NETLOC provide access to the current protocol and
814 # location (host and port).
815 - name: get the thing we just posted
817 GET: $LOCATION
818 request_headers:
819 x-magic-exchange: $HEADERS['x-magic-exchange']
820 x-token: $ENVIRON['OS_TOKEN']
821 response_json_paths:
822 $.name: $RESPONSE['$.name']
823 $.abode: $RESPONSE['$.abode']
824 response_headers:
825 content-location: /$SCHEME://$NETLOC/
826 # For APIs where resource creation is asynchronous it can be
828 # necessary to poll for the resulting resource. First we create the
829 # resource in one test. The next test uses the "poll" key to loop
830 # with a delay for a set number of times.
831 - name: create asynch
833 POST: /async_creator
834 request_headers:
835 content-type: application/json
836 data:
837 name: jones
838 abode: bungalow
839 status: 202
840 - name: poll for created resource
842 GET: $LOCATION
843 poll:
844 count: 10 # try up to ten times
845 delay: .5 # wait .5 seconds between each try
846 response_json_paths:
847 $.name: $RESPONSE['$.name']
848 $.abode: $RESPONSE['$.abode']
849 Gabbi supports JSONPath both for validating JSON response bodies and
852 within substitutions.
853 JSONPath expressions are provided by jsonpath_rw, with jsonpath_rw_ext
855 custom extensions to address common requirements:
856 1. Sorting via sorted and [/property].
858 2. Filtering via [?property = value].
860 3. Returning the respective length via len.
862 (These apply both to arrays and key-value pairs.)
864 Here is a JSONPath example demonstrating some of these features. Given
866 JSON data as follows:
867 {
869 "pets": [
870 {"type": "cat", "sound": "meow"},
871 {"type": "dog", "sound": "woof"}
872 ]
873 }
874 If the ordering of the list in pets is predictable and reliable it is
876 relatively straightforward to test values:
877 response_json_paths:
879 # length of list is two
880 $.pets.`len`: 2
881 # sound of second item in list is woof
882 $.pets[1].sound: woof
883 If the ordering is not predictable additional effort is required:
885 response_json_paths:
887 # sort by type
888 $.pets[/type][0].sound: meow
889 # sort by type, reversed
890 $.pets[\type][0].sound: woof
891 # all the sounds
892 $.pets[/type]..sound: ['meow', 'woof']
893 # filter by type = dog
894 $.pets[?type = "dog"].sound: woof
895 If it is necessary to validate the entire JSON response use a JSONPath
897 of $:
898 response_json_paths:
900 $:
901 pets:
902 - type: cat
903 sound: meow
904 - type: dog
905 sound: woof
906 This is not a technique that should be used frequently as it can lead
908 to difficult to read tests and it also indicates that your gabbi tests
909 are being used to test your serializers and data models, not just your
910 API interactions.
911 It is also possible to read raw JSON from disk for either all or some
913 of a JSON response:
914 response_json_paths:
916 $: @<data.json
917 or:
919 response_json_paths:
921 $.pets: <@pets.json
922 $.pets[0]: <@cat.json
923 Examples like this can be found in one of gabbi's own tests.
925 There are more JSONPath examples in example and in the jsonpath_rw and
927 jsonpath_rw_ext documentation.
928
930 Substitutions can be made in both the left (query) and right (expected)
931 hand sides of the json path expression. When subtitutions are used in
932 the query, care must be taken to ensure proper quoting of the resulting
933 value. For example if there is a uuid (with hyphens) at
934 $RESPONSE['$.id'] then this expression may fail:
935 $.nested.structure.$RESPONSE['$.id'].name: foobar
937 as it will evaluate to something like:
939 $.nested.structure.ADC8AAFC-D564-40D1-9724-7680D3C010C2.name: foobar
941 which may be treated as an arithemtic expression by the json path
943 parser. The test author should write:
944 $.nested.structure["$RESPONSE['$.id']"].name: foobar
946 to quote the result of the substitution.
948 The target host is the host on which the API to be tested can be found.
950 Gabbi intends to preserve the flow and semantics of HTTP interactions
951 as much as possible, and every HTTP request needs to be directed at a
952 host of some form. Gabbi provides three ways to control this:
953 · Using wsgi-intercept to provide a fake socket and WSGI environment on
955 an arbitrary host and port attached to a WSGI application (see
956 intercept examples).
957 · Using fully qualified url values in the YAML defined tests (see full
959 examples).
960 · Using a host and (optionally) port defined at test build time (see
962 live examples).
963 The intercept and live methods are mutually exclusive per test builder,
965 but either kind of test can freely intermix fully qualified URLs into
966 the sequence of tests in a YAML file.
967 For test driven development and local tests the intercept style of
969 testing lowers test requirements (no web server required) and is fast.
970 Interception is performed as part of fixtures processing as the most
971 deeply nested fixture. This allows any configuration or database setup
972 to be performed prior to the WSGI application being created.
973 For the implementation of the above see build_tests().
975 Each suite of tests is represented by a single YAML file, and may
977 optionally use one or more fixtures to provide the necessary environ‐
978 ment required by the tests in that file.
979 Fixtures are implemented as nested context managers. Subclasses of
981 GabbiFixture must implement start_fixture and stop_fixture methods for
982 creating and destroying, respectively, any resources managed by the
983 fixture. While the subclass may choose to implement __init__ it is
984 important that no exceptions are thrown in that method, otherwise the
985 stack of context managers will fail in unexpected ways. Instead ini‐
986 tialization of real resources should happen in start_fixture.
987 At this time there is no mechanism for the individual tests to have any
989 direct awareness of the fixtures. The fixtures exist, conceptually, on
990 the server side of the API being tested.
991 Fixtures may do whatever is required by the testing environment, how‐
993 ever there are two common scenarios:
994 · Establishing (and then resetting when a test suite has finished) any
996 baseline configuration settings and persistence systems required for
997 the tests.
998 · Creating sample data for use by the tests.
1000 If a fixture raises unittest.case.SkipTest during start_fixture all the
1002 tests in the current file will be skipped. This makes it possible to
1003 skip the tests if some optional configuration (such as a particular
1004 type of database) is not available.
1005 If an exception is raised while a fixture is being used, information
1007 about the exception will be stored on the fixture so that the stop_fix‐
1008 ture method can decide if the exception should change how the fixture
1009 should clean up. The exception information can be found on exc_type,
1010 exc_value and traceback method attributes.
1011 If an exception is raised when a fixture is started (in start_fixture)
1013 the first test in the suite using the fixture will be marked with an
1014 error using the traceback from the exception and all the tests in the
1015 suite will be skipped. This ensures that fixture failure is adequately
1016 captured and reported by test runners.
1017 In some contexts (for example CI environments with a large number of
1019 tests being run in a broadly concurrent environment where output is
1020 logged to a single file) it can be important to capture and consolidate
1021 stray output that is produced during the tests and display it associ‐
1022 ated with an individual test. This can help debugging and avoids unus‐
1023 able output that is the result of multiple streams being interleaved.
1024 Inner fixtures have been added to support this. These are fixtures more
1026 in line with the tradtional unittest concept of fixtures: a class on
1027 which setUp and cleanUp is automatically called.
1028 build_tests() accepts a named parameter arguments of inner_fixtures.
1030 The value of that argument may be an ordered list of fixtures.Fixture
1031 classes that will be called when each individual test is set up.
1032 An example fixture that could be useful is the FakeLogger.
1034 NOTE:
1036 At this time inner_fixtures are not supported when using the pytest
1037 loader.
1038 Content handlers are responsible for preparing request data and evalu‐
1040 ating response data based on the content-type of the request and
1041 response. A content handler operates as follows:
1042 · Structured YAML data provided via the data attribute is converted to
1044 a string or bytes sequence and used as request body.
1045 · The response body (a string or sequence of bytes) is transformed into
1047 a content-type dependent structure and stored in an internal
1048 attribute named response_data that is:
1049 · used when evaluating the response body
1051 · used in $RESPONSE[] substitutions
1053 By default, gabbi provides content handlers for JSON. In that content
1055 handler the data test key is converted from structured YAML into a JSON
1056 string. Response bodies are converted from a JSON string into a data
1057 structure in response_data that is used when evaluating
1058 response_json_paths entries in a test or doing JSONPath-based
1059 $RESPONSE[] substitutions.
1060 Further content handlers can be added as extensions. Test authors may
1062 need these extensions for their own suites, or enterprising developers
1063 may wish to create and distribute extensions for others to use.
1064 NOTE:
1066 One extension that is likely to be useful is a content handler that
1067 turns data into url-encoded form data suitable for POST and turns an
1068 HTML response into a DOM object.
1069
1071 Content handlers are an evolution of the response handler concept in
1072 earlier versions gabbi. To preserve backwards compatibility with exist‐
1073 ing response handlers, old style response handlers are still allowed,
1074 but new handlers should implement the content handler interface
1075 (described below).
1076 Registering additional custom handlers is done by passing a subclass of
1078 ContentHandler to build_tests():
1079 driver.build_tests(test_dir, loader, host=None,
1081 intercept=simple_wsgi.SimpleWsgi,
1082 content_handlers=[MyContentHandler])
1083 If pytest is being used:
1085 driver.py_test_generator(test_dir, intercept=simple_wsgi.SimpleWsgi,
1087 content_handlers=[MyContenHandler])
1088 WARNING:
1090 When there are multiple handlers listed that accept the same con‐
1091 tent-type, the one that is earliest in the list will be used.
1092 With gabbi-run, custom handlers can be loaded via the --response-han‐
1094 dler option -- see load_response_handlers() for details.
1095 NOTE:
1097 The use of the --response-handler argument is done to preserve back‐
1098 wards compatibility and avoid excessive arguments. Both types of
1099 handler may be passed to the argument.
1100 Implementation Details
1102 Creating a content handler requires subclassing ContentHandler and
1103 implementing several methods. These methods are described below, but
1104 inspecting JSONHandler will be instructive in highlighting required
1105 arguments and techniques.
1106 To provide a response_<something> response-body evaluator a subclass
1108 must define:
1109 · test_key_suffix: This, along with the prefix response_, forms the key
1111 used in the test structure. It is a class level string.
1112 · test_key_value: The key's default value, either an empty list ([]) or
1114 empty dict ({}). It is a class level value.
1115 · action: An instance method which tests the expected values against
1117 the HTTP response - it is invoked for each entry, with the parameters
1118 depending on the default value. The arguments to action are (in
1119 order):
1120 · self: The current instance.
1122 · test: The currently active HTTPTestCase
1124 · item: The current entry if test_key_value is a list, otherwise the
1126 key half of the key/value pair at this entry.
1127 · value: None if test_key_value is a list, otherwise the value half
1129 of the key/value pair at this entry.
1130 To translate request or response bodies to or from structured data a
1132 subclass must define an accepts method. This should return True if this
1133 class is willing to translate the provided content-type. During request
1134 processing it is given the value of the content-type header that will
1135 be sent in the request. During response processing it is given the
1136 value of the content-type header of the response. This makes it possi‐
1137 ble to handle different request and response bodies in the same han‐
1138 dler, if desired. For example a handler might accept applica‐
1139 tion/x-www-form-urlencoded and text/html.
1140 If accepts is defined two additional static methods should be defined:
1142 · dumps: Turn structured Python data from the data key in a test into a
1144 string or byte stream. The optional test param allows you to access
1145 the current test case which may help with manipulations for custom
1146 content handlers, e.g. multipart/form-data needs to add a boundary to
1147 the Content-Type header in order to mark the appropriate sections of
1148 the body.
1149 · loads: Turn a string or byte stream in a response into a Python data
1151 structure. Gabbi will put this data on the response_data attribute on
1152 the test, where it can be used in the evaluations described above (in
1153 the action method) or in $RESPONSE handling. An example usage here
1154 would be to turn HTML into a DOM.
1155 Finally if a replacer class method is defined, then when a $RESPONSE
1157 substitution is encountered, replacer will be passed the response_data
1158 of the prior test and the argument within the $RESPONSE.
1159 Please see the JSONHandler source for additional detail.
1161 If there is a running web service that needs to be tested and creating
1163 a test loader with build_tests() is either inconvenient or overkill it
1164 is possible to run YAML test files directly from the command line with
1165 the console-script gabbi-run. It accepts YAML on stdin or as multiple
1166 file arguments, and generates and runs tests and outputs a summary of
1167 the results.
1168 The provided YAML may not use custom fixtures but otherwise uses the
1170 default format. host information is either expressed directly in the
1171 YAML file or provided on the command line:
1172 gabbi-run [host[:port]] < /my/test.yaml
1174 or:
1176 gabbi-run http://host:port < /my/test.yaml
1178 To test with one or more files the following command syntax may be
1180 used:
1181 gabbi-run http://host:port -- /my/test.yaml /my/other.yaml
1183 NOTE:
1185 The filename arguments must come after a -- and all other arguments
1186 (host, port, prefix, failfast) must come before the --.
1187 NOTE:
1189 If files are provided, test output will use names including the name
1190 of the file. If any single file includes an error, the name of the
1191 file will be included in a summary of failed files at the end of the
1192 test report.
1193 To facilitate using the same tests against the same application mounted
1195 in different locations in a WSGI server, a prefix may be provided as a
1196 second argument:
1197 gabbi-run host[:port] [prefix] < /my/test.yaml
1199 or in the target URL:
1201 gabbi-run http://host:port/prefix < /my/test.yaml
1203 The value of prefix will be prepended to the path portion of URLs that
1205 are not fully qualified.
1206 Anywhere host is used, if it is a raw IPV6 address it should be wrapped
1208 in [ and ].
1209 If https is used in the target, then the tests in the provided YAML
1211 will default to ssl: True.
1212 If a -x or --failfast argument is provided then gabbi-run will exit
1214 after the first test failure.
1215 Use -v or --verbose with a value of all, headers or body to turn on
1217 verbosity for all tests being run.
1218 These are informal release notes for gabbi since version 1.0.0, high‐
1220 lighting major features and changes. For more detail see the commit
1221 logs on GitHub.
1222
1224 · Allow listing of tests with no host configured. When host is an empty
1225 string, tests can be listed (for discovery), but will be skipped on
1226 run.
1227
1229 · JSON $RESPONSE substitutions in the data field may be complex types
1230 (lists and dicts), not solely strings.
1231
1233 · When the HTTP response begins with a bad status line, have BadSta‐
1234 tusLine be raised from urllib3.
1235
1237 · Allow substitutions in the key portion of request and response head‐
1238 ers, not just the value.
1239
1241 · Remove support for Python 3.3.
1242 · Make handling of fixture-level skips in pytest actually work.
1244
1246 · Add safe_yaml parameter to build_tests().
1247
1249 · use_prior_test is added to test metadata.
1250 · Extensive cleanups in regular expression handling when constructing
1252 tests from YAML.
1253
1255 jsonpath handling gets two improvements:
1256 · The value side of a response_json_paths entry can be loaded from a
1258 file using the <@file.json syntax also used in data.
1259 · The key side of a response_json_paths entry can use substitutions.
1261 This was already true for the value side.
1262
1264 Substitutions in $RESPONSE handling now preserve numeric types instead
1265 of casting to a string. This is useful when servers are expecting
1266 strong types and tests want to send response data back to the server.
1267
1269 count and delay test keys allow substitutions.
1270 gabbi.driver.build_tests() accepts a verbose parameter to set test ver‐
1271 bosity for an entire session.
1272
1274 Better failure reporting when using gabbi-run with multiple files. Test
1275 names are based on the files and a summary of failed files is provided
1276 at the end of the report.
1277
1279 Effectively capture a failure in a fixture and report the traceback.
1280 Without this some test runners swallow the error and discovering prob‐
1281 lems when developing fixtures can be quite challenging.
1282
1284 Thanks to Samuel Fekete, tests can use the $HISTORY dictionary to refer
1285 to any prior test in the same file, not just the one immediately prior,
1286 when doing substitutions.
1287
1289 Filenames used to read data into tests using the <@ syntax may now use
1290 pathnames relative to the YAML file. See data.
1291 gabbi-run gains a --verbose parameter to force all tests run in a ses‐
1293 sion to run with verbose set.
1294 When using pytest to load tests, a new mechanism is available which
1296 avoids warnings produced in when using a version of pytest greater than
1297 3.0.
1298
1300 When verbosely displaying request and response bodies that are JSON,
1301 pretty print for improved readability.
1302
1304 Allow gabbi-run to accept multiple filenames as command line arguments
1305 instead of reading tests from stdin.
1306
1308 Switch from response handlers to handlers to allow more flexible pro‐
1309 cessing of both response _and_ request bodies.
1310 Add inner fixtures for per test fixtures, useful for output capturing.
1312
1314 Allow the test_loader_name arg to gabbi.driver.build_tests() to over‐
1315 ride the prefix of the pretty printed name of generated tests.
1316
1318 String values in JSONPath matches may be wrapped in /.../` to be
1319 treated as regular expressions.
1320
1322 Better documentation of how to run gabbi in a concurrent environment.
1323 Improved handling of pytest fixtures and test counts.
1324
1326 Add url to gabbi.driver.build_tests() to use instead of host, port and
1327 prefix.
1328
1330 Add require_ssl to gabbi.driver.build_tests() to force use of SSL.
1331
1333 Add $COOKIE substitution.
1334
1336 Correctly support IPV6 hosts.
1337
1339 Add $LAST_URL substitution.
1340
1342 Introduce support for loading and running tests with pytest.
1343
1345 Use urllib3 instead of httplib2 for driving HTTP requests.
1346
1348 Add sorting and filtering to jsonpath handling.
1349
1351 Add the response_forbidden_headers to response expectations.
1352
1354 Instead of:
1355 tests:
1357 - name: a simple get
1358 url: /some/path
1359 method: get
1360 1.7.0 also makes it possible to:
1362 tests:
1364 - name: a simple get
1365 GET: /some/path
1366 Any upper case key is treated as a method.
1368
1370 Enhanced flexibility and colorization when setting tests to be verbose.
1371
1373 Adds the query_parameters key to request parameters.
1374
1376 The start of improvements and extensions to jsonpath handling. In this
1377 case the addition of the len function.
1378
1380 Vastly improved output and behavior in gabbi-run.
1381
1383 Version 1 was the first release with a commitment to a stable format.
1384 Since then new fields have been added but have not been taken away.
1385 The following people have contributed code to gabbi. Thanks to them.
1387 Thanks also to all the people who have made gabbi better by reporting
1388 issues and their successes and failures with using gabbi.
1389 · Chris Dent
1391 · FND
1393 · Mehdi Abaakouk
1395 · Tom Viner
1397 · Jason Myers
1399 · Josh Leeb-du Toit
1401 · Duc Truong
1403 · Zane Bitter
1405 · Ryan Spencer
1407 · Kim Raymoure
1409 · Travis Truman
1411 · Samuel Fekete
1413 · Michael McCune
1415 · Imran Hayder
1417 · Julien Danjou
1419 · Danek Duvall
1421 · Marc Abramowitz
1423 NOTE:
1425 This section provides a collection of questions with answers that
1426 don't otherwise fit in the rest of the documentation. If something
1427 is missing, please create an issue.
1428 As this document grows it will gain a more refined structure.
1430
1432 Is gabbi only for testing Python-based APIs?
1433 No, you can use gabbi-run to test an HTTP service built in any program‐
1434 ming language.
1435 How do I run just one test?
1437 Each YAML file contains a sequence of tests, each test within each file
1438 has a name. That name is translated to the name of the test by replac‐
1439 ing spaces with an _.
1440 When running tests that are generated dynamically, filtering based on
1442 the test name prior to the test being collected will not work in some
1443 test runners. Test runners that use a --load-list functionality can be
1444 convinced to filter after discovery.
1445 pytest does this directly with the -k keyword flag.
1447 When using testrepository with tox as used in gabbi's own tests it is
1449 possible to pass a filter in the tox command:
1450 tox -epy27 -- get_the_widget
1452 When using testtools.run and similar test runners it's a bit more com‐
1454 plicated. It is necessary to provide the full name of the test as a
1455 list to --load-list:
1456 python -m testtools.run --load-list \
1458 <(echo package.tests.test_api.yamlfile_get_the_widge.test_request)
1459 How do I run just one test, without running prior tests in a sequence?
1461 By default, when you select a single test to run, all tests prior to
1462 that one in a file will be run as well: the file is treated as as
1463 sequence of dependent tests. If you do not want this you can adjust the
1464 use_prior_test test metadata in one of three ways:
1465 · Set it in the YAML file for the one test you are concerned with.
1467 · Set the defaults for all tests in that file.
1469 · set use_prior_test to false when calling build_tests()
1471 Be aware that doing this breaks a fundamental assumption that gabbi
1473 makes about how tests work. Any substitutions will fail.
1474
1476 Can I have variables in my YAML file?
1477 Gabbi provides the $ENVIRON substitution which can operate a bit like
1478 variables that are set elsewhere and then used in the tests defined by
1479 the YAML.
1480 If you find it necessary to have variables within a single YAML file
1482 you take advantage of YAML alias nodes list this:
1483 vars:
1485 - &uuid_1 5613AABF-BAED-4BBA-887A-252B2D3543F8
1486 tests:
1488 - name: send a uuid to a post
1489 POST: /resource
1490 request_headers:
1491 content-type: application/json
1492 data:
1493 uuid: *uuid_1
1494 You can alias all sorts of nodes, not just single items. Be aware that
1496 the replacement of an alias node happens while the YAML is being
1497 loaded, before gabbi does any processing.
1498 How many tests should be put in one YAML file?
1500 For the sake of readability it is best to keep each YAML file rela‐
1501 tively short. Since each YAML file represents a sequence of requests,
1502 it usually makes sense to create a new file when a test is not depen‐
1503 dent on any before it.
1504 It's tempting to put all the tests for any resource or URL in the same
1506 file, but this eventually leads to files that are too long and are thus
1507 difficult to read.
1508
1510 A single HTTP request represented as a subclass of testtools.TestCase
1511 The test case encapsulates the request headers and body and expected
1513 response headers and body. When the test is run an HTTP request is made
1514 using urllib3. Assertions are made against the response.
1515 class gabbi.case.HTTPTestCase(*args, **kwargs)
1517 Bases: testtools.testcase.TestCase
1518 Encapsulate a single HTTP request as a TestCase.
1520 If the test is a member of a sequence of requests, ensure that
1522 prior tests are run.
1523 To keep the test harness happy we need to make sure the setUp
1525 and tearDown are only run once.
1526 assert_in_or_print_output(expected, iterable)
1528 Assert the iterable contains expected or print some out‐
1529 put.
1530 If the output is long, it is limited by either
1532 GABBI_MAX_CHARS_OUTPUT in the environment or the
1533 MAX_CHARS_OUTPUT constant.
1534 base_test = {'data': '', 'desc': '', 'method': 'GET', 'name':
1536 '', 'poll': {}, 'query_parameters': {}, 'redirects': False,
1537 'request_headers': {}, 'skip': '', 'ssl': False, 'status':
1538 '200', 'url': '', 'use_prior_test': True, 'verbose': False,
1539 'xfail': False}
1540 get_content_handler(content_type)
1542 Determine the content handler for this media type.
1543 load_data_file(filename)
1545 Read a file from the current test directory.
1546 replace_template(message, escape_regex=False)
1548 Replace magic strings in message.
1549 run(result=None)
1551 Store the current result handler on this test.
1552 setUp()
1554 Hook method for setting up the test fixture before exer‐
1555 cising it.
1556 tearDown()
1558 Hook method for deconstructing the test fixture after
1559 testing it.
1560 test_request()
1562 Run this request if it has not yet run.
1563 If there is a prior test in the sequence, run it first.
1565 gabbi.case.potentialFailure(func)
1567 Decorate a test method that is expected to fail if 'xfail' is
1568 true.
1569
1571 Generate HTTP tests from YAML files
1572 Each HTTP request is its own TestCase and can be requested to be run in
1574 isolation from other tests. If it is a member of a sequence of
1575 requests, prior requests will be run.
1576 A sequence is represented by an ordered list in a single YAML file.
1578 Each sequence becomes a TestSuite.
1580 An entire directory of YAML files is a TestSuite of TestSuites.
1582 gabbi.driver.build_tests(path, loader, host=None, port=8001, inter‐
1584 cept=None, test_loader_name=None, fixture_module=None, response_han‐
1585 dlers=None, content_handlers=None, prefix='', require_ssl=False,
1586 url=None, inner_fixtures=None, verbose=False, use_prior_test=True,
1587 safe_yaml=True)
1588 Read YAML files from a directory to create tests.
1589 Each YAML file represents a list of HTTP requests.
1591 Parameters
1593 · path -- The directory where yaml files are located.
1595 · loader -- The TestLoader.
1597 · host -- The host to test against. Do not use with
1599 intercept.
1600 · port -- The port to test against. Used with host.
1602 · intercept -- WSGI app factory for wsgi-intercept.
1604 · test_loader_name -- Base name for test classes. Use
1606 this to align the naming of the tests with other tests
1607 in a system.
1608 · fixture_module -- Python module containing fixture
1610 classes.
1611 · response_handers -- ResponseHandler classes.
1613 · content_handlers (List of ContentHandler classes.) --
1615 ContentHandler classes.
1616 · prefix -- A URL prefix for all URLs that are not fully
1618 qualified.
1619 · url -- A full URL to test against. Replaces host, port
1621 and prefix.
1622 · require_ssl -- If True, make all tests default to using
1624 SSL.
1625 · inner_fixtures (List of fixtures.Fixture classes.) -- A
1627 list of Fixtures to use with each individual test
1628 request.
1629 · verbose -- If True or 'all', make tests verbose by
1631 default 'headers' and 'body' are also accepted.
1632 · use_prior_test -- If True, uses prior test to create
1634 ordered sequence of tests
1635 · safe_yaml -- If True, recognizes only standard YAML
1637 tags and not Python object
1638 Return type
1640 TestSuite containing multiple TestSuites (one for each
1641 YAML file)
1642 gabbi.driver.py_test_generator(test_dir, host=None, port=8001, inter‐
1644 cept=None, prefix=None, test_loader_name=None, fixture_module=None,
1645 response_handlers=None, content_handlers=None, require_ssl=False,
1646 url=None, metafunc=None, use_prior_test=True, inner_fixtures=None,
1647 safe_yaml=True)
1648 Generate tests cases for py.test
1649 This uses build_tests to create TestCases and then yields them
1651 in a way that pytest can handle.
1652 gabbi.driver.test_pytest(test, result)
1654 gabbi.driver.test_suite_from_yaml(loader, test_base_name, test_yaml,
1656 test_directory, host, port, fixture_module, intercept, prefix='')
1657 Legacy wrapper retained for backwards compatibility.
1658
1660 The code that creates a suite of tests.
1661 The key piece of code is test_suite_from_dict(). It produces a
1663 gabbi.suite.GabbiSuite containing one or more gabbi.case.HTTPTestCase.
1664 class gabbi.suitemaker.TestBuilder
1666 Bases: type
1667 Metaclass to munge a dynamically created test.
1669 required_attributes = {'has_run': False}
1671 class gabbi.suitemaker.TestMaker(test_base_name, test_defaults,
1673 test_directory, fixture_classes, loader, host, port, intercept, prefix,
1674 response_handlers, content_handlers, test_loader_name=None, inner_fix‐
1675 tures=None)
1676 Bases: object
1677 A class for encapsulating test invariants.
1679 All of the tests in a single gabbi file have invariants which
1681 are provided when creating each HTTPTestCase. It is not useful
1682 to pass these around when making each test case. So they are
1683 wrapped in this class which then has make_one_test called multi‐
1684 ple times to generate all the tests in the suite.
1685 make_one_test(test_dict, prior_test)
1687 Create one single HTTPTestCase.
1688 The returned HTTPTestCase is added to the TestSuite cur‐
1690 rently being built (one per YAML file).
1691 gabbi.suitemaker.test_suite_from_dict(loader, test_base_name,
1693 suite_dict, test_directory, host, port, fixture_module, intercept, pre‐
1694 fix='', handlers=None, test_loader_name=None, inner_fixtures=None)
1695 Generate a GabbiSuite from a dict represent a list of tests.
1696 The dict takes the form:
1698 Parameters
1700 · fixtures -- An optional list of fixture classes that
1702 this suite can use.
1703 · defaults -- An optional dictionary of default values to
1705 be used in each test.
1706 · tests -- A list of individual tests, themselves each
1708 being a dictionary. See gabbi.case.BASE_TEST.
1709 gabbi.suitemaker.test_update(orig_dict, new_dict)
1711 Modify test in place to update with new data.
1712
1714 Manage fixtures for gabbi at the test suite level.
1715 class gabbi.fixture.GabbiFixture
1717 Bases: object
1718 A context manager that operates as a fixture.
1720 Subclasses must implement start_fixture and stop_fixture, each
1722 of which contain the logic for stopping and starting whatever
1723 the fixture is. What a fixture is is left as an exercise for the
1724 implementor.
1725 These context managers will be nested so any actual work needs
1727 to happen in start_fixture and stop_fixture and not in __init__.
1728 Otherwise exception handling will not work properly.
1729 start_fixture()
1731 Implement the actual workings of starting the fixture
1732 here.
1733 stop_fixture()
1735 Implement the actual workings of stopping the fixture
1736 here.
1737 exception gabbi.fixture.GabbiFixtureError
1739 Bases: Exception
1740 Generic exception for GabbiFixture.
1742 class gabbi.fixture.SkipAllFixture
1744 Bases: gabbi.fixture.GabbiFixture
1745 A fixture that skips all the tests in the current suite.
1747 start_fixture()
1749 Implement the actual workings of starting the fixture
1750 here.
1751 gabbi.fixture.nest(fixtures)
1753 Nest a series of fixtures.
1754 This is duplicated from nested in the stdlib, which has been
1756 deprecated because of issues with how exceptions are difficult
1757 to handle during __init__. Gabbi needs to nest an unknown number
1758 of fixtures dynamically, so the with syntax that replaces nested
1759 will not work.
1760
1762 Package for response and content handlers that process the body of a
1763 response in various ways.
1764 handlers.base Module
1766 Base classes for response and content handlers.
1767 class gabbi.handlers.base.ContentHandler
1769 Bases: gabbi.handlers.base.ResponseHandler
1770 A subclass of ResponseHandlers that adds content handling.
1772 static accepts(content_type)
1774 Return True if this handler can handler this type.
1775 static dumps(data, pretty=False, test=None)
1777 Return structured data as a string.
1778 If pretty is true, prettify.
1780 static loads(data)
1782 Create structured (Python) data from a stream.
1783 classmethod replacer(response_data, path)
1785 Return the string that is replacing RESPONSE.
1786 class gabbi.handlers.base.ResponseHandler
1788 Bases: object
1789 Add functionality for making assertions about an HTTP response.
1791 A subclass may implement two methods: action and preprocess.
1793 preprocess takes one argument, the TestCase. It is called
1795 exactly once for each test before looping across the assertions.
1796 It is used, rarely, to copy the test.output into a useful form
1797 (such as a parsed DOM).
1798 action takes two or three arguments. If test_key_value is a list
1800 action is called with the test case and a single list item. If
1801 test_key_value is a dict then action is called with the test
1802 case and a key and value pair.
1803 action(test, item, value=None)
1805 Test an individual entry for this response handler.
1806 If the entry is a key value pair the key is in item and
1808 the value in value. Otherwise the entry is considered a
1809 single item from a list.
1810 preprocess(test)
1812 Do any pre-single-test preprocessing.
1813 test_key_suffix = ''
1815 test_key_value = []
1817 handlers.core Module
1819 Core response handlers.
1820 class gabbi.handlers.core.ForbiddenHeadersResponseHandler
1822 Bases: gabbi.handlers.base.ResponseHandler
1823 Test that listed headers are not in the response.
1825 action(test, forbidden, value=None)
1827 Test an individual entry for this response handler.
1828 If the entry is a key value pair the key is in item and
1830 the value in value. Otherwise the entry is considered a
1831 single item from a list.
1832 test_key_suffix = 'forbidden_headers'
1834 test_key_value = []
1836 class gabbi.handlers.core.HeadersResponseHandler
1838 Bases: gabbi.handlers.base.ResponseHandler
1839 Compare expected headers with actual headers.
1841 If a header value is wrapped in / it is treated as a raw regular
1843 expression.
1844 Headers values are always treated as strings.
1846 action(test, header, value=None)
1848 Test an individual entry for this response handler.
1849 If the entry is a key value pair the key is in item and
1851 the value in value. Otherwise the entry is considered a
1852 single item from a list.
1853 test_key_suffix = 'headers'
1855 test_key_value = {}
1857 class gabbi.handlers.core.StringResponseHandler
1859 Bases: gabbi.handlers.base.ResponseHandler
1860 Test for matching strings in the the response body.
1862 action(test, expected, value=None)
1864 Test an individual entry for this response handler.
1865 If the entry is a key value pair the key is in item and
1867 the value in value. Otherwise the entry is considered a
1868 single item from a list.
1869 test_key_suffix = 'strings'
1871 test_key_value = []
1873 handlers.jsonhandler Module
1875 JSON-related content handling.
1876 class gabbi.handlers.jsonhandler.JSONHandler
1878 Bases: gabbi.handlers.base.ContentHandler
1879 A ContentHandler for JSON
1881 · Structured test data is turned into JSON when request con‐
1883 tent-type is JSON.
1884 · Response bodies that are JSON strings are made into Python
1886 data on the test response_data attribute when the response
1887 content-type is JSON.
1888 · A response_json_paths response handler is added.
1890 · JSONPaths in $RESPONSE substitutions are supported.
1892 static accepts(content_type)
1894 Return True if this handler can handler this type.
1895 action(test, path, value=None)
1897 Test json_paths against json data.
1898 static dumps(data, pretty=False, test=None)
1900 Return structured data as a string.
1901 If pretty is true, prettify.
1903 static extract_json_path_value(data, path)
1905 Extract the value at JSON Path path from the data.
1906 The input data is a Python datastructure, not a JSON
1908 string.
1909 static loads(data)
1911 Create structured (Python) data from a stream.
1912 classmethod replacer(response_data, match)
1914 Return the string that is replacing RESPONSE.
1915 test_key_suffix = 'json_paths'
1917 test_key_value = {}
1919
1921 A TestSuite for containing gabbi tests.
1922 This suite has two features: the contained tests are ordered and there
1924 are suite-level fixtures that operate as context managers.
1925 class gabbi.suite.GabbiSuite(tests=())
1927 Bases: unittest.suite.TestSuite
1928 A TestSuite with fixtures.
1930 The suite wraps the tests with a set of nested context managers
1932 that operate as fixtures.
1933 If a fixture raises unittest.case.SkipTest during setup, all the
1935 tests in this suite will be skipped.
1936 run(result, debug=False)
1938 Override TestSuite run to start suite-level fixtures.
1939 To avoid exception confusion, use a null Fixture when
1941 there are no fixtures.
1942 start(result, tests)
1944 Start fixtures when using pytest.
1945 stop() Stop fixtures when using pytest.
1947 gabbi.suite.noop(*args)
1949 A noop method used to disable collected tests.
1950
1952 Implementation of a command-line runner for gabbi files (AKA suites).
1953 gabbi.runner.extract_file_paths(argv)
1955 Extract command-line arguments following the -- end-of-options
1956 delimiter, if any.
1957 gabbi.runner.initialize_handlers(response_handlers)
1959 gabbi.runner.load_response_handlers(import_path)
1961 Load and return custom response handlers from the given Python
1962 package or module.
1963 The import path references either a specific response handler
1965 class ("package.module:class") or a module that contains one or
1966 more response handler classes ("package.module").
1967 For the latter, the module is expected to contain a
1969 gabbi_response_handlers object, which is either a list of
1970 response handler classes or a function returning such a list.
1971 gabbi.runner.run()
1973 Run simple tests from STDIN.
1974 This command provides a way to run a set of tests encoded in
1976 YAML that is provided on STDIN. No fixtures are supported, so
1977 this is primarily designed for use with real running services.
1978 Host and port information may be provided in three different
1980 ways:
1981 · In the URL value of the tests.
1983 · In a host or host:port argument on the command line.
1985 · In a URL on the command line.
1987 An example run might looks like this:
1989 gabbi-run example.com:9999 < mytest.yaml
1991 or:
1993 gabbi-run http://example.com:999 < mytest.yaml
1995 It is also possible to provide a URL prefix which can be useful
1997 if the target application might be mounted in different loca‐
1998 tions. An example:
1999 gabbi-run example.com:9999 /mountpoint < mytest.yaml
2001 or:
2003 gabbi-run http://example.com:9999/mountpoint < mytest.yaml
2005 Use -x or --failfast to abort after the first error or failure:
2007 gabbi-run -x example.com:9999 /mountpoint < mytest.yaml
2009 Use -v or --verbose with a value of all, headers or body to turn
2011 on verbosity for all tests being run.
2012 Multiple files may be named as arguments, separated from other
2014 arguments by a --. Each file will be run as a separate test
2015 suite:
2016 gabbi-run http://example.com -- /path/to/x.yaml /path/to/y.yaml
2018 Output is formatted as unittest summary information.
2020 gabbi.runner.run_suite(handle, handler_objects, host, port, prefix,
2022 force_ssl=False, failfast=False, data_dir='.', verbosity=False,
2023 name='input', safe_yaml=True)
2024 Run the tests from the YAML in handle.
2025
2027 TestRunner and TestResult for gabbi-run.
2028 class gabbi.reporter.ConciseTestResult(stream, descriptions, verbosity)
2030 Bases: unittest.runner.TextTestResult
2031 A TextTestResult with simple but useful output.
2033 If the output is a tty or GABBI_FORCE_COLOR is set in the envi‐
2035 ronment, output will be colorized.
2036 addError(test, err)
2038 Called when an error has occurred. 'err' is a tuple of
2039 values as returned by sys.exc_info().
2040 addExpectedFailure(test, err)
2042 Called when an expected failure/error occurred.
2043 addFailure(test, err)
2045 Called when an error has occurred. 'err' is a tuple of
2046 values as returned by sys.exc_info().
2047 addSkip(test, reason)
2049 Called when a test is skipped.
2050 addSuccess(test)
2052 Called when a test has completed successfully
2053 addUnexpectedSuccess(test)
2055 Called when a test was expected to fail, but succeed.
2056 getDescription(test)
2058 printErrorList(flavor, errors)
2060 startTest(test)
2062 Called when the given test is about to be run
2063 class gabbi.reporter.ConciseTestRunner(stream=None, descriptions=True,
2065 verbosity=1, failfast=False, buffer=False, resultclass=None, warn‐
2066 ings=None, *, tb_locals=False)
2067 Bases: unittest.runner.TextTestRunner
2068 A TextTestRunner that uses ConciseTestResult for reporting
2070 results.
2071 resultclass
2073 alias of ConciseTestResult
2074 class gabbi.reporter.PyTestResult(stream=None, descriptions=None, ver‐
2076 bosity=None)
2077 Bases: unittest.result.TestResult
2078 Wrap a test result to allow it to work with pytest.
2080 The main behaviors here are:
2082 · to turn what had been exceptions back into exceptions
2084 · use pytest's skip and xfail methods
2086 addError(test, err)
2088 Called when an error has occurred. 'err' is a tuple of
2089 values as returned by sys.exc_info().
2090 addExpectedFailure(test, err)
2092 Called when an expected failure/error occurred.
2093 addFailure(test, err)
2095 Called when an error has occurred. 'err' is a tuple of
2096 values as returned by sys.exc_info().
2097 addSkip(test, reason)
2099 Called when a test is skipped.
2100
2102 Utility functions grab bag.
2103 gabbi.utils.create_url(base_url, host, port=None, prefix='', ssl=False)
2105 Given pieces of a path-based url, return a fully qualified url.
2106 gabbi.utils.decode_response_content(header_dict, content)
2108 Decode content to a proper string.
2109 gabbi.utils.extract_content_type(header_dict, default='applica‐
2111 tion/binary')
2112 Extract parsed content-type from headers.
2113 gabbi.utils.get_colorizer(stream)
2115 Return a function to colorize a string.
2116 Only if stream is a tty .
2118 gabbi.utils.host_info_from_target(target, prefix=None)
2120 Turn url or host:port and target into test destination.
2121 gabbi.utils.load_yaml(handle=None, yaml_file=None, safe=True)
2123 Read and parse any YAML file or filehandle.
2124 Let exceptions flow where they may.
2126 If no file or handle is provided, read from STDIN.
2128 gabbi.utils.not_binary(content_type)
2130 Decide if something is content we'd like to treat as a string.
2131 gabbi.utils.parse_content_type(content_type, default_charset='utf-8')
2133 Parse content type value for media type and charset.
2134
2136 Gabbi specific exceptions.
2137 exception gabbi.exception.GabbiFormatError
2139 Bases: ValueError
2140 An exception to encapsulate poorly formed test data.
2142 exception gabbi.exception.GabbiSyntaxWarning
2144 Bases: SyntaxWarning
2145 A warning about syntax that is not desirable.
2147
2149 class gabbi.httpclient.Http(num_pools=10, headers=None, **connec‐
2150 tion_pool_kw)
2151 Bases: urllib3.poolmanager.PoolManager
2152 A subclass of the urllib3.PoolManager to munge the data.
2154 This transforms the response to look more like what httplib2
2156 provided when it was used as the HTTP client.
2157 request(absolute_uri, method, body, headers, redirect)
2159 Make a request using urlopen() with the appropriate
2160 encoding of fields based on the method used.
2161 This is a convenience method that requires the least
2163 amount of manual effort. It can be used in most situa‐
2164 tions, while still having the option to drop down to more
2165 specific methods when necessary, such as
2166 request_encode_url(), request_encode_body(), or even the
2167 lowest level urlopen().
2168 class gabbi.httpclient.VerboseHttp(**kwargs)
2170 Bases: gabbi.httpclient.Http
2171 A subclass of Http that verbosely reports on activity.
2173 If the output is a tty or GABBI_FORCE_COLOR is set in the envi‐
2175 ronment, then output will be colorized according to COLORMAP.
2176 Output can include request and response headers, request and
2178 response body content (if of a printable content type), or both.
2179 The color of the output has reasonable defaults. These may be
2181 overridden by setting the following environment variables
2182 · GABBI_CAPTION_COLOR
2184 · GABBI_HEADER_COLOR
2186 · GABBI_REQUEST_COLOR
2188 · GABBI_STATUS_COLOR
2190 to any of: BLACK RED GREEN YELLOW BLUE MAGENTA CYAN WHITE
2192 COLORMAP = {'caption': 'BLUE', 'header': 'YELLOW', 'request':
2194 'CYAN', 'status': 'CYAN'}
2195 HEADER_BLACKLIST = ['status', 'reason']
2197 REQUEST_PREFIX = '>'
2199 RESPONSE_PREFIX = '<'
2201 request(absolute_uri, method, body, headers, redirect)
2203 Display request parameters before requesting.
2204 gabbi.httpclient.get_http(verbose=False, caption='')
2206 Return an Http class for making requests.
2207
2209 Keep one single global jsonpath parser.
2210 gabbi.json_parser.parse(path)
2212 Parse a JSONPath expression use the global parser.
2213 Gabbi is a tool for running HTTP tests where requests and responses are
2215 expressed as declarations in a collection of YAML files. The simplest
2216 test looks like this:
2217 tests:
2219 - name: A test
2220 GET: /api/resources/id
2221 See the rest of these docs for more details on the many features and
2223 formats for setting request headers and bodies and evaluating
2224 responses.
2225 The name is derived from "gabby": excessively talkative. In a test
2227 environment having visibility of what a test is actually doing is a
2228 good thing. This is especially true when the goal of a test is to test
2229 the HTTP, not the testing infrastructure. Gabbi tries to put the HTTP
2230 interaction in the foreground of testing.
2231 Tests can be run using unittest style test runners or py.test or from
2233 the command line with a gabbi-run script.
2234 If you want to get straight to creating tests look at example, the test
2236 files in the source distribution and format. A gabbi-demo repository
2237 provides a tutorial of using gabbi to build an API, via the commit his‐
2238 tory of the repo.
2239
2241 Gabbi works to bridge the gap between human readable YAML files (see
2242 format for details) that represent HTTP requests and expected responses
2243 and the rather complex world of automated testing.
2244 Each YAML file represents an ordered list of HTTP requests along with
2246 the expected responses. This allows a single file to represent a
2247 process in the API being tested. For example:
2248 · Create a resource.
2250 · Retrieve a resource.
2252 · Delete a resource.
2254 · Retrieve a resource again to confirm it is gone.
2256 At the same time it is still possible to ask gabbi to run just one
2258 request. If it is in a sequence of tests, those tests prior to it in
2259 the YAML file will be run (in order). In any single process any test
2260 will only be run once. Concurrency is handled such that one file runs
2261 in one process.
2262 These features mean that it is possible to create tests that are useful
2264 for both humans (as tools for learning, improving and developing APIs)
2265 and automated CI systems.
2266 Significant flexibility and power is available in the format to make it
2268 relatively straightforward to test existing complex APIs. This
2269 extended functionality includes the use of JSONPath to query response
2270 bodies and templating of test data to allow access to the prior HTTP
2271 response in the current request. For APIs which do not use JSON addi‐
2272 tional handlers can be created.
2273 Care should be taken when using this functionality when you are creat‐
2275 ing a new API. If your API is so complex that it needs complex test
2276 files then you may wish to take that as a sign that your API itself too
2277 complex. One goal of gabbi is to encourage transparent and comprehensi‐
2278 ble APIs.
2279 Though gabbi is written in Python and under the covers uses unittest
2281 data structures and processes, there is no requirement that the host be
2282 a Python-based service. Anything talking HTTP can be tested. A runner
2283 makes it possible to simply create YAML files and point them at a run‐
2284 ning server.
2285
2287 Chris Dent
2288 Jul 26, 2019 GABBI(1)