1GABBI(1) Gabbi GABBI(1)
2
6 gabbi - Gabbi Documentation
7 Gabbi tests are expressed in YAML as a series of HTTP requests with
9 their expected response:
10 tests:
12 - name: retrieve root
13 GET: /
14 status: 200
15 This will trigger a GET request to / on the configured host. The test
17 will pass if the response's status code is 200.
18
20 The top-level tests category contains an ordered sequence of test dec‐
21 larations, each describing the expected response to a given request:
22 Metadata
24 ┌───────────────┬─────────────────────┬───────────────────┐
25 │Key │ Description │ Notes │
26 ├───────────────┼─────────────────────┼───────────────────┤
27 │name │ The test's name. │ required │
28 │ │ Must be unique │ │
29 │ │ within a file. │ │
30 ├───────────────┼─────────────────────┼───────────────────┤
31 │desc │ An arbitrary string │ │
32 │ │ describing the │ │
33 │ │ test. │ │
34 ├───────────────┼─────────────────────┼───────────────────┤
35 │verbose │ If True or all │ defaults to False │
36 │ │ (synonymous), │ │
37 │ │ prints a represen‐ │ │
38 │ │ tation of the cur‐ │ │
39 │ │ rent request and │ │
40 │ │ response to stdout, │ │
41 │ │ including both │ │
42 │ │ headers and body. │ │
43 │ │ If set to headers │ │
44 │ │ or body, only the │ │
45 │ │ corresponding part │ │
46 │ │ of the request and │ │
47 │ │ response will be │ │
48 │ │ printed. If the │ │
49 │ │ output is a TTY, │ │
50 │ │ colors will be │ │
51 │ │ used. If the body │ │
52 │ │ content-type is │ │
53 │ │ JSON it will be │ │
54 │ │ formatted for │ │
55 │ │ improved readabil‐ │ │
56 │ │ ity. See Verbose‐ │ │
57 │ │ Http for details. │ │
58 ├───────────────┼─────────────────────┼───────────────────┤
59 │skip │ A string message │ defaults to False │
60 │ │ which if set will │ │
61 │ │ cause the test to │ │
62 │ │ be skipped with the │ │
63 │ │ provided message. │ │
64 └───────────────┴─────────────────────┴───────────────────┘
65 │xfail │ Determines whether │ defaults to False │
68 │ │ to expect this test │ │
69 │ │ to fail. Note that │ │
70 │ │ the test will be │ │
71 │ │ run anyway. │ │
72 ├───────────────┼─────────────────────┼───────────────────┤
73 │use_prior_test │ Determines if this │ defaults to True │
74 │ │ test will be run in │ │
75 │ │ sequence (after) │ │
76 │ │ the test prior to │ │
77 │ │ it in the list of │ │
78 │ │ tests within a │ │
79 │ │ file. To be con‐ │ │
80 │ │ crete, when this is │ │
81 │ │ True the test is │ │
82 │ │ dependent on the │ │
83 │ │ prior test and if │ │
84 │ │ that prior has not │ │
85 │ │ yet run, it wil be │ │
86 │ │ run, even if only │ │
87 │ │ the current test │ │
88 │ │ has been selected. │ │
89 │ │ Set this to False │ │
90 │ │ to allow selecting │ │
91 │ │ a test without │ │
92 │ │ dependencies. │ │
93 ├───────────────┼─────────────────────┼───────────────────┤
94 │cert_validate │ States whether the │ defaults to True │
95 │ │ underlying HTTP │ │
96 │ │ client should │ │
97 │ │ attempt to validate │ │
98 │ │ SSL certificates. │ │
99 │ │ In many test envi‐ │ │
100 │ │ ronment certifi‐ │ │
101 │ │ cates will be │ │
102 │ │ self-signed so │ │
103 │ │ changing this may │ │
104 │ │ be requried. It can │ │
105 │ │ also be changed │ │
106 │ │ when loader or │ │
107 │ │ using gabbi-run. │ │
108 └───────────────┴─────────────────────┴───────────────────┘
109 NOTE:
111 When tests are generated dynamically, the TestCase name will include
112 the respective test's name, lowercased with spaces transformed to _.
113 In at least some test runners this will allow you to select and fil‐
114 ter on test name.
115 Request Parameters
117 ┌────────────────────┬───────────────────────┬─────────────────────┐
118 │Key │ Description │ Notes │
119 └────────────────────┴───────────────────────┴─────────────────────┘
120 │any uppercase │ Any such key is │ │
134 │string │ considered an HTTP │ │
135 │ │ method, with the │ │
136 │ │ corresponding value │ │
137 │ │ expressing the URL. │ │
138 │ │ │ │
139 │ │ This is a shortcut │ │
140 │ │ combining method │ │
141 │ │ and url into a sin‐ │ │
142 │ │ gle statement: │ │
143 │ │ │ │
144 │ │ GET: /index │ │
145 │ │ │ │
146 │ │ corresponds │ │
147 │ │ to: │ │
148 │ │ │ │
149 │ │ method: GET │ │
150 │ │ url: /index │ │
151 ├────────────────────┼───────────────────────┼─────────────────────┤
152 │method │ The HTTP request │ defaults to GET │
153 │ │ method. │ │
154 ├────────────────────┼───────────────────────┼─────────────────────┤
155 │url │ The URL to request. │ Either this or the │
156 │ │ This can either be a │ shortcut above is │
157 │ │ full path (e.g. │ required │
158 │ │ "/index") or a fully │ │
159 │ │ qualified URL (i.e. │ │
160 │ │ including host and │ │
161 │ │ scheme, e.g. "‐ │ │
162 │ │ http://exam‐ │ │
163 │ │ ple.org/index") — see │ │
164 │ │ host for details. │ │
165 ├────────────────────┼───────────────────────┼─────────────────────┤
166 │request_headers │ A dictionary of │ │
167 │ │ key-value pairs rep‐ │ │
168 │ │ resenting request │ │
169 │ │ header names and val‐ │ │
170 │ │ ues. These will be │ │
171 │ │ added to the con‐ │ │
172 │ │ structed request. │ │
173 ├────────────────────┼───────────────────────┼─────────────────────┤
174 │query_parameters │ A dictionary of query │ │
175 │ │ parameters that will │ │
176 │ │ be added to the url │ │
177 │ │ as query string. If │ │
178 │ │ that URL already con‐ │ │
179 │ │ tains a set of query │ │
180 │ │ parameters, those wil │ │
181 │ │ be extended. See │ │
182 │ │ example for a demon‐ │ │
183 │ │ stration of how the │ │
184 │ │ data is structured. │ │
185 ├────────────────────┼───────────────────────┼─────────────────────┤
186 │data │ A representation to │ │
187 │ │ pass as the body of a │ │
188 │ │ request. Note that │ │
189 │ │ content-type in │ │
190 │ │ request_headers │ │
191 │ │ should also be set — │ │
192 │ │ see Data for details. │ │
193 ├────────────────────┼───────────────────────┼─────────────────────┤
194 │redirects │ If True, redirects │ defaults to False │
195 │ │ will automatically be │ │
196 │ │ followed. │ │
197 └────────────────────┴───────────────────────┴─────────────────────┘
198 │ssl │ Determines whether │ defaults to False │
200 │ │ the request uses SSL │ │
201 │ │ (i.e. HTTPS). Note │ │
202 │ │ that the url's scheme │ │
203 │ │ takes precedence if │ │
204 │ │ present — see host │ │
205 │ │ for details. │ │
206 └────────────────────┴───────────────────────┴─────────────────────┘
207 Response Expectations
209 ┌────────────────────┬─────────────────────┬─────────────────┐
210 │Key │ Description │ Notes │
211 ├────────────────────┼─────────────────────┼─────────────────┤
212 │status │ The expected │ defaults to 200 │
213 │ │ response status │ │
214 │ │ code. Multiple │ │
215 │ │ acceptable response │ │
216 │ │ codes may be pro‐ │ │
217 │ │ vided, separated by │ │
218 │ │ || (e.g. 302 || 301 │ │
219 │ │ — note, however, │ │
220 │ │ that this indicates │ │
221 │ │ ambiguity, which is │ │
222 │ │ generally undesir‐ │ │
223 │ │ able). │ │
224 ├────────────────────┼─────────────────────┼─────────────────┤
225 │response_headers │ A dictionary of │ │
226 │ │ key-value pairs │ │
227 │ │ representing │ │
228 │ │ expected response │ │
229 │ │ header names and │ │
230 │ │ values. If a │ │
231 │ │ header's value is │ │
232 │ │ wrapped in /.../, │ │
233 │ │ it will be treated │ │
234 │ │ as a regular │ │
235 │ │ expression to │ │
236 │ │ search for in the │ │
237 │ │ response header. │ │
238 ├────────────────────┼─────────────────────┼─────────────────┤
239 │response_forbid‐ │ A list of headers │ │
240 │den_headers │ which must not be │ │
241 │ │ present. │ │
242 ├────────────────────┼─────────────────────┼─────────────────┤
243 │response_strings │ A list of string │ │
244 │ │ fragments expected │ │
245 │ │ to be present in │ │
246 │ │ the response body. │ │
247 └────────────────────┴─────────────────────┴─────────────────┘
248 │response_json_paths │ A dictionary of │ │
266 │ │ JSONPath rules │ │
267 │ │ paired with │ │
268 │ │ expected matches. │ │
269 │ │ Using this rule │ │
270 │ │ requires that the │ │
271 │ │ content being sent │ │
272 │ │ from the server is │ │
273 │ │ JSON (i.e. a con‐ │ │
274 │ │ tent type of appli‐ │ │
275 │ │ cation/json or con‐ │ │
276 │ │ taining +json) │ │
277 │ │ │ │
278 │ │ If the value is │ │
279 │ │ wrapped in /.../ │ │
280 │ │ the result of the │ │
281 │ │ JSONPath query will │ │
282 │ │ be searched for the │ │
283 │ │ value as a regular │ │
284 │ │ expression. │ │
285 ├────────────────────┼─────────────────────┼─────────────────┤
286 │poll │ A dictionary of two │ │
287 │ │ keys: │ │
288 │ │ │ │
289 │ │ · count: An │ │
290 │ │ integer │ │
291 │ │ stating │ │
292 │ │ the number │ │
293 │ │ of times │ │
294 │ │ to attempt │ │
295 │ │ this test │ │
296 │ │ before │ │
297 │ │ giving up. │ │
298 │ │ │ │
299 │ │ · delay: A │ │
300 │ │ floating │ │
301 │ │ point num‐ │ │
302 │ │ ber of │ │
303 │ │ seconds to │ │
304 │ │ delay │ │
305 │ │ between │ │
306 │ │ attempts. │ │
307 │ │ │ │
308 │ │ This makes │ │
309 │ │ it possible │ │
310 │ │ to poll for │ │
311 │ │ a resource │ │
312 │ │ created via │ │
313 │ │ an asynchro‐ │ │
314 │ │ nous │ │
315 │ │ request. Use │ │
316 │ │ with cau‐ │ │
317 │ │ tion. │ │
318 └────────────────────┴─────────────────────┴─────────────────┘
319 Note that many of these items allow substitutions.
321 Default values for a file's tests may be provided via the top-level
323 defaults category. These take precedence over the global defaults
324 (explained below).
325 For examples see the gabbi tests, example and the gabbi-demo tutorial.
327
329 The top-level fixtures category contains a sequence of named fixtures.
330
332 response_* keys are examples of Response Handlers. Custom handlers may
333 be created by test authors for specific use cases. See handlers for
334 more information.
335
337 There are a number of magical variables that can be used to make refer‐
338 ence to the state of a current test, the one just prior or any test
339 prior to the current one. The variables are replaced with real values
340 during test processing.
341 Global
343 · $ENVIRON['<environment variable>']: The name of an environment vari‐
344 able. Its value will replace the magical variable. If the string
345 value of the environment variable is "True" or "False" then the
346 resulting value will be the corresponding boolean, not a string.
347 Current Test
349 · $SCHEME: The current scheme/protocol (usually http or https).
350 · $NETLOC: The host and potentially port of the request.
352 Immediately Prior Test
354 · $COOKIE: All the cookies set by any Set-Cookie headers in the prior
355 response, including only the cookie key and value pairs and no meta‐
356 data (e.g. expires or domain).
357 · $URL: The URL defined in the prior request, after substitutions have
359 been made. For backwards compatibility with earlier releases
360 $LAST_URL may also be used, but if $HISTORY (see below) is being
361 used, $URL must be used.
362 · $LOCATION: The location header returned in the prior response.
364 · $HEADERS['<header>']: The value of any header from the prior
366 response.
367 · $RESPONSE['<json path>']: A JSONPath query into the prior response.
369 See jsonpath for more on formatting.
370 Any Previous Test
372 · $HISTORY['<test name>'].<magical variable expression>: Any variable
373 which refers to a prior test may be used in an expression that refers
374 to any earlier test in the same file by identifying the target test
375 by its name in a $HISTORY dictionary. For example, to refer to a
376 value in a JSON object in the response of a test named post json:
377 $HISTORY['post json'].$RESPONSE['$.key']
379 This is a very powerful feature that could lead to test that are dif‐
381 ficult for humans to read. Take care to optimize for the maintainers
382 that will come after you, not yourself.
383 NOTE:
385 Where a single-quote character, ', is shown in the variables above
386 you may also use a double-quote character, ", but in any given
387 expression the same character must be used at both ends.
388 All of these variables may be used in all of the following fields:
390 · url
392 · query_parameters
394 · data
396 · request_headers (in both the key and value)
398 · response_strings
400 · response_json_paths (in both the key and value, see json path substi‐
402 tution for more info)
403 · response_headers (in both the key and value)
405 · response_forbidden_headers
407 · count and delay fields of poll
409 With these variables it ought to be possible to traverse an API without
411 any explicit statements about the URLs being used. If you need a
412 replacement on a field that is not currently supported please raise an
413 issue or provide a patch.
414 As all of these features needed to be tested in the development of
416 gabbi itself, the gabbi tests are a good source of examples on how to
417 use the functionality. See also example for a collection of examples
418 and the gabbi-demo tutorial.
419
421 The data key has some special handing to allow for a bit more flexibil‐
422 ity when doing a POST or PUT:
423 · If the value is not a string (that is, it is a sequence or structure)
425 it is treated as a data structure that will be turned into a string
426 by the dumps method on the relevant content handler. For example if
427 the content-type of the body is application/json the data structure
428 will be turned into a JSON string.
429 · If the value is a string that begins with <@ then the rest of the
431 string is treated as a filepath to be loaded. The path is relative to
432 the test directory and may not traverse up into parent directories.
433 · If the value is an undecorated string, that's the value.
435 NOTE:
437 When reading from a file care should be taken to ensure that a rea‐
438 sonable content-type is set for the data as this will control if any
439 encoding is done of the resulting string value. If it is text, json,
440 xml or javascript it will be encoded to UTF-8.
441 To run gabbi tests with a test harness they must be generated in some
443 fashion and then run. This is accomplished by a test loader. Initially
444 gabbi only supported those test harnesses that supported the load_tests
445 protocol in UnitTest. It now possible to also build and run tests with
446 pytest with some limitations described below.
447 NOTE:
449 It is also possible to run gabbi tests from the command line. See
450 runner.
451 NOTE:
453 By default gabbi will load YAML files using the safe_load function.
454 This means only basic YAML types are allowed in the file. For most
455 use cases this is fine. If you need custom types (for example, to
456 match NaN) it is possible to set the safe_yaml parameter of
457 build_tests() to False. If custom types are used, please keep in
458 mind that this can limit the portability of the YAML files to other
459 contexts.
460 WARNING:
462 If test are being run with a runner that supports concurrency (such
463 as testrepository) it is critical that the test runner is informed
464 of how to group the tests into their respective suites. The usual
465 way to do this is to use a regular expression that groups based on
466 the name of the yaml files. For example, when using testrepository
467 the .testr.conf file needs an entry similar to the following:
468 group_regex=gabbi\.suitemaker\.(test_[^_]+_[^_]+)
470
472 To run the tests with a load_tests style loader a test file containing
473 a load_tests method is required. That will look a bit like:
474 """A sample test module."""
476 # For pathname munging
478 import os
479 # The module that build_tests comes from.
481 from gabbi import driver
482 # We need access to the WSGI application that hosts our service
484 from myapp import wsgiapp
485 # We're using fixtures in the YAML files, we need to know where to
488 # load them from.
489 from myapp.test import fixtures
490 # By convention the YAML files are put in a directory named
492 # "gabbits" that is in the same directory as the Python test file.
493 TESTS_DIR = 'gabbits'
494 def load_tests(loader, tests, pattern):
497 """Provide a TestSuite to the discovery process."""
498 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
499 # Pass "require_ssl=True" as an argument to force all tests
500 # to use SSL in requests.
501 return driver.build_tests(test_dir, loader,
502 intercept=wsgiapp.app,
503 fixture_module=fixtures)
504 For details on the arguments available when building tests see
507 build_tests().
508 Once the test loader has been created, it needs to be run. There are
510 many options. Which is appropriate depends very much on your environ‐
511 ment. Here are some examples using unittest or testtools that require
512 minimal knowledge to get started.
513 By file:
515 python -m testtools.run -v test/test_loader.py
517 By module:
519 python -m testttols.run -v test.test_loader
521 python -m unittest -v test.test_loader
523 Using test discovery to locate all tests in a directory tree:
525 python -m testtools.run discover
527 python -m unittest discover test
529 See the source distribution and the tutorial repo for more advanced
531 options, including using testrepository and subunit.
532
534 Since pytest does not support the load_tests system, a different way of
535 generating tests is required. Two techniques are supported.
536 The original method (described below) used yield statements to generate
538 tests which pytest would collect. This style of tests is deprecated as
539 of pytest>=3.0 so a new style using pytest fixtures has been developed.
540 pytest >= 3.0
542 In the newer technique, a test file is created that uses the
543 pytest_generate_tests hook. Special care must be taken to always import
544 the test_pytest method which is the base test that the pytest hook
545 parametrizes to generate the tests from the YAML files. Without the
546 method, the hook will not be called and no tests generated.
547 Here is a simple example file:
549 """A sample pytest module for pytest >= 3.0."""
551 # For pathname munging
553 import os
554 # The module that py_test_generator comes from.
556 from gabbi import driver
557 # We need test_pytest so that pytest test collection works properly.
559 # Without this, the pytest_generate_tests method below will not be
560 # called.
561 from gabbi.driver import test_pytest # noqa
562 # We need access to the WSGI application that hosts our service
564 from myapp import wsgiapp
565 # We're using fixtures in the YAML files, we need to know where to
567 # load them from.
568 from myapp.test import fixtures
569 # By convention the YAML files are put in a directory named
571 # "gabbits" that is in the same directory as the Python test file.
572 TESTS_DIR = 'gabbits'
573 def pytest_generate_tests(metafunc):
576 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
577 driver.py_test_generator(
578 test_dir, intercept=wsgiapp.app,
579 fixture_module=fixtures, metafunc=metafunc)
580 This can then be run with the usual pytest commands. For example:
583 py.test -svx pytest3.0-example.py
585 pytest < 3.0
587 When using the older technique, test file must be created that calls
588 py_test_generator() and yields the generated tests. That will look a
589 bit like this:
590 """A sample pytest module."""
592 # For pathname munging
594 import os
595 # The module that build_tests comes from.
597 from gabbi import driver
598 # We need access to the WSGI application that hosts our service
600 from myapp import wsgiapp
601 # We're using fixtures in the YAML files, we need to know where to
603 # load them from.
604 from myapp.test import fixtures
605 # By convention the YAML files are put in a directory named
607 # "gabbits" that is in the same directory as the Python test file.
608 TESTS_DIR = 'gabbits'
609 def test_gabbits():
612 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
613 # Pass "require_ssl=True" as an argument to force all tests
614 # to use SSL in requests.
615 test_generator = driver.py_test_generator(
616 test_dir, intercept=wsgiapp.app,
617 fixture_module=fixtures)
618 for test in test_generator:
620 yield test
621 This can then be run with the usual pytest commands. For example:
624 py.test -svx pytest-example.py
626 The older technique will continue to work with all versions of
628 pytest<4.0 but >=3.0 will produce warnings. If you want to use the
629 older technique but not see the warnings add --disable-pytest-warnings
630 parameter to the invocation of py.test.
631 What follows is a commented example of some tests in a single file
633 demonstrating many of the format features. See loader for the Python
634 needed to integrate with a testing harness.
635 # Fixtures can be used to set any necessary configuration, such as a
638 # persistence layer, and establish sample data. They operate per
639 # file. They are context managers, each one wrapping the next in the
640 # sequence.
641 fixtures:
643 - ConfigFixture
644 - SampleDataFixture
645 # There is an included fixture named "SkipAllFixture" which can be
647 # used to declare that all the tests in the given file are to be
648 # skipped.
649 # Each test file can specify a set of defaults that will be used for
651 # every request. This is useful for always specifying a particular
652 # header or always requiring SSL. These values will be used on every
653 # test in the file unless overriden. Lists and dicts are merged one
654 # level deep, except for "data" which is copied verbatim whether it
655 # is a string, list or dict (it can be all three).
656 defaults:
658 ssl: True
659 request_headers:
660 x-my-token: zoom
661 # The tests themselves are a list under a "tests" key. It's useful
663 # to use plenty of whitespace to help readability.
664 tests:
666 # Each request *must* have a name which is unique to the file. When it
668 # becomes a TestCase the name will be lowercased and spaces will
669 # become "_". Use that generated name when limiting test runs.
670 - name: a test for root
672 desc: Some explanatory text that could be used by other tooling
673 # The URL can either be relative to a host specified elsewhere or
675 # be a fully qualified "http" or "https" URL. *You* are responsible
676 # for url-encoding the URL.
677 url: /
679 method: GET
680 # If no status or method are provided they default to "200" and
682 # "GET".
683 # Instead of explicitly stating "url" and "method" you can join
685 # those two keys into one key representing the method. The method
686 # *must* be uppercase.
687 - name: another test for root
689 desc: Same test as above but with GET key
690 GET: /
691 # A single test can override settings in defaults (set above).
693 - name: root without ssl redirects
695 ssl: False
696 GET: /
697 status: 302
698 # When evaluating response headers it is possible to use a regular
700 # expression to not have to test the whole value. Regular expressions match
701 # anywhere in the output, not just at the beginning.
702 response_headers:
704 location: /^https/
705 # By default redirects will not be followed. This can be changed.
707 - name: follow root without ssl redirect
709 ssl: False
710 redirects: True
711 GET: /
712 status: 200 # This is the response code after the redirect.
713 # URLs can express query parameters in two ways: either in the url
715 # value directly, or as query_parameters. If both are used then
716 # query_parameters are appended. In this example the resulting URL
717 # will be equivalient to
718 # /foo?section=news&article=1&article=2&date=yesterday
719 # but not necessarily in that order.
720 - name: create a url with parameters
722 GET: /foo?section=news
723 query_parameters:
724 article:
725 - 1
726 - 2
727 date: yesterday
728 # Request headers can be used to declare media-type choices and
730 # experiment with authorization handling (amongst other things).
731 # Response headers allow evaluating headers in the response. These
732 # two together form the core value of gabbi.
733 - name: test accept
735 GET: /resource
736 request_headers:
737 accept: application/json
738 response_headers:
739 content-type: /application/json/
740 # If a header must not be present in a response at all that can be
742 # expressed in a test as follows.
743 - name: test forbidden headers
745 GET: /resource
746 response_forbidden_headers:
747 - x-special-header
748 # All of the above requests have defaulted to a "GET" method. When
750 # using "POST", "PUT" or "PATCH", the "data" key provides the
751 # request body.
752 - name: post some text
754 POST: /text_repo
755 request_headers:
756 content-type: text/plain
757 data: "I'm storing this"
758 status: 201
759 # If the data is not a string, it will be transformed into JSON.
761 # You must supply an appropriate content-type request header.
762 - name: post some json
764 POST: /json_repo
765 request_headers:
766 content-type: application/json
767 data:
768 name: smith
769 abode: castle
770 status: 201
771 # If the data is a string prepended with "<@" the value will be
773 # treated as the name of a file in the same directory as the YAML
774 # file. Again, you must supply an appropriate content-type. If the
775 # content-type is one of several "text-like" types, the content will
776 # be assumed to be UTF-8 encoded.
777 - name: post an image
779 POST: /image_repo
780 request_headers:
781 content-type: image/png
782 data: <@kittens.png
783 # A single request can be marked to be skipped.
785 - name: patch an image
787 skip: patching images not yet implemented
788 PATCH: /image_repo/12d96fb8-e78c-11e4-8c03-685b35afa334
789 # Or a single request can be marked that it is expected to fail.
791 - name: check allow headers
793 desc: the framework doesn't do allow yet
794 xfail: True
795 PUT: /post_only_url
796 status: 405
797 response_headers:
798 allow: POST
799 # The body of a response can be evaluated with response handlers.
801 # The most simple checks for a set of strings anywhere in the
802 # response. Note that the strings are members of a list.
803 - name: check for css file
805 GET: /blog/posts/12
806 response_strings:
807 - normalize.css
808 # For JSON responses, JSONPath rules can be used.
810 - name: post some json get back json
812 POST: /json_repo
813 request_headers:
814 content-type: application/json
815 data:
816 name: smith
817 abode: castle
818 status: 201
819 response_json_paths:
820 $.name: smith
821 $.abode: castle
822 # Requests run in sequence. One test can make reference to the test
824 # immediately prior using some special variables.
825 # "$LOCATION" contains the "location" header in the previous
826 # response.
827 # "$HEADERS" is a pseudo dictionary containing all the headers of
828 # the previous response.
829 # "$ENVIRON" is a pseudo dictionary providing access to the current
830 # environment.
831 # "$RESPONSE" provides access to the JSON in the prior response, via
832 # JSONPath. See http://jsonpath-rw.readthedocs.io/ for
833 # jsonpath-rw formatting.
834 # $SCHEME and $NETLOC provide access to the current protocol and
835 # location (host and port).
836 - name: get the thing we just posted
838 GET: $LOCATION
839 request_headers:
840 x-magic-exchange: $HEADERS['x-magic-exchange']
841 x-token: $ENVIRON['OS_TOKEN']
842 response_json_paths:
843 $.name: $RESPONSE['$.name']
844 $.abode: $RESPONSE['$.abode']
845 response_headers:
846 content-location: /$SCHEME://$NETLOC/
847 # For APIs where resource creation is asynchronous it can be
849 # necessary to poll for the resulting resource. First we create the
850 # resource in one test. The next test uses the "poll" key to loop
851 # with a delay for a set number of times.
852 - name: create asynch
854 POST: /async_creator
855 request_headers:
856 content-type: application/json
857 data:
858 name: jones
859 abode: bungalow
860 status: 202
861 - name: poll for created resource
863 GET: $LOCATION
864 poll:
865 count: 10 # try up to ten times
866 delay: .5 # wait .5 seconds between each try
867 response_json_paths:
868 $.name: $RESPONSE['$.name']
869 $.abode: $RESPONSE['$.abode']
870 Gabbi supports JSONPath both for validating JSON response bodies and
873 within substitutions.
874 JSONPath expressions are provided by jsonpath_rw, with jsonpath_rw_ext
876 custom extensions to address common requirements:
877 1. Sorting via sorted and [/property].
879 2. Filtering via [?property = value].
881 3. Returning the respective length via len.
883 (These apply both to arrays and key-value pairs.)
885 Here is a JSONPath example demonstrating some of these features. Given
887 JSON data as follows:
888 {
890 "pets": [
891 {"type": "cat", "sound": "meow"},
892 {"type": "dog", "sound": "woof"}
893 ]
894 }
895 If the ordering of the list in pets is predictable and reliable it is
897 relatively straightforward to test values:
898 response_json_paths:
900 # length of list is two
901 $.pets.`len`: 2
902 # sound of second item in list is woof
903 $.pets[1].sound: woof
904 If the ordering is not predictable additional effort is required:
906 response_json_paths:
908 # sort by type
909 $.pets[/type][0].sound: meow
910 # sort by type, reversed
911 $.pets[\type][0].sound: woof
912 # all the sounds
913 $.pets[/type]..sound: ['meow', 'woof']
914 # filter by type = dog
915 $.pets[?type = "dog"].sound: woof
916 If it is necessary to validate the entire JSON response use a JSONPath
918 of $:
919 response_json_paths:
921 $:
922 pets:
923 - type: cat
924 sound: meow
925 - type: dog
926 sound: woof
927 This is not a technique that should be used frequently as it can lead
929 to difficult to read tests and it also indicates that your gabbi tests
930 are being used to test your serializers and data models, not just your
931 API interactions.
932 It is also possible to read raw JSON from disk for either all or some
934 of a JSON response:
935 response_json_paths:
937 $: @<data.json
938 or:
940 response_json_paths:
942 $.pets: <@pets.json
943 $.pets[0]: <@cat.json
944 Examples like this can be found in one of gabbi's own tests.
946 If it is desired to load YAML files like the JSON ones above, two
948 things must be done:
949 1. The YAMLDiskLoadingJSONHandler custom content handler must be passed
951 to the driver through the content_handlers argument. See Extensions
952 on how to do this.
953 2. The YAML files to load must be placed in a subdirectory to prevent
955 the test runner from consuming them as test files to run:
956 response_json_paths:
958 $: @<subdir/values.yaml
959 When reading from disk you can apply the same JSONPath by adding a ':'
961 to the end of your file name. This allows you to store multiple API
962 responses into a single file to reduce file management when construct‐
963 ing your tests.
964 Given JSON data as follows:
966 {
968 "values": [{
969 "pets": [{
970 "type": "cat",
971 "sound": "meow"
972 }, {
973 "type": "dog",
974 "sound": "woof"
975 }]
976 }, {
977 "people": [{
978 "name": "chris",
979 "id": 1
980 }, {
981 "name": "justin",
982 "id": 2
983 }]
984 }]
985 }
986 You can write your tests like the following:
988 response_json_paths:
990 $.pets: <@pets.json
991 $.pets[?type = "cat"].sound: <@values.json:$.values[0].pets[?type = "cat"].sound
992 Although placing more than one API response into a single JSON file may
994 seem convenient, keep in mind there is a tradeoff in readability that
995 should not be overlooked before implementing this technique.
996 Examples like this can be found in one of gabbi's yaml-from-disk tests.
998 There are more JSONPath examples in example and in the jsonpath_rw and
1000 jsonpath_rw_ext documentation.
1001
1003 Substitutions can be made in both the left (query) and right (expected)
1004 hand sides of the json path expression. When subtitutions are used in
1005 the query, care must be taken to ensure proper quoting of the resulting
1006 value. For example if there is a uuid (with hyphens) at
1007 $RESPONSE['$.id'] then this expression may fail:
1008 $.nested.structure.$RESPONSE['$.id'].name: foobar
1010 as it will evaluate to something like:
1012 $.nested.structure.ADC8AAFC-D564-40D1-9724-7680D3C010C2.name: foobar
1014 which may be treated as an arithemtic expression by the json path
1016 parser. The test author should write:
1017 $.nested.structure["$RESPONSE['$.id']"].name: foobar
1019 to quote the result of the substitution.
1021 The target host is the host on which the API to be tested can be found.
1023 Gabbi intends to preserve the flow and semantics of HTTP interactions
1024 as much as possible, and every HTTP request needs to be directed at a
1025 host of some form. Gabbi provides three ways to control this:
1026 · Using wsgi-intercept to provide a fake socket and WSGI environment on
1028 an arbitrary host and port attached to a WSGI application (see
1029 intercept examples).
1030 · Using fully qualified url values in the YAML defined tests (see full
1032 examples).
1033 · Using a host and (optionally) port defined at test build time (see
1035 live examples).
1036 The intercept and live methods are mutually exclusive per test builder,
1038 but either kind of test can freely intermix fully qualified URLs into
1039 the sequence of tests in a YAML file.
1040 For test driven development and local tests the intercept style of
1042 testing lowers test requirements (no web server required) and is fast.
1043 Interception is performed as part of fixtures processing as the most
1044 deeply nested fixture. This allows any configuration or database setup
1045 to be performed prior to the WSGI application being created.
1046 For the implementation of the above see build_tests().
1048 Each suite of tests is represented by a single YAML file, and may
1050 optionally use one or more fixtures to provide the necessary environ‐
1051 ment required by the tests in that file.
1052 Fixtures are implemented as nested context managers. Subclasses of
1054 GabbiFixture must implement start_fixture and stop_fixture methods for
1055 creating and destroying, respectively, any resources managed by the
1056 fixture. While the subclass may choose to implement __init__ it is
1057 important that no exceptions are thrown in that method, otherwise the
1058 stack of context managers will fail in unexpected ways. Instead ini‐
1059 tialization of real resources should happen in start_fixture.
1060 At this time there is no mechanism for the individual tests to have any
1062 direct awareness of the fixtures. The fixtures exist, conceptually, on
1063 the server side of the API being tested.
1064 Fixtures may do whatever is required by the testing environment, how‐
1066 ever there are two common scenarios:
1067 · Establishing (and then resetting when a test suite has finished) any
1069 baseline configuration settings and persistence systems required for
1070 the tests.
1071 · Creating sample data for use by the tests.
1073 If a fixture raises unittest.case.SkipTest during start_fixture all the
1075 tests in the current file will be skipped. This makes it possible to
1076 skip the tests if some optional configuration (such as a particular
1077 type of database) is not available.
1078 If an exception is raised while a fixture is being used, information
1080 about the exception will be stored on the fixture so that the stop_fix‐
1081 ture method can decide if the exception should change how the fixture
1082 should clean up. The exception information can be found on exc_type,
1083 exc_value and traceback method attributes.
1084 If an exception is raised when a fixture is started (in start_fixture)
1086 the first test in the suite using the fixture will be marked with an
1087 error using the traceback from the exception and all the tests in the
1088 suite will be skipped. This ensures that fixture failure is adequately
1089 captured and reported by test runners.
1090 In some contexts (for example CI environments with a large number of
1092 tests being run in a broadly concurrent environment where output is
1093 logged to a single file) it can be important to capture and consolidate
1094 stray output that is produced during the tests and display it associ‐
1095 ated with an individual test. This can help debugging and avoids unus‐
1096 able output that is the result of multiple streams being interleaved.
1097 Inner fixtures have been added to support this. These are fixtures more
1099 in line with the tradtional unittest concept of fixtures: a class on
1100 which setUp and cleanUp is automatically called.
1101 build_tests() accepts a named parameter arguments of inner_fixtures.
1103 The value of that argument may be an ordered list of fixtures.Fixture
1104 classes that will be called when each individual test is set up.
1105 An example fixture that could be useful is the FakeLogger.
1107 NOTE:
1109 At this time inner_fixtures are not supported when using the pytest
1110 loader.
1111 Content handlers are responsible for preparing request data and evalu‐
1113 ating response data based on the content-type of the request and
1114 response. A content handler operates as follows:
1115 · Structured YAML data provided via the data attribute is converted to
1117 a string or bytes sequence and used as request body.
1118 · The response body (a string or sequence of bytes) is transformed into
1120 a content-type dependent structure and stored in an internal
1121 attribute named response_data that is:
1122 · used when evaluating the response body
1124 · used in $RESPONSE[] substitutions
1126 By default, gabbi provides content handlers for JSON. In that content
1128 handler the data test key is converted from structured YAML into a JSON
1129 string. Response bodies are converted from a JSON string into a data
1130 structure in response_data that is used when evaluating
1131 response_json_paths entries in a test or doing JSONPath-based
1132 $RESPONSE[] substitutions.
1133 A YAMLDiskLoadingJSONHandler has been added to extend the JSON handler.
1135 It works the same way as the JSON handler except for when evaluating
1136 the response_json_paths handle, data that is read from disk can be
1137 either in JSON or YAML format. The YAMLDiskLoadingJSONHandler is not
1138 enabled by default and must be added as shown in the Extensions section
1139 in order to be used in the tests.
1140 Further content handlers can be added as extensions. Test authors may
1142 need these extensions for their own suites, or enterprising developers
1143 may wish to create and distribute extensions for others to use.
1144 NOTE:
1146 One extension that is likely to be useful is a content handler that
1147 turns data into url-encoded form data suitable for POST and turns an
1148 HTML response into a DOM object.
1149
1151 Content handlers are an evolution of the response handler concept in
1152 earlier versions gabbi. To preserve backwards compatibility with exist‐
1153 ing response handlers, old style response handlers are still allowed,
1154 but new handlers should implement the content handler interface
1155 (described below).
1156 Registering additional custom handlers is done by passing a subclass of
1158 ContentHandler to build_tests():
1159 driver.build_tests(test_dir, loader, host=None,
1161 intercept=simple_wsgi.SimpleWsgi,
1162 content_handlers=[MyContentHandler])
1163 If pytest is being used:
1165 driver.py_test_generator(test_dir, intercept=simple_wsgi.SimpleWsgi,
1167 content_handlers=[MyContenHandler])
1168 Gabbi provides an additional custom handler named YAMLDiskLoadingJSON‐
1170 Handler. This can be used for loading JSON and YAML files from disk
1171 when evaluating the response_json_paths handle.
1172 WARNING:
1174 YAMLDiskLoadingJSONHandler shares the same content-type as the
1175 default JSONHandler. When there are multiple handlers listed that
1176 accept the same content-type, the one that is earliest in the list
1177 will be used.
1178 With gabbi-run, custom handlers can be loaded via the --response-han‐
1180 dler option -- see load_response_handlers() for details.
1181 NOTE:
1183 The use of the --response-handler argument is done to preserve back‐
1184 wards compatibility and avoid excessive arguments. Both types of
1185 handler may be passed to the argument.
1186 Implementation Details
1188 Creating a content handler requires subclassing ContentHandler and
1189 implementing several methods. These methods are described below, but
1190 inspecting JSONHandler will be instructive in highlighting required
1191 arguments and techniques.
1192 To provide a response_<something> response-body evaluator a subclass
1194 must define:
1195 · test_key_suffix: This, along with the prefix response_, forms the key
1197 used in the test structure. It is a class level string.
1198 · test_key_value: The key's default value, either an empty list ([]) or
1200 empty dict ({}). It is a class level value.
1201 · action: An instance method which tests the expected values against
1203 the HTTP response - it is invoked for each entry, with the parameters
1204 depending on the default value. The arguments to action are (in
1205 order):
1206 · self: The current instance.
1208 · test: The currently active HTTPTestCase
1210 · item: The current entry if test_key_value is a list, otherwise the
1212 key half of the key/value pair at this entry.
1213 · value: None if test_key_value is a list, otherwise the value half
1215 of the key/value pair at this entry.
1216 To translate request or response bodies to or from structured data a
1218 subclass must define an accepts method. This should return True if this
1219 class is willing to translate the provided content-type. During request
1220 processing it is given the value of the content-type header that will
1221 be sent in the request. During response processing it is given the
1222 value of the content-type header of the response. This makes it possi‐
1223 ble to handle different request and response bodies in the same han‐
1224 dler, if desired. For example a handler might accept applica‐
1225 tion/x-www-form-urlencoded and text/html.
1226 If accepts is defined two additional static methods should be defined:
1228 · dumps: Turn structured Python data from the data key in a test into a
1230 string or byte stream. The optional test param allows you to access
1231 the current test case which may help with manipulations for custom
1232 content handlers, e.g. multipart/form-data needs to add a boundary to
1233 the Content-Type header in order to mark the appropriate sections of
1234 the body.
1235 · loads: Turn a string or byte stream in a response into a Python data
1237 structure. Gabbi will put this data on the response_data attribute on
1238 the test, where it can be used in the evaluations described above (in
1239 the action method) or in $RESPONSE handling. An example usage here
1240 would be to turn HTML into a DOM.
1241 · load_data_file: Load data from disk into a Python data structure.
1243 Gabbi will call this method when response_<something> contains an
1244 item where the right hand side value starts with <@. The test param
1245 allows you to access the current test case and provides a
1246 load_data_file method which should be used because it verifies the
1247 data is loaded within the test diectory and returns the file source
1248 as a string. The load_data_file method was introduced to re-use the
1249 JSONHandler in order to support loading YAML files from disk through
1250 the implementation of an additional custom handler, see YAMLDiskLoad‐
1251 ingJSONHandler for details.
1252 Finally if a replacer class method is defined, then when a $RESPONSE
1254 substitution is encountered, replacer will be passed the response_data
1255 of the prior test and the argument within the $RESPONSE.
1256 Please see the JSONHandler source for additional detail.
1258 If there is a running web service that needs to be tested and creating
1260 a test loader with build_tests() is either inconvenient or overkill it
1261 is possible to run YAML test files directly from the command line with
1262 the console-script gabbi-run. It accepts YAML on stdin or as multiple
1263 file arguments, and generates and runs tests and outputs a summary of
1264 the results.
1265 The provided YAML may not use custom fixtures but otherwise uses the
1267 default format. host information is either expressed directly in the
1268 YAML file or provided on the command line:
1269 gabbi-run [host[:port]] < /my/test.yaml
1271 or:
1273 gabbi-run http://host:port < /my/test.yaml
1275 To test with one or more files the following command syntax may be
1277 used:
1278 gabbi-run http://host:port -- /my/test.yaml /my/other.yaml
1280 NOTE:
1282 The filename arguments must come after a -- and all other arguments
1283 (host, port, prefix, failfast) must come before the --.
1284 NOTE:
1286 If files are provided, test output will use names including the name
1287 of the file. If any single file includes an error, the name of the
1288 file will be included in a summary of failed files at the end of the
1289 test report.
1290 To facilitate using the same tests against the same application mounted
1292 in different locations in a WSGI server, a prefix may be provided as a
1293 second argument:
1294 gabbi-run host[:port] [prefix] < /my/test.yaml
1296 or in the target URL:
1298 gabbi-run http://host:port/prefix < /my/test.yaml
1300 The value of prefix will be prepended to the path portion of URLs that
1302 are not fully qualified.
1303 Anywhere host is used, if it is a raw IPV6 address it should be wrapped
1305 in [ and ].
1306 If https is used in the target, then the tests in the provided YAML
1308 will default to ssl: True.
1309 Use -k or --insecure to not validate certificates when making https
1311 connections.
1312 If a -x or --failfast argument is provided then gabbi-run will exit
1314 after the first test failure.
1315 Use -v or --verbose with a value of all, headers or body to turn on
1317 verbosity for all tests being run.
1318 Use -q or --quiet to silence test runner output.
1320 These are informal release notes for gabbi since version 1.0.0, high‐
1322 lighting major features and changes. For more detail see the commit
1323 logs on GitHub.
1324
1326 · Add support for not validating certificates in https requests. Con‐
1327 trolled by the cert_validate attribute in individual tests and
1328 build_tests() and the -k or --insecure argument to gabbi-run.
1329
1331 · Support pytest 5.0.0 in Python >=3.5. For earlier versions of Python,
1332 pytest<5.0.0 will be used; the pytest project is dropping support for
1333 older versions of Python.
1334
1336 · Use pytest<5.0.0 until gabbi has solutions for the changes in 5.0.0.
1337
1339 · A -q argument is added to gabbi-run to suppress output from the test
1340 runner.
1341
1343 · Adjust loading of YAML to be ready for new release of PyYAML.
1344
1346 · Provide the YAMLDiskLoadingJSONHandler class that allows test result
1347 data for response_json_path checks to be loaded from YAML-on-disk.
1348
1350 · Use jsonpath to select a portion of data-on-disk in
1351 response_json_path checks.
1352 · Restrict PyYAML to <4.0.
1354
1356 · Allow listing of tests with no host configured. When host is an empty
1357 string, tests can be listed (for discovery), but will be skipped on
1358 run.
1359
1361 · JSON $RESPONSE substitutions in the data field may be complex types
1362 (lists and dicts), not solely strings.
1363
1365 · When the HTTP response begins with a bad status line, have BadSta‐
1366 tusLine be raised from urllib3.
1367
1369 · Allow substitutions in the key portion of request and response head‐
1370 ers, not just the value.
1371
1373 · Remove support for Python 3.3.
1374 · Make handling of fixture-level skips in pytest actually work.
1376
1378 · Add safe_yaml parameter to build_tests().
1379
1381 · use_prior_test is added to test metadata.
1382 · Extensive cleanups in regular expression handling when constructing
1384 tests from YAML.
1385
1387 jsonpath handling gets two improvements:
1388 · The value side of a response_json_paths entry can be loaded from a
1390 file using the <@file.json syntax also used in data.
1391 · The key side of a response_json_paths entry can use substitutions.
1393 This was already true for the value side.
1394
1396 Substitutions in $RESPONSE handling now preserve numeric types instead
1397 of casting to a string. This is useful when servers are expecting
1398 strong types and tests want to send response data back to the server.
1399
1401 count and delay test keys allow substitutions.
1402 gabbi.driver.build_tests() accepts a verbose parameter to set test ver‐
1403 bosity for an entire session.
1404
1406 Better failure reporting when using gabbi-run with multiple files. Test
1407 names are based on the files and a summary of failed files is provided
1408 at the end of the report.
1409
1411 Effectively capture a failure in a fixture and report the traceback.
1412 Without this some test runners swallow the error and discovering prob‐
1413 lems when developing fixtures can be quite challenging.
1414
1416 Thanks to Samuel Fekete, tests can use the $HISTORY dictionary to refer
1417 to any prior test in the same file, not just the one immediately prior,
1418 when doing substitutions.
1419
1421 Filenames used to read data into tests using the <@ syntax may now use
1422 pathnames relative to the YAML file. See data.
1423 gabbi-run gains a --verbose parameter to force all tests run in a ses‐
1425 sion to run with verbose set.
1426 When using pytest to load tests, a new mechanism is available which
1428 avoids warnings produced in when using a version of pytest greater than
1429 3.0.
1430
1432 When verbosely displaying request and response bodies that are JSON,
1433 pretty print for improved readability.
1434
1436 Allow gabbi-run to accept multiple filenames as command line arguments
1437 instead of reading tests from stdin.
1438
1440 Switch from response handlers to handlers to allow more flexible pro‐
1441 cessing of both response _and_ request bodies.
1442 Add inner fixtures for per test fixtures, useful for output capturing.
1444
1446 Allow the test_loader_name arg to gabbi.driver.build_tests() to over‐
1447 ride the prefix of the pretty printed name of generated tests.
1448
1450 String values in JSONPath matches may be wrapped in /.../` to be
1451 treated as regular expressions.
1452
1454 Better documentation of how to run gabbi in a concurrent environment.
1455 Improved handling of pytest fixtures and test counts.
1456
1458 Add url to gabbi.driver.build_tests() to use instead of host, port and
1459 prefix.
1460
1462 Add require_ssl to gabbi.driver.build_tests() to force use of SSL.
1463
1465 Add $COOKIE substitution.
1466
1468 Correctly support IPV6 hosts.
1469
1471 Add $LAST_URL substitution.
1472
1474 Introduce support for loading and running tests with pytest.
1475
1477 Use urllib3 instead of httplib2 for driving HTTP requests.
1478
1480 Add sorting and filtering to jsonpath handling.
1481
1483 Add the response_forbidden_headers to response expectations.
1484
1486 Instead of:
1487 tests:
1489 - name: a simple get
1490 url: /some/path
1491 method: get
1492 1.7.0 also makes it possible to:
1494 tests:
1496 - name: a simple get
1497 GET: /some/path
1498 Any upper case key is treated as a method.
1500
1502 Enhanced flexibility and colorization when setting tests to be verbose.
1503
1505 Adds the query_parameters key to request parameters.
1506
1508 The start of improvements and extensions to jsonpath handling. In this
1509 case the addition of the len function.
1510
1512 Vastly improved output and behavior in gabbi-run.
1513
1515 Version 1 was the first release with a commitment to a stable format.
1516 Since then new fields have been added but have not been taken away.
1517 The following people have contributed code to gabbi. Thanks to them.
1519 Thanks also to all the people who have made gabbi better by reporting
1520 issues and their successes and failures with using gabbi.
1521 · Chris Dent
1523 · FND
1525 · Mehdi Abaakouk
1527 · Tom Viner
1529 · Jason Myers
1531 · Josh Leeb-du Toit
1533 · Duc Truong
1535 · Zane Bitter
1537 · Ryan Spencer
1539 · Kim Raymoure
1541 · Travis Truman
1543 · Samuel Fekete
1545 · Michael McCune
1547 · Imran Hayder
1549 · Julien Danjou
1551 · Trevor McCasland
1553 · Danek Duvall
1555 · Marc Abramowitz
1557 NOTE:
1559 This section provides a collection of questions with answers that
1560 don't otherwise fit in the rest of the documentation. If something
1561 is missing, please create an issue.
1562 As this document grows it will gain a more refined structure.
1564
1566 Is gabbi only for testing Python-based APIs?
1567 No, you can use gabbi-run to test an HTTP service built in any program‐
1568 ming language.
1569 How do I run just one test?
1571 Each YAML file contains a sequence of tests, each test within each file
1572 has a name. That name is translated to the name of the test by replac‐
1573 ing spaces with an _.
1574 When running tests that are generated dynamically, filtering based on
1576 the test name prior to the test being collected will not work in some
1577 test runners. Test runners that use a --load-list functionality can be
1578 convinced to filter after discovery.
1579 pytest does this directly with the -k keyword flag.
1581 When using testrepository with tox as used in gabbi's own tests it is
1583 possible to pass a filter in the tox command:
1584 tox -epy27 -- get_the_widget
1586 When using testtools.run and similar test runners it's a bit more com‐
1588 plicated. It is necessary to provide the full name of the test as a
1589 list to --load-list:
1590 python -m testtools.run --load-list \
1592 <(echo package.tests.test_api.yamlfile_get_the_widge.test_request)
1593 How do I run just one test, without running prior tests in a sequence?
1595 By default, when you select a single test to run, all tests prior to
1596 that one in a file will be run as well: the file is treated as as
1597 sequence of dependent tests. If you do not want this you can adjust the
1598 use_prior_test test metadata in one of three ways:
1599 · Set it in the YAML file for the one test you are concerned with.
1601 · Set the defaults for all tests in that file.
1603 · set use_prior_test to false when calling build_tests()
1605 Be aware that doing this breaks a fundamental assumption that gabbi
1607 makes about how tests work. Any substitutions will fail.
1608
1610 Can I have variables in my YAML file?
1611 Gabbi provides the $ENVIRON substitution which can operate a bit like
1612 variables that are set elsewhere and then used in the tests defined by
1613 the YAML.
1614 If you find it necessary to have variables within a single YAML file
1616 you take advantage of YAML alias nodes list this:
1617 vars:
1619 - &uuid_1 5613AABF-BAED-4BBA-887A-252B2D3543F8
1620 tests:
1622 - name: send a uuid to a post
1623 POST: /resource
1624 request_headers:
1625 content-type: application/json
1626 data:
1627 uuid: *uuid_1
1628 You can alias all sorts of nodes, not just single items. Be aware that
1630 the replacement of an alias node happens while the YAML is being
1631 loaded, before gabbi does any processing.
1632 How many tests should be put in one YAML file?
1634 For the sake of readability it is best to keep each YAML file rela‐
1635 tively short. Since each YAML file represents a sequence of requests,
1636 it usually makes sense to create a new file when a test is not depen‐
1637 dent on any before it.
1638 It's tempting to put all the tests for any resource or URL in the same
1640 file, but this eventually leads to files that are too long and are thus
1641 difficult to read.
1642
1644 A single HTTP request represented as a subclass of testtools.TestCase
1645 The test case encapsulates the request headers and body and expected
1647 response headers and body. When the test is run an HTTP request is made
1648 using urllib3. Assertions are made against the response.
1649 class gabbi.case.HTTPTestCase(*args, **kwargs)
1651 Bases: testtools.testcase.TestCase
1652 Encapsulate a single HTTP request as a TestCase.
1654 If the test is a member of a sequence of requests, ensure that
1656 prior tests are run.
1657 To keep the test harness happy we need to make sure the setUp
1659 and tearDown are only run once.
1660 assert_in_or_print_output(expected, iterable)
1662 Assert the iterable contains expected or print some out‐
1663 put.
1664 If the output is long, it is limited by either
1666 GABBI_MAX_CHARS_OUTPUT in the environment or the
1667 MAX_CHARS_OUTPUT constant.
1668 base_test = {'cert_validate': True, 'data': '', 'desc': '',
1670 'method': 'GET', 'name': '', 'poll': {}, 'query_parameters': {},
1671 'redirects': False, 'request_headers': {}, 'skip': '', 'ssl':
1672 False, 'status': '200', 'url': '', 'use_prior_test': True, 'ver‐
1673 bose': False, 'xfail': False}
1674 get_content_handler(content_type)
1676 Determine the content handler for this media type.
1677 load_data_file(filename)
1679 Read a file from the current test directory.
1680 replace_template(message, escape_regex=False)
1682 Replace magic strings in message.
1683 run(result=None)
1685 Store the current result handler on this test.
1686 setUp()
1688 Hook method for setting up the test fixture before exer‐
1689 cising it.
1690 tearDown()
1692 Hook method for deconstructing the test fixture after
1693 testing it.
1694 test_request()
1696 Run this request if it has not yet run.
1697 If there is a prior test in the sequence, run it first.
1699 gabbi.case.potentialFailure(func)
1701 Decorate a test method that is expected to fail if 'xfail' is
1702 true.
1703
1707 Manage fixtures for gabbi at the test suite level.
1708 class gabbi.fixture.GabbiFixture
1710 Bases: object
1711 A context manager that operates as a fixture.
1713 Subclasses must implement start_fixture and stop_fixture, each
1715 of which contain the logic for stopping and starting whatever
1716 the fixture is. What a fixture is is left as an exercise for the
1717 implementor.
1718 These context managers will be nested so any actual work needs
1720 to happen in start_fixture and stop_fixture and not in __init__.
1721 Otherwise exception handling will not work properly.
1722 start_fixture()
1724 Implement the actual workings of starting the fixture
1725 here.
1726 stop_fixture()
1728 Implement the actual workings of stopping the fixture
1729 here.
1730 exception gabbi.fixture.GabbiFixtureError
1732 Bases: Exception
1733 Generic exception for GabbiFixture.
1735 class gabbi.fixture.SkipAllFixture
1737 Bases: gabbi.fixture.GabbiFixture
1738 A fixture that skips all the tests in the current suite.
1740 start_fixture()
1742 Implement the actual workings of starting the fixture
1743 here.
1744 gabbi.fixture.nest(fixtures)
1746 Nest a series of fixtures.
1747 This is duplicated from nested in the stdlib, which has been
1749 deprecated because of issues with how exceptions are difficult
1750 to handle during __init__. Gabbi needs to nest an unknown number
1751 of fixtures dynamically, so the with syntax that replaces nested
1752 will not work.
1753
1755 Package for response and content handlers that process the body of a
1756 response in various ways.
1757 handlers.base Module
1759 Base classes for response and content handlers.
1760 class gabbi.handlers.base.ContentHandler
1762 Bases: gabbi.handlers.base.ResponseHandler
1763 A subclass of ResponseHandlers that adds content handling.
1765 static accepts(content_type)
1767 Return True if this handler can handler this type.
1768 static dumps(data, pretty=False, test=None)
1770 Return structured data as a string.
1771 If pretty is true, prettify.
1773 static load_data_file(test, file_path)
1775 Return the string content of the file specified by the
1776 file_path.
1777 static loads(data)
1779 Create structured (Python) data from a stream.
1780 classmethod replacer(response_data, path)
1782 Return the string that is replacing RESPONSE.
1783 class gabbi.handlers.base.ResponseHandler
1785 Bases: object
1786 Add functionality for making assertions about an HTTP response.
1788 A subclass may implement two methods: action and preprocess.
1790 preprocess takes one argument, the TestCase. It is called
1792 exactly once for each test before looping across the assertions.
1793 It is used, rarely, to copy the test.output into a useful form
1794 (such as a parsed DOM).
1795 action takes two or three arguments. If test_key_value is a list
1797 action is called with the test case and a single list item. If
1798 test_key_value is a dict then action is called with the test
1799 case and a key and value pair.
1800 action(test, item, value=None)
1802 Test an individual entry for this response handler.
1803 If the entry is a key value pair the key is in item and
1805 the value in value. Otherwise the entry is considered a
1806 single item from a list.
1807 preprocess(test)
1809 Do any pre-single-test preprocessing.
1810 test_key_suffix = ''
1812 test_key_value = []
1814 handlers.core Module
1816 Core response handlers.
1817 class gabbi.handlers.core.ForbiddenHeadersResponseHandler
1819 Bases: gabbi.handlers.base.ResponseHandler
1820 Test that listed headers are not in the response.
1822 action(test, forbidden, value=None)
1824 Test an individual entry for this response handler.
1825 If the entry is a key value pair the key is in item and
1827 the value in value. Otherwise the entry is considered a
1828 single item from a list.
1829 test_key_suffix = 'forbidden_headers'
1831 test_key_value = []
1833 class gabbi.handlers.core.HeadersResponseHandler
1835 Bases: gabbi.handlers.base.ResponseHandler
1836 Compare expected headers with actual headers.
1838 If a header value is wrapped in / it is treated as a raw regular
1840 expression.
1841 Headers values are always treated as strings.
1843 action(test, header, value=None)
1845 Test an individual entry for this response handler.
1846 If the entry is a key value pair the key is in item and
1848 the value in value. Otherwise the entry is considered a
1849 single item from a list.
1850 test_key_suffix = 'headers'
1852 test_key_value = {}
1854 class gabbi.handlers.core.StringResponseHandler
1856 Bases: gabbi.handlers.base.ResponseHandler
1857 Test for matching strings in the the response body.
1859 action(test, expected, value=None)
1861 Test an individual entry for this response handler.
1862 If the entry is a key value pair the key is in item and
1864 the value in value. Otherwise the entry is considered a
1865 single item from a list.
1866 test_key_suffix = 'strings'
1868 test_key_value = []
1870 handlers.jsonhandler Module
1872 JSON-related content handling.
1873 class gabbi.handlers.jsonhandler.JSONHandler
1875 Bases: gabbi.handlers.base.ContentHandler
1876 A ContentHandler for JSON
1878 · Structured test data is turned into JSON when request con‐
1880 tent-type is JSON.
1881 · Response bodies that are JSON strings are made into Python
1883 data on the test response_data attribute when the response
1884 content-type is JSON.
1885 · A response_json_paths response handler is added.
1887 · JSONPaths in $RESPONSE substitutions are supported.
1889 static accepts(content_type)
1891 Return True if this handler can handler this type.
1892 action(test, path, value=None)
1894 Test json_paths against json data.
1895 static dumps(data, pretty=False, test=None)
1897 Return structured data as a string.
1898 If pretty is true, prettify.
1900 static extract_json_path_value(data, path)
1902 Extract the value at JSON Path path from the data.
1903 The input data is a Python datastructure, not a JSON
1905 string.
1906 static load_data_file(test, file_path)
1908 Return the string content of the file specified by the
1909 file_path.
1910 static loads(data)
1912 Create structured (Python) data from a stream.
1913 classmethod replacer(response_data, match)
1915 Return the string that is replacing RESPONSE.
1916 test_key_suffix = 'json_paths'
1918 test_key_value = {}
1920 handlers.yaml_disk_loading_jsonhandler Module
1922 JSON-related content handling with YAML data disk loading.
1923 class gabbi.handlers.yaml_disk_loading_jsonhandler.YAMLDiskLoadingJSON‐
1925 Handler
1926 Bases: gabbi.handlers.jsonhandler.JSONHandler
1927 A ContentHandler for JSON responses that loads YAML from disk
1929 · Structured test data is turned into JSON when request con‐
1931 tent-type is JSON.
1932 · Response bodies that are JSON strings are made into Python
1934 data on the test response_data attribute when the response
1935 content-type is JSON.
1936 · A response_json_paths response handler is added. Data read
1938 from disk during this handle will be loaded with the
1939 yaml.safe_load method to support both JSON and YAML data
1940 sources from disk.
1941 · JSONPaths in $RESPONSE substitutions are supported.
1943 static load_data_file(test, file_path)
1945 Return the string content of the file specified by the
1946 file_path.
1947
1949 A TestSuite for containing gabbi tests.
1950 This suite has two features: the contained tests are ordered and there
1952 are suite-level fixtures that operate as context managers.
1953 class gabbi.suite.GabbiSuite(tests=())
1955 Bases: unittest.suite.TestSuite
1956 A TestSuite with fixtures.
1958 The suite wraps the tests with a set of nested context managers
1960 that operate as fixtures.
1961 If a fixture raises unittest.case.SkipTest during setup, all the
1963 tests in this suite will be skipped.
1964 run(result, debug=False)
1966 Override TestSuite run to start suite-level fixtures.
1967 To avoid exception confusion, use a null Fixture when
1969 there are no fixtures.
1970 start(result, tests)
1972 Start fixtures when using pytest.
1973 stop() Stop fixtures when using pytest.
1975 gabbi.suite.noop(*args)
1977 A noop method used to disable collected tests.
1978
1981 TestRunner and TestResult for gabbi-run.
1982 class gabbi.reporter.ConciseTestResult(stream, descriptions, verbosity)
1984 Bases: unittest.runner.TextTestResult
1985 A TextTestResult with simple but useful output.
1987 If the output is a tty or GABBI_FORCE_COLOR is set in the envi‐
1989 ronment, output will be colorized.
1990 addError(test, err)
1992 Called when an error has occurred. 'err' is a tuple of
1993 values as returned by sys.exc_info().
1994 addExpectedFailure(test, err)
1996 Called when an expected failure/error occurred.
1997 addFailure(test, err)
1999 Called when an error has occurred. 'err' is a tuple of
2000 values as returned by sys.exc_info().
2001 addSkip(test, reason)
2003 Called when a test is skipped.
2004 addSuccess(test)
2006 Called when a test has completed successfully
2007 addUnexpectedSuccess(test)
2009 Called when a test was expected to fail, but succeed.
2010 getDescription(test)
2012 printErrorList(flavor, errors)
2014 startTest(test)
2016 Called when the given test is about to be run
2017 class gabbi.reporter.ConciseTestRunner(stream=None, descriptions=True,
2019 verbosity=1, failfast=False, buffer=False, resultclass=None, warn‐
2020 ings=None, *, tb_locals=False)
2021 Bases: unittest.runner.TextTestRunner
2022 A TextTestRunner that uses ConciseTestResult for reporting
2024 results.
2025 resultclass
2027 alias of ConciseTestResult
2028 class gabbi.reporter.PyTestResult(stream=None, descriptions=None, ver‐
2030 bosity=None)
2031 Bases: unittest.result.TestResult
2032 Wrap a test result to allow it to work with pytest.
2034 The main behaviors here are:
2036 · to turn what had been exceptions back into exceptions
2038 · use pytest's skip and xfail methods
2040 addError(test, err)
2042 Called when an error has occurred. 'err' is a tuple of
2043 values as returned by sys.exc_info().
2044 addExpectedFailure(test, err)
2046 Called when an expected failure/error occurred.
2047 addFailure(test, err)
2049 Called when an error has occurred. 'err' is a tuple of
2050 values as returned by sys.exc_info().
2051 addSkip(test, reason)
2053 Called when a test is skipped.
2054
2056 Utility functions grab bag.
2057 gabbi.utils.create_url(base_url, host, port=None, prefix='', ssl=False)
2059 Given pieces of a path-based url, return a fully qualified url.
2060 gabbi.utils.decode_response_content(header_dict, content)
2062 Decode content to a proper string.
2063 gabbi.utils.extract_content_type(header_dict, default='applica‐
2065 tion/binary')
2066 Extract parsed content-type from headers.
2067 gabbi.utils.get_colorizer(stream)
2069 Return a function to colorize a string.
2070 Only if stream is a tty .
2072 gabbi.utils.host_info_from_target(target, prefix=None)
2074 Turn url or host:port and target into test destination.
2075 gabbi.utils.load_yaml(handle=None, yaml_file=None, safe=True)
2077 Read and parse any YAML file or filehandle.
2078 Let exceptions flow where they may.
2080 If no file or handle is provided, read from STDIN.
2082 gabbi.utils.not_binary(content_type)
2084 Decide if something is content we'd like to treat as a string.
2085 gabbi.utils.parse_content_type(content_type, default_charset='utf-8')
2087 Parse content type value for media type and charset.
2088
2090 Gabbi specific exceptions.
2091 exception gabbi.exception.GabbiFormatError
2093 Bases: ValueError
2094 An exception to encapsulate poorly formed test data.
2096 exception gabbi.exception.GabbiSyntaxWarning
2098 Bases: SyntaxWarning
2099 A warning about syntax that is not desirable.
2101
2104 Keep one single global jsonpath parser.
2105 gabbi.json_parser.parse(path)
2107 Parse a JSONPath expression use the global parser.
2108 Gabbi is a tool for running HTTP tests where requests and responses are
2110 expressed as declarations in YAML files:
2111 tests:
2113 - name: retrieve items
2114 GET: /items
2115 See the rest of these docs for more details on the many features and
2117 formats for setting request headers and bodies and evaluating
2118 responses.
2119 Tests can be run from the command line with gabbi-run or programmati‐
2121 cally using either py.test or unittest-style test runners. See
2122 installation instructions below.
2123 The name is derived from "gabby": excessively talkative. In a test
2125 environment having visibility of what a test is actually doing is a
2126 good thing. This is especially true when the goal of a test is to test
2127 the HTTP, not the testing infrastructure. Gabbi tries to put the HTTP
2128 interaction in the foreground of testing.
2129 If you want to get straight to creating tests look at example, the test
2131 files in the source distribution and format. A gabbi-demo repository
2132 provides a tutorial of using gabbi to build an API, via the commit his‐
2133 tory of the repo.
2134
2136 Gabbi works to bridge the gap between human readable YAML files (see
2137 format for details) that represent HTTP requests and expected responses
2138 and the rather complex world of automated testing.
2139 Each YAML file represents an ordered list of HTTP requests along with
2141 the expected responses. This allows a single file to represent a
2142 process in the API being tested. For example:
2143 · Create a resource.
2145 · Retrieve a resource.
2147 · Delete a resource.
2149 · Retrieve a resource again to confirm it is gone.
2151 At the same time it is still possible to ask gabbi to run just one
2153 request. If it is in a sequence of tests, those tests prior to it in
2154 the YAML file will be run (in order). In any single process any test
2155 will only be run once. Concurrency is handled such that one file runs
2156 in one process.
2157 These features mean that it is possible to create tests that are useful
2159 for both humans (as tools for learning, improving and developing APIs)
2160 and automated CI systems.
2161 Significant flexibility and power is available in the format to make it
2163 relatively straightforward to test existing complex APIs. This
2164 extended functionality includes the use of JSONPath to query response
2165 bodies and templating of test data to allow access to the prior HTTP
2166 response in the current request. For APIs which do not use JSON addi‐
2167 tional handlers can be created.
2168 Care should be taken when using this functionality when you are creat‐
2170 ing a new API. If your API is so complex that it needs complex test
2171 files then you may wish to take that as a sign that your API itself too
2172 complex. One goal of gabbi is to encourage transparent and comprehensi‐
2173 ble APIs.
2174 Though gabbi is written in Python and under the covers uses unittest
2176 data structures and processes, there is no requirement that the host be
2177 a Python-based service. Anything talking HTTP can be tested. A runner
2178 makes it possible to simply create YAML files and point them at a run‐
2179 ning server.
2180
2182 As a Python package, gabbi is typically installed via pip:
2183 pip install gabbi
2185 (both Python 2 and Python 3 are supported)
2187 You might want to create a virtual environment; an isolated context for
2189 Python packages, keeping gabbi cleany separated from the rest of your
2190 system.
2191 Python 3 comes with a built-in tool to create virtual environments:
2193 python3 -m venv venv
2195 . venv/bin/activate
2196 pip install gabbi
2198 Alternatively, with Python 2 we can use virtualenv:
2200 pip install virtualenv
2202 virtualenv venv
2203 . venv/bin/activate
2204 pip install gabbi
2206 This way we can later use deactivate and safely remove the venv direc‐
2208 tory, thus erasing any trace of gabbi from the system.
2209
2211 Chris Dent
2212 Jul 29, 2020 GABBI(1)