1GABBI(1) Gabbi GABBI(1)
2
6 gabbi - Gabbi Documentation
7 Gabbi tests are expressed in YAML as a series of HTTP requests with
9 their expected response:
10 tests:
12 - name: retrieve root
13 GET: /
14 status: 200
15 This will trigger a GET request to / on the configured host. The test
17 will pass if the response's status code is 200.
18
20 The top-level tests category contains an ordered sequence of test dec‐
21 larations, each describing the expected response to a given request:
22 Metadata
24 ┌───────────────┬─────────────────────┬───────────────────┐
25 │Key │ Description │ Notes │
26 ├───────────────┼─────────────────────┼───────────────────┤
27 │name │ The test's name. │ required │
28 │ │ Must be unique │ │
29 │ │ within a file. │ │
30 ├───────────────┼─────────────────────┼───────────────────┤
31 │desc │ An arbitrary string │ │
32 │ │ describing the │ │
33 │ │ test. │ │
34 ├───────────────┼─────────────────────┼───────────────────┤
35 │verbose │ If True or all │ defaults to False │
36 │ │ (synonymous), │ │
37 │ │ prints a represen‐ │ │
38 │ │ tation of the cur‐ │ │
39 │ │ rent request and │ │
40 │ │ response to stdout, │ │
41 │ │ including both │ │
42 │ │ headers and body. │ │
43 │ │ If set to headers │ │
44 │ │ or body, only the │ │
45 │ │ corresponding part │ │
46 │ │ of the request and │ │
47 │ │ response will be │ │
48 │ │ printed. If the │ │
49 │ │ output is a TTY, │ │
50 │ │ colors will be │ │
51 │ │ used. If the body │ │
52 │ │ content-type is │ │
53 │ │ JSON it will be │ │
54 │ │ formatted for im‐ │ │
55 │ │ proved readability. │ │
56 │ │ See VerboseHttp for │ │
57 │ │ details. │ │
58 ├───────────────┼─────────────────────┼───────────────────┤
59 │skip │ A string message │ defaults to False │
60 │ │ which if set will │ │
61 │ │ cause the test to │ │
62 │ │ be skipped with the │ │
63 │ │ provided message. │ │
64 └───────────────┴─────────────────────┴───────────────────┘
65 │xfail │ Determines whether │ defaults to False │
68 │ │ to expect this test │ │
69 │ │ to fail. Note that │ │
70 │ │ the test will be │ │
71 │ │ run anyway. │ │
72 ├───────────────┼─────────────────────┼───────────────────┤
73 │use_prior_test │ Determines if this │ defaults to True │
74 │ │ test will be run in │ │
75 │ │ sequence (after) │ │
76 │ │ the test prior to │ │
77 │ │ it in the list of │ │
78 │ │ tests within a │ │
79 │ │ file. To be con‐ │ │
80 │ │ crete, when this is │ │
81 │ │ True the test is │ │
82 │ │ dependent on the │ │
83 │ │ prior test and if │ │
84 │ │ that prior has not │ │
85 │ │ yet run, it wil be │ │
86 │ │ run, even if only │ │
87 │ │ the current test │ │
88 │ │ has been selected. │ │
89 │ │ Set this to False │ │
90 │ │ to allow selecting │ │
91 │ │ a test without de‐ │ │
92 │ │ pendencies. │ │
93 ├───────────────┼─────────────────────┼───────────────────┤
94 │cert_validate │ States whether the │ defaults to True │
95 │ │ underlying HTTP │ │
96 │ │ client should at‐ │ │
97 │ │ tempt to validate │ │
98 │ │ SSL certificates. │ │
99 │ │ In many test envi‐ │ │
100 │ │ ronment certifi‐ │ │
101 │ │ cates will be │ │
102 │ │ self-signed so │ │
103 │ │ changing this may │ │
104 │ │ be requried. It can │ │
105 │ │ also be changed │ │
106 │ │ when loader or us‐ │ │
107 │ │ ing gabbi-run. │ │
108 └───────────────┴─────────────────────┴───────────────────┘
109 NOTE:
111 When tests are generated dynamically, the TestCase name will include
112 the respective test's name, lowercased with spaces transformed to _.
113 In at least some test runners this will allow you to select and fil‐
114 ter on test name.
115 Request Parameters
117 ┌────────────────────┬───────────────────────┬─────────────────────┐
118 │Key │ Description │ Notes │
119 └────────────────────┴───────────────────────┴─────────────────────┘
120 │any uppercase │ Any such key is │ │
134 │string │ considered an HTTP │ │
135 │ │ method, with the │ │
136 │ │ corresponding value │ │
137 │ │ expressing the URL. │ │
138 │ │ │ │
139 │ │ This is a shortcut │ │
140 │ │ combining method │ │
141 │ │ and url into a sin‐ │ │
142 │ │ gle statement: │ │
143 │ │ │ │
144 │ │ GET: /index │ │
145 │ │ │ │
146 │ │ corresponds │ │
147 │ │ to: │ │
148 │ │ │ │
149 │ │ method: GET │ │
150 │ │ url: /index │ │
151 ├────────────────────┼───────────────────────┼─────────────────────┤
152 │method │ The HTTP request │ defaults to GET │
153 │ │ method. │ │
154 ├────────────────────┼───────────────────────┼─────────────────────┤
155 │url │ The URL to request. │ Either this or the │
156 │ │ This can either be a │ shortcut above is │
157 │ │ full path (e.g. "/in‐ │ required │
158 │ │ dex") or a fully │ │
159 │ │ qualified URL (i.e. │ │
160 │ │ including host and │ │
161 │ │ scheme, e.g. "‐ │ │
162 │ │ http://exam‐ │ │
163 │ │ ple.org/index") — see │ │
164 │ │ host for details. │ │
165 ├────────────────────┼───────────────────────┼─────────────────────┤
166 │request_headers │ A dictionary of │ │
167 │ │ key-value pairs rep‐ │ │
168 │ │ resenting request │ │
169 │ │ header names and val‐ │ │
170 │ │ ues. These will be │ │
171 │ │ added to the con‐ │ │
172 │ │ structed request. │ │
173 ├────────────────────┼───────────────────────┼─────────────────────┤
174 │query_parameters │ A dictionary of query │ │
175 │ │ parameters that will │ │
176 │ │ be added to the url │ │
177 │ │ as query string. If │ │
178 │ │ that URL already con‐ │ │
179 │ │ tains a set of query │ │
180 │ │ parameters, those wil │ │
181 │ │ be extended. See ex‐ │ │
182 │ │ ample for a demon‐ │ │
183 │ │ stration of how the │ │
184 │ │ data is structured. │ │
185 ├────────────────────┼───────────────────────┼─────────────────────┤
186 │data │ A representation to │ │
187 │ │ pass as the body of a │ │
188 │ │ request. Note that │ │
189 │ │ content-type in re‐ │ │
190 │ │ quest_headers should │ │
191 │ │ also be set — see │ │
192 │ │ Data for details. │ │
193 ├────────────────────┼───────────────────────┼─────────────────────┤
194 │redirects │ If True, redirects │ defaults to False │
195 │ │ will automatically be │ │
196 │ │ followed. │ │
197 └────────────────────┴───────────────────────┴─────────────────────┘
198 │ssl │ Determines whether │ defaults to False │
200 │ │ the request uses SSL │ │
201 │ │ (i.e. HTTPS). Note │ │
202 │ │ that the url's scheme │ │
203 │ │ takes precedence if │ │
204 │ │ present — see host │ │
205 │ │ for details. │ │
206 └────────────────────┴───────────────────────┴─────────────────────┘
207 Response Expectations
209 ┌────────────────────┬─────────────────────┬─────────────────┐
210 │Key │ Description │ Notes │
211 ├────────────────────┼─────────────────────┼─────────────────┤
212 │status │ The expected re‐ │ defaults to 200 │
213 │ │ sponse status code. │ │
214 │ │ Multiple acceptable │ │
215 │ │ response codes may │ │
216 │ │ be provided, sepa‐ │ │
217 │ │ rated by || (e.g. │ │
218 │ │ 302 || 301 — note, │ │
219 │ │ however, that this │ │
220 │ │ indicates ambigu‐ │ │
221 │ │ ity, which is gen‐ │ │
222 │ │ erally undesir‐ │ │
223 │ │ able). │ │
224 ├────────────────────┼─────────────────────┼─────────────────┤
225 │response_headers │ A dictionary of │ │
226 │ │ key-value pairs │ │
227 │ │ representing ex‐ │ │
228 │ │ pected response │ │
229 │ │ header names and │ │
230 │ │ values. If a │ │
231 │ │ header's value is │ │
232 │ │ wrapped in /.../, │ │
233 │ │ it will be treated │ │
234 │ │ as a regular ex‐ │ │
235 │ │ pression to search │ │
236 │ │ for in the response │ │
237 │ │ header. │ │
238 ├────────────────────┼─────────────────────┼─────────────────┤
239 │response_forbid‐ │ A list of headers │ │
240 │den_headers │ which must not be │ │
241 │ │ present. │ │
242 ├────────────────────┼─────────────────────┼─────────────────┤
243 │response_strings │ A list of string │ │
244 │ │ fragments expected │ │
245 │ │ to be present in │ │
246 │ │ the response body. │ │
247 └────────────────────┴─────────────────────┴─────────────────┘
248 │response_json_paths │ A dictionary of │ │
266 │ │ JSONPath rules │ │
267 │ │ paired with ex‐ │ │
268 │ │ pected matches. Us‐ │ │
269 │ │ ing this rule re‐ │ │
270 │ │ quires that the │ │
271 │ │ content being sent │ │
272 │ │ from the server is │ │
273 │ │ JSON (i.e. a con‐ │ │
274 │ │ tent type of appli‐ │ │
275 │ │ cation/json or con‐ │ │
276 │ │ taining +json) │ │
277 │ │ │ │
278 │ │ If the value is │ │
279 │ │ wrapped in /.../ │ │
280 │ │ the result of the │ │
281 │ │ JSONPath query will │ │
282 │ │ be searched for the │ │
283 │ │ value as a regular │ │
284 │ │ expression. │ │
285 ├────────────────────┼─────────────────────┼─────────────────┤
286 │poll │ A dictionary of two │ │
287 │ │ keys: │ │
288 │ │ │ │
289 │ │ • count: An │ │
290 │ │ integer │ │
291 │ │ stating │ │
292 │ │ the number │ │
293 │ │ of times │ │
294 │ │ to attempt │ │
295 │ │ this test │ │
296 │ │ before │ │
297 │ │ giving up. │ │
298 │ │ │ │
299 │ │ • delay: A │ │
300 │ │ floating │ │
301 │ │ point num‐ │ │
302 │ │ ber of │ │
303 │ │ seconds to │ │
304 │ │ delay be‐ │ │
305 │ │ tween at‐ │ │
306 │ │ tempts. │ │
307 │ │ │ │
308 │ │ This makes │ │
309 │ │ it possible │ │
310 │ │ to poll for │ │
311 │ │ a resource │ │
312 │ │ created via │ │
313 │ │ an asynchro‐ │ │
314 │ │ nous re‐ │ │
315 │ │ quest. Use │ │
316 │ │ with cau‐ │ │
317 │ │ tion. │ │
318 └────────────────────┴─────────────────────┴─────────────────┘
319 Note that many of these items allow substitutions.
321 Default values for a file's tests may be provided via the top-level de‐
323 faults category. These take precedence over the global defaults (ex‐
324 plained below).
325 For examples see the gabbi tests, example and the gabbi-demo tutorial.
327
329 The top-level fixtures category contains a sequence of named fixtures.
330
332 response_* keys are examples of Response Handlers. Custom handlers may
333 be created by test authors for specific use cases. See handlers for
334 more information.
335
337 There are a number of magical variables that can be used to make refer‐
338 ence to the state of a current test, the one just prior or any test
339 prior to the current one. The variables are replaced with real values
340 during test processing.
341 Global
343 • $ENVIRON['<environment variable>']: The name of an environment vari‐
344 able. Its value will replace the magical variable. If the string
345 value of the environment variable is "True" or "False" then the re‐
346 sulting value will be the corresponding boolean, not a string.
347 Current Test
349 • $SCHEME: The current scheme/protocol (usually http or https).
350 • $NETLOC: The host and potentially port of the request.
352 Immediately Prior Test
354 • $COOKIE: All the cookies set by any Set-Cookie headers in the prior
355 response, including only the cookie key and value pairs and no meta‐
356 data (e.g. expires or domain).
357 • $URL: The URL defined in the prior request, after substitutions have
359 been made. For backwards compatibility with earlier releases
360 $LAST_URL may also be used, but if $HISTORY (see below) is being
361 used, $URL must be used.
362 • $LOCATION: The location header returned in the prior response.
364 • $HEADERS['<header>']: The value of any header from the prior re‐
366 sponse.
367 • $RESPONSE['<json path>']: A JSONPath query into the prior response.
369 See jsonpath for more on formatting.
370 Any Previous Test
372 • $HISTORY['<test name>'].<magical variable expression>: Any variable
373 which refers to a prior test may be used in an expression that refers
374 to any earlier test in the same file by identifying the target test
375 by its name in a $HISTORY dictionary. For example, to refer to a
376 value in a JSON object in the response of a test named post json:
377 $HISTORY['post json'].$RESPONSE['$.key']
379 This is a very powerful feature that could lead to test that are dif‐
381 ficult for humans to read. Take care to optimize for the maintainers
382 that will come after you, not yourself.
383 NOTE:
385 Where a single-quote character, ', is shown in the variables above
386 you may also use a double-quote character, ", but in any given ex‐
387 pression the same character must be used at both ends.
388 All of these variables may be used in all of the following fields:
390 • skip
392 • url
394 • query_parameters
396 • data
398 • request_headers (in both the key and value)
400 • response_strings
402 • response_json_paths (in both the key and value, see json path substi‐
404 tution for more info)
405 • response_headers (in both the key and value)
407 • response_forbidden_headers
409 • count and delay fields of poll
411 With these variables it ought to be possible to traverse an API without
413 any explicit statements about the URLs being used. If you need a re‐
414 placement on a field that is not currently supported please raise an
415 issue or provide a patch.
416 As all of these features needed to be tested in the development of
418 gabbi itself, the gabbi tests are a good source of examples on how to
419 use the functionality. See also example for a collection of examples
420 and the gabbi-demo tutorial.
421
423 The data key has some special handing to allow for a bit more flexibil‐
424 ity when doing a POST or PUT:
425 • If the value is not a string (that is, it is a sequence or structure)
427 it is treated as a data structure that will be turned into a string
428 by the dumps method on the relevant content handler. For example if
429 the content-type of the body is application/json the data structure
430 will be turned into a JSON string.
431 • If the value is a string that begins with <@ then the rest of the
433 string is treated as a filepath to be loaded. The path is relative to
434 the test directory and may not traverse up into parent directories.
435 • If the value is an undecorated string, that's the value.
437 NOTE:
439 When reading from a file care should be taken to ensure that a rea‐
440 sonable content-type is set for the data as this will control if any
441 encoding is done of the resulting string value. If it is text, json,
442 xml or javascript it will be encoded to UTF-8.
443 To run gabbi tests with a test harness they must be generated in some
445 fashion and then run. This is accomplished by a test loader. Initially
446 gabbi only supported those test harnesses that supported the load_tests
447 protocol in UnitTest. It now possible to also build and run tests with
448 pytest with some limitations described below.
449 NOTE:
451 It is also possible to run gabbi tests from the command line. See
452 runner.
453 NOTE:
455 By default gabbi will load YAML files using the safe_load function.
456 This means only basic YAML types are allowed in the file. For most
457 use cases this is fine. If you need custom types (for example, to
458 match NaN) it is possible to set the safe_yaml parameter of
459 build_tests() to False. If custom types are used, please keep in
460 mind that this can limit the portability of the YAML files to other
461 contexts.
462 WARNING:
464 If test are being run with a runner that supports concurrency (such
465 as testrepository) it is critical that the test runner is informed
466 of how to group the tests into their respective suites. The usual
467 way to do this is to use a regular expression that groups based on
468 the name of the yaml files. For example, when using testrepository
469 the .testr.conf file needs an entry similar to the following:
470 group_regex=gabbi\.suitemaker\.(test_[^_]+_[^_]+)
472
474 To run the tests with a load_tests style loader a test file containing
475 a load_tests method is required. That will look a bit like:
476 """A sample test module."""
478 # For pathname munging
480 import os
481 # The module that build_tests comes from.
483 from gabbi import driver
484 # We need access to the WSGI application that hosts our service
486 from myapp import wsgiapp
487 # We're using fixtures in the YAML files, we need to know where to
490 # load them from.
491 from myapp.test import fixtures
492 # By convention the YAML files are put in a directory named
494 # "gabbits" that is in the same directory as the Python test file.
495 TESTS_DIR = 'gabbits'
496 def load_tests(loader, tests, pattern):
499 """Provide a TestSuite to the discovery process."""
500 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
501 # Pass "require_ssl=True" as an argument to force all tests
502 # to use SSL in requests.
503 return driver.build_tests(test_dir, loader,
504 intercept=wsgiapp.app,
505 fixture_module=fixtures)
506 For details on the arguments available when building tests see
509 build_tests().
510 Once the test loader has been created, it needs to be run. There are
512 many options. Which is appropriate depends very much on your environ‐
513 ment. Here are some examples using unittest or testtools that require
514 minimal knowledge to get started.
515 By file:
517 python -m testtools.run -v test/test_loader.py
519 By module:
521 python -m testttols.run -v test.test_loader
523 python -m unittest -v test.test_loader
525 Using test discovery to locate all tests in a directory tree:
527 python -m testtools.run discover
529 python -m unittest discover test
531 See the source distribution and the tutorial repo for more advanced op‐
533 tions, including using testrepository and subunit.
534
536 Since pytest does not support the load_tests system, a different way of
537 generating tests is required. Two techniques are supported.
538 The original method (described below) used yield statements to generate
540 tests which pytest would collect. This style of tests is deprecated as
541 of pytest>=3.0 so a new style using pytest fixtures has been developed.
542 pytest >= 3.0
544 In the newer technique, a test file is created that uses the
545 pytest_generate_tests hook. Special care must be taken to always import
546 the test_pytest method which is the base test that the pytest hook
547 parametrizes to generate the tests from the YAML files. Without the
548 method, the hook will not be called and no tests generated.
549 Here is a simple example file:
551 """A sample pytest module for pytest >= 3.0."""
553 # For pathname munging
555 import os
556 # The module that py_test_generator comes from.
558 from gabbi import driver
559 # We need test_pytest so that pytest test collection works properly.
561 # Without this, the pytest_generate_tests method below will not be
562 # called.
563 from gabbi.driver import test_pytest # noqa
564 # We need access to the WSGI application that hosts our service
566 from myapp import wsgiapp
567 # We're using fixtures in the YAML files, we need to know where to
569 # load them from.
570 from myapp.test import fixtures
571 # By convention the YAML files are put in a directory named
573 # "gabbits" that is in the same directory as the Python test file.
574 TESTS_DIR = 'gabbits'
575 def pytest_generate_tests(metafunc):
578 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
579 driver.py_test_generator(
580 test_dir, intercept=wsgiapp.app,
581 fixture_module=fixtures, metafunc=metafunc)
582 This can then be run with the usual pytest commands. For example:
585 py.test -svx pytest3.0-example.py
587 pytest < 3.0
589 When using the older technique, test file must be created that calls
590 py_test_generator() and yields the generated tests. That will look a
591 bit like this:
592 """A sample pytest module."""
594 # For pathname munging
596 import os
597 # The module that build_tests comes from.
599 from gabbi import driver
600 # We need access to the WSGI application that hosts our service
602 from myapp import wsgiapp
603 # We're using fixtures in the YAML files, we need to know where to
605 # load them from.
606 from myapp.test import fixtures
607 # By convention the YAML files are put in a directory named
609 # "gabbits" that is in the same directory as the Python test file.
610 TESTS_DIR = 'gabbits'
611 def test_gabbits():
614 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
615 # Pass "require_ssl=True" as an argument to force all tests
616 # to use SSL in requests.
617 test_generator = driver.py_test_generator(
618 test_dir, intercept=wsgiapp.app,
619 fixture_module=fixtures)
620 for test in test_generator:
622 yield test
623 This can then be run with the usual pytest commands. For example:
626 py.test -svx pytest-example.py
628 The older technique will continue to work with all versions of
630 pytest<4.0 but >=3.0 will produce warnings. If you want to use the
631 older technique but not see the warnings add --disable-pytest-warnings
632 parameter to the invocation of py.test.
633 What follows is a commented example of some tests in a single file
635 demonstrating many of the format features. See loader for the Python
636 needed to integrate with a testing harness.
637 # Fixtures can be used to set any necessary configuration, such as a
640 # persistence layer, and establish sample data. They operate per
641 # file. They are context managers, each one wrapping the next in the
642 # sequence.
643 fixtures:
645 - ConfigFixture
646 - SampleDataFixture
647 # There is an included fixture named "SkipAllFixture" which can be
649 # used to declare that all the tests in the given file are to be
650 # skipped.
651 # Each test file can specify a set of defaults that will be used for
653 # every request. This is useful for always specifying a particular
654 # header or always requiring SSL. These values will be used on every
655 # test in the file unless overriden. Lists and dicts are merged one
656 # level deep, except for "data" which is copied verbatim whether it
657 # is a string, list or dict (it can be all three).
658 defaults:
660 ssl: True
661 request_headers:
662 x-my-token: zoom
663 # The tests themselves are a list under a "tests" key. It's useful
665 # to use plenty of whitespace to help readability.
666 tests:
668 # Each request *must* have a name which is unique to the file. When it
670 # becomes a TestCase the name will be lowercased and spaces will
671 # become "_". Use that generated name when limiting test runs.
672 - name: a test for root
674 desc: Some explanatory text that could be used by other tooling
675 # The URL can either be relative to a host specified elsewhere or
677 # be a fully qualified "http" or "https" URL. *You* are responsible
678 # for url-encoding the URL.
679 url: /
681 method: GET
682 # If no status or method are provided they default to "200" and
684 # "GET".
685 # Instead of explicitly stating "url" and "method" you can join
687 # those two keys into one key representing the method. The method
688 # *must* be uppercase.
689 - name: another test for root
691 desc: Same test as above but with GET key
692 GET: /
693 # A single test can override settings in defaults (set above).
695 - name: root without ssl redirects
697 ssl: False
698 GET: /
699 status: 302
700 # When evaluating response headers it is possible to use a regular
702 # expression to not have to test the whole value. Regular expressions match
703 # anywhere in the output, not just at the beginning.
704 response_headers:
706 location: /^https/
707 # By default redirects will not be followed. This can be changed.
709 - name: follow root without ssl redirect
711 ssl: False
712 redirects: True
713 GET: /
714 status: 200 # This is the response code after the redirect.
715 # URLs can express query parameters in two ways: either in the url
717 # value directly, or as query_parameters. If both are used then
718 # query_parameters are appended. In this example the resulting URL
719 # will be equivalient to
720 # /foo?section=news&article=1&article=2&date=yesterday
721 # but not necessarily in that order.
722 - name: create a url with parameters
724 GET: /foo?section=news
725 query_parameters:
726 article:
727 - 1
728 - 2
729 date: yesterday
730 # Request headers can be used to declare media-type choices and
732 # experiment with authorization handling (amongst other things).
733 # Response headers allow evaluating headers in the response. These
734 # two together form the core value of gabbi.
735 - name: test accept
737 GET: /resource
738 request_headers:
739 accept: application/json
740 response_headers:
741 content-type: /application/json/
742 # If a header must not be present in a response at all that can be
744 # expressed in a test as follows.
745 - name: test forbidden headers
747 GET: /resource
748 response_forbidden_headers:
749 - x-special-header
750 # All of the above requests have defaulted to a "GET" method. When
752 # using "POST", "PUT" or "PATCH", the "data" key provides the
753 # request body.
754 - name: post some text
756 POST: /text_repo
757 request_headers:
758 content-type: text/plain
759 data: "I'm storing this"
760 status: 201
761 # If the data is not a string, it will be transformed into JSON.
763 # You must supply an appropriate content-type request header.
764 - name: post some json
766 POST: /json_repo
767 request_headers:
768 content-type: application/json
769 data:
770 name: smith
771 abode: castle
772 status: 201
773 # If the data is a string prepended with "<@" the value will be
775 # treated as the name of a file in the same directory as the YAML
776 # file. Again, you must supply an appropriate content-type. If the
777 # content-type is one of several "text-like" types, the content will
778 # be assumed to be UTF-8 encoded.
779 - name: post an image
781 POST: /image_repo
782 request_headers:
783 content-type: image/png
784 data: <@kittens.png
785 # A single request can be marked to be skipped.
787 - name: patch an image
789 skip: patching images not yet implemented
790 PATCH: /image_repo/12d96fb8-e78c-11e4-8c03-685b35afa334
791 # Or a single request can be marked that it is expected to fail.
793 - name: check allow headers
795 desc: the framework doesn't do allow yet
796 xfail: True
797 PUT: /post_only_url
798 status: 405
799 response_headers:
800 allow: POST
801 # The body of a response can be evaluated with response handlers.
803 # The most simple checks for a set of strings anywhere in the
804 # response. Note that the strings are members of a list.
805 - name: check for css file
807 GET: /blog/posts/12
808 response_strings:
809 - normalize.css
810 # For JSON responses, JSONPath rules can be used.
812 - name: post some json get back json
814 POST: /json_repo
815 request_headers:
816 content-type: application/json
817 data:
818 name: smith
819 abode: castle
820 status: 201
821 response_json_paths:
822 $.name: smith
823 $.abode: castle
824 # Requests run in sequence. One test can make reference to the test
826 # immediately prior using some special variables.
827 # "$LOCATION" contains the "location" header in the previous
828 # response.
829 # "$HEADERS" is a pseudo dictionary containing all the headers of
830 # the previous response.
831 # "$ENVIRON" is a pseudo dictionary providing access to the current
832 # environment.
833 # "$RESPONSE" provides access to the JSON in the prior response, via
834 # JSONPath. See http://jsonpath-rw.readthedocs.io/ for
835 # jsonpath-rw formatting.
836 # $SCHEME and $NETLOC provide access to the current protocol and
837 # location (host and port).
838 - name: get the thing we just posted
840 GET: $LOCATION
841 request_headers:
842 x-magic-exchange: $HEADERS['x-magic-exchange']
843 x-token: $ENVIRON['OS_TOKEN']
844 response_json_paths:
845 $.name: $RESPONSE['$.name']
846 $.abode: $RESPONSE['$.abode']
847 response_headers:
848 content-location: /$SCHEME://$NETLOC/
849 # For APIs where resource creation is asynchronous it can be
851 # necessary to poll for the resulting resource. First we create the
852 # resource in one test. The next test uses the "poll" key to loop
853 # with a delay for a set number of times.
854 - name: create asynch
856 POST: /async_creator
857 request_headers:
858 content-type: application/json
859 data:
860 name: jones
861 abode: bungalow
862 status: 202
863 - name: poll for created resource
865 GET: $LOCATION
866 poll:
867 count: 10 # try up to ten times
868 delay: .5 # wait .5 seconds between each try
869 response_json_paths:
870 $.name: $RESPONSE['$.name']
871 $.abode: $RESPONSE['$.abode']
872 Gabbi supports JSONPath both for validating JSON response bodies and
875 within substitutions.
876 JSONPath expressions are provided by jsonpath_rw, with jsonpath_rw_ext
878 custom extensions to address common requirements:
879 1. Sorting via sorted and [/property].
881 2. Filtering via [?property = value].
883 3. Returning the respective length via len.
885 (These apply both to arrays and key-value pairs.)
887 Here is a JSONPath example demonstrating some of these features. Given
889 JSON data as follows:
890 {
892 "pets": [
893 {"type": "cat", "sound": "meow"},
894 {"type": "dog", "sound": "woof"}
895 ]
896 }
897 If the ordering of the list in pets is predictable and reliable it is
899 relatively straightforward to test values:
900 response_json_paths:
902 # length of list is two
903 $.pets.`len`: 2
904 # sound of second item in list is woof
905 $.pets[1].sound: woof
906 If the ordering is not predictable additional effort is required:
908 response_json_paths:
910 # sort by type
911 $.pets[/type][0].sound: meow
912 # sort by type, reversed
913 $.pets[\type][0].sound: woof
914 # all the sounds
915 $.pets[/type]..sound: ['meow', 'woof']
916 # filter by type = dog
917 $.pets[?type = "dog"].sound: woof
918 If it is necessary to validate the entire JSON response use a JSONPath
920 of $:
921 response_json_paths:
923 $:
924 pets:
925 - type: cat
926 sound: meow
927 - type: dog
928 sound: woof
929 This is not a technique that should be used frequently as it can lead
931 to difficult to read tests and it also indicates that your gabbi tests
932 are being used to test your serializers and data models, not just your
933 API interactions.
934 It is also possible to read raw JSON from disk for either all or some
936 of a JSON response:
937 response_json_paths:
939 $: @<data.json
940 or:
942 response_json_paths:
944 $.pets: <@pets.json
945 $.pets[0]: <@cat.json
946 Examples like this can be found in one of gabbi's own tests.
948 If it is desired to load YAML files like the JSON ones above, two
950 things must be done:
951 1. The YAMLDiskLoadingJSONHandler custom content handler must be passed
953 to the driver through the content_handlers argument. See Extensions
954 on how to do this.
955 2. The YAML files to load must be placed in a subdirectory to prevent
957 the test runner from consuming them as test files to run:
958 response_json_paths:
960 $: @<subdir/values.yaml
961 When reading from disk you can apply the same JSONPath by adding a ':'
963 to the end of your file name. This allows you to store multiple API re‐
964 sponses into a single file to reduce file management when constructing
965 your tests.
966 Given JSON data as follows:
968 {
970 "values": [{
971 "pets": [{
972 "type": "cat",
973 "sound": "meow"
974 }, {
975 "type": "dog",
976 "sound": "woof"
977 }]
978 }, {
979 "people": [{
980 "name": "chris",
981 "id": 1
982 }, {
983 "name": "justin",
984 "id": 2
985 }]
986 }]
987 }
988 You can write your tests like the following:
990 response_json_paths:
992 $.pets: <@pets.json
993 $.pets[?type = "cat"].sound: <@values.json:$.values[0].pets[?type = "cat"].sound
994 Although placing more than one API response into a single JSON file may
996 seem convenient, keep in mind there is a tradeoff in readability that
997 should not be overlooked before implementing this technique.
998 Examples like this can be found in one of gabbi's yaml-from-disk tests.
1000 There are more JSONPath examples in example and in the jsonpath_rw and
1002 jsonpath_rw_ext documentation.
1003
1005 Substitutions can be made in both the left (query) and right (expected)
1006 hand sides of the json path expression. When subtitutions are used in
1007 the query, care must be taken to ensure proper quoting of the resulting
1008 value. For example if there is a uuid (with hyphens) at $RE‐
1009 SPONSE['$.id'] then this expression may fail:
1010 $.nested.structure.$RESPONSE['$.id'].name: foobar
1012 as it will evaluate to something like:
1014 $.nested.structure.ADC8AAFC-D564-40D1-9724-7680D3C010C2.name: foobar
1016 which may be treated as an arithemtic expression by the json path
1018 parser. The test author should write:
1019 $.nested.structure["$RESPONSE['$.id']"].name: foobar
1021 to quote the result of the substitution.
1023 The target host is the host on which the API to be tested can be found.
1025 Gabbi intends to preserve the flow and semantics of HTTP interactions
1026 as much as possible, and every HTTP request needs to be directed at a
1027 host of some form. Gabbi provides three ways to control this:
1028 • Using wsgi-intercept to provide a fake socket and WSGI environment on
1030 an arbitrary host and port attached to a WSGI application (see
1031 intercept examples).
1032 • Using fully qualified url values in the YAML defined tests (see full
1034 examples).
1035 • Using a host and (optionally) port defined at test build time (see
1037 live examples).
1038 The intercept and live methods are mutually exclusive per test builder,
1040 but either kind of test can freely intermix fully qualified URLs into
1041 the sequence of tests in a YAML file.
1042 For test driven development and local tests the intercept style of
1044 testing lowers test requirements (no web server required) and is fast.
1045 Interception is performed as part of fixtures processing as the most
1046 deeply nested fixture. This allows any configuration or database setup
1047 to be performed prior to the WSGI application being created.
1048 For the implementation of the above see build_tests().
1050 Each suite of tests is represented by a single YAML file, and may op‐
1052 tionally use one or more fixtures to provide the necessary environment
1053 required by the tests in that file.
1054 Fixtures are implemented as nested context managers. Subclasses of
1056 GabbiFixture must implement start_fixture and stop_fixture methods for
1057 creating and destroying, respectively, any resources managed by the
1058 fixture. While the subclass may choose to implement __init__ it is im‐
1059 portant that no exceptions are thrown in that method, otherwise the
1060 stack of context managers will fail in unexpected ways. Instead ini‐
1061 tialization of real resources should happen in start_fixture.
1062 At this time there is no mechanism for the individual tests to have any
1064 direct awareness of the fixtures. The fixtures exist, conceptually, on
1065 the server side of the API being tested.
1066 Fixtures may do whatever is required by the testing environment, how‐
1068 ever there are two common scenarios:
1069 • Establishing (and then resetting when a test suite has finished) any
1071 baseline configuration settings and persistence systems required for
1072 the tests.
1073 • Creating sample data for use by the tests.
1075 If a fixture raises unittest.case.SkipTest during start_fixture all the
1077 tests in the current file will be skipped. This makes it possible to
1078 skip the tests if some optional configuration (such as a particular
1079 type of database) is not available.
1080 If an exception is raised while a fixture is being used, information
1082 about the exception will be stored on the fixture so that the stop_fix‐
1083 ture method can decide if the exception should change how the fixture
1084 should clean up. The exception information can be found on exc_type,
1085 exc_value and traceback method attributes.
1086 If an exception is raised when a fixture is started (in start_fixture)
1088 the first test in the suite using the fixture will be marked with an
1089 error using the traceback from the exception and all the tests in the
1090 suite will be skipped. This ensures that fixture failure is adequately
1091 captured and reported by test runners.
1092 In some contexts (for example CI environments with a large number of
1094 tests being run in a broadly concurrent environment where output is
1095 logged to a single file) it can be important to capture and consolidate
1096 stray output that is produced during the tests and display it associ‐
1097 ated with an individual test. This can help debugging and avoids unus‐
1098 able output that is the result of multiple streams being interleaved.
1099 Inner fixtures have been added to support this. These are fixtures more
1101 in line with the tradtional unittest concept of fixtures: a class on
1102 which setUp and cleanUp is automatically called.
1103 build_tests() accepts a named parameter arguments of inner_fixtures.
1105 The value of that argument may be an ordered list of fixtures.Fixture
1106 classes that will be called when each individual test is set up.
1107 An example fixture that could be useful is the FakeLogger.
1109 NOTE:
1111 At this time inner_fixtures are not supported when using the pytest
1112 loader.
1113 Content handlers are responsible for preparing request data and evalu‐
1115 ating response data based on the content-type of the request and re‐
1116 sponse. A content handler operates as follows:
1117 • Structured YAML data provided via the data attribute is converted to
1119 a string or bytes sequence and used as request body.
1120 • The response body (a string or sequence of bytes) is transformed into
1122 a content-type dependent structure and stored in an internal attri‐
1123 bute named response_data that is:
1124 • used when evaluating the response body
1126 • used in $RESPONSE[] substitutions
1128 By default, gabbi provides content handlers for JSON. In that content
1130 handler the data test key is converted from structured YAML into a JSON
1131 string. Response bodies are converted from a JSON string into a data
1132 structure in response_data that is used when evaluating re‐
1133 sponse_json_paths entries in a test or doing JSONPath-based $RESPONSE[]
1134 substitutions.
1135 A YAMLDiskLoadingJSONHandler has been added to extend the JSON handler.
1137 It works the same way as the JSON handler except for when evaluating
1138 the response_json_paths handle, data that is read from disk can be ei‐
1139 ther in JSON or YAML format. The YAMLDiskLoadingJSONHandler is not en‐
1140 abled by default and must be added as shown in the Extensions section
1141 in order to be used in the tests.
1142 Further content handlers can be added as extensions. Test authors may
1144 need these extensions for their own suites, or enterprising developers
1145 may wish to create and distribute extensions for others to use.
1146 NOTE:
1148 One extension that is likely to be useful is a content handler that
1149 turns data into url-encoded form data suitable for POST and turns an
1150 HTML response into a DOM object.
1151
1153 Content handlers are an evolution of the response handler concept in
1154 earlier versions gabbi. To preserve backwards compatibility with exist‐
1155 ing response handlers, old style response handlers are still allowed,
1156 but new handlers should implement the content handler interface (de‐
1157 scribed below).
1158 Registering additional custom handlers is done by passing a subclass of
1160 ContentHandler to build_tests():
1161 driver.build_tests(test_dir, loader, host=None,
1163 intercept=simple_wsgi.SimpleWsgi,
1164 content_handlers=[MyContentHandler])
1165 If pytest is being used:
1167 driver.py_test_generator(test_dir, intercept=simple_wsgi.SimpleWsgi,
1169 content_handlers=[MyContenHandler])
1170 Gabbi provides an additional custom handler named YAMLDiskLoadingJSON‐
1172 Handler. This can be used for loading JSON and YAML files from disk
1173 when evaluating the response_json_paths handle.
1174 WARNING:
1176 YAMLDiskLoadingJSONHandler shares the same content-type as the de‐
1177 fault JSONHandler. When there are multiple handlers listed that ac‐
1178 cept the same content-type, the one that is earliest in the list
1179 will be used.
1180 With gabbi-run, custom handlers can be loaded via the --response-han‐
1182 dler option -- see load_response_handlers() for details.
1183 NOTE:
1185 The use of the --response-handler argument is done to preserve back‐
1186 wards compatibility and avoid excessive arguments. Both types of
1187 handler may be passed to the argument.
1188 Implementation Details
1190 Creating a content handler requires subclassing ContentHandler and im‐
1191 plementing several methods. These methods are described below, but in‐
1192 specting JSONHandler will be instructive in highlighting required argu‐
1193 ments and techniques.
1194 To provide a response_<something> response-body evaluator a subclass
1196 must define:
1197 • test_key_suffix: This, along with the prefix response_, forms the key
1199 used in the test structure. It is a class level string.
1200 • test_key_value: The key's default value, either an empty list ([]) or
1202 empty dict ({}). It is a class level value.
1203 • action: An instance method which tests the expected values against
1205 the HTTP response - it is invoked for each entry, with the parameters
1206 depending on the default value. The arguments to action are (in or‐
1207 der):
1208 • self: The current instance.
1210 • test: The currently active HTTPTestCase
1212 • item: The current entry if test_key_value is a list, otherwise the
1214 key half of the key/value pair at this entry.
1215 • value: None if test_key_value is a list, otherwise the value half
1217 of the key/value pair at this entry.
1218 To translate request or response bodies to or from structured data a
1220 subclass must define an accepts method. This should return True if this
1221 class is willing to translate the provided content-type. During request
1222 processing it is given the value of the content-type header that will
1223 be sent in the request. During response processing it is given the
1224 value of the content-type header of the response. This makes it possi‐
1225 ble to handle different request and response bodies in the same han‐
1226 dler, if desired. For example a handler might accept applica‐
1227 tion/x-www-form-urlencoded and text/html.
1228 If accepts is defined two additional static methods should be defined:
1230 • dumps: Turn structured Python data from the data key in a test into a
1232 string or byte stream. The optional test param allows you to access
1233 the current test case which may help with manipulations for custom
1234 content handlers, e.g. multipart/form-data needs to add a boundary to
1235 the Content-Type header in order to mark the appropriate sections of
1236 the body.
1237 • loads: Turn a string or byte stream in a response into a Python data
1239 structure. Gabbi will put this data on the response_data attribute on
1240 the test, where it can be used in the evaluations described above (in
1241 the action method) or in $RESPONSE handling. An example usage here
1242 would be to turn HTML into a DOM.
1243 • load_data_file: Load data from disk into a Python data structure.
1245 Gabbi will call this method when response_<something> contains an
1246 item where the right hand side value starts with <@. The test param
1247 allows you to access the current test case and provides a
1248 load_data_file method which should be used because it verifies the
1249 data is loaded within the test diectory and returns the file source
1250 as a string. The load_data_file method was introduced to re-use the
1251 JSONHandler in order to support loading YAML files from disk through
1252 the implementation of an additional custom handler, see YAMLDiskLoad‐
1253 ingJSONHandler for details.
1254 Finally if a replacer class method is defined, then when a $RESPONSE
1256 substitution is encountered, replacer will be passed the response_data
1257 of the prior test and the argument within the $RESPONSE.
1258 Please see the JSONHandler source for additional detail.
1260 If there is a running web service that needs to be tested and creating
1262 a test loader with build_tests() is either inconvenient or overkill it
1263 is possible to run YAML test files directly from the command line with
1264 the console-script gabbi-run. It accepts YAML on stdin or as multiple
1265 file arguments, and generates and runs tests and outputs a summary of
1266 the results.
1267 The provided YAML may not use custom fixtures but otherwise uses the
1269 default format. host information is either expressed directly in the
1270 YAML file or provided on the command line:
1271 gabbi-run [host[:port]] < /my/test.yaml
1273 or:
1275 gabbi-run http://host:port < /my/test.yaml
1277 To test with one or more files the following command syntax may be
1279 used:
1280 gabbi-run http://host:port -- /my/test.yaml /my/other.yaml
1282 NOTE:
1284 The filename arguments must come after a -- and all other arguments
1285 (host, port, prefix, failfast) must come before the --.
1286 NOTE:
1288 If files are provided, test output will use names including the name
1289 of the file. If any single file includes an error, the name of the
1290 file will be included in a summary of failed files at the end of the
1291 test report.
1292 To facilitate using the same tests against the same application mounted
1294 in different locations in a WSGI server, a prefix may be provided as a
1295 second argument:
1296 gabbi-run host[:port] [prefix] < /my/test.yaml
1298 or in the target URL:
1300 gabbi-run http://host:port/prefix < /my/test.yaml
1302 The value of prefix will be prepended to the path portion of URLs that
1304 are not fully qualified.
1305 Anywhere host is used, if it is a raw IPV6 address it should be wrapped
1307 in [ and ].
1308 If https is used in the target, then the tests in the provided YAML
1310 will default to ssl: True.
1311 Use -k or --insecure to not validate certificates when making https
1313 connections.
1314 If a -x or --failfast argument is provided then gabbi-run will exit af‐
1316 ter the first test failure.
1317 Use -v or --verbose with a value of all, headers or body to turn on
1319 verbosity for all tests being run.
1320 Use -q or --quiet to silence test runner output.
1322 These are informal release notes for gabbi since version 1.0.0, high‐
1324 lighting major features and changes. For more detail see the commit
1325 logs on GitHub.
1326
1328 • Allow substitutions in skip for more flexible skipping, notablly with
1329 $ENVIRON.
1330 • Better test name output when running with unittest and verbose.
1332
1334 • Remove support for Python 3.5. Ensure testing support for 3.8, 3.9,
1335 and pypy3.
1336 • Adapt to new behavior in urllib.parse.
1338 • Correct fixture start when using pytest.
1340
1342 • If no content-type is provided with a response and verbose is on for
1343 responses, display the response body as if it were text.
1344
1346 • Properly declare that gabbi 2.x is Python 3 only.
1347
1349 • Correct management of response handler default fields.
1350
1352 • Drop support for Python 2. If you need Python 2 support, use an older
1353 version.
1354 • Stop using testtools and fixtures. These two modules present several
1356 difficulties and their maintenance situation suggests those difficul‐
1357 ties will not be resolved. Since Python 2 support is being removed,
1358 the need for the modules can be removed as well without losing func‐
1359 tionality. "Inner fixtures" that use the fixtures.Fixture interface
1360 should continue to work.
1361
1363 • Add support for not validating certificates in https requests. Con‐
1364 trolled by the cert_validate attribute in individual tests and
1365 build_tests() and the -k or --insecure argument to gabbi-run.
1366
1368 • Support pytest 5.0.0 in Python >=3.5. For earlier versions of Python,
1369 pytest<5.0.0 will be used; the pytest project is dropping support for
1370 older versions of Python.
1371
1373 • Use pytest<5.0.0 until gabbi has solutions for the changes in 5.0.0.
1374
1376 • A -q argument is added to gabbi-run to suppress output from the test
1377 runner.
1378
1380 • Adjust loading of YAML to be ready for new release of PyYAML.
1381
1383 • Provide the YAMLDiskLoadingJSONHandler class that allows test result
1384 data for response_json_path checks to be loaded from YAML-on-disk.
1385
1387 • Use jsonpath to select a portion of data-on-disk in re‐
1388 sponse_json_path checks.
1389 • Restrict PyYAML to <4.0.
1391
1393 • Allow listing of tests with no host configured. When host is an empty
1394 string, tests can be listed (for discovery), but will be skipped on
1395 run.
1396
1398 • JSON $RESPONSE substitutions in the data field may be complex types
1399 (lists and dicts), not solely strings.
1400
1402 • When the HTTP response begins with a bad status line, have BadSta‐
1403 tusLine be raised from urllib3.
1404
1406 • Allow substitutions in the key portion of request and response head‐
1407 ers, not just the value.
1408
1410 • Remove support for Python 3.3.
1411 • Make handling of fixture-level skips in pytest actually work.
1413
1415 • Add safe_yaml parameter to build_tests().
1416
1418 • use_prior_test is added to test metadata.
1419 • Extensive cleanups in regular expression handling when constructing
1421 tests from YAML.
1422
1424 jsonpath handling gets two improvements:
1425 • The value side of a response_json_paths entry can be loaded from a
1427 file using the <@file.json syntax also used in data.
1428 • The key side of a response_json_paths entry can use substitutions.
1430 This was already true for the value side.
1431
1433 Substitutions in $RESPONSE handling now preserve numeric types instead
1434 of casting to a string. This is useful when servers are expecting
1435 strong types and tests want to send response data back to the server.
1436
1438 count and delay test keys allow substitutions.
1439 gabbi.driver.build_tests() accepts a verbose parameter to set test ver‐
1440 bosity for an entire session.
1441
1443 Better failure reporting when using gabbi-run with multiple files. Test
1444 names are based on the files and a summary of failed files is provided
1445 at the end of the report.
1446
1448 Effectively capture a failure in a fixture and report the traceback.
1449 Without this some test runners swallow the error and discovering prob‐
1450 lems when developing fixtures can be quite challenging.
1451
1453 Thanks to Samuel Fekete, tests can use the $HISTORY dictionary to refer
1454 to any prior test in the same file, not just the one immediately prior,
1455 when doing substitutions.
1456
1458 Filenames used to read data into tests using the <@ syntax may now use
1459 pathnames relative to the YAML file. See data.
1460 gabbi-run gains a --verbose parameter to force all tests run in a ses‐
1462 sion to run with verbose set.
1463 When using pytest to load tests, a new mechanism is available which
1465 avoids warnings produced in when using a version of pytest greater than
1466 3.0.
1467
1469 When verbosely displaying request and response bodies that are JSON,
1470 pretty print for improved readability.
1471
1473 Allow gabbi-run to accept multiple filenames as command line arguments
1474 instead of reading tests from stdin.
1475
1477 Switch from response handlers to handlers to allow more flexible pro‐
1478 cessing of both response _and_ request bodies.
1479 Add inner fixtures for per test fixtures, useful for output capturing.
1481
1483 Allow the test_loader_name arg to gabbi.driver.build_tests() to over‐
1484 ride the prefix of the pretty printed name of generated tests.
1485
1487 String values in JSONPath matches may be wrapped in /.../` to be
1488 treated as regular expressions.
1489
1491 Better documentation of how to run gabbi in a concurrent environment.
1492 Improved handling of pytest fixtures and test counts.
1493
1495 Add url to gabbi.driver.build_tests() to use instead of host, port and
1496 prefix.
1497
1499 Add require_ssl to gabbi.driver.build_tests() to force use of SSL.
1500
1502 Add $COOKIE substitution.
1503
1505 Correctly support IPV6 hosts.
1506
1508 Add $LAST_URL substitution.
1509
1511 Introduce support for loading and running tests with pytest.
1512
1514 Use urllib3 instead of httplib2 for driving HTTP requests.
1515
1517 Add sorting and filtering to jsonpath handling.
1518
1520 Add the response_forbidden_headers to response expectations.
1521
1523 Instead of:
1524 tests:
1526 - name: a simple get
1527 url: /some/path
1528 method: get
1529 1.7.0 also makes it possible to:
1531 tests:
1533 - name: a simple get
1534 GET: /some/path
1535 Any upper case key is treated as a method.
1537
1539 Enhanced flexibility and colorization when setting tests to be verbose.
1540
1542 Adds the query_parameters key to request parameters.
1543
1545 The start of improvements and extensions to jsonpath handling. In this
1546 case the addition of the len function.
1547
1549 Vastly improved output and behavior in gabbi-run.
1550
1552 Version 1 was the first release with a commitment to a stable format.
1553 Since then new fields have been added but have not been taken away.
1554 The following people have contributed code to gabbi. Thanks to them.
1556 Thanks also to all the people who have made gabbi better by reporting
1557 issues and their successes and failures with using gabbi.
1558 • Chris Dent
1560 • FND
1562 • Mehdi Abaakouk
1564 • Tom Viner
1566 • Jason Myers
1568 • Josh Leeb-du Toit
1570 • Duc Truong
1572 • Zane Bitter
1574 • Ryan Spencer
1576 • Kim Raymoure
1578 • Travis Truman
1580 • Samuel Fekete
1582 • Michael McCune
1584 • Imran Hayder
1586 • Julien Danjou
1588 • Trevor McCasland
1590 • Danek Duvall
1592 • Marc Abramowitz
1594 NOTE:
1596 This section provides a collection of questions with answers that
1597 don't otherwise fit in the rest of the documentation. If something
1598 is missing, please create an issue.
1599 As this document grows it will gain a more refined structure.
1601
1603 Is gabbi only for testing Python-based APIs?
1604 No, you can use gabbi-run to test an HTTP service built in any program‐
1605 ming language.
1606 How do I run just one test?
1608 Each YAML file contains a sequence of tests, each test within each file
1609 has a name. That name is translated to the name of the test by replac‐
1610 ing spaces with an _.
1611 When running tests that are generated dynamically, filtering based on
1613 the test name prior to the test being collected will not work in some
1614 test runners. Test runners that use a --load-list functionality can be
1615 convinced to filter after discovery.
1616 pytest does this directly with the -k keyword flag.
1618 When using testrepository with tox as used in gabbi's own tests it is
1620 possible to pass a filter in the tox command:
1621 tox -epy27 -- get_the_widget
1623 When using testtools.run and similar test runners it's a bit more com‐
1625 plicated. It is necessary to provide the full name of the test as a
1626 list to --load-list:
1627 python -m testtools.run --load-list \
1629 <(echo package.tests.test_api.yamlfile_get_the_widge.test_request)
1630 How do I run just one test, without running prior tests in a sequence?
1632 By default, when you select a single test to run, all tests prior to
1633 that one in a file will be run as well: the file is treated as as se‐
1634 quence of dependent tests. If you do not want this you can adjust the
1635 use_prior_test test metadata in one of three ways:
1636 • Set it in the YAML file for the one test you are concerned with.
1638 • Set the defaults for all tests in that file.
1640 • set use_prior_test to false when calling build_tests()
1642 Be aware that doing this breaks a fundamental assumption that gabbi
1644 makes about how tests work. Any substitutions will fail.
1645
1647 Can I have variables in my YAML file?
1648 Gabbi provides the $ENVIRON substitution which can operate a bit like
1649 variables that are set elsewhere and then used in the tests defined by
1650 the YAML.
1651 If you find it necessary to have variables within a single YAML file
1653 you take advantage of YAML alias nodes list this:
1654 vars:
1656 - &uuid_1 5613AABF-BAED-4BBA-887A-252B2D3543F8
1657 tests:
1659 - name: send a uuid to a post
1660 POST: /resource
1661 request_headers:
1662 content-type: application/json
1663 data:
1664 uuid: *uuid_1
1665 You can alias all sorts of nodes, not just single items. Be aware that
1667 the replacement of an alias node happens while the YAML is being
1668 loaded, before gabbi does any processing.
1669 How many tests should be put in one YAML file?
1671 For the sake of readability it is best to keep each YAML file rela‐
1672 tively short. Since each YAML file represents a sequence of requests,
1673 it usually makes sense to create a new file when a test is not depen‐
1674 dent on any before it.
1675 It's tempting to put all the tests for any resource or URL in the same
1677 file, but this eventually leads to files that are too long and are thus
1678 difficult to read.
1679
1681 A single HTTP request represented as a subclass of unittest.TestCase
1682 The test case encapsulates the request headers and body and expected
1684 response headers and body. When the test is run an HTTP request is made
1685 using urllib3. Assertions are made against the response.
1686 class gabbi.case.HTTPTestCase(methodName='runTest')
1688 Bases: unittest.case.TestCase
1689 Encapsulate a single HTTP request as a TestCase.
1691 If the test is a member of a sequence of requests, ensure that
1693 prior tests are run.
1694 To keep the test harness happy we need to make sure the setUp
1696 and tearDown are only run once.
1697 assert_in_or_print_output(expected, iterable)
1699 Assert the iterable contains expected or print some out‐
1700 put.
1701 If the output is long, it is limited by either
1703 GABBI_MAX_CHARS_OUTPUT in the environment or the
1704 MAX_CHARS_OUTPUT constant.
1705 base_test = {'cert_validate': True, 'data': '', 'desc': '',
1707 'method': 'GET', 'name': '', 'poll': {}, 'query_parameters': {},
1708 'redirects': False, 'request_headers': {}, 'skip': '', 'ssl':
1709 False, 'status': '200', 'url': '', 'use_prior_test': True, 'ver‐
1710 bose': False, 'xfail': False}
1711 get_content_handler(content_type)
1713 Determine the content handler for this media type.
1714 load_data_file(filename)
1716 Read a file from the current test directory.
1717 replace_template(message, escape_regex=False)
1719 Replace magic strings in message.
1720 run(result=None)
1722 Store the current result handler on this test.
1723 setUp()
1725 Hook method for setting up the test fixture before exer‐
1726 cising it.
1727 shortDescription()
1729 Returns a one-line description of the test, or None if no
1730 description has been provided.
1731 The default implementation of this method returns the
1733 first line of the specified test method's docstring.
1734 tearDown()
1736 Hook method for deconstructing the test fixture after
1737 testing it.
1738 test_request()
1740 Run this request if it has not yet run.
1741 If there is a prior test in the sequence, run it first.
1743 gabbi.case.potentialFailure(func)
1745 Decorate a test method that is expected to fail if 'xfail' is
1746 true.
1747
1751 Manage fixtures for gabbi at the test suite level.
1752 class gabbi.fixture.GabbiFixture
1754 Bases: object
1755 A context manager that operates as a fixture.
1757 Subclasses must implement start_fixture and stop_fixture, each
1759 of which contain the logic for stopping and starting whatever
1760 the fixture is. What a fixture is is left as an exercise for the
1761 implementor.
1762 These context managers will be nested so any actual work needs
1764 to happen in start_fixture and stop_fixture and not in __init__.
1765 Otherwise exception handling will not work properly.
1766 start_fixture()
1768 Implement the actual workings of starting the fixture
1769 here.
1770 stop_fixture()
1772 Implement the actual workings of stopping the fixture
1773 here.
1774 exception gabbi.fixture.GabbiFixtureError
1776 Bases: Exception
1777 Generic exception for GabbiFixture.
1779 class gabbi.fixture.SkipAllFixture
1781 Bases: gabbi.fixture.GabbiFixture
1782 A fixture that skips all the tests in the current suite.
1784 start_fixture()
1786 Implement the actual workings of starting the fixture
1787 here.
1788 gabbi.fixture.nest(fixtures)
1790 Nest a series of fixtures.
1791 This is duplicated from nested in the stdlib, which has been
1793 deprecated because of issues with how exceptions are difficult
1794 to handle during __init__. Gabbi needs to nest an unknown number
1795 of fixtures dynamically, so the with syntax that replaces nested
1796 will not work.
1797
1799 Package for response and content handlers that process the body of a
1800 response in various ways.
1801 handlers.base Module
1803 Base classes for response and content handlers.
1804 class gabbi.handlers.base.ContentHandler
1806 Bases: gabbi.handlers.base.ResponseHandler
1807 A subclass of ResponseHandlers that adds content handling.
1809 static accepts(content_type)
1811 Return True if this handler can handler this type.
1812 static dumps(data, pretty=False, test=None)
1814 Return structured data as a string.
1815 If pretty is true, prettify.
1817 static load_data_file(test, file_path)
1819 Return the string content of the file specified by the
1820 file_path.
1821 static loads(data)
1823 Create structured (Python) data from a stream.
1824 classmethod replacer(response_data, path)
1826 Return the string that is replacing RESPONSE.
1827 class gabbi.handlers.base.ResponseHandler
1829 Bases: object
1830 Add functionality for making assertions about an HTTP response.
1832 A subclass may implement two methods: action and preprocess.
1834 preprocess takes one argument, the TestCase. It is called ex‐
1836 actly once for each test before looping across the assertions.
1837 It is used, rarely, to copy the test.output into a useful form
1838 (such as a parsed DOM).
1839 action takes two or three arguments. If test_key_value is a list
1841 action is called with the test case and a single list item. If
1842 test_key_value is a dict then action is called with the test
1843 case and a key and value pair.
1844 action(test, item, value=None)
1846 Test an individual entry for this response handler.
1847 If the entry is a key value pair the key is in item and
1849 the value in value. Otherwise the entry is considered a
1850 single item from a list.
1851 preprocess(test)
1853 Do any pre-single-test preprocessing.
1854 test_key_suffix = ''
1856 test_key_value = []
1858 handlers.core Module
1860 Core response handlers.
1861 class gabbi.handlers.core.ForbiddenHeadersResponseHandler
1863 Bases: gabbi.handlers.base.ResponseHandler
1864 Test that listed headers are not in the response.
1866 action(test, forbidden, value=None)
1868 Test an individual entry for this response handler.
1869 If the entry is a key value pair the key is in item and
1871 the value in value. Otherwise the entry is considered a
1872 single item from a list.
1873 test_key_suffix = 'forbidden_headers'
1875 test_key_value = []
1877 class gabbi.handlers.core.HeadersResponseHandler
1879 Bases: gabbi.handlers.base.ResponseHandler
1880 Compare expected headers with actual headers.
1882 If a header value is wrapped in / it is treated as a raw regular
1884 expression.
1885 Headers values are always treated as strings.
1887 action(test, header, value=None)
1889 Test an individual entry for this response handler.
1890 If the entry is a key value pair the key is in item and
1892 the value in value. Otherwise the entry is considered a
1893 single item from a list.
1894 test_key_suffix = 'headers'
1896 test_key_value = {}
1898 class gabbi.handlers.core.StringResponseHandler
1900 Bases: gabbi.handlers.base.ResponseHandler
1901 Test for matching strings in the the response body.
1903 action(test, expected, value=None)
1905 Test an individual entry for this response handler.
1906 If the entry is a key value pair the key is in item and
1908 the value in value. Otherwise the entry is considered a
1909 single item from a list.
1910 test_key_suffix = 'strings'
1912 test_key_value = []
1914 handlers.jsonhandler Module
1916 JSON-related content handling.
1917 class gabbi.handlers.jsonhandler.JSONHandler
1919 Bases: gabbi.handlers.base.ContentHandler
1920 A ContentHandler for JSON
1922 • Structured test data is turned into JSON when request con‐
1924 tent-type is JSON.
1925 • Response bodies that are JSON strings are made into Python
1927 data on the test response_data attribute when the response
1928 content-type is JSON.
1929 • A response_json_paths response handler is added.
1931 • JSONPaths in $RESPONSE substitutions are supported.
1933 static accepts(content_type)
1935 Return True if this handler can handler this type.
1936 action(test, path, value=None)
1938 Test json_paths against json data.
1939 static dumps(data, pretty=False, test=None)
1941 Return structured data as a string.
1942 If pretty is true, prettify.
1944 static extract_json_path_value(data, path)
1946 Extract the value at JSON Path path from the data.
1947 The input data is a Python datastructure, not a JSON
1949 string.
1950 static load_data_file(test, file_path)
1952 Return the string content of the file specified by the
1953 file_path.
1954 static loads(data)
1956 Create structured (Python) data from a stream.
1957 classmethod replacer(response_data, match)
1959 Return the string that is replacing RESPONSE.
1960 test_key_suffix = 'json_paths'
1962 test_key_value = {}
1964 handlers.yaml_disk_loading_jsonhandler Module
1966 JSON-related content handling with YAML data disk loading.
1967 class gabbi.handlers.yaml_disk_loading_jsonhandler.YAMLDiskLoadingJSON‐
1969 Handler
1970 Bases: gabbi.handlers.jsonhandler.JSONHandler
1971 A ContentHandler for JSON responses that loads YAML from disk
1973 • Structured test data is turned into JSON when request con‐
1975 tent-type is JSON.
1976 • Response bodies that are JSON strings are made into Python
1978 data on the test response_data attribute when the response
1979 content-type is JSON.
1980 • A response_json_paths response handler is added. Data read
1982 from disk during this handle will be loaded with the
1983 yaml.safe_load method to support both JSON and YAML data
1984 sources from disk.
1985 • JSONPaths in $RESPONSE substitutions are supported.
1987 static load_data_file(test, file_path)
1989 Return the string content of the file specified by the
1990 file_path.
1991
1993 A TestSuite for containing gabbi tests.
1994 This suite has two features: the contained tests are ordered and there
1996 are suite-level fixtures that operate as context managers.
1997 class gabbi.suite.GabbiSuite(tests=())
1999 Bases: unittest.suite.TestSuite
2000 A TestSuite with fixtures.
2002 The suite wraps the tests with a set of nested context managers
2004 that operate as fixtures.
2005 If a fixture raises unittest.case.SkipTest during setup, all the
2007 tests in this suite will be skipped.
2008 run(result, debug=False)
2010 Override TestSuite run to start suite-level fixtures.
2011 To avoid exception confusion, use a null Fixture when
2013 there are no fixtures.
2014 start(result, tests=None)
2016 Start fixtures when using pytest.
2017 stop() Stop fixtures when using pytest.
2019 gabbi.suite.noop(*args)
2021 A noop method used to disable collected tests.
2022
2025 TestRunner and TestResult for gabbi-run.
2026 class gabbi.reporter.ConciseTestResult(stream, descriptions, verbosity)
2028 Bases: unittest.runner.TextTestResult
2029 A TextTestResult with simple but useful output.
2031 If the output is a tty or GABBI_FORCE_COLOR is set in the envi‐
2033 ronment, output will be colorized.
2034 addError(test, err)
2036 Called when an error has occurred. 'err' is a tuple of
2037 values as returned by sys.exc_info().
2038 addExpectedFailure(test, err)
2040 Called when an expected failure/error occurred.
2041 addFailure(test, err)
2043 Called when an error has occurred. 'err' is a tuple of
2044 values as returned by sys.exc_info().
2045 addSkip(test, reason)
2047 Called when a test is skipped.
2048 addSuccess(test)
2050 Called when a test has completed successfully
2051 addUnexpectedSuccess(test)
2053 Called when a test was expected to fail, but succeed.
2054 getDescription(test)
2056 printErrorList(flavor, errors)
2058 startTest(test)
2060 Called when the given test is about to be run
2061 class gabbi.reporter.ConciseTestRunner(stream=None, descriptions=True,
2063 verbosity=1, failfast=False, buffer=False, resultclass=None, warn‐
2064 ings=None, *, tb_locals=False)
2065 Bases: unittest.runner.TextTestRunner
2066 A TextTestRunner that uses ConciseTestResult for reporting re‐
2068 sults.
2069 resultclass
2071 alias of gabbi.reporter.ConciseTestResult
2072 class gabbi.reporter.PyTestResult(stream=None, descriptions=None, ver‐
2074 bosity=None)
2075 Bases: unittest.result.TestResult
2076 Wrap a test result to allow it to work with pytest.
2078 The main behaviors here are:
2080 • to turn what had been exceptions back into exceptions
2082 • use pytest's skip and xfail methods
2084 addError(test, err)
2086 Called when an error has occurred. 'err' is a tuple of
2087 values as returned by sys.exc_info().
2088 addExpectedFailure(test, err)
2090 Called when an expected failure/error occurred.
2091 addFailure(test, err)
2093 Called when an error has occurred. 'err' is a tuple of
2094 values as returned by sys.exc_info().
2095 addSkip(test, reason)
2097 Called when a test is skipped.
2098
2100 Utility functions grab bag.
2101 gabbi.utils.create_url(base_url, host, port=None, prefix='', ssl=False)
2103 Given pieces of a path-based url, return a fully qualified url.
2104 gabbi.utils.decode_response_content(header_dict, content)
2106 Decode content to a proper string.
2107 gabbi.utils.extract_content_type(header_dict, default='application/bi‐
2109 nary')
2110 Extract parsed content-type from headers.
2111 gabbi.utils.get_colorizer(stream)
2113 Return a function to colorize a string.
2114 Only if stream is a tty .
2116 gabbi.utils.host_info_from_target(target, prefix=None)
2118 Turn url or host:port and target into test destination.
2119 gabbi.utils.load_yaml(handle=None, yaml_file=None, safe=True)
2121 Read and parse any YAML file or filehandle.
2122 Let exceptions flow where they may.
2124 If no file or handle is provided, read from STDIN.
2126 gabbi.utils.not_binary(content_type)
2128 Decide if something is content we'd like to treat as a string.
2129 gabbi.utils.parse_content_type(content_type, default_charset='utf-8')
2131 Parse content type value for media type and charset.
2132
2134 Gabbi specific exceptions.
2135 exception gabbi.exception.GabbiFormatError
2137 Bases: ValueError
2138 An exception to encapsulate poorly formed test data.
2140 exception gabbi.exception.GabbiSyntaxWarning
2142 Bases: SyntaxWarning
2143 A warning about syntax that is not desirable.
2145
2148 Keep one single global jsonpath parser.
2149 gabbi.json_parser.parse(path)
2151 Parse a JSONPath expression use the global parser.
2152 Gabbi is a tool for running HTTP tests where requests and responses are
2154 expressed as declarations in YAML files:
2155 tests:
2157 - name: retrieve items
2158 GET: /items
2159 See the rest of these docs for more details on the many features and
2161 formats for setting request headers and bodies and evaluating re‐
2162 sponses.
2163 Tests can be run from the command line with gabbi-run or programmati‐
2165 cally using either py.test or unittest-style test runners. See
2166 installation instructions below.
2167 The name is derived from "gabby": excessively talkative. In a test en‐
2169 vironment having visibility of what a test is actually doing is a good
2170 thing. This is especially true when the goal of a test is to test the
2171 HTTP, not the testing infrastructure. Gabbi tries to put the HTTP in‐
2172 teraction in the foreground of testing.
2173 If you want to get straight to creating tests look at example, the test
2175 files in the source distribution and format. A gabbi-demo repository
2176 provides a tutorial of using gabbi to build an API, via the commit his‐
2177 tory of the repo.
2178
2180 Gabbi works to bridge the gap between human readable YAML files (see
2181 format for details) that represent HTTP requests and expected responses
2182 and the rather complex world of automated testing.
2183 Each YAML file represents an ordered list of HTTP requests along with
2185 the expected responses. This allows a single file to represent a
2186 process in the API being tested. For example:
2187 • Create a resource.
2189 • Retrieve a resource.
2191 • Delete a resource.
2193 • Retrieve a resource again to confirm it is gone.
2195 At the same time it is still possible to ask gabbi to run just one re‐
2197 quest. If it is in a sequence of tests, those tests prior to it in the
2198 YAML file will be run (in order). In any single process any test will
2199 only be run once. Concurrency is handled such that one file runs in one
2200 process.
2201 These features mean that it is possible to create tests that are useful
2203 for both humans (as tools for learning, improving and developing APIs)
2204 and automated CI systems.
2205 Significant flexibility and power is available in the format to make it
2207 relatively straightforward to test existing complex APIs. This ex‐
2208 tended functionality includes the use of JSONPath to query response
2209 bodies and templating of test data to allow access to the prior HTTP
2210 response in the current request. For APIs which do not use JSON addi‐
2211 tional handlers can be created.
2212 Care should be taken when using this functionality when you are creat‐
2214 ing a new API. If your API is so complex that it needs complex test
2215 files then you may wish to take that as a sign that your API itself too
2216 complex. One goal of gabbi is to encourage transparent and comprehensi‐
2217 ble APIs.
2218 Though gabbi is written in Python and under the covers uses unittest
2220 data structures and processes, there is no requirement that the host be
2221 a Python-based service. Anything talking HTTP can be tested. A runner
2222 makes it possible to simply create YAML files and point them at a run‐
2223 ning server.
2224
2226 As a Python package, gabbi is typically installed via pip:
2227 pip install gabbi
2229 You might want to create a virtual environment; an isolated context for
2231 Python packages, keeping gabbi cleany separated from the rest of your
2232 system.
2233 Python 3 comes with a built-in tool to create virtual environments:
2235 python3 -m venv venv
2237 . venv/bin/activate
2238 pip install gabbi
2240 This way we can later use deactivate and safely remove the venv direc‐
2242 tory, thus erasing any trace of gabbi from the system.
2243
2245 Chris Dent
2246 Jul 23, 2021 GABBI(1)