1GABBI(1) Gabbi GABBI(1)
2
6 gabbi - Gabbi Documentation
7 Gabbi tests are expressed in YAML as a series of HTTP requests with
9 their expected response:
10 tests:
12 - name: retrieve root
13 GET: /
14 status: 200
15 This will trigger a GET request to / on the configured Target Host. The
17 test will pass if the response's status code is 200.
18
20 The top-level tests category contains an ordered sequence of test dec‐
21 larations, each describing the expected response to a given request:
22 Metadata
24 ┌───────────────┬─────────────────────┬───────────────────┐
25 │Key │ Description │ Notes │
26 ├───────────────┼─────────────────────┼───────────────────┤
27 │name │ The test's name. │ required │
28 │ │ Must be unique │ │
29 │ │ within a file. │ │
30 ├───────────────┼─────────────────────┼───────────────────┤
31 │desc │ An arbitrary string │ │
32 │ │ describing the │ │
33 │ │ test. │ │
34 ├───────────────┼─────────────────────┼───────────────────┤
35 │verbose │ If True or all │ defaults to False │
36 │ │ (synonymous), │ │
37 │ │ prints a represen‐ │ │
38 │ │ tation of the cur‐ │ │
39 │ │ rent request and │ │
40 │ │ response to stdout, │ │
41 │ │ including both │ │
42 │ │ headers and body. │ │
43 │ │ If set to headers │ │
44 │ │ or body, only the │ │
45 │ │ corresponding part │ │
46 │ │ of the request and │ │
47 │ │ response will be │ │
48 │ │ printed. If the │ │
49 │ │ output is a TTY, │ │
50 │ │ colors will be │ │
51 │ │ used. If the body │ │
52 │ │ content-type is │ │
53 │ │ JSON it will be │ │
54 │ │ formatted for im‐ │ │
55 │ │ proved readability. │ │
56 │ │ See VerboseHttp for │ │
57 │ │ details. │ │
58 ├───────────────┼─────────────────────┼───────────────────┤
59 │skip │ A string message │ defaults to False │
60 │ │ which if set will │ │
61 │ │ cause the test to │ │
62 │ │ be skipped with the │ │
63 │ │ provided message. │ │
64 └───────────────┴─────────────────────┴───────────────────┘
65 │xfail │ Determines whether │ defaults to False │
68 │ │ to expect this test │ │
69 │ │ to fail. Note that │ │
70 │ │ the test will be │ │
71 │ │ run anyway. │ │
72 ├───────────────┼─────────────────────┼───────────────────┤
73 │use_prior_test │ Determines if this │ defaults to True │
74 │ │ test will be run in │ │
75 │ │ sequence (after) │ │
76 │ │ the test prior to │ │
77 │ │ it in the list of │ │
78 │ │ tests within a │ │
79 │ │ file. To be con‐ │ │
80 │ │ crete, when this is │ │
81 │ │ True the test is │ │
82 │ │ dependent on the │ │
83 │ │ prior test and if │ │
84 │ │ that prior has not │ │
85 │ │ yet run, it wil be │ │
86 │ │ run, even if only │ │
87 │ │ the current test │ │
88 │ │ has been selected. │ │
89 │ │ Set this to False │ │
90 │ │ to allow selecting │ │
91 │ │ a test without de‐ │ │
92 │ │ pendencies. │ │
93 ├───────────────┼─────────────────────┼───────────────────┤
94 │cert_validate │ States whether the │ defaults to True │
95 │ │ underlying HTTP │ │
96 │ │ client should at‐ │ │
97 │ │ tempt to validate │ │
98 │ │ SSL certificates. │ │
99 │ │ In many test envi‐ │ │
100 │ │ ronment certifi‐ │ │
101 │ │ cates will be │ │
102 │ │ self-signed so │ │
103 │ │ changing this may │ │
104 │ │ be requried. It can │ │
105 │ │ also be changed │ │
106 │ │ when Loading and │ │
107 │ │ Running Tests or │ │
108 │ │ using gabbi-run. │ │
109 └───────────────┴─────────────────────┴───────────────────┘
110 NOTE:
112 When tests are generated dynamically, the TestCase name will include
113 the respective test's name, lowercased with spaces transformed to _.
114 In at least some test runners this will allow you to select and fil‐
115 ter on test name.
116 Request Parameters
118 ┌────────────────────┬───────────────────────┬─────────────────────┐
119 │Key │ Description │ Notes │
120 └────────────────────┴───────────────────────┴─────────────────────┘
121 │any uppercase │ Any such key is │ │
134 │string │ considered an HTTP │ │
135 │ │ method, with the │ │
136 │ │ corresponding value │ │
137 │ │ expressing the URL. │ │
138 │ │ │ │
139 │ │ This is a shortcut │ │
140 │ │ combining method │ │
141 │ │ and url into a sin‐ │ │
142 │ │ gle statement: │ │
143 │ │ │ │
144 │ │ GET: /index │ │
145 │ │ │ │
146 │ │ corresponds │ │
147 │ │ to: │ │
148 │ │ │ │
149 │ │ method: GET │ │
150 │ │ url: /index │ │
151 ├────────────────────┼───────────────────────┼─────────────────────┤
152 │method │ The HTTP request │ defaults to GET │
153 │ │ method. │ │
154 ├────────────────────┼───────────────────────┼─────────────────────┤
155 │url │ The URL to request. │ Either this or the │
156 │ │ This can either be a │ shortcut above is │
157 │ │ full path (e.g. "/in‐ │ required │
158 │ │ dex") or a fully │ │
159 │ │ qualified URL (i.e. │ │
160 │ │ including host and │ │
161 │ │ scheme, e.g. "‐ │ │
162 │ │ http://exam‐ │ │
163 │ │ ple.org/index") — see │ │
164 │ │ Target Host for de‐ │ │
165 │ │ tails. │ │
166 ├────────────────────┼───────────────────────┼─────────────────────┤
167 │request_headers │ A dictionary of │ │
168 │ │ key-value pairs rep‐ │ │
169 │ │ resenting request │ │
170 │ │ header names and val‐ │ │
171 │ │ ues. These will be │ │
172 │ │ added to the con‐ │ │
173 │ │ structed request. │ │
174 ├────────────────────┼───────────────────────┼─────────────────────┤
175 │query_parameters │ A dictionary of query │ │
176 │ │ parameters that will │ │
177 │ │ be added to the url │ │
178 │ │ as query string. If │ │
179 │ │ that URL already con‐ │ │
180 │ │ tains a set of query │ │
181 │ │ parameters, those wil │ │
182 │ │ be extended. See │ │
183 │ │ Example Tests for a │ │
184 │ │ demonstration of how │ │
185 │ │ the data is struc‐ │ │
186 │ │ tured. │ │
187 ├────────────────────┼───────────────────────┼─────────────────────┤
188 │data │ A representation to │ │
189 │ │ pass as the body of a │ │
190 │ │ request. Note that │ │
191 │ │ content-type in re‐ │ │
192 │ │ quest_headers should │ │
193 │ │ also be set — see │ │
194 │ │ Data for details. │ │
195 └────────────────────┴───────────────────────┴─────────────────────┘
196 │redirects │ If True, redirects │ defaults to False │
200 │ │ will automatically be │ │
201 │ │ followed. │ │
202 ├────────────────────┼───────────────────────┼─────────────────────┤
203 │ssl │ Determines whether │ defaults to False │
204 │ │ the request uses SSL │ │
205 │ │ (i.e. HTTPS). Note │ │
206 │ │ that the url's scheme │ │
207 │ │ takes precedence if │ │
208 │ │ present — see Target │ │
209 │ │ Host for details. │ │
210 └────────────────────┴───────────────────────┴─────────────────────┘
211 Response Expectations
213 ┌────────────────────┬─────────────────────┬─────────────────┐
214 │Key │ Description │ Notes │
215 ├────────────────────┼─────────────────────┼─────────────────┤
216 │status │ The expected re‐ │ defaults to 200 │
217 │ │ sponse status code. │ │
218 │ │ Multiple acceptable │ │
219 │ │ response codes may │ │
220 │ │ be provided, sepa‐ │ │
221 │ │ rated by || (e.g. │ │
222 │ │ 302 || 301 — note, │ │
223 │ │ however, that this │ │
224 │ │ indicates ambigu‐ │ │
225 │ │ ity, which is gen‐ │ │
226 │ │ erally undesir‐ │ │
227 │ │ able). │ │
228 ├────────────────────┼─────────────────────┼─────────────────┤
229 │response_headers │ A dictionary of │ │
230 │ │ key-value pairs │ │
231 │ │ representing ex‐ │ │
232 │ │ pected response │ │
233 │ │ header names and │ │
234 │ │ values. If a │ │
235 │ │ header's value is │ │
236 │ │ wrapped in /.../, │ │
237 │ │ it will be treated │ │
238 │ │ as a regular ex‐ │ │
239 │ │ pression to search │ │
240 │ │ for in the response │ │
241 │ │ header. │ │
242 ├────────────────────┼─────────────────────┼─────────────────┤
243 │response_forbid‐ │ A list of headers │ │
244 │den_headers │ which must not be │ │
245 │ │ present. │ │
246 ├────────────────────┼─────────────────────┼─────────────────┤
247 │response_strings │ A list of string │ │
248 │ │ fragments expected │ │
249 │ │ to be present in │ │
250 │ │ the response body. │ │
251 └────────────────────┴─────────────────────┴─────────────────┘
252 │response_json_paths │ A dictionary of │ │
266 │ │ JSONPath rules │ │
267 │ │ paired with ex‐ │ │
268 │ │ pected matches. Us‐ │ │
269 │ │ ing this rule re‐ │ │
270 │ │ quires that the │ │
271 │ │ content being sent │ │
272 │ │ from the server is │ │
273 │ │ JSON (i.e. a con‐ │ │
274 │ │ tent type of appli‐ │ │
275 │ │ cation/json or con‐ │ │
276 │ │ taining +json) │ │
277 │ │ │ │
278 │ │ If the value is │ │
279 │ │ wrapped in /.../ │ │
280 │ │ the result of the │ │
281 │ │ JSONPath query will │ │
282 │ │ be searched for the │ │
283 │ │ value as a regular │ │
284 │ │ expression. │ │
285 ├────────────────────┼─────────────────────┼─────────────────┤
286 │poll │ A dictionary of two │ │
287 │ │ keys: │ │
288 │ │ │ │
289 │ │ • count: An │ │
290 │ │ integer │ │
291 │ │ stating │ │
292 │ │ the number │ │
293 │ │ of times │ │
294 │ │ to attempt │ │
295 │ │ this test │ │
296 │ │ before │ │
297 │ │ giving up. │ │
298 │ │ │ │
299 │ │ • delay: A │ │
300 │ │ floating │ │
301 │ │ point num‐ │ │
302 │ │ ber of │ │
303 │ │ seconds to │ │
304 │ │ delay be‐ │ │
305 │ │ tween at‐ │ │
306 │ │ tempts. │ │
307 │ │ │ │
308 │ │ This makes │ │
309 │ │ it possible │ │
310 │ │ to poll for │ │
311 │ │ a resource │ │
312 │ │ created via │ │
313 │ │ an asynchro‐ │ │
314 │ │ nous re‐ │ │
315 │ │ quest. Use │ │
316 │ │ with cau‐ │ │
317 │ │ tion. │ │
318 └────────────────────┴─────────────────────┴─────────────────┘
319 Note that many of these items allow substitutions.
321 Default values for a file's tests may be provided via the top-level de‐
323 faults category. These take precedence over the global defaults (ex‐
324 plained below).
325 For examples see the gabbi tests, Example Tests and the gabbi-demo tu‐
327 torial.
328
330 The top-level fixtures category contains a sequence of named Fixtures.
331
333 response_* keys are examples of Response Handlers. Custom handlers may
334 be created by test authors for specific use cases. See Content Handlers
335 for more information.
336
338 There are a number of magical variables that can be used to make refer‐
339 ence to the state of a current test, the one just prior or any test
340 prior to the current one. The variables are replaced with real values
341 during test processing.
342 Global
344 • $ENVIRON['<environment variable>']: The name of an environment vari‐
345 able. Its value will replace the magical variable. If the string
346 value of the environment variable is "True" or "False" then the re‐
347 sulting value will be the corresponding boolean, not a string.
348 Current Test
350 • $SCHEME: The current scheme/protocol (usually http or https).
351 • $NETLOC: The host and potentially port of the request.
353 Immediately Prior Test
355 • $COOKIE: All the cookies set by any Set-Cookie headers in the prior
356 response, including only the cookie key and value pairs and no meta‐
357 data (e.g. expires or domain).
358 • $URL: The URL defined in the prior request, after substitutions have
360 been made. For backwards compatibility with earlier releases
361 $LAST_URL may also be used, but if $HISTORY (see below) is being
362 used, $URL must be used.
363 • $LOCATION: The location header returned in the prior response.
365 • $HEADERS['<header>']: The value of any header from the prior re‐
367 sponse.
368 • $RESPONSE['<json path>']: A JSONPath query into the prior response.
370 See JSONPath for more on formatting.
371 Any Previous Test
373 • $HISTORY['<test name>'].<magical variable expression>: Any variable
374 which refers to a prior test may be used in an expression that refers
375 to any earlier test in the same file by identifying the target test
376 by its name in a $HISTORY dictionary. For example, to refer to a
377 value in a JSON object in the response of a test named post json:
378 $HISTORY['post json'].$RESPONSE['$.key']
380 This is a very powerful feature that could lead to test that are dif‐
382 ficult for humans to read. Take care to optimize for the maintainers
383 that will come after you, not yourself.
384 NOTE:
386 Where a single-quote character, ', is shown in the variables above
387 you may also use a double-quote character, ", but in any given ex‐
388 pression the same character must be used at both ends.
389 All of these variables may be used in all of the following fields:
391 • skip
393 • url
395 • query_parameters
397 • data
399 • request_headers (in both the key and value)
401 • response_strings
403 • response_json_paths (in both the key and value, see json path substi‐
405 tution for more info)
406 • response_headers (in both the key and value)
408 • response_forbidden_headers
410 • count and delay fields of poll
412 With these variables it ought to be possible to traverse an API without
414 any explicit statements about the URLs being used. If you need a re‐
415 placement on a field that is not currently supported please raise an
416 issue or provide a patch.
417 As all of these features needed to be tested in the development of
419 gabbi itself, the gabbi tests are a good source of examples on how to
420 use the functionality. See also Example Tests for a collection of exam‐
421 ples and the gabbi-demo tutorial.
422
424 The data key has some special handing to allow for a bit more flexibil‐
425 ity when doing a POST or PUT:
426 • If the value is not a string (that is, it is a sequence or structure)
428 it is treated as a data structure that will be turned into a string
429 by the dumps method on the relevant content handler. For example if
430 the content-type of the body is application/json the data structure
431 will be turned into a JSON string.
432 • If the value is a string that begins with <@ then the rest of the
434 string is treated as a filepath to be loaded. The path is relative to
435 the test directory and may not traverse up into parent directories.
436 • If the value is an undecorated string, that's the value.
438 NOTE:
440 When reading from a file care should be taken to ensure that a rea‐
441 sonable content-type is set for the data as this will control if any
442 encoding is done of the resulting string value. If it is text, json,
443 xml or javascript it will be encoded to UTF-8.
444 To run gabbi tests with a test harness they must be generated in some
446 fashion and then run. This is accomplished by a test loader. Initially
447 gabbi only supported those test harnesses that supported the load_tests
448 protocol in UnitTest. It now possible to also build and run tests with
449 pytest with some limitations described below.
450 NOTE:
452 It is also possible to run gabbi tests from the command line. See
453 YAML Runner.
454 NOTE:
456 By default gabbi will load YAML files using the safe_load function.
457 This means only basic YAML types are allowed in the file. For most
458 use cases this is fine. If you need custom types (for example, to
459 match NaN) it is possible to set the safe_yaml parameter of
460 build_tests() to False. If custom types are used, please keep in
461 mind that this can limit the portability of the YAML files to other
462 contexts.
463 WARNING:
465 If test are being run with a runner that supports concurrency (such
466 as testrepository) it is critical that the test runner is informed
467 of how to group the tests into their respective suites. The usual
468 way to do this is to use a regular expression that groups based on
469 the name of the yaml files. For example, when using testrepository
470 the .testr.conf file needs an entry similar to the following:
471 group_regex=gabbi\.suitemaker\.(test_[^_]+_[^_]+)
473
475 To run the tests with a load_tests style loader a test file containing
476 a load_tests method is required. That will look a bit like:
477 """A sample test module."""
479 # For pathname munging
481 import os
482 # The module that build_tests comes from.
484 from gabbi import driver
485 # We need access to the WSGI application that hosts our service
487 from myapp import wsgiapp
488 # We're using fixtures in the YAML files, we need to know where to
491 # load them from.
492 from myapp.test import fixtures
493 # By convention the YAML files are put in a directory named
495 # "gabbits" that is in the same directory as the Python test file.
496 TESTS_DIR = 'gabbits'
497 def load_tests(loader, tests, pattern):
500 """Provide a TestSuite to the discovery process."""
501 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
502 # Pass "require_ssl=True" as an argument to force all tests
503 # to use SSL in requests.
504 return driver.build_tests(test_dir, loader,
505 intercept=wsgiapp.app,
506 fixture_module=fixtures)
507 For details on the arguments available when building tests see
510 build_tests().
511 Once the test loader has been created, it needs to be run. There are
513 many options. Which is appropriate depends very much on your environ‐
514 ment. Here are some examples using unittest or testtools that require
515 minimal knowledge to get started.
516 By file:
518 python -m testtools.run -v test/test_loader.py
520 By module:
522 python -m testttols.run -v test.test_loader
524 python -m unittest -v test.test_loader
526 Using test discovery to locate all tests in a directory tree:
528 python -m testtools.run discover
530 python -m unittest discover test
532 See the source distribution and the tutorial repo for more advanced op‐
534 tions, including using testrepository and subunit.
535
537 Since pytest does not support the load_tests system, a different way of
538 generating tests is required. Two techniques are supported.
539 The original method (described below) used yield statements to generate
541 tests which pytest would collect. This style of tests is deprecated as
542 of pytest>=3.0 so a new style using pytest fixtures has been developed.
543 pytest >= 3.0
545 In the newer technique, a test file is created that uses the
546 pytest_generate_tests hook. Special care must be taken to always import
547 the test_pytest method which is the base test that the pytest hook
548 parametrizes to generate the tests from the YAML files. Without the
549 method, the hook will not be called and no tests generated.
550 Here is a simple example file:
552 """A sample pytest module for pytest >= 3.0."""
554 # For pathname munging
556 import os
557 # The module that py_test_generator comes from.
559 from gabbi import driver
560 # We need test_pytest so that pytest test collection works properly.
562 # Without this, the pytest_generate_tests method below will not be
563 # called.
564 from gabbi.driver import test_pytest # noqa
565 # We need access to the WSGI application that hosts our service
567 from myapp import wsgiapp
568 # We're using fixtures in the YAML files, we need to know where to
570 # load them from.
571 from myapp.test import fixtures
572 # By convention the YAML files are put in a directory named
574 # "gabbits" that is in the same directory as the Python test file.
575 TESTS_DIR = 'gabbits'
576 def pytest_generate_tests(metafunc):
579 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
580 driver.py_test_generator(
581 test_dir, intercept=wsgiapp.app,
582 fixture_module=fixtures, metafunc=metafunc)
583 This can then be run with the usual pytest commands. For example:
586 py.test -svx pytest3.0-example.py
588 pytest < 3.0
590 When using the older technique, test file must be created that calls
591 py_test_generator() and yields the generated tests. That will look a
592 bit like this:
593 """A sample pytest module."""
595 # For pathname munging
597 import os
598 # The module that build_tests comes from.
600 from gabbi import driver
601 # We need access to the WSGI application that hosts our service
603 from myapp import wsgiapp
604 # We're using fixtures in the YAML files, we need to know where to
606 # load them from.
607 from myapp.test import fixtures
608 # By convention the YAML files are put in a directory named
610 # "gabbits" that is in the same directory as the Python test file.
611 TESTS_DIR = 'gabbits'
612 def test_gabbits():
615 test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
616 # Pass "require_ssl=True" as an argument to force all tests
617 # to use SSL in requests.
618 test_generator = driver.py_test_generator(
619 test_dir, intercept=wsgiapp.app,
620 fixture_module=fixtures)
621 for test in test_generator:
623 yield test
624 This can then be run with the usual pytest commands. For example:
627 py.test -svx pytest-example.py
629 The older technique will continue to work with all versions of
631 pytest<4.0 but >=3.0 will produce warnings. If you want to use the
632 older technique but not see the warnings add --disable-pytest-warnings
633 parameter to the invocation of py.test.
634 What follows is a commented example of some tests in a single file
636 demonstrating many of the Test Format features. See Loading and Running
637 Tests for the Python needed to integrate with a testing harness.
638 # Fixtures can be used to set any necessary configuration, such as a
641 # persistence layer, and establish sample data. They operate per
642 # file. They are context managers, each one wrapping the next in the
643 # sequence.
644 fixtures:
646 - ConfigFixture
647 - SampleDataFixture
648 # There is an included fixture named "SkipAllFixture" which can be
650 # used to declare that all the tests in the given file are to be
651 # skipped.
652 # Each test file can specify a set of defaults that will be used for
654 # every request. This is useful for always specifying a particular
655 # header or always requiring SSL. These values will be used on every
656 # test in the file unless overriden. Lists and dicts are merged one
657 # level deep, except for "data" which is copied verbatim whether it
658 # is a string, list or dict (it can be all three).
659 defaults:
661 ssl: True
662 request_headers:
663 x-my-token: zoom
664 # The tests themselves are a list under a "tests" key. It's useful
666 # to use plenty of whitespace to help readability.
667 tests:
669 # Each request *must* have a name which is unique to the file. When it
671 # becomes a TestCase the name will be lowercased and spaces will
672 # become "_". Use that generated name when limiting test runs.
673 - name: a test for root
675 desc: Some explanatory text that could be used by other tooling
676 # The URL can either be relative to a host specified elsewhere or
678 # be a fully qualified "http" or "https" URL. *You* are responsible
679 # for url-encoding the URL.
680 url: /
682 method: GET
683 # If no status or method are provided they default to "200" and
685 # "GET".
686 # Instead of explicitly stating "url" and "method" you can join
688 # those two keys into one key representing the method. The method
689 # *must* be uppercase.
690 - name: another test for root
692 desc: Same test as above but with GET key
693 GET: /
694 # A single test can override settings in defaults (set above).
696 - name: root without ssl redirects
698 ssl: False
699 GET: /
700 status: 302
701 # When evaluating response headers it is possible to use a regular
703 # expression to not have to test the whole value. Regular expressions match
704 # anywhere in the output, not just at the beginning.
705 response_headers:
707 location: /^https/
708 # By default redirects will not be followed. This can be changed.
710 - name: follow root without ssl redirect
712 ssl: False
713 redirects: True
714 GET: /
715 status: 200 # This is the response code after the redirect.
716 # URLs can express query parameters in two ways: either in the url
718 # value directly, or as query_parameters. If both are used then
719 # query_parameters are appended. In this example the resulting URL
720 # will be equivalient to
721 # /foo?section=news&article=1&article=2&date=yesterday
722 # but not necessarily in that order.
723 - name: create a url with parameters
725 GET: /foo?section=news
726 query_parameters:
727 article:
728 - 1
729 - 2
730 date: yesterday
731 # Request headers can be used to declare media-type choices and
733 # experiment with authorization handling (amongst other things).
734 # Response headers allow evaluating headers in the response. These
735 # two together form the core value of gabbi.
736 - name: test accept
738 GET: /resource
739 request_headers:
740 accept: application/json
741 response_headers:
742 content-type: /application/json/
743 # If a header must not be present in a response at all that can be
745 # expressed in a test as follows.
746 - name: test forbidden headers
748 GET: /resource
749 response_forbidden_headers:
750 - x-special-header
751 # All of the above requests have defaulted to a "GET" method. When
753 # using "POST", "PUT" or "PATCH", the "data" key provides the
754 # request body.
755 - name: post some text
757 POST: /text_repo
758 request_headers:
759 content-type: text/plain
760 data: "I'm storing this"
761 status: 201
762 # If the data is not a string, it will be transformed into JSON.
764 # You must supply an appropriate content-type request header.
765 - name: post some json
767 POST: /json_repo
768 request_headers:
769 content-type: application/json
770 data:
771 name: smith
772 abode: castle
773 status: 201
774 # If the data is a string prepended with "<@" the value will be
776 # treated as the name of a file in the same directory as the YAML
777 # file. Again, you must supply an appropriate content-type. If the
778 # content-type is one of several "text-like" types, the content will
779 # be assumed to be UTF-8 encoded.
780 - name: post an image
782 POST: /image_repo
783 request_headers:
784 content-type: image/png
785 data: <@kittens.png
786 # A single request can be marked to be skipped.
788 - name: patch an image
790 skip: patching images not yet implemented
791 PATCH: /image_repo/12d96fb8-e78c-11e4-8c03-685b35afa334
792 # Or a single request can be marked that it is expected to fail.
794 - name: check allow headers
796 desc: the framework doesn't do allow yet
797 xfail: True
798 PUT: /post_only_url
799 status: 405
800 response_headers:
801 allow: POST
802 # The body of a response can be evaluated with response handlers.
804 # The most simple checks for a set of strings anywhere in the
805 # response. Note that the strings are members of a list.
806 - name: check for css file
808 GET: /blog/posts/12
809 response_strings:
810 - normalize.css
811 # For JSON responses, JSONPath rules can be used.
813 - name: post some json get back json
815 POST: /json_repo
816 request_headers:
817 content-type: application/json
818 data:
819 name: smith
820 abode: castle
821 status: 201
822 response_json_paths:
823 $.name: smith
824 $.abode: castle
825 # Requests run in sequence. One test can make reference to the test
827 # immediately prior using some special variables.
828 # "$LOCATION" contains the "location" header in the previous
829 # response.
830 # "$HEADERS" is a pseudo dictionary containing all the headers of
831 # the previous response.
832 # "$ENVIRON" is a pseudo dictionary providing access to the current
833 # environment.
834 # "$RESPONSE" provides access to the JSON in the prior response, via
835 # JSONPath. See http://jsonpath-rw.readthedocs.io/ for
836 # jsonpath-rw formatting.
837 # $SCHEME and $NETLOC provide access to the current protocol and
838 # location (host and port).
839 - name: get the thing we just posted
841 GET: $LOCATION
842 request_headers:
843 x-magic-exchange: $HEADERS['x-magic-exchange']
844 x-token: $ENVIRON['OS_TOKEN']
845 response_json_paths:
846 $.name: $RESPONSE['$.name']
847 $.abode: $RESPONSE['$.abode']
848 response_headers:
849 content-location: /$SCHEME://$NETLOC/
850 # For APIs where resource creation is asynchronous it can be
852 # necessary to poll for the resulting resource. First we create the
853 # resource in one test. The next test uses the "poll" key to loop
854 # with a delay for a set number of times.
855 - name: create asynch
857 POST: /async_creator
858 request_headers:
859 content-type: application/json
860 data:
861 name: jones
862 abode: bungalow
863 status: 202
864 - name: poll for created resource
866 GET: $LOCATION
867 poll:
868 count: 10 # try up to ten times
869 delay: .5 # wait .5 seconds between each try
870 response_json_paths:
871 $.name: $RESPONSE['$.name']
872 $.abode: $RESPONSE['$.abode']
873 Gabbi supports JSONPath both for validating JSON response bodies and
876 within substitutions.
877 JSONPath expressions are provided by jsonpath_rw, with jsonpath_rw_ext
879 custom extensions to address common requirements:
880 1. Sorting via sorted and [/property].
882 2. Filtering via [?property = value].
884 3. Returning the respective length via len.
886 (These apply both to arrays and key-value pairs.)
888 Here is a JSONPath example demonstrating some of these features. Given
890 JSON data as follows:
891 {
893 "pets": [
894 {"type": "cat", "sound": "meow"},
895 {"type": "dog", "sound": "woof"}
896 ]
897 }
898 If the ordering of the list in pets is predictable and reliable it is
900 relatively straightforward to test values:
901 response_json_paths:
903 # length of list is two
904 $.pets.`len`: 2
905 # sound of second item in list is woof
906 $.pets[1].sound: woof
907 If the ordering is not predictable additional effort is required:
909 response_json_paths:
911 # sort by type
912 $.pets[/type][0].sound: meow
913 # sort by type, reversed
914 $.pets[\type][0].sound: woof
915 # all the sounds
916 $.pets[/type]..sound: ['meow', 'woof']
917 # filter by type = dog
918 $.pets[?type = "dog"].sound: woof
919 If it is necessary to validate the entire JSON response use a JSONPath
921 of $:
922 response_json_paths:
924 $:
925 pets:
926 - type: cat
927 sound: meow
928 - type: dog
929 sound: woof
930 This is not a technique that should be used frequently as it can lead
932 to difficult to read tests and it also indicates that your gabbi tests
933 are being used to test your serializers and data models, not just your
934 API interactions.
935 It is also possible to read raw JSON from disk for either all or some
937 of a JSON response:
938 response_json_paths:
940 $: @<data.json
941 or:
943 response_json_paths:
945 $.pets: <@pets.json
946 $.pets[0]: <@cat.json
947 Examples like this can be found in one of gabbi's own tests.
949 If it is desired to load YAML files like the JSON ones above, two
951 things must be done:
952 1. The YAMLDiskLoadingJSONHandler custom content handler must be passed
954 to the driver through the content_handlers argument. See Extensions
955 on how to do this.
956 2. The YAML files to load must be placed in a subdirectory to prevent
958 the test runner from consuming them as test files to run:
959 response_json_paths:
961 $: @<subdir/values.yaml
962 When reading from disk you can apply the same JSONPath by adding a ':'
964 to the end of your file name. This allows you to store multiple API re‐
965 sponses into a single file to reduce file management when constructing
966 your tests.
967 Given JSON data as follows:
969 {
971 "values": [{
972 "pets": [{
973 "type": "cat",
974 "sound": "meow"
975 }, {
976 "type": "dog",
977 "sound": "woof"
978 }]
979 }, {
980 "people": [{
981 "name": "chris",
982 "id": 1
983 }, {
984 "name": "justin",
985 "id": 2
986 }]
987 }]
988 }
989 You can write your tests like the following:
991 response_json_paths:
993 $.pets: <@pets.json
994 $.pets[?type = "cat"].sound: <@values.json:$.values[0].pets[?type = "cat"].sound
995 Although placing more than one API response into a single JSON file may
997 seem convenient, keep in mind there is a tradeoff in readability that
998 should not be overlooked before implementing this technique.
999 Examples like this can be found in one of gabbi's yaml-from-disk tests.
1001 There are more JSONPath examples in Example Tests and in the
1003 jsonpath_rw and jsonpath_rw_ext documentation.
1004
1006 Substitutions can be made in both the left (query) and right (expected)
1007 hand sides of the json path expression. When subtitutions are used in
1008 the query, care must be taken to ensure proper quoting of the resulting
1009 value. For example if there is a uuid (with hyphens) at $RE‐
1010 SPONSE['$.id'] then this expression may fail:
1011 $.nested.structure.$RESPONSE['$.id'].name: foobar
1013 as it will evaluate to something like:
1015 $.nested.structure.ADC8AAFC-D564-40D1-9724-7680D3C010C2.name: foobar
1017 which may be treated as an arithemtic expression by the json path
1019 parser. The test author should write:
1020 $.nested.structure["$RESPONSE['$.id']"].name: foobar
1022 to quote the result of the substitution.
1024 The target host is the host on which the API to be tested can be found.
1026 Gabbi intends to preserve the flow and semantics of HTTP interactions
1027 as much as possible, and every HTTP request needs to be directed at a
1028 host of some form. Gabbi provides three ways to control this:
1029 • Using wsgi-intercept to provide a fake socket and WSGI environment on
1031 an arbitrary host and port attached to a WSGI application (see
1032 intercept examples).
1033 • Using fully qualified url values in the YAML defined tests (see full
1035 examples).
1036 • Using a host and (optionally) port defined at test build time (see
1038 live examples).
1039 The intercept and live methods are mutually exclusive per test builder,
1041 but either kind of test can freely intermix fully qualified URLs into
1042 the sequence of tests in a YAML file.
1043 For test driven development and local tests the intercept style of
1045 testing lowers test requirements (no web server required) and is fast.
1046 Interception is performed as part of Fixtures processing as the most
1047 deeply nested fixture. This allows any configuration or database setup
1048 to be performed prior to the WSGI application being created.
1049 For the implementation of the above see build_tests().
1051 Each suite of tests is represented by a single YAML file, and may op‐
1053 tionally use one or more fixtures to provide the necessary environment
1054 required by the tests in that file.
1055 Fixtures are implemented as nested context managers. Subclasses of
1057 GabbiFixture must implement start_fixture and stop_fixture methods for
1058 creating and destroying, respectively, any resources managed by the
1059 fixture. While the subclass may choose to implement __init__ it is im‐
1060 portant that no exceptions are thrown in that method, otherwise the
1061 stack of context managers will fail in unexpected ways. Instead ini‐
1062 tialization of real resources should happen in start_fixture.
1063 At this time there is no mechanism for the individual tests to have any
1065 direct awareness of the fixtures. The fixtures exist, conceptually, on
1066 the server side of the API being tested.
1067 Fixtures may do whatever is required by the testing environment, how‐
1069 ever there are two common scenarios:
1070 • Establishing (and then resetting when a test suite has finished) any
1072 baseline configuration settings and persistence systems required for
1073 the tests.
1074 • Creating sample data for use by the tests.
1076 If a fixture raises unittest.case.SkipTest during start_fixture all the
1078 tests in the current file will be skipped. This makes it possible to
1079 skip the tests if some optional configuration (such as a particular
1080 type of database) is not available.
1081 If an exception is raised while a fixture is being used, information
1083 about the exception will be stored on the fixture so that the stop_fix‐
1084 ture method can decide if the exception should change how the fixture
1085 should clean up. The exception information can be found on exc_type,
1086 exc_value and traceback method attributes.
1087 If an exception is raised when a fixture is started (in start_fixture)
1089 the first test in the suite using the fixture will be marked with an
1090 error using the traceback from the exception and all the tests in the
1091 suite will be skipped. This ensures that fixture failure is adequately
1092 captured and reported by test runners.
1093 In some contexts (for example CI environments with a large number of
1095 tests being run in a broadly concurrent environment where output is
1096 logged to a single file) it can be important to capture and consolidate
1097 stray output that is produced during the tests and display it associ‐
1098 ated with an individual test. This can help debugging and avoids unus‐
1099 able output that is the result of multiple streams being interleaved.
1100 Inner fixtures have been added to support this. These are fixtures more
1102 in line with the tradtional unittest concept of fixtures: a class on
1103 which setUp and cleanUp is automatically called.
1104 build_tests() accepts a named parameter arguments of inner_fixtures.
1106 The value of that argument may be an ordered list of fixtures.Fixture
1107 classes that will be called when each individual test is set up.
1108 An example fixture that could be useful is the FakeLogger.
1110 NOTE:
1112 At this time inner_fixtures are not supported when using the pytest
1113 loader.
1114 Content handlers are responsible for preparing request data and evalu‐
1116 ating response data based on the content-type of the request and re‐
1117 sponse. A content handler operates as follows:
1118 • Structured YAML data provided via the data attribute is converted to
1120 a string or bytes sequence and used as request body.
1121 • The response body (a string or sequence of bytes) is transformed into
1123 a content-type dependent structure and stored in an internal attri‐
1124 bute named response_data that is:
1125 • used when evaluating the response body
1127 • used in $RESPONSE[] substitutions
1129 By default, gabbi provides content handlers for JSON. In that content
1131 handler the data test key is converted from structured YAML into a JSON
1132 string. Response bodies are converted from a JSON string into a data
1133 structure in response_data that is used when evaluating re‐
1134 sponse_json_paths entries in a test or doing JSONPath-based $RESPONSE[]
1135 substitutions.
1136 A YAMLDiskLoadingJSONHandler has been added to extend the JSON handler.
1138 It works the same way as the JSON handler except for when evaluating
1139 the response_json_paths handle, data that is read from disk can be ei‐
1140 ther in JSON or YAML format. The YAMLDiskLoadingJSONHandler is not en‐
1141 abled by default and must be added as shown in the Extensions section
1142 in order to be used in the tests.
1143 Further content handlers can be added as extensions. Test authors may
1145 need these extensions for their own suites, or enterprising developers
1146 may wish to create and distribute extensions for others to use.
1147 NOTE:
1149 One extension that is likely to be useful is a content handler that
1150 turns data into url-encoded form data suitable for POST and turns an
1151 HTML response into a DOM object.
1152
1154 Content handlers are an evolution of the response handler concept in
1155 earlier versions gabbi. To preserve backwards compatibility with exist‐
1156 ing response handlers, old style response handlers are still allowed,
1157 but new handlers should implement the content handler interface (de‐
1158 scribed below).
1159 Registering additional custom handlers is done by passing a subclass of
1161 ContentHandler to build_tests():
1162 driver.build_tests(test_dir, loader, host=None,
1164 intercept=simple_wsgi.SimpleWsgi,
1165 content_handlers=[MyContentHandler])
1166 If pytest is being used:
1168 driver.py_test_generator(test_dir, intercept=simple_wsgi.SimpleWsgi,
1170 content_handlers=[MyContenHandler])
1171 Gabbi provides an additional custom handler named YAMLDiskLoadingJSON‐
1173 Handler. This can be used for loading JSON and YAML files from disk
1174 when evaluating the response_json_paths handle.
1175 WARNING:
1177 YAMLDiskLoadingJSONHandler shares the same content-type as the de‐
1178 fault JSONHandler. When there are multiple handlers listed that ac‐
1179 cept the same content-type, the one that is earliest in the list
1180 will be used.
1181 With gabbi-run, custom handlers can be loaded via the --response-han‐
1183 dler option -- see load_response_handlers() for details.
1184 NOTE:
1186 The use of the --response-handler argument is done to preserve back‐
1187 wards compatibility and avoid excessive arguments. Both types of
1188 handler may be passed to the argument.
1189 Implementation Details
1191 Creating a content handler requires subclassing ContentHandler and im‐
1192 plementing several methods. These methods are described below, but in‐
1193 specting JSONHandler will be instructive in highlighting required argu‐
1194 ments and techniques.
1195 To provide a response_<something> response-body evaluator a subclass
1197 must define:
1198 • test_key_suffix: This, along with the prefix response_, forms the key
1200 used in the test structure. It is a class level string.
1201 • test_key_value: The key's default value, either an empty list ([]) or
1203 empty dict ({}). It is a class level value.
1204 • action: An instance method which tests the expected values against
1206 the HTTP response - it is invoked for each entry, with the parameters
1207 depending on the default value. The arguments to action are (in or‐
1208 der):
1209 • self: The current instance.
1211 • test: The currently active HTTPTestCase
1213 • item: The current entry if test_key_value is a list, otherwise the
1215 key half of the key/value pair at this entry.
1216 • value: None if test_key_value is a list, otherwise the value half
1218 of the key/value pair at this entry.
1219 To translate request or response bodies to or from structured data a
1221 subclass must define an accepts method. This should return True if this
1222 class is willing to translate the provided content-type. During request
1223 processing it is given the value of the content-type header that will
1224 be sent in the request. During response processing it is given the
1225 value of the content-type header of the response. This makes it possi‐
1226 ble to handle different request and response bodies in the same han‐
1227 dler, if desired. For example a handler might accept applica‐
1228 tion/x-www-form-urlencoded and text/html.
1229 If accepts is defined two additional static methods should be defined:
1231 • dumps: Turn structured Python data from the data key in a test into a
1233 string or byte stream. The optional test param allows you to access
1234 the current test case which may help with manipulations for custom
1235 content handlers, e.g. multipart/form-data needs to add a boundary to
1236 the Content-Type header in order to mark the appropriate sections of
1237 the body.
1238 • loads: Turn a string or byte stream in a response into a Python data
1240 structure. Gabbi will put this data on the response_data attribute on
1241 the test, where it can be used in the evaluations described above (in
1242 the action method) or in $RESPONSE handling. An example usage here
1243 would be to turn HTML into a DOM.
1244 • load_data_file: Load data from disk into a Python data structure.
1246 Gabbi will call this method when response_<something> contains an
1247 item where the right hand side value starts with <@. The test param
1248 allows you to access the current test case and provides a
1249 load_data_file method which should be used because it verifies the
1250 data is loaded within the test diectory and returns the file source
1251 as a string. The load_data_file method was introduced to re-use the
1252 JSONHandler in order to support loading YAML files from disk through
1253 the implementation of an additional custom handler, see
1254 YAMLDiskLoadingJSONHandler for details.
1255 Finally if a replacer class method is defined, then when a $RESPONSE
1257 substitution is encountered, replacer will be passed the response_data
1258 of the prior test and the argument within the $RESPONSE.
1259 Please see the JSONHandler source for additional detail.
1261 If there is a running web service that needs to be tested and creating
1263 a test loader with build_tests() is either inconvenient or overkill it
1264 is possible to run YAML test files directly from the command line with
1265 the console-script gabbi-run. It accepts YAML on stdin or as multiple
1266 file arguments, and generates and runs tests and outputs a summary of
1267 the results.
1268 The provided YAML may not use custom Fixtures but otherwise uses the
1270 default Test Format. Target Host information is either expressed di‐
1271 rectly in the YAML file or provided on the command line:
1272 gabbi-run [host[:port]] < /my/test.yaml
1274 or:
1276 gabbi-run http://host:port < /my/test.yaml
1278 To test with one or more files the following command syntax may be
1280 used:
1281 gabbi-run http://host:port -- /my/test.yaml /my/other.yaml
1283 NOTE:
1285 The filename arguments must come after a -- and all other arguments
1286 (host, port, prefix, failfast) must come before the --.
1287 NOTE:
1289 If files are provided, test output will use names including the name
1290 of the file. If any single file includes an error, the name of the
1291 file will be included in a summary of failed files at the end of the
1292 test report.
1293 To facilitate using the same tests against the same application mounted
1295 in different locations in a WSGI server, a prefix may be provided as a
1296 second argument:
1297 gabbi-run host[:port] [prefix] < /my/test.yaml
1299 or in the target URL:
1301 gabbi-run http://host:port/prefix < /my/test.yaml
1303 The value of prefix will be prepended to the path portion of URLs that
1305 are not fully qualified.
1306 Anywhere host is used, if it is a raw IPV6 address it should be wrapped
1308 in [ and ].
1309 If https is used in the target, then the tests in the provided YAML
1311 will default to ssl: True.
1312 Use -k or --insecure to not validate certificates when making https
1314 connections.
1315 If a -x or --failfast argument is provided then gabbi-run will exit af‐
1317 ter the first test failure.
1318 Use -v or --verbose with a value of all, headers or body to turn on
1320 verbosity for all tests being run.
1321 Use -q or --quiet to silence test runner output.
1323 These are informal release notes for gabbi since version 1.0.0, high‐
1325 lighting major features and changes. For more detail see the commit
1326 logs on GitHub.
1327
1329 • Allow substitutions in skip for more flexible skipping, notablly with
1330 $ENVIRON.
1331 • Better test name output when running with unittest and verbose.
1333
1335 • Remove support for Python 3.5. Ensure testing support for 3.8, 3.9,
1336 and pypy3.
1337 • Adapt to new behavior in urllib.parse.
1339 • Correct fixture start when using pytest.
1341
1343 • If no content-type is provided with a response and verbose is on for
1344 responses, display the response body as if it were text.
1345
1347 • Properly declare that gabbi 2.x is Python 3 only.
1348
1350 • Correct management of response handler default fields.
1351
1353 • Drop support for Python 2. If you need Python 2 support, use an older
1354 version.
1355 • Stop using testtools and fixtures. These two modules present several
1357 difficulties and their maintenance situation suggests those difficul‐
1358 ties will not be resolved. Since Python 2 support is being removed,
1359 the need for the modules can be removed as well without losing func‐
1360 tionality. "Inner fixtures" that use the fixtures.Fixture interface
1361 should continue to work.
1362
1364 • Add support for not validating certificates in https requests. Con‐
1365 trolled by the cert_validate attribute in individual tests and
1366 build_tests() and the -k or --insecure argument to gabbi-run.
1367
1369 • Support pytest 5.0.0 in Python >=3.5. For earlier versions of Python,
1370 pytest<5.0.0 will be used; the pytest project is dropping support for
1371 older versions of Python.
1372
1374 • Use pytest<5.0.0 until gabbi has solutions for the changes in 5.0.0.
1375
1377 • A -q argument is added to gabbi-run to suppress output from the test
1378 runner.
1379
1381 • Adjust loading of YAML to be ready for new release of PyYAML.
1382
1384 • Provide the YAMLDiskLoadingJSONHandler class that allows test result
1385 data for response_json_path checks to be loaded from YAML-on-disk.
1386
1388 • Use JSONPath to select a portion of data-on-disk in re‐
1389 sponse_json_path checks.
1390 • Restrict PyYAML to <4.0.
1392
1394 • Allow listing of tests with no host configured. When host is an empty
1395 string, tests can be listed (for discovery), but will be skipped on
1396 run.
1397
1399 • JSON $RESPONSE substitutions in the data field may be complex types
1400 (lists and dicts), not solely strings.
1401
1403 • When the HTTP response begins with a bad status line, have BadSta‐
1404 tusLine be raised from urllib3.
1405
1407 • Allow substitutions in the key portion of request and response head‐
1408 ers, not just the value.
1409
1411 • Remove support for Python 3.3.
1412 • Make handling of fixture-level skips in pytest actually work.
1414
1416 • Add safe_yaml parameter to build_tests().
1417
1419 • use_prior_test is added to test Metadata.
1420 • Extensive cleanups in regular expression handling when constructing
1422 tests from YAML.
1423
1425 JSONPath handling gets two improvements:
1426 • The value side of a response_json_paths entry can be loaded from a
1428 file using the <@file.json syntax also used in Data.
1429 • The key side of a response_json_paths entry can use substitutions.
1431 This was already true for the value side.
1432
1434 Substitutions in $RESPONSE handling now preserve numeric types instead
1435 of casting to a string. This is useful when servers are expecting
1436 strong types and tests want to send response data back to the server.
1437
1439 count and delay test keys allow substitutions.
1440 gabbi.driver.build_tests() accepts a verbose parameter to set test
1441 verbosity for an entire session.
1442
1444 Better failure reporting when using gabbi-run with multiple files. Test
1445 names are based on the files and a summary of failed files is provided
1446 at the end of the report.
1447
1449 Effectively capture a failure in a fixture and report the traceback.
1450 Without this some test runners swallow the error and discovering prob‐
1451 lems when developing fixtures can be quite challenging.
1452
1454 Thanks to Samuel Fekete, tests can use the $HISTORY dictionary to refer
1455 to any prior test in the same file, not just the one immediately prior,
1456 when doing substitutions.
1457
1459 Filenames used to read data into tests using the <@ syntax may now use
1460 pathnames relative to the YAML file. See Data.
1461 gabbi-run gains a --verbose parameter to force all tests run in a ses‐
1463 sion to run with verbose set.
1464 When using pytest to load tests, a new mechanism is available which
1466 avoids warnings produced in when using a version of pytest greater than
1467 3.0.
1468
1470 When verbosely displaying request and response bodies that are JSON,
1471 pretty print for improved readability.
1472
1474 Allow gabbi-run to accept multiple filenames as command line arguments
1475 instead of reading tests from stdin.
1476
1478 Switch from response handlers to Content Handlers to allow more flexi‐
1479 ble processing of both response _and_ request bodies.
1480 Add inner fixtures for per test fixtures, useful for output capturing.
1482
1484 Allow the test_loader_name arg to gabbi.driver.build_tests() to over‐
1485 ride the prefix of the pretty printed name of generated tests.
1486
1488 String values in JSONPath matches may be wrapped in /.../` to be
1489 treated as regular expressions.
1490
1492 Better documentation of how to run gabbi in a concurrent environment.
1493 Improved handling of pytest fixtures and test counts.
1494
1496 Add url to gabbi.driver.build_tests() to use instead of host, port and
1497 prefix.
1498
1500 Add require_ssl to gabbi.driver.build_tests() to force use of SSL.
1501
1503 Add $COOKIE substitution.
1504
1506 Correctly support IPV6 hosts.
1507
1509 Add $LAST_URL substitution.
1510
1512 Introduce support for loading and running tests with pytest.
1513
1515 Use urllib3 instead of httplib2 for driving HTTP requests.
1516
1518 Add sorting and filtering to JSONPath handling.
1519
1521 Add the response_forbidden_headers to response expectations.
1522
1524 Instead of:
1525 tests:
1527 - name: a simple get
1528 url: /some/path
1529 method: get
1530 1.7.0 also makes it possible to:
1532 tests:
1534 - name: a simple get
1535 GET: /some/path
1536 Any upper case key is treated as a method.
1538
1540 Enhanced flexibility and colorization when setting tests to be verbose.
1541
1543 Adds the query_parameters key to request parameters.
1544
1546 The start of improvements and extensions to JSONPath handling. In this
1547 case the addition of the len function.
1548
1550 Vastly improved output and behavior in gabbi-run.
1551
1553 Version 1 was the first release with a commitment to a stable Test For‐
1554 mat. Since then new fields have been added but have not been taken
1555 away.
1556 The following people have contributed code to gabbi. Thanks to them.
1558 Thanks also to all the people who have made gabbi better by reporting
1559 issues and their successes and failures with using gabbi.
1560 • Chris Dent
1562 • FND
1564 • Mehdi Abaakouk
1566 • Tom Viner
1568 • Jason Myers
1570 • Josh Leeb-du Toit
1572 • Duc Truong
1574 • Zane Bitter
1576 • Ryan Spencer
1578 • Kim Raymoure
1580 • Travis Truman
1582 • Samuel Fekete
1584 • Michael McCune
1586 • Imran Hayder
1588 • Julien Danjou
1590 • Trevor McCasland
1592 • Danek Duvall
1594 • Marc Abramowitz
1596 NOTE:
1598 This section provides a collection of questions with answers that
1599 don't otherwise fit in the rest of the documentation. If something
1600 is missing, please create an issue.
1601 As this document grows it will gain a more refined structure.
1603
1605 Is gabbi only for testing Python-based APIs?
1606 No, you can use gabbi-run to test an HTTP service built in any program‐
1607 ming language.
1608 How do I run just one test?
1610 Each YAML file contains a sequence of tests, each test within each file
1611 has a name. That name is translated to the name of the test by replac‐
1612 ing spaces with an _.
1613 When running tests that are generated dynamically, filtering based on
1615 the test name prior to the test being collected will not work in some
1616 test runners. Test runners that use a --load-list functionality can be
1617 convinced to filter after discovery.
1618 pytest does this directly with the -k keyword flag.
1620 When using testrepository with tox as used in gabbi's own tests it is
1622 possible to pass a filter in the tox command:
1623 tox -epy27 -- get_the_widget
1625 When using testtools.run and similar test runners it's a bit more com‐
1627 plicated. It is necessary to provide the full name of the test as a
1628 list to --load-list:
1629 python -m testtools.run --load-list \
1631 <(echo package.tests.test_api.yamlfile_get_the_widge.test_request)
1632 How do I run just one test, without running prior tests in a sequence?
1634 By default, when you select a single test to run, all tests prior to
1635 that one in a file will be run as well: the file is treated as as se‐
1636 quence of dependent tests. If you do not want this you can adjust the
1637 use_prior_test test metadata in one of three ways:
1638 • Set it in the YAML file for the one test you are concerned with.
1640 • Set the defaults for all tests in that file.
1642 • set use_prior_test to false when calling build_tests()
1644 Be aware that doing this breaks a fundamental assumption that gabbi
1646 makes about how tests work. Any substitutions will fail.
1647
1649 Can I have variables in my YAML file?
1650 Gabbi provides the $ENVIRON substitution which can operate a bit like
1651 variables that are set elsewhere and then used in the tests defined by
1652 the YAML.
1653 If you find it necessary to have variables within a single YAML file
1655 you take advantage of YAML alias nodes list this:
1656 vars:
1658 - &uuid_1 5613AABF-BAED-4BBA-887A-252B2D3543F8
1659 tests:
1661 - name: send a uuid to a post
1662 POST: /resource
1663 request_headers:
1664 content-type: application/json
1665 data:
1666 uuid: *uuid_1
1667 You can alias all sorts of nodes, not just single items. Be aware that
1669 the replacement of an alias node happens while the YAML is being
1670 loaded, before gabbi does any processing.
1671 How many tests should be put in one YAML file?
1673 For the sake of readability it is best to keep each YAML file rela‐
1674 tively short. Since each YAML file represents a sequence of requests,
1675 it usually makes sense to create a new file when a test is not depen‐
1676 dent on any before it.
1677 It's tempting to put all the tests for any resource or URL in the same
1679 file, but this eventually leads to files that are too long and are thus
1680 difficult to read.
1681
1683 A single HTTP request represented as a subclass of unittest.TestCase
1684 The test case encapsulates the request headers and body and expected
1686 response headers and body. When the test is run an HTTP request is made
1687 using urllib3. Assertions are made against the response.
1688 class gabbi.case.HTTPTestCase(methodName='runTest')
1690 Bases: TestCase
1691 Encapsulate a single HTTP request as a TestCase.
1693 If the test is a member of a sequence of requests, ensure that
1695 prior tests are run.
1696 To keep the test harness happy we need to make sure the setUp
1698 and tearDown are only run once.
1699 assert_in_or_print_output(expected, iterable)
1701 Assert the iterable contains expected or print some out‐
1702 put.
1703 If the output is long, it is limited by either
1705 GABBI_MAX_CHARS_OUTPUT in the environment or the
1706 MAX_CHARS_OUTPUT constant.
1707 base_test = {'cert_validate': True, 'data': '', 'desc': '',
1709 'method': 'GET', 'name': '', 'poll': {}, 'query_parameters': {},
1710 'redirects': False, 'request_headers': {}, 'skip': '', 'ssl':
1711 False, 'status': '200', 'url': '', 'use_prior_test': True, 'ver‐
1712 bose': False, 'xfail': False}
1713 get_content_handler(content_type)
1715 Determine the content handler for this media type.
1716 load_data_file(filename)
1718 Read a file from the current test directory.
1719 replace_template(message, escape_regex=False)
1721 Replace magic strings in message.
1722 run(result=None)
1724 Store the current result handler on this test.
1725 setUp()
1727 Hook method for setting up the test fixture before exer‐
1728 cising it.
1729 shortDescription()
1731 Returns a one-line description of the test, or None if no
1732 description has been provided.
1733 The default implementation of this method returns the
1735 first line of the specified test method's docstring.
1736 tearDown()
1738 Hook method for deconstructing the test fixture after
1739 testing it.
1740 test_request()
1742 Run this request if it has not yet run.
1743 If there is a prior test in the sequence, run it first.
1745 gabbi.case.potentialFailure(func)
1747 Decorate a test method that is expected to fail if 'xfail' is
1748 true.
1749
1753 Manage fixtures for gabbi at the test suite level.
1754 class gabbi.fixture.GabbiFixture
1756 Bases: object
1757 A context manager that operates as a fixture.
1759 Subclasses must implement start_fixture and stop_fixture, each
1761 of which contain the logic for stopping and starting whatever
1762 the fixture is. What a fixture is is left as an exercise for the
1763 implementor.
1764 These context managers will be nested so any actual work needs
1766 to happen in start_fixture and stop_fixture and not in __init__.
1767 Otherwise exception handling will not work properly.
1768 start_fixture()
1770 Implement the actual workings of starting the fixture
1771 here.
1772 stop_fixture()
1774 Implement the actual workings of stopping the fixture
1775 here.
1776 exception gabbi.fixture.GabbiFixtureError
1778 Bases: Exception
1779 Generic exception for GabbiFixture.
1781 class gabbi.fixture.SkipAllFixture
1783 Bases: GabbiFixture
1784 A fixture that skips all the tests in the current suite.
1786 start_fixture()
1788 Implement the actual workings of starting the fixture
1789 here.
1790 gabbi.fixture.nest(fixtures)
1792 Nest a series of fixtures.
1793 This is duplicated from nested in the stdlib, which has been
1795 deprecated because of issues with how exceptions are difficult
1796 to handle during __init__. Gabbi needs to nest an unknown number
1797 of fixtures dynamically, so the with syntax that replaces nested
1798 will not work.
1799
1801 Package for response and content handlers that process the body of a
1802 response in various ways.
1803 handlers.base Module
1805 Base classes for response and content handlers.
1806 class gabbi.handlers.base.ContentHandler
1808 Bases: ResponseHandler
1809 A subclass of ResponseHandlers that adds content handling.
1811 static accepts(content_type)
1813 Return True if this handler can handler this type.
1814 static dumps(data, pretty=False, test=None)
1816 Return structured data as a string.
1817 If pretty is true, prettify.
1819 static load_data_file(test, file_path)
1821 Return the string content of the file specified by the
1822 file_path.
1823 static loads(data)
1825 Create structured (Python) data from a stream.
1826 classmethod replacer(response_data, path)
1828 Return the string that is replacing RESPONSE.
1829 class gabbi.handlers.base.ResponseHandler
1831 Bases: object
1832 Add functionality for making assertions about an HTTP response.
1834 A subclass may implement two methods: action and preprocess.
1836 preprocess takes one argument, the TestCase. It is called ex‐
1838 actly once for each test before looping across the assertions.
1839 It is used, rarely, to copy the test.output into a useful form
1840 (such as a parsed DOM).
1841 action takes two or three arguments. If test_key_value is a list
1843 action is called with the test case and a single list item. If
1844 test_key_value is a dict then action is called with the test
1845 case and a key and value pair.
1846 action(test, item, value=None)
1848 Test an individual entry for this response handler.
1849 If the entry is a key value pair the key is in item and
1851 the value in value. Otherwise the entry is considered a
1852 single item from a list.
1853 preprocess(test)
1855 Do any pre-single-test preprocessing.
1856 test_key_suffix = ''
1858 test_key_value = []
1860 handlers.core Module
1862 Core response handlers.
1863 class gabbi.handlers.core.ForbiddenHeadersResponseHandler
1865 Bases: ResponseHandler
1866 Test that listed headers are not in the response.
1868 action(test, forbidden, value=None)
1870 Test an individual entry for this response handler.
1871 If the entry is a key value pair the key is in item and
1873 the value in value. Otherwise the entry is considered a
1874 single item from a list.
1875 test_key_suffix = 'forbidden_headers'
1877 test_key_value = []
1879 class gabbi.handlers.core.HeadersResponseHandler
1881 Bases: ResponseHandler
1882 Compare expected headers with actual headers.
1884 If a header value is wrapped in / it is treated as a raw regular
1886 expression.
1887 Headers values are always treated as strings.
1889 action(test, header, value=None)
1891 Test an individual entry for this response handler.
1892 If the entry is a key value pair the key is in item and
1894 the value in value. Otherwise the entry is considered a
1895 single item from a list.
1896 test_key_suffix = 'headers'
1898 test_key_value = {}
1900 class gabbi.handlers.core.StringResponseHandler
1902 Bases: ResponseHandler
1903 Test for matching strings in the the response body.
1905 action(test, expected, value=None)
1907 Test an individual entry for this response handler.
1908 If the entry is a key value pair the key is in item and
1910 the value in value. Otherwise the entry is considered a
1911 single item from a list.
1912 test_key_suffix = 'strings'
1914 test_key_value = []
1916 handlers.jsonhandler Module
1918 JSON-related content handling.
1919 class gabbi.handlers.jsonhandler.JSONHandler
1921 Bases: ContentHandler
1922 A ContentHandler for JSON
1924 • Structured test data is turned into JSON when request con‐
1926 tent-type is JSON.
1927 • Response bodies that are JSON strings are made into Python
1929 data on the test response_data attribute when the response
1930 content-type is JSON.
1931 • A response_json_paths response handler is added.
1933 • JSONPaths in $RESPONSE substitutions are supported.
1935 static accepts(content_type)
1937 Return True if this handler can handler this type.
1938 action(test, path, value=None)
1940 Test json_paths against json data.
1941 static dumps(data, pretty=False, test=None)
1943 Return structured data as a string.
1944 If pretty is true, prettify.
1946 static extract_json_path_value(data, path)
1948 Extract the value at JSON Path path from the data.
1949 The input data is a Python datastructure, not a JSON
1951 string.
1952 static load_data_file(test, file_path)
1954 Return the string content of the file specified by the
1955 file_path.
1956 static loads(data)
1958 Create structured (Python) data from a stream.
1959 classmethod replacer(response_data, match)
1961 Return the string that is replacing RESPONSE.
1962 test_key_suffix = 'json_paths'
1964 test_key_value = {}
1966 handlers.yaml_disk_loading_jsonhandler Module
1968 JSON-related content handling with YAML data disk loading.
1969 class gabbi.handlers.yaml_disk_loading_jsonhandler.YAMLDiskLoadingJSON‐
1971 Handler
1972 Bases: JSONHandler
1973 A ContentHandler for JSON responses that loads YAML from disk
1975 • Structured test data is turned into JSON when request con‐
1977 tent-type is JSON.
1978 • Response bodies that are JSON strings are made into Python
1980 data on the test response_data attribute when the response
1981 content-type is JSON.
1982 • A response_json_paths response handler is added. Data read
1984 from disk during this handle will be loaded with the
1985 yaml.safe_load method to support both JSON and YAML data
1986 sources from disk.
1987 • JSONPaths in $RESPONSE substitutions are supported.
1989 static load_data_file(test, file_path)
1991 Return the string content of the file specified by the
1992 file_path.
1993
1995 A TestSuite for containing gabbi tests.
1996 This suite has two features: the contained tests are ordered and there
1998 are suite-level fixtures that operate as context managers.
1999 class gabbi.suite.GabbiSuite(tests=())
2001 Bases: TestSuite
2002 A TestSuite with fixtures.
2004 The suite wraps the tests with a set of nested context managers
2006 that operate as fixtures.
2007 If a fixture raises unittest.case.SkipTest during setup, all the
2009 tests in this suite will be skipped.
2010 run(result, debug=False)
2012 Override TestSuite run to start suite-level fixtures.
2013 To avoid exception confusion, use a null Fixture when
2015 there are no fixtures.
2016 start(result, tests=None)
2018 Start fixtures when using pytest.
2019 stop() Stop fixtures when using pytest.
2021 gabbi.suite.noop(*args)
2023 A noop method used to disable collected tests.
2024
2027 TestRunner and TestResult for gabbi-run.
2028 class gabbi.reporter.ConciseTestResult(stream, descriptions, verbosity)
2030 Bases: TextTestResult
2031 A TextTestResult with simple but useful output.
2033 If the output is a tty or GABBI_FORCE_COLOR is set in the envi‐
2035 ronment, output will be colorized.
2036 addError(test, err)
2038 Called when an error has occurred. 'err' is a tuple of
2039 values as returned by sys.exc_info().
2040 addExpectedFailure(test, err)
2042 Called when an expected failure/error occurred.
2043 addFailure(test, err)
2045 Called when an error has occurred. 'err' is a tuple of
2046 values as returned by sys.exc_info().
2047 addSkip(test, reason)
2049 Called when a test is skipped.
2050 addSuccess(test)
2052 Called when a test has completed successfully
2053 addUnexpectedSuccess(test)
2055 Called when a test was expected to fail, but succeed.
2056 getDescription(test)
2058 printErrorList(flavor, errors)
2060 startTest(test)
2062 Called when the given test is about to be run
2063 class gabbi.reporter.ConciseTestRunner(stream=None, descriptions=True,
2065 verbosity=1, failfast=False, buffer=False, resultclass=None, warn‐
2066 ings=None, *, tb_locals=False)
2067 Bases: TextTestRunner
2068 A TextTestRunner that uses ConciseTestResult for reporting re‐
2070 sults.
2071 resultclass
2073 alias of ConciseTestResult
2074 class gabbi.reporter.PyTestResult(stream=None, descriptions=None, ver‐
2076 bosity=None)
2077 Bases: TestResult
2078 Wrap a test result to allow it to work with pytest.
2080 The main behaviors here are:
2082 • to turn what had been exceptions back into exceptions
2084 • use pytest's skip and xfail methods
2086 addError(test, err)
2088 Called when an error has occurred. 'err' is a tuple of
2089 values as returned by sys.exc_info().
2090 addExpectedFailure(test, err)
2092 Called when an expected failure/error occurred.
2093 addFailure(test, err)
2095 Called when an error has occurred. 'err' is a tuple of
2096 values as returned by sys.exc_info().
2097 addSkip(test, reason)
2099 Called when a test is skipped.
2100
2102 Utility functions grab bag.
2103 gabbi.utils.create_url(base_url, host, port=None, prefix='', ssl=False)
2105 Given pieces of a path-based url, return a fully qualified url.
2106 gabbi.utils.decode_response_content(header_dict, content)
2108 Decode content to a proper string.
2109 gabbi.utils.extract_content_type(header_dict, default='application/bi‐
2111 nary')
2112 Extract parsed content-type from headers.
2113 gabbi.utils.get_colorizer(stream)
2115 Return a function to colorize a string.
2116 Only if stream is a tty .
2118 gabbi.utils.host_info_from_target(target, prefix=None)
2120 Turn url or host:port and target into test destination.
2121 gabbi.utils.load_yaml(handle=None, yaml_file=None, safe=True)
2123 Read and parse any YAML file or filehandle.
2124 Let exceptions flow where they may.
2126 If no file or handle is provided, read from STDIN.
2128 gabbi.utils.not_binary(content_type)
2130 Decide if something is content we'd like to treat as a string.
2131 gabbi.utils.parse_content_type(content_type, default_charset='utf-8')
2133 Parse content type value for media type and charset.
2134
2136 Gabbi specific exceptions.
2137 exception gabbi.exception.GabbiFormatError
2139 Bases: ValueError
2140 An exception to encapsulate poorly formed test data.
2142 exception gabbi.exception.GabbiSyntaxWarning
2144 Bases: SyntaxWarning
2145 A warning about syntax that is not desirable.
2147
2150 Keep one single global jsonpath parser.
2151 gabbi.json_parser.parse(path)
2153 Parse a JSONPath expression use the global parser.
2154 Gabbi is a tool for running HTTP tests where requests and responses are
2156 expressed as declarations in YAML files:
2157 tests:
2159 - name: retrieve items
2160 GET: /items
2161 See the rest of these docs for more details on the many features and
2163 formats for setting request headers and bodies and evaluating re‐
2164 sponses.
2165 Tests can be run from the command line with gabbi-run or programmati‐
2167 cally using either py.test or unittest-style test runners. See
2168 installation instructions below.
2169 The name is derived from "gabby": excessively talkative. In a test en‐
2171 vironment having visibility of what a test is actually doing is a good
2172 thing. This is especially true when the goal of a test is to test the
2173 HTTP, not the testing infrastructure. Gabbi tries to put the HTTP in‐
2174 teraction in the foreground of testing.
2175 If you want to get straight to creating tests look at Example Tests,
2177 the test files in the source distribution and Test Format. A gabbi-demo
2178 repository provides a tutorial of using gabbi to build an API, via the
2179 commit history of the repo.
2180
2182 Gabbi works to bridge the gap between human readable YAML files (see
2183 Test Format for details) that represent HTTP requests and expected re‐
2184 sponses and the rather complex world of automated testing.
2185 Each YAML file represents an ordered list of HTTP requests along with
2187 the expected responses. This allows a single file to represent a
2188 process in the API being tested. For example:
2189 • Create a resource.
2191 • Retrieve a resource.
2193 • Delete a resource.
2195 • Retrieve a resource again to confirm it is gone.
2197 At the same time it is still possible to ask gabbi to run just one re‐
2199 quest. If it is in a sequence of tests, those tests prior to it in the
2200 YAML file will be run (in order). In any single process any test will
2201 only be run once. Concurrency is handled such that one file runs in one
2202 process.
2203 These features mean that it is possible to create tests that are useful
2205 for both humans (as tools for learning, improving and developing APIs)
2206 and automated CI systems.
2207 Significant flexibility and power is available in the Test Format to
2209 make it relatively straightforward to test existing complex APIs. This
2210 extended functionality includes the use of JSONPath to query response
2211 bodies and templating of test data to allow access to the prior HTTP
2212 response in the current request. For APIs which do not use JSON addi‐
2213 tional Content Handlers can be created.
2214 Care should be taken when using this functionality when you are creat‐
2216 ing a new API. If your API is so complex that it needs complex test
2217 files then you may wish to take that as a sign that your API itself too
2218 complex. One goal of gabbi is to encourage transparent and comprehensi‐
2219 ble APIs.
2220 Though gabbi is written in Python and under the covers uses unittest
2222 data structures and processes, there is no requirement that the Target
2223 Host be a Python-based service. Anything talking HTTP can be tested. A
2224 YAML Runner makes it possible to simply create YAML files and point
2225 them at a running server.
2226
2228 As a Python package, gabbi is typically installed via pip:
2229 pip install gabbi
2231 You might want to create a virtual environment; an isolated context for
2233 Python packages, keeping gabbi cleany separated from the rest of your
2234 system.
2235 Python 3 comes with a built-in tool to create virtual environments:
2237 python3 -m venv venv
2239 . venv/bin/activate
2240 pip install gabbi
2242 This way we can later use deactivate and safely remove the venv direc‐
2244 tory, thus erasing any trace of gabbi from the system.
2245
2247 Chris Dent
2248 Jul 22, 2022 GABBI(1)