1GABBI(1)                             Gabbi                            GABBI(1)
2
3
4

NAME

6       gabbi - Gabbi Documentation
7
8       Gabbi  tests  are  expressed  in YAML as a series of HTTP requests with
9       their expected response:
10
11          tests:
12             - name: retrieve root
13               GET: /
14               status: 200
15
16       This will trigger a GET request to / on the configured Target Host. The
17       test will pass if the response's status code is 200.
18

TEST STRUCTURE

20       The  top-level tests category contains an ordered sequence of test dec‐
21       larations, each describing the expected response to a given request:
22
23   Metadata
24             ┌───────────────┬─────────────────────┬───────────────────┐
25             │Key            │ Description         │ Notes             │
26             ├───────────────┼─────────────────────┼───────────────────┤
27name           │ The  test's   name. │ required          
28             │               │ Must    be   unique │                   │
29             │               │ within a file.      │                   │
30             ├───────────────┼─────────────────────┼───────────────────┤
31desc           │ An arbitrary string │                   │
32             │               │ describing      the │                   │
33             │               │ test.               │                   │
34             ├───────────────┼─────────────────────┼───────────────────┤
35verbose        │ If  True   or   all │ defaults to False 
36             │               │ (synonymous),       │                   │
37             │               │ prints a  represen‐ │                   │
38             │               │ tation  of the cur‐ │                   │
39             │               │ rent  request   and │                   │
40             │               │ response to stdout, │                   │
41             │               │ including      both │                   │
42             │               │ headers  and  body. │                   │
43             │               │ If set  to  headers │                   │
44             │               │ or  body,  only the │                   │
45             │               │ corresponding  part │                   │
46             │               │ of  the request and │                   │
47             │               │ response  will   be │                   │
48             │               │ printed.   If   the │                   │
49             │               │ output  is  a  TTY, │                   │
50             │               │ colors    will   be │                   │
51             │               │ used. If  the  body │                   │
52             │               │ content-type     is │                   │
53             │               │ JSON  it  will   be │                   │
54             │               │ formatted  for  im‐ │                   │
55             │               │ proved readability. │                   │
56             │               │ See VerboseHttp for │                   │
57             │               │ details.            │                   │
58             ├───────────────┼─────────────────────┼───────────────────┤
59skip           │ A  string   message │ defaults to False 
60             │               │ which  if  set will │                   │
61             │               │ cause the  test  to │                   │
62             │               │ be skipped with the │                   │
63             │               │ provided message.   │                   │
64             └───────────────┴─────────────────────┴───────────────────┘
65
66
67xfail          │ Determines  whether │ defaults to False 
68             │               │ to expect this test │                   │
69             │               │ to fail. Note  that │                   │
70             │               │ the  test  will  be │                   │
71             │               │ run anyway.         │                   │
72             ├───────────────┼─────────────────────┼───────────────────┤
73use_prior_test │ Determines if  this │ defaults to True  
74             │               │ test will be run in │                   │
75             │               │ sequence    (after) │                   │
76             │               │ the  test  prior to │                   │
77             │               │ it in the  list  of │                   │
78             │               │ tests    within   a │                   │
79             │               │ file.  To  be  con‐ │                   │
80             │               │ crete, when this is │                   │
81             │               │ True  the  test  is │                   │
82             │               │ dependent   on  the │                   │
83             │               │ prior test  and  if │                   │
84             │               │ that  prior has not │                   │
85             │               │ yet run, it wil  be │                   │
86             │               │ run,  even  if only │                   │
87             │               │ the  current   test │                   │
88             │               │ has  been selected. │                   │
89             │               │ Set this  to  False │                   │
90             │               │ to  allow selecting │                   │
91             │               │ a test without  de‐ │                   │
92             │               │ pendencies.         │                   │
93             ├───────────────┼─────────────────────┼───────────────────┤
94cert_validate  │ States  whether the │ defaults to True  
95             │               │ underlying     HTTP │                   │
96             │               │ client  should  at‐ │                   │
97             │               │ tempt  to  validate │                   │
98             │               │ SSL   certificates. │                   │
99             │               │ In many test  envi‐ │                   │
100             │               │ ronment    certifi‐ │                   │
101             │               │ cates    will    be │                   │
102             │               │ self-signed      so │                   │
103             │               │ changing  this  may │                   │
104             │               │ be requried. It can │                   │
105             │               │ also   be   changed │                   │
106             │               │ when   Loading  and │                   │
107             │               │ Running  Tests   or │                   │
108             │               │ using gabbi-run.    │                   │
109             ├───────────────┼─────────────────────┼───────────────────┤
110disable_re‐    │ If True, means that │ defaults to False 
111sponse_handler │ the  response  body │                   │
112             │               │ will  not  be  pro‐ │                   │
113             │               │ cessed   to  Python │                   │
114             │               │ data. This  can  be │                   │
115             │               │ necessary  if a re‐ │                   │
116             │               │ sponse   claims   a │                   │
117             │               │ content-type    but │                   │
118             │               │ the body is not ac‐ │                   │
119             │               │ tually   that  type │                   │
120             │               │ but  it  is   still │                   │
121             │               │ necessary   to  run │                   │
122             │               │ tests  against  the │                   │
123             │               │ response.  In  that │                   │
124             │               │ situation, if  dis‐ │                   │
125             │               │ able_response_han‐  │                   │
126             │               │ dler is  False  the │                   │
127             │               │ test     will    be │                   │
128             │               │ treated as a  fail‐ │                   │
129             │               │ ure.                │                   │
130             └───────────────┴─────────────────────┴───────────────────┘
131
132       NOTE:
133          When tests are generated dynamically, the TestCase name will include
134          the respective test's name, lowercased with spaces transformed to _.
135          In at least some test runners this will allow you to select and fil‐
136          ter on test name.
137
138   Request Parameters
139         ┌────────────────────┬───────────────────────┬─────────────────────┐
140         │Key                 │ Description           │ Notes               │
141         ├────────────────────┼───────────────────────┼─────────────────────┤
142         │any       uppercase │ Any   such  key  is   │                     │
143         │string              │ considered an  HTTP   │                     │
144         │                    │ method,   with  the   │                     │
145         │                    │ corresponding value   │                     │
146         │                    │ expressing the URL.   │                     │
147         │                    │                       │                     │
148         │                    │ This  is a shortcut   │                     │
149         │                    │ combining    method   │                     │
150         │                    │ and url into a sin‐   │                     │
151         │                    │ gle statement:        │                     │
152         │                    │                       │                     │
153         │                    │           GET: /index │                     │
154         │                    │                       │                     │
155         │                    │        corresponds    │                     │
156         │                    │        to:            │                     │
157         │                    │                       │                     │
158         │                    │           method: GET │                     │
159         │                    │           url: /index │                     │
160         ├────────────────────┼───────────────────────┼─────────────────────┤
161method              │ The    HTTP   request │ defaults to GET     
162         │                    │ method.               │                     │
163         ├────────────────────┼───────────────────────┼─────────────────────┤
164url                 │ The URL  to  request. │ Either this or  the │
165         │                    │ This  can either be a │ shortcut  above  is │
166         │                    │ full path (e.g. "/in‐ │ required            
167         │                    │ dex")   or   a  fully │                     │
168         │                    │ qualified  URL  (i.e. │                     │
169         │                    │ including   host  and │                     │
170         │                    │ scheme,   e.g.     "‐ │                     │
171         │                    │ http://exam          │                     │
172         │                    │ ple.org/index") — see │                     │
173         │                    │ Target  Host  for de‐ │                     │
174         │                    │ tails.                │                     │
175         ├────────────────────┼───────────────────────┼─────────────────────┤
176request_headers     │ A    dictionary    of │                     │
177         │                    │ key-value  pairs rep‐ │                     │
178         │                    │ resenting     request │                     │
179         │                    │ header names and val‐ │                     │
180         │                    │ ues.  These  will  be │                     │
181         │                    │ added   to  the  con‐ │                     │
182         │                    │ structed request.     │                     │
183         ├────────────────────┼───────────────────────┼─────────────────────┤
184query_parameters    │ A dictionary of query │                     │
185         │                    │ parameters  that will │                     │
186         │                    │ be added to  the  url │                     │
187         │                    │ as  query  string. If │                     │
188         │                    │ that URL already con‐ │                     │
189         │                    │ tains  a set of query │                     │
190         │                    │ parameters, those wil │                     │
191         │                    │ be    extended.   See │                     │
192         │                    │ Example Tests  for  a │                     │
193         │                    │ demonstration  of how │                     │
194         │                    │ the  data  is  struc‐ │                     │
195         │                    │ tured.                │                     │
196         └────────────────────┴───────────────────────┴─────────────────────┘
197
198
199data                │ A  representation  to │                     │
200         │                    │ pass as the body of a │                     │
201         │                    │ request.   Note  that │                     │
202         │                    │ content-type  in  re‐ │                     │
203         │                    │ quest_headers  should │                     │
204         │                    │ also  be  set  —  see │                     │
205         │                    │ Data for details.     │                     │
206         ├────────────────────┼───────────────────────┼─────────────────────┤
207redirects           │ If   True,  redirects │ defaults to False   
208         │                    │ will automatically be │                     │
209         │                    │ followed.             │                     │
210         ├────────────────────┼───────────────────────┼─────────────────────┤
211ssl                 │ Determines    whether │ defaults to False   
212         │                    │ the request uses  SSL │                     │
213         │                    │ (i.e.   HTTPS).  Note │                     │
214         │                    │ that the url's scheme │                     │
215         │                    │ takes  precedence  if │                     │
216         │                    │ present — see  Target │                     │
217         │                    │ Host for details.     │                     │
218         └────────────────────┴───────────────────────┴─────────────────────┘
219
220   Response Expectations
221            ┌────────────────────┬─────────────────────┬─────────────────┐
222            │Key                 │ Description         │ Notes           │
223            ├────────────────────┼─────────────────────┼─────────────────┤
224status              │ The   expected  re‐ │ defaults to 200 
225            │                    │ sponse status code. │                 │
226            │                    │ Multiple acceptable │                 │
227            │                    │ response codes  may │                 │
228            │                    │ be  provided, sepa‐ │                 │
229            │                    │ rated by  ||  (e.g. │                 │
230            │                    │ 302  || 301 — note, │                 │
231            │                    │ however, that  this │                 │
232            │                    │ indicates   ambigu‐ │                 │
233            │                    │ ity, which is  gen‐ │                 │
234            │                    │ erally     undesir‐ │                 │
235            │                    │ able).              │                 │
236            ├────────────────────┼─────────────────────┼─────────────────┤
237response_headers    │ A   dictionary   of │                 │
238            │                    │ key-value     pairs │                 │
239            │                    │ representing    ex‐ │                 │
240            │                    │ pected     response │                 │
241            │                    │ header  names   and │                 │
242            │                    │ values.     If    a │                 │
243            │                    │ header's  value  is │                 │
244            │                    │ wrapped  in  /.../, │                 │
245            │                    │ it will be  treated │                 │
246            │                    │ as  a  regular  ex‐ │                 │
247            │                    │ pression to  search │                 │
248            │                    │ for in the response │                 │
249            │                    │ header.             │                 │
250            ├────────────────────┼─────────────────────┼─────────────────┤
251response_forbid‐    │ A  list  of headers │                 │
252den_headers         │ which must  not  be │                 │
253            │                    │ present.            │                 │
254            ├────────────────────┼─────────────────────┼─────────────────┤
255response_strings    │ A  list  of  string │                 │
256            │                    │ fragments  expected │                 │
257            │                    │ to  be  present  in │                 │
258            │                    │ the response body.  │                 │
259            └────────────────────┴─────────────────────┴─────────────────┘
260
261
262
263
264
265response_json_paths │ A   dictionary   of │                 │
266            │                    │ JSONPath      rules │                 │
267            │                    │ paired   with   ex‐ │                 │
268            │                    │ pected matches. Us‐ │                 │
269            │                    │ ing this  rule  re‐ │                 │
270            │                    │ quires   that   the │                 │
271            │                    │ content being  sent │                 │
272            │                    │ from  the server is │                 │
273            │                    │ JSON (i.e.  a  con‐ │                 │
274            │                    │ tent type of appli‐ │                 │
275            │                    │ cation/json or con‐ │                 │
276            │                    │ taining +json)      │                 │
277            │                    │                     │                 │
278            │                    │ If   the  value  is │                 │
279            │                    │ wrapped  in   /.../ │                 │
280            │                    │ the  result  of the │                 │
281            │                    │ JSONPath query will │                 │
282            │                    │ be searched for the │                 │
283            │                    │ value as a  regular │                 │
284            │                    │ expression.         │                 │
285            ├────────────────────┼─────────────────────┼─────────────────┤
286poll                │ A dictionary of two │                 │
287            │                    │ keys:               │                 │
288            │                    │                     │                 │
289            │                    │        • count:  An │                 │
290            │                    │          integer    │                 │
291            │                    │          stating    │                 │
292            │                    │          the number │                 │
293            │                    │          of   times │                 │
294            │                    │          to attempt │                 │
295            │                    │          this  test │                 │
296            │                    │          before     │                 │
297            │                    │          giving up. │                 │
298            │                    │                     │                 │
299            │                    │        • delay:   A │                 │
300            │                    │          floating   │                 │
301            │                    │          point num‐ │                 │
302            │                    │          ber     of │                 │
303            │                    │          seconds to │                 │
304            │                    │          delay  be‐ │                 │
305            │                    │          tween  at‐ │                 │
306            │                    │          tempts.    │                 │
307            │                    │                     │                 │
308            │                    │        This   makes │                 │
309            │                    │        it  possible │                 │
310            │                    │        to  poll for │                 │
311            │                    │        a   resource │                 │
312            │                    │        created  via │                 │
313            │                    │        an asynchro‐ │                 │
314            │                    │        nous     re‐ │                 │
315            │                    │        quest.   Use │                 │
316            │                    │        with    cau‐ │                 │
317            │                    │        tion.        │                 │
318            └────────────────────┴─────────────────────┴─────────────────┘
319
320       Note that many of these items allow substitutions.
321
322       Default values for a file's tests may be provided via the top-level de‐
323       faults  category.  These  take precedence over the global defaults (ex‐
324       plained below).
325
326       For examples see the gabbi tests, Example Tests and the gabbi-demo  tu‐
327       torial.
328

FIXTURES

330       The top-level fixtures category contains a sequence of named Fixtures.
331

RESPONSE HANDLERS

333       response_*  keys are examples of Response Handlers. Custom handlers may
334       be created by test authors for specific use cases. See Content Handlers
335       for more information.
336

SUBSTITUTION

338       There are a number of magical variables that can be used to make refer‐
339       ence to the state of a current test, the one just  prior  or  any  test
340       prior  to  the current one. The variables are replaced with real values
341       during test processing.
342
343   Global
344$ENVIRON['<environment variable>']: The name of an environment  vari‐
345         able.  Its  value  will  replace  the magical variable. If the string
346         value of the environment variable is "True" or "False" then  the  re‐
347         sulting value will be the corresponding boolean, not a string.
348
349   Current Test
350$SCHEME: The current scheme/protocol (usually http or https).
351
352$NETLOC: The host and potentially port of the request.
353
354   Immediately Prior Test
355$COOKIE:  All  the cookies set by any Set-Cookie headers in the prior
356         response, including only the cookie key and value pairs and no  meta‐
357         data (e.g. expires or domain).
358
359$URL:  The URL defined in the prior request, after substitutions have
360         been  made.  For  backwards  compatibility  with   earlier   releases
361         $LAST_URL  may  also  be  used,  but if $HISTORY (see below) is being
362         used, $URL must be used.
363
364$LOCATION: The location header returned in the prior response.
365
366$HEADERS['<header>']: The value of any  header  from  the  prior  re‐
367         sponse.
368
369$RESPONSE['<json  path>']:  A JSONPath query into the prior response.
370         See JSONPath for more on formatting.
371
372   Any Previous Test
373$HISTORY['<test name>'].<magical variable expression>:  Any  variable
374         which refers to a prior test may be used in an expression that refers
375         to any earlier test in the same file by identifying the  target  test
376         by  its  name  in  a  $HISTORY dictionary. For example, to refer to a
377         value in a JSON object in the response of a test named post json:
378
379            $HISTORY['post json'].$RESPONSE['$.key']
380
381         This is a very powerful feature that could lead to test that are dif‐
382         ficult  for humans to read. Take care to optimize for the maintainers
383         that will come after you, not yourself.
384
385   Casting
386       For $ENVIRON and $RESPONSE it is possible to attempt to cast the  value
387       to  another type: int, float, str, or bool. If the cast fails an excep‐
388       tion will be raised and the test will fail.
389
390       This functionality only works when the magical variable  is  the  whole
391       value  of  a YAML entry. If the variable is intermixed with other data,
392       an exception will be raised and the test will fail.
393
394       The format for a cast is to append a : and the cast type after the type
395       of the magical variable. For example:
396
397          $RESPONSE:int['$.some_string_value']
398
399       WARNING:
400          Prior to the introduction of this feature, $ENVIRON would already do
401          some automatic casting of numbers to ints and floats and the strings
402          True  and False to booleans. This continues to be the case, but only
403          if no cast is provided.
404
405       NOTE:
406          Where a single-quote character, ', is shown in the  variables  above
407          you  may  also use a double-quote character, ", but in any given ex‐
408          pression the same character must be used at both ends.
409
410       All of these variables may be used in all of the following fields:
411
412skip
413
414url
415
416query_parameters
417
418data
419
420request_headers (in both the key and value)
421
422response_strings
423
424response_json_paths (in both the key and value, see json path substi‐
425         tution for more info)
426
427response_headers (in both the key and value)
428
429response_forbidden_headers
430
431count and delay fields of poll
432
433       With these variables it ought to be possible to traverse an API without
434       any explicit statements about the URLs being used. If you  need  a  re‐
435       placement  on  a  field that is not currently supported please raise an
436       issue or provide a patch.
437
438       As all of these features needed to be  tested  in  the  development  of
439       gabbi  itself,  the gabbi tests are a good source of examples on how to
440       use the functionality. See also Example Tests for a collection of exam‐
441       ples and the gabbi-demo tutorial.
442

DATA

444       The data key has some special handing to allow for a bit more flexibil‐
445       ity when doing a POST or PUT:
446
447       • If the value is not a string (that is, it is a sequence or structure)
448         it  is  treated as a data structure that will be turned into a string
449         by the dumps method on the relevant content handler. For  example  if
450         the  content-type  of the body is application/json the data structure
451         will be turned into a JSON string.
452
453       • If the value is a string that begins with <@ then  the  rest  of  the
454         string is treated as a filepath to be loaded. The path is relative to
455         the test directory and may not traverse up into parent directories.
456
457       • If the value is an undecorated string, that's the value.
458
459       NOTE:
460          When reading from a file care should be taken to ensure that a  rea‐
461          sonable content-type is set for the data as this will control if any
462          encoding is done of the resulting string value. If it is text, json,
463          xml or javascript it will be encoded to UTF-8.
464
465       To  run  gabbi tests with a test harness they must be generated in some
466       fashion and then run. This is accomplished by a test loader.  Initially
467       gabbi only supported those test harnesses that supported the load_tests
468       protocol in UnitTest. It now possible to also build and run tests  with
469       pytest with some limitations described below.
470
471       NOTE:
472          It  is  also  possible to run gabbi tests from the command line. See
473          YAML Runner.
474
475       NOTE:
476          By default gabbi will load YAML files using the safe_load  function.
477          This  means  only basic YAML types are allowed in the file. For most
478          use cases this is fine. If you need custom types  (for  example,  to
479          match  NaN)  it  is  possible  to  set  the  safe_yaml  parameter of
480          build_tests() to False.  If custom types are used,  please  keep  in
481          mind  that this can limit the portability of the YAML files to other
482          contexts.
483
484       WARNING:
485          If test are being run with a runner that supports concurrency  (such
486          as  testrepository)  it is critical that the test runner is informed
487          of how to group the tests into their respective  suites.  The  usual
488          way  to  do this is to use a regular expression that groups based on
489          the name of the yaml files. For example, when  using  testrepository
490          the .testr.conf file needs an entry similar to the following:
491
492              group_regex=gabbi\.suitemaker\.(test_[^_]+_[^_]+)
493

UNITTEST STYLE LOADER

495       To  run the tests with a load_tests style loader a test file containing
496       a load_tests method is required. That will look a bit like:
497
498          """A sample test module."""
499
500          # For pathname munging
501          import os
502
503          # The module that build_tests comes from.
504          from gabbi import driver
505
506          # We need access to the WSGI application that hosts our service
507          from myapp import wsgiapp
508
509
510          # We're using fixtures in the YAML files, we need to know where to
511          # load them from.
512          from myapp.test import fixtures
513
514          # By convention the YAML files are put in a directory named
515          # "gabbits" that is in the same directory as the Python test file.
516          TESTS_DIR = 'gabbits'
517
518
519          def load_tests(loader, tests, pattern):
520              """Provide a TestSuite to the discovery process."""
521              test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
522              # Pass "require_ssl=True" as an argument to force all tests
523              # to use SSL in requests.
524              return driver.build_tests(test_dir, loader,
525                                        intercept=wsgiapp.app,
526                                        fixture_module=fixtures)
527
528
529       For  details  on  the  arguments  available  when  building  tests  see
530       build_tests().
531
532       Once  the  test  loader has been created, it needs to be run. There are
533       many options. Which is appropriate depends very much on  your  environ‐
534       ment.  Here  are some examples using unittest or testtools that require
535       minimal knowledge to get started.
536
537       By file:
538
539          python -m testtools.run -v test/test_loader.py
540
541       By module:
542
543          python -m testttols.run -v test.test_loader
544
545          python -m unittest -v test.test_loader
546
547       Using test discovery to locate all tests in a directory tree:
548
549          python -m testtools.run discover
550
551          python -m unittest discover test
552
553       See the source distribution and the tutorial repo for more advanced op‐
554       tions, including using testrepository and subunit.
555

PYTEST

557       Since pytest does not support the load_tests system, a different way of
558       generating tests is required. Two techniques are supported.
559
560       The original method (described below) used yield statements to generate
561       tests  which pytest would collect. This style of tests is deprecated as
562       of pytest>=3.0 so a new style using pytest fixtures has been developed.
563
564       WARNING:
565          The pytest loader now requires that  test_loader_name  be  set  when
566          gabbi.driver.py_test_generator() is called.
567
568   pytest >= 3.0
569       In  the  newer  technique,  a  test  file  is  created  that  uses  the
570       pytest_generate_tests hook. Special care must be taken to always import
571       the  test_pytest  method  which  is  the base test that the pytest hook
572       parametrizes to generate the tests from the YAML  files.   Without  the
573       method, the hook will not be called and no tests generated.
574
575       Here is a simple example file:
576
577          """A sample pytest module for pytest >= 3.0."""
578
579          # For pathname munging
580          import os
581
582          # The module that py_test_generator comes from.
583          from gabbi import driver
584
585          # We need test_pytest so that pytest test collection works properly.
586          # Without this, the pytest_generate_tests method below will not be
587          # called.
588          from gabbi.driver import test_pytest  # noqa
589
590          # We need access to the WSGI application that hosts our service
591          from myapp import wsgiapp
592
593          # We're using fixtures in the YAML files, we need to know where to
594          # load them from.
595          from myapp.test import fixtures
596
597          # By convention the YAML files are put in a directory named
598          # "gabbits" that is in the same directory as the Python test file.
599          TESTS_DIR = 'gabbits'
600
601
602          def pytest_generate_tests(metafunc):
603              test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
604              driver.py_test_generator(
605                  test_dir, intercept=wsgiapp.app,
606                  test_loader_name=__name__,
607                  fixture_module=fixtures, metafunc=metafunc)
608
609
610       This can then be run with the usual pytest commands. For example:
611
612          py.test -svx pytest3.0-example.py
613
614   pytest < 3.0
615       When  using  the  older technique, test file must be created that calls
616       py_test_generator() and yields the generated tests. That  will  look  a
617       bit like this:
618
619          """A sample pytest module."""
620
621          # For pathname munging
622          import os
623
624          # The module that build_tests comes from.
625          from gabbi import driver
626
627          # We need access to the WSGI application that hosts our service
628          from myapp import wsgiapp
629
630          # We're using fixtures in the YAML files, we need to know where to
631          # load them from.
632          from myapp.test import fixtures
633
634          # By convention the YAML files are put in a directory named
635          # "gabbits" that is in the same directory as the Python test file.
636          TESTS_DIR = 'gabbits'
637
638
639          def test_gabbits():
640              test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
641              # Pass "require_ssl=True" as an argument to force all tests
642              # to use SSL in requests.
643              test_generator = driver.py_test_generator(
644                  test_loader_name=__name__,
645                  test_dir, intercept=wsgiapp.app,
646                  fixture_module=fixtures)
647
648              for test in test_generator:
649                  yield test
650
651
652       This can then be run with the usual pytest commands. For example:
653
654          py.test -svx pytest-example.py
655
656       The  older  technique  will  continue  to  work  with  all  versions of
657       pytest<4.0 but >=3.0 will produce warnings. If  you  want  to  use  the
658       older  technique but not see the warnings add --disable-pytest-warnings
659       parameter to the invocation of py.test.
660
661       What follows is a commented example of some  tests  in  a  single  file
662       demonstrating many of the Test Format features. See Loading and Running
663       Tests for the Python needed to integrate with a testing harness.
664
665
666          # Fixtures can be used to set any necessary configuration, such as a
667          # persistence layer, and establish sample data. They operate per
668          # file. They are context managers, each one wrapping the next in the
669          # sequence.
670
671          fixtures:
672              - ConfigFixture
673              - SampleDataFixture
674
675          # There is an included fixture named "SkipAllFixture" which can be
676          # used to declare that all the tests in the given file are to be
677          # skipped.
678
679          # Each test file can specify a set of defaults that will be used for
680          # every request. This is useful for always specifying a particular
681          # header or always requiring SSL. These values will be used on every
682          # test in the file unless overriden. Lists and dicts are merged one
683          # level deep, except for "data" which is copied verbatim whether it
684          # is a string, list or dict (it can be all three).
685
686          defaults:
687              ssl: True
688              request_headers:
689                  x-my-token: zoom
690
691          # The tests themselves are a list under a "tests" key. It's useful
692          # to use plenty of whitespace to help readability.
693
694          tests:
695
696          # Each request *must* have a name which is unique to the file. When it
697          # becomes a TestCase the name will be lowercased and spaces will
698          # become "_". Use that generated name when limiting test runs.
699
700              - name: a test for root
701                desc: Some explanatory text that could be used by other tooling
702
703          # The URL can either be relative to a host specified elsewhere or
704          # be a fully qualified "http" or "https" URL. *You* are responsible
705          # for url-encoding the URL.
706
707                url: /
708                method: GET
709
710          # If no status or method are provided they default to "200" and
711          # "GET".
712
713          # Instead of explicitly stating "url" and "method" you can join
714          # those two keys into one key representing the method. The method
715          # *must* be uppercase.
716
717              - name: another test for root
718                desc: Same test as above but with GET key
719                GET: /
720
721          # A single test can override settings in defaults (set above).
722
723              - name: root without ssl redirects
724                ssl: False
725                GET: /
726                status: 302
727
728          # When evaluating response headers it is possible to use a regular
729          # expression to not have to test the whole value. Regular expressions match
730          # anywhere in the output, not just at the beginning.
731
732                response_headers:
733                    location: /^https/
734
735          # By default redirects will not be followed. This can be changed.
736
737              - name: follow root without ssl redirect
738                ssl: False
739                redirects: True
740                GET: /
741                status: 200 # This is the response code after the redirect.
742
743          # URLs can express query parameters in two ways: either in the url
744          # value directly, or as query_parameters. If both are used then
745          # query_parameters are appended. In this example the resulting URL
746          # will be equivalient to
747          # /foo?section=news&article=1&article=2&date=yesterday
748          # but not necessarily in that order.
749
750              - name: create a url with parameters
751                GET: /foo?section=news
752                query_parameters:
753                    article:
754                        - 1
755                        - 2
756                    date: yesterday
757
758          # Request headers can be used to declare media-type choices and
759          # experiment with authorization handling (amongst other things).
760          # Response headers allow evaluating headers in the response. These
761          # two together form the core value of gabbi.
762
763              - name: test accept
764                GET: /resource
765                request_headers:
766                    accept: application/json
767                response_headers:
768                    content-type: /application/json/
769
770          # If a header must not be present in a response at all that can be
771          # expressed in a test as follows.
772
773              - name: test forbidden headers
774                GET: /resource
775                response_forbidden_headers:
776                    - x-special-header
777
778          # All of the above requests have defaulted to a "GET" method. When
779          # using "POST", "PUT" or "PATCH", the "data" key provides the
780          # request body.
781
782              - name: post some text
783                POST: /text_repo
784                request_headers:
785                    content-type: text/plain
786                data: "I'm storing this"
787                status: 201
788
789          # If the data is not a string, it will be transformed into JSON.
790          # You must supply an appropriate content-type request header.
791
792              - name: post some json
793                POST: /json_repo
794                request_headers:
795                    content-type: application/json
796                data:
797                    name: smith
798                    abode: castle
799                status: 201
800
801          # If the data is a string prepended with "<@" the value will be
802          # treated as the name of a file in the same directory as the YAML
803          # file. Again, you must supply an appropriate content-type. If the
804          # content-type is one of several "text-like" types, the content will
805          # be assumed to be UTF-8 encoded.
806
807              - name: post an image
808                POST: /image_repo
809                request_headers:
810                    content-type: image/png
811                data: <@kittens.png
812
813          # A single request can be marked to be skipped.
814
815              - name: patch an image
816                skip: patching images not yet implemented
817                PATCH: /image_repo/12d96fb8-e78c-11e4-8c03-685b35afa334
818
819          # Or a single request can be marked that it is expected to fail.
820
821              - name: check allow headers
822                desc: the framework doesn't do allow yet
823                xfail: True
824                PUT: /post_only_url
825                status: 405
826                response_headers:
827                    allow: POST
828
829          # The body of a response can be evaluated with response handlers.
830          # The most simple checks for a set of strings anywhere in the
831          # response. Note that the strings are members of a list.
832
833              - name: check for css file
834                GET: /blog/posts/12
835                response_strings:
836                    - normalize.css
837
838          # For JSON responses, JSONPath rules can be used.
839
840              - name: post some json get back json
841                POST: /json_repo
842                request_headers:
843                    content-type: application/json
844                data:
845                    name: smith
846                    abode: castle
847                status: 201
848                response_json_paths:
849                    $.name: smith
850                    $.abode: castle
851
852          # Requests run in sequence. One test can make reference to the test
853          # immediately prior using some special variables.
854          # "$LOCATION" contains the "location" header in the previous
855          # response.
856          # "$HEADERS" is a pseudo dictionary containing all the headers of
857          # the previous response.
858          # "$ENVIRON" is a pseudo dictionary providing access to the current
859          # environment.
860          # "$RESPONSE" provides access to the JSON in the prior response, via
861          # JSONPath. See http://jsonpath-rw.readthedocs.io/ for
862          # jsonpath-rw formatting.
863          # $SCHEME and $NETLOC provide access to the current protocol and
864          # location (host and port).
865
866              - name: get the thing we just posted
867                GET: $LOCATION
868                request_headers:
869                    x-magic-exchange: $HEADERS['x-magic-exchange']
870                    x-token: $ENVIRON['OS_TOKEN']
871                response_json_paths:
872                    $.name: $RESPONSE['$.name']
873                    $.abode: $RESPONSE['$.abode']
874                response_headers:
875                    content-location: /$SCHEME://$NETLOC/
876
877          # For APIs where resource creation is asynchronous it can be
878          # necessary to poll for the resulting resource. First we create the
879          # resource in one test. The next test uses the "poll" key to loop
880          # with a delay for a set number of times.
881
882              - name: create asynch
883                POST: /async_creator
884                request_headers:
885                    content-type: application/json
886                data:
887                    name: jones
888                    abode: bungalow
889                status: 202
890
891              - name: poll for created resource
892                GET: $LOCATION
893                poll:
894                    count: 10 # try up to ten times
895                    delay: .5 # wait .5 seconds between each try
896                response_json_paths:
897                    $.name: $RESPONSE['$.name']
898                    $.abode: $RESPONSE['$.abode']
899
900
901       Gabbi supports JSONPath both for validating JSON  response  bodies  and
902       within substitutions.
903
904       JSONPath  expressions are provided by jsonpath_rw, with jsonpath_rw_ext
905       custom extensions to address common requirements:
906
907       1. Sorting via sorted and [/property].
908
909       2. Filtering via [?property = value].
910
911       3. Returning the respective length via len.
912
913       (These apply both to arrays and key-value pairs.)
914
915       Here is a JSONPath example demonstrating some of these features.  Given
916       JSON data as follows:
917
918          {
919              "pets": [
920                  {"type": "cat", "sound": "meow"},
921                  {"type": "dog", "sound": "woof"}
922              ]
923          }
924
925       If  the  ordering of the list in pets is predictable and reliable it is
926       relatively straightforward to test values:
927
928          response_json_paths:
929              # length of list is two
930              $.pets.`len`: 2
931              # sound of second item in list is woof
932              $.pets[1].sound: woof
933
934       If the ordering is not predictable additional effort is required:
935
936          response_json_paths:
937              # sort by type
938              $.pets[/type][0].sound: meow
939              # sort by type, reversed
940              $.pets[\type][0].sound: woof
941              # all the sounds
942              $.pets[/type]..sound: ['meow', 'woof']
943              # filter by type = dog
944              $.pets[?type = "dog"].sound: woof
945
946       If it is necessary to validate the entire JSON response use a  JSONPath
947       of $:
948
949          response_json_paths:
950              $:
951                  pets:
952                      - type: cat
953                        sound: meow
954                      - type: dog
955                        sound: woof
956
957       This  is  not a technique that should be used frequently as it can lead
958       to difficult to read tests and it also indicates that your gabbi  tests
959       are  being used to test your serializers and data models, not just your
960       API interactions.
961
962       It is also possible to read raw JSON from disk for either all  or  some
963       of a JSON response:
964
965          response_json_paths:
966              $: @<data.json
967
968       or:
969
970          response_json_paths:
971              $.pets: <@pets.json
972              $.pets[0]: <@cat.json
973
974       Examples like this can be found in one of gabbi's own tests.
975
976       If  it  is  desired  to  load  YAML files like the JSON ones above, two
977       things must be done:
978
979       1. The YAMLDiskLoadingJSONHandler custom content handler must be passed
980          to  the driver through the content_handlers argument. See Extensions
981          on how to do this.
982
983       2. The YAML files to load must be placed in a subdirectory  to  prevent
984          the test runner from consuming them as test files to run:
985
986             response_json_paths:
987                 $: @<subdir/values.yaml
988
989       When  reading from disk you can apply the same JSONPath by adding a ':'
990       to the end of your file name. This allows you to store multiple API re‐
991       sponses  into a single file to reduce file management when constructing
992       your tests.
993
994       Given JSON data as follows:
995
996          {
997              "values": [{
998                  "pets": [{
999                      "type": "cat",
1000                      "sound": "meow"
1001                  }, {
1002                      "type": "dog",
1003                      "sound": "woof"
1004                  }]
1005              }, {
1006                  "people": [{
1007                      "name": "chris",
1008                      "id": 1
1009                  }, {
1010                      "name": "justin",
1011                      "id": 2
1012                  }]
1013              }]
1014          }
1015
1016       You can write your tests like the following:
1017
1018          response_json_paths:
1019              $.pets: <@pets.json
1020              $.pets[?type = "cat"].sound: <@values.json:$.values[0].pets[?type = "cat"].sound
1021
1022       Although placing more than one API response into a single JSON file may
1023       seem  convenient,  keep in mind there is a tradeoff in readability that
1024       should not be overlooked before implementing this technique.
1025
1026       Examples like this can be found in one of gabbi's yaml-from-disk tests.
1027
1028       There  are  more  JSONPath  examples  in  Example  Tests  and  in   the
1029       jsonpath_rw and jsonpath_rw_ext documentation.
1030

SUBSTITUTION

1032       Substitutions can be made in both the left (query) and right (expected)
1033       hand sides of the json path expression. When subtitutions are  used  in
1034       the query, care must be taken to ensure proper quoting of the resulting
1035       value.  For  example  if  there  is  a  uuid  (with  hyphens)  at  $RE‐
1036       SPONSE['$.id'] then this expression may fail:
1037
1038          $.nested.structure.$RESPONSE['$.id'].name: foobar
1039
1040       as it will evaluate to something like:
1041
1042          $.nested.structure.ADC8AAFC-D564-40D1-9724-7680D3C010C2.name: foobar
1043
1044       which  may  be  treated  as  an  arithemtic expression by the json path
1045       parser. The test author should write:
1046
1047          $.nested.structure["$RESPONSE['$.id']"].name: foobar
1048
1049       to quote the result of the substitution.
1050
1051       The target host is the host on which the API to be tested can be found.
1052       Gabbi  intends  to preserve the flow and semantics of HTTP interactions
1053       as much as possible, and every HTTP request needs to be directed  at  a
1054       host of some form. Gabbi provides three ways to control this:
1055
1056       • Using wsgi-intercept to provide a fake socket and WSGI environment on
1057         an arbitrary host and  port  attached  to  a  WSGI  application  (see
1058         intercept examples).
1059
1060       • Using  fully qualified url values in the YAML defined tests (see full
1061         examples).
1062
1063       • Using a host and (optionally) port defined at test  build  time  (see
1064         live examples).
1065
1066       The intercept and live methods are mutually exclusive per test builder,
1067       but either kind of test can freely intermix fully qualified  URLs  into
1068       the sequence of tests in a YAML file.
1069
1070       For  test  driven  development  and  local tests the intercept style of
1071       testing lowers test requirements (no web server required) and is  fast.
1072       Interception  is  performed  as part of Fixtures processing as the most
1073       deeply nested fixture. This allows any configuration or database  setup
1074       to be performed prior to the WSGI application being created.
1075
1076       For the implementation of the above see build_tests().
1077
1078       Each  suite  of tests is represented by a single YAML file, and may op‐
1079       tionally use one or more fixtures to provide the necessary  environment
1080       required by the tests in that file.
1081
1082       Fixtures  are  implemented  as  nested  context managers. Subclasses of
1083       GabbiFixture must implement start_fixture and stop_fixture methods  for
1084       creating  and  destroying,  respectively,  any resources managed by the
1085       fixture.  While the subclass may choose to implement __init__ it is im‐
1086       portant  that  no  exceptions  are thrown in that method, otherwise the
1087       stack of context managers will fail in unexpected  ways.  Instead  ini‐
1088       tialization of real resources should happen in start_fixture.
1089
1090       At this time there is no mechanism for the individual tests to have any
1091       direct awareness of the fixtures. The fixtures exist, conceptually,  on
1092       the server side of the API being tested.
1093
1094       Fixtures  may  do whatever is required by the testing environment, how‐
1095       ever there are two common scenarios:
1096
1097       • Establishing (and then resetting when a test suite has finished)  any
1098         baseline  configuration settings and persistence systems required for
1099         the tests.
1100
1101       • Creating sample data for use by the tests.
1102
1103       If a fixture raises unittest.case.SkipTest during start_fixture all the
1104       tests  in  the current file will be skipped.  This makes it possible to
1105       skip the tests if some optional configuration  (such  as  a  particular
1106       type of database) is not available.
1107
1108       If  an  exception  is raised while a fixture is being used, information
1109       about the exception will be stored on the fixture so that the stop_fix‐
1110       ture  method  can decide if the exception should change how the fixture
1111       should clean up. The exception information can be  found  on  exc_type,
1112       exc_value and traceback method attributes.
1113
1114       If  an exception is raised when a fixture is started (in start_fixture)
1115       the first test in the suite using the fixture will be  marked  with  an
1116       error  using  the traceback from the exception and all the tests in the
1117       suite will be skipped. This ensures that fixture failure is  adequately
1118       captured and reported by test runners.
1119
1120       In  some  contexts  (for example CI environments with a large number of
1121       tests being run in a broadly concurrent  environment  where  output  is
1122       logged to a single file) it can be important to capture and consolidate
1123       stray output that is produced during the tests and display  it  associ‐
1124       ated  with an individual test. This can help debugging and avoids unus‐
1125       able output that is the result of multiple streams being interleaved.
1126
1127       Inner fixtures have been added to support this. These are fixtures more
1128       in  line  with  the tradtional unittest concept of fixtures: a class on
1129       which setUp and cleanUp is automatically called.
1130
1131       build_tests() accepts a named parameter  arguments  of  inner_fixtures.
1132       The  value  of that argument may be an ordered list of fixtures.Fixture
1133       classes that will be called when each individual test is set up.
1134
1135       An example fixture that could be useful is the FakeLogger.
1136
1137       NOTE:
1138          At this time inner_fixtures are not supported when using the  pytest
1139          loader.
1140
1141       Content  handlers are responsible for preparing request data and evalu‐
1142       ating response data based on the content-type of the  request  and  re‐
1143       sponse. A content handler operates as follows:
1144
1145       • Structured  YAML data provided via the data attribute is converted to
1146         a string or bytes sequence and used as request body.
1147
1148       • The response body (a string or sequence of bytes) is transformed into
1149         a  content-type  dependent structure and stored in an internal attri‐
1150         bute named response_data that is:
1151
1152         • used when evaluating the response body
1153
1154         • used in $RESPONSE[] substitutions
1155
1156       By default, gabbi provides content handlers for JSON. In  that  content
1157       handler the data test key is converted from structured YAML into a JSON
1158       string. Response bodies are converted from a JSON string  into  a  data
1159       structure   in   response_data   that   is  used  when  evaluating  re‐
1160       sponse_json_paths entries in a test or doing JSONPath-based $RESPONSE[]
1161       substitutions.
1162
1163       A YAMLDiskLoadingJSONHandler has been added to extend the JSON handler.
1164       It works the same way as the JSON handler except  for  when  evaluating
1165       the  response_json_paths handle, data that is read from disk can be ei‐
1166       ther in JSON or YAML format. The YAMLDiskLoadingJSONHandler is not  en‐
1167       abled  by  default and must be added as shown in the Extensions section
1168       in order to be used in the tests.
1169
1170       Further content handlers can be added as extensions. Test  authors  may
1171       need  these extensions for their own suites, or enterprising developers
1172       may wish to create and distribute extensions for others to use.
1173
1174       NOTE:
1175          One extension that is likely to be useful is a content handler  that
1176          turns data into url-encoded form data suitable for POST and turns an
1177          HTML response into a DOM object.
1178

EXTENSIONS

1180       Content handlers are an evolution of the response  handler  concept  in
1181       earlier versions gabbi. To preserve backwards compatibility with exist‐
1182       ing response handlers, old style response handlers are  still  allowed,
1183       but  new  handlers  should implement the content handler interface (de‐
1184       scribed below).
1185
1186       Registering additional custom handlers is done by passing a subclass of
1187       ContentHandler to build_tests():
1188
1189          driver.build_tests(test_dir, loader, host=None,
1190                             intercept=simple_wsgi.SimpleWsgi,
1191                             content_handlers=[MyContentHandler])
1192
1193       If pytest is being used:
1194
1195          driver.py_test_generator(test_dir, intercept=simple_wsgi.SimpleWsgi,
1196                                   test_loader_name=__name__,
1197                                   content_handlers=[MyContenHandler])
1198
1199       Gabbi  provides an additional custom handler named YAMLDiskLoadingJSON‐
1200       Handler.  This can be used for loading JSON and YAML  files  from  disk
1201       when evaluating the response_json_paths handle.
1202
1203       WARNING:
1204          YAMLDiskLoadingJSONHandler  shares  the same content-type as the de‐
1205          fault JSONHandler. When there are multiple handlers listed that  ac‐
1206          cept  the  same  content-type,  the one that is earliest in the list
1207          will be used.
1208
1209       With gabbi-run, custom handlers can be loaded via  the  --response-han‐
1210       dler option -- see load_response_handlers() for details.
1211
1212       NOTE:
1213          The use of the --response-handler argument is done to preserve back‐
1214          wards compatibility and avoid excessive arguments.   Both  types  of
1215          handler may be passed to the argument.
1216
1217   Implementation Details
1218       Creating  a content handler requires subclassing ContentHandler and im‐
1219       plementing several methods.  These methods are described below, but in‐
1220       specting JSONHandler will be instructive in highlighting required argu‐
1221       ments and techniques.
1222
1223       To provide a response_<something> response-body  evaluator  a  subclass
1224       must define:
1225
1226test_key_suffix: This, along with the prefix response_, forms the key
1227         used in the test structure. It is a class level string.
1228
1229test_key_value: The key's default value, either an empty list ([]) or
1230         empty dict ({}). It is a class level value.
1231
1232action:  An  instance  method which tests the expected values against
1233         the HTTP response - it is invoked for each entry, with the parameters
1234         depending  on  the default value. The arguments to action are (in or‐
1235         der):
1236
1237self: The current instance.
1238
1239test: The currently active HTTPTestCase
1240
1241item: The current entry if test_key_value is a list, otherwise  the
1242           key half of the key/value pair at this entry.
1243
1244value:  None  if test_key_value is a list, otherwise the value half
1245           of the key/value pair at this entry.
1246
1247       To translate request or response bodies to or from  structured  data  a
1248       subclass must define an accepts method. This should return True if this
1249       class is willing to translate the provided content-type. During request
1250       processing  it  is given the value of the content-type header that will
1251       be sent in the request. During response  processing  it  is  given  the
1252       value  of the content-type header of the response. This makes it possi‐
1253       ble to handle different request and response bodies in  the  same  han‐
1254       dler,   if  desired.  For  example  a  handler  might  accept  applica‐
1255       tion/x-www-form-urlencoded and text/html.
1256
1257       If accepts is defined two additional static methods should be defined:
1258
1259dumps: Turn structured Python data from the data key in a test into a
1260         string  or  byte stream. The optional test param allows you to access
1261         the current test case which may help with  manipulations  for  custom
1262         content handlers, e.g. multipart/form-data needs to add a boundary to
1263         the Content-Type header in order to mark the appropriate sections  of
1264         the body.
1265
1266loads:  Turn a string or byte stream in a response into a Python data
1267         structure. Gabbi will put this data on the response_data attribute on
1268         the test, where it can be used in the evaluations described above (in
1269         the  action method) or in $RESPONSE handling.  An example usage  here
1270         would be to turn HTML into a DOM.
1271
1272load_data_file:  Load  data  from  disk into a Python data structure.
1273         Gabbi will call this method  when  response_<something>  contains  an
1274         item  where  the right hand side value starts with <@. The test param
1275         allows  you  to  access  the  current  test  case  and   provides   a
1276         load_data_file  method  which  should be used because it verifies the
1277         data is loaded within the test diectory and returns the  file  source
1278         as  a  string. The load_data_file method was introduced to re-use the
1279         JSONHandler in order to support loading YAML files from disk  through
1280         the   implementation   of   an   additional   custom   handler,   see
1281         YAMLDiskLoadingJSONHandler for details.
1282
1283       Finally if a replacer class method is defined, then  when  a  $RESPONSE
1284       substitution  is encountered, replacer will be passed the response_data
1285       of the prior test and the argument within the $RESPONSE.
1286
1287       Please see the JSONHandler source for additional detail.
1288
1289       If there is a running web service that needs to be tested and  creating
1290       a  test loader with build_tests() is either inconvenient or overkill it
1291       is possible to run YAML test files directly from the command line  with
1292       the  console-script  gabbi-run. It accepts YAML on stdin or as multiple
1293       file arguments, and generates and runs tests and outputs a  summary  of
1294       the results.
1295
1296       The  provided  YAML  may not use custom Fixtures but otherwise uses the
1297       default Test Format. Target Host information is  either  expressed  di‐
1298       rectly in the YAML file or provided on the command line:
1299
1300          gabbi-run [host[:port]] < /my/test.yaml
1301
1302       or:
1303
1304          gabbi-run http://host:port < /my/test.yaml
1305
1306       To  test  with  one  or  more files the following command syntax may be
1307       used:
1308
1309          gabbi-run http://host:port -- /my/test.yaml /my/other.yaml
1310
1311       NOTE:
1312          The filename arguments must come after a -- and all other  arguments
1313          (host, port, prefix, failfast) must come before the --.
1314
1315       NOTE:
1316          If files are provided, test output will use names including the name
1317          of the file. If any single file includes an error, the name  of  the
1318          file will be included in a summary of failed files at the end of the
1319          test report.
1320
1321       To facilitate using the same tests against the same application mounted
1322       in  different locations in a WSGI server, a prefix may be provided as a
1323       second argument:
1324
1325          gabbi-run host[:port] [prefix] < /my/test.yaml
1326
1327       or in the target URL:
1328
1329          gabbi-run http://host:port/prefix < /my/test.yaml
1330
1331       The value of prefix will be prepended to the path portion of URLs  that
1332       are not fully qualified.
1333
1334       Anywhere host is used, if it is a raw IPV6 address it should be wrapped
1335       in [ and ].
1336
1337       If https is used in the target, then the tests  in  the  provided  YAML
1338       will default to ssl: True.
1339
1340       Use  -k  or  --insecure  to not validate certificates when making https
1341       connections.
1342
1343       If a -x or --failfast argument is provided then gabbi-run will exit af‐
1344       ter the first test failure.
1345
1346       Use  -v  or  --verbose  with a value of all, headers or body to turn on
1347       verbosity for all tests being run.
1348
1349       Use -q or --quiet to silence test runner output.
1350
1351       Use -r or --response-handler to load a custom response or content  han‐
1352       dler for use with tests.
1353
1354       Use -l to load response handlers relative to the current working direc‐
1355       tory.
1356
1357       For example, to load a handler named HTMLHandler from the handlers.html
1358       module relative to the current directory:
1359          gabbi-run   -l  -r  handlers.html:HTMLHandler  http://example.com  <
1360          my.yaml
1361
1362       These are informal release notes for gabbi since version  1.0.0,  high‐
1363       lighting  major  features  and  changes. For more detail see the commit
1364       logs on GitHub.
1365

2.7.2

1367       • Handle SNI when host request header is any case.
1368

2.7.1

1370       • Clear the PoolManager after each test to avoid running out of  socket
1371         file descriptors.
1372

2.7.0

1374       • Add  -l  argument  to gabbi-run to allow loading content and response
1375         handlers relative to the local directory.
1376

2.6.0

1378       • Fix naming of tests when using pytest. This changes means  that  set‐
1379         ting test_loader_name is now required when using pytest.
1380

2.5.0

1382       • Support SNI in https requests when host request header is set.
1383

2.4.0

1385       • Publish  a  docker container that runs gabbi-run with each tagged re‐
1386         lease.
1387

2.3.0

1389       • For the $ENVIRON and $RESPONSE substitutions it is  now  possible  to
1390         cast the value to a type of int, float, str, or bool.
1391
1392       • The  JSONHandler  is now more strict about how it detects that a body
1393         content is JSON, avoiding some errors where the  content-type  header
1394         suggests JSON but the content cannot be decoded as such.
1395
1396       • Better error message when content cannot be decoded.
1397
1398       • Addition of the disable_response_handler test setting for those cases
1399         when the test author has no control over the content-type header  and
1400         it is wrong.
1401

2.2.0

1403       • Allow substitutions in skip for more flexible skipping, notablly with
1404         $ENVIRON.
1405
1406       • Better test name output when running with unittest and verbose.
1407

2.1.0

1409       • Remove support for Python 3.5. Ensure testing support for  3.8,  3.9,
1410         and pypy3.
1411
1412       • Adapt to new behavior in urllib.parse.
1413
1414       • Correct fixture start when using pytest.
1415

2.0.4

1417       • If  no content-type is provided with a response and verbose is on for
1418         responses, display the response body as if it were text.
1419

2.0.3

1421       • Properly declare that gabbi 2.x is Python 3 only.
1422

2.0.1

1424       • Correct management of response handler default fields.
1425

2.0.0

1427       • Drop support for Python 2. If you need Python 2 support, use an older
1428         version.
1429
1430       • Stop  using testtools and fixtures. These two modules present several
1431         difficulties and their maintenance situation suggests those difficul‐
1432         ties  will  not be resolved. Since Python 2 support is being removed,
1433         the need for the modules can be removed as well without losing  func‐
1434         tionality.  "Inner  fixtures" that use the fixtures.Fixture interface
1435         should continue to work.
1436

1.49.0

1438       • Add support for not validating certificates in https  requests.  Con‐
1439         trolled  by  the  cert_validate  attribute  in  individual  tests and
1440         build_tests() and the -k or --insecure argument to gabbi-run.
1441

1.48.0

1443       • Support pytest 5.0.0 in Python >=3.5. For earlier versions of Python,
1444         pytest<5.0.0 will be used; the pytest project is dropping support for
1445         older versions of Python.
1446

1.47.0

1448       • Use pytest<5.0.0 until gabbi has solutions for the changes in 5.0.0.
1449

1.46.0

1451       • A -q argument is added to gabbi-run to suppress output from the  test
1452         runner.
1453

1.45.0

1455       • Adjust loading of YAML to be ready for new release of PyYAML.
1456

1.44.0

1458       • Provide  the YAMLDiskLoadingJSONHandler class that allows test result
1459         data for response_json_path checks to be loaded from YAML-on-disk.
1460

1.43.0

1462       • Use  JSONPath  to  select  a   portion   of   data-on-disk   in   re‐
1463         sponse_json_path checks.
1464
1465       • Restrict PyYAML to <4.0.
1466

1.42.0

1468       • Allow listing of tests with no host configured. When host is an empty
1469         string, tests can be listed (for discovery), but will be  skipped  on
1470         run.
1471

1.41.0

1473       • JSON  $RESPONSE  substitutions in the data field may be complex types
1474         (lists and dicts), not solely strings.
1475

1.40.0

1477       • When the HTTP response begins with a bad status  line,  have  BadSta‐
1478         tusLine be raised from urllib3.
1479

1.39.0

1481       • Allow  substitutions in the key portion of request and response head‐
1482         ers, not just the value.
1483

1.38.0

1485       • Remove support for Python 3.3.
1486
1487       • Make handling of fixture-level skips in pytest actually work.
1488

1.37.0

1490       • Add safe_yaml parameter to build_tests().
1491

1.36.0

1493use_prior_test is added to test Metadata.
1494
1495       • Extensive cleanups in regular expression handling  when  constructing
1496         tests from YAML.
1497

1.35.0

1499       JSONPath handling gets two improvements:
1500
1501       • The  value  side  of a response_json_paths entry can be loaded from a
1502         file using the <@file.json syntax also used in Data.
1503
1504       • The key side of a response_json_paths entry  can  use  substitutions.
1505         This was already true for the value side.
1506

1.34.0

1508       Substitutions  in $RESPONSE handling now preserve numeric types instead
1509       of casting to a string. This  is  useful  when  servers  are  expecting
1510       strong types and tests want to send response data back to the server.
1511

1.33.0

1513       count      and     delay     test     keys     allow     substitutions.
1514       gabbi.driver.build_tests() accepts a  verbose  parameter  to  set  test
1515       verbosity for an entire session.
1516

1.32.0

1518       Better failure reporting when using gabbi-run with multiple files. Test
1519       names are based on the files and a summary of failed files is  provided
1520       at the end of the report.
1521

1.31.0

1523       Effectively  capture  a  failure in a fixture and report the traceback.
1524       Without this some test runners swallow the error and discovering  prob‐
1525       lems when developing fixtures can be quite challenging.
1526

1.30.0

1528       Thanks to Samuel Fekete, tests can use the $HISTORY dictionary to refer
1529       to any prior test in the same file, not just the one immediately prior,
1530       when doing substitutions.
1531

1.29.0

1533       Filenames  used to read data into tests using the <@ syntax may now use
1534       pathnames relative to the YAML file. See Data.
1535
1536       gabbi-run gains a --verbose parameter to force all tests run in a  ses‐
1537       sion to run with verbose set.
1538
1539       When  using  pytest  to  load tests, a new mechanism is available which
1540       avoids warnings produced in when using a version of pytest greater than
1541       3.0.
1542

1.28.0

1544       When  verbosely  displaying  request and response bodies that are JSON,
1545       pretty print for improved readability.
1546

1.27.0

1548       Allow gabbi-run to accept multiple filenames as command line  arguments
1549       instead of reading tests from stdin.
1550

1.26.0

1552       Switch  from response handlers to Content Handlers to allow more flexi‐
1553       ble processing of both response _and_ request bodies.
1554
1555       Add inner fixtures for per test fixtures, useful for output capturing.
1556

1.25.0

1558       Allow the test_loader_name arg to gabbi.driver.build_tests()  to  over‐
1559       ride the prefix of the pretty printed name of generated tests.
1560

1.24.0

1562       String  values  in  JSONPath  matches  may  be  wrapped in /.../` to be
1563       treated as regular expressions.
1564

1.23.0

1566       Better documentation of how to run gabbi in a  concurrent  environment.
1567       Improved handling of pytest fixtures and test counts.
1568

1.22.0

1570       Add  url to gabbi.driver.build_tests() to use instead of host, port and
1571       prefix.
1572

1.21.0

1574       Add require_ssl to gabbi.driver.build_tests() to force use of SSL.
1575

1.20.0

1577       Add $COOKIE substitution.
1578

1.19.1

1580       Correctly support IPV6 hosts.
1581

1.19.0

1583       Add $LAST_URL substitution.
1584

1.17.0

1586       Introduce support for loading and running tests with pytest.
1587

1.16.0

1589       Use urllib3 instead of httplib2 for driving HTTP requests.
1590

1.13.0

1592       Add sorting and filtering to JSONPath handling.
1593

1.11.0

1595       Add the response_forbidden_headers to response expectations.
1596

1.7.0

1598       Instead of:
1599
1600          tests:
1601          - name: a simple get
1602            url: /some/path
1603            method: get
1604
1605       1.7.0 also makes it possible to:
1606
1607          tests:
1608          - name: a simple get
1609            GET: /some/path
1610
1611       Any upper case key is treated as a method.
1612

1.4.0 AND 1.5.0

1614       Enhanced flexibility and colorization when setting tests to be verbose.
1615

1.3.0

1617       Adds the query_parameters key to request parameters.
1618

1.2.0

1620       The start of improvements and extensions to JSONPath handling. In  this
1621       case the addition of the len function.
1622

1.1.0

1624       Vastly improved output and behavior in gabbi-run.
1625

1.0.0

1627       Version 1 was the first release with a commitment to a stable Test For‐
1628       mat. Since then new fields have been added  but  have  not  been  taken
1629       away.
1630
1631       The  following  people  have contributed code to gabbi. Thanks to them.
1632       Thanks also to all the people who have made gabbi better  by  reporting
1633       issues and their successes and failures with using gabbi.
1634
1635       • Chris Dent
1636
1637       • FND
1638
1639       • Mehdi Abaakouk
1640
1641       • Tom Viner
1642
1643       • Jason Myers
1644
1645       • Josh Leeb-du Toit
1646
1647       • Duc Truong
1648
1649       • Zane Bitter
1650
1651       • Ryan Spencer
1652
1653       • Kim Raymoure
1654
1655       • Travis Truman
1656
1657       • Samuel Fekete
1658
1659       • Michael McCune
1660
1661       • Imran Hayder
1662
1663       • Julien Danjou
1664
1665       • Trevor McCasland
1666
1667       • Danek Duvall
1668
1669       • Marc Abramowitz
1670
1671       • Scott Wallace
1672
1673       NOTE:
1674          This  section  provides  a collection of questions with answers that
1675          don't otherwise fit in the rest of the documentation.  If  something
1676          is missing, please create an issue.
1677
1678          As this document grows it will gain a more refined structure.
1679

GENERAL

1681   Is gabbi only for testing Python-based APIs?
1682       No, you can use gabbi-run to test an HTTP service built in any program‐
1683       ming language.
1684
1685   How do I run just one test?
1686       Each YAML file contains a sequence of tests, each test within each file
1687       has  a name. That name is translated to the name of the test by replac‐
1688       ing spaces with an _.
1689
1690       When running tests that are generated dynamically, filtering  based  on
1691       the  test  name prior to the test being collected will not work in some
1692       test runners.  Test runners that use a --load-list functionality can be
1693       convinced to filter after discovery.
1694
1695       pytest does this directly with the -k keyword flag.
1696
1697       When  using  testrepository with tox as used in gabbi's own tests it is
1698       possible to pass a filter in the tox command:
1699
1700          tox -epy27 -- get_the_widget
1701
1702       When using testtools.run and similar test runners it's a bit more  com‐
1703       plicated.  It  is  necessary  to provide the full name of the test as a
1704       list to --load-list:
1705
1706          python -m testtools.run --load-list \
1707              <(echo package.tests.test_api.yamlfile_get_the_widge.test_request)
1708
1709   How do I run just one test, without running prior tests in a sequence?
1710       By default, when you select a single test to run, all  tests  prior  to
1711       that  one  in a file will be run as well: the file is treated as as se‐
1712       quence of dependent tests. If you do not want this you can  adjust  the
1713       use_prior_test test metadata in one of three ways:
1714
1715       • Set it in the YAML file for the one test you are concerned with.
1716
1717       • Set the defaults for all tests in that file.
1718
1719       • set use_prior_test to false when calling build_tests()
1720
1721       Be  aware  that  doing  this breaks a fundamental assumption that gabbi
1722       makes about how tests work. Any substitutions will fail.
1723

TESTING STYLE

1725   Can I have variables in my YAML file?
1726       Gabbi provides the $ENVIRON substitution which can operate a  bit  like
1727       variables  that are set elsewhere and then used in the tests defined by
1728       the YAML.
1729
1730       If you find it necessary to have variables within a  single  YAML  file
1731       you take advantage of YAML alias nodes list this:
1732
1733          vars:
1734            - &uuid_1 5613AABF-BAED-4BBA-887A-252B2D3543F8
1735
1736          tests:
1737          - name: send a uuid to a post
1738            POST: /resource
1739            request_headers:
1740              content-type: application/json
1741            data:
1742              uuid: *uuid_1
1743
1744       You  can alias all sorts of nodes, not just single items. Be aware that
1745       the replacement of an alias  node  happens  while  the  YAML  is  being
1746       loaded, before gabbi does any processing.
1747
1748   How many tests should be put in one YAML file?
1749       For  the  sake  of  readability it is best to keep each YAML file rela‐
1750       tively short. Since each YAML file represents a sequence  of  requests,
1751       it  usually  makes sense to create a new file when a test is not depen‐
1752       dent on any before it.
1753
1754       It's tempting to put all the tests for any resource or URL in the  same
1755       file, but this eventually leads to files that are too long and are thus
1756       difficult to read.
1757

CASE MODULE

1759       A single HTTP request represented as a subclass of unittest.TestCase
1760
1761       The test case encapsulates the request headers and  body  and  expected
1762       response headers and body. When the test is run an HTTP request is made
1763       using urllib3. Assertions are made against the response.
1764
1765       class gabbi.case.HTTPTestCase(methodName='runTest')
1766              Bases: TestCase
1767
1768              Encapsulate a single HTTP request as a TestCase.
1769
1770              If the test is a member of a sequence of requests,  ensure  that
1771              prior tests are run.
1772
1773              To  keep  the  test harness happy we need to make sure the setUp
1774              and tearDown are only run once.
1775
1776              assert_in_or_print_output(expected, iterable)
1777                     Assert the iterable contains expected or print some  out‐
1778                     put.
1779
1780                     If   the   output  is  long,  it  is  limited  by  either
1781                     GABBI_MAX_CHARS_OUTPUT  in   the   environment   or   the
1782                     MAX_CHARS_OUTPUT constant.
1783
1784              base_test  =  {'cert_validate':  True,  'data':  '', 'desc': '',
1785              'disable_response_handler': False, 'method': 'GET', 'name':  '',
1786              'poll':  {},  'query_parameters':  {},  'redirects': False, 're‐
1787              quest_headers': {}, 'skip': '', 'ssl': False,  'status':  '200',
1788              'url':  '',  'use_prior_test':  True, 'verbose': False, 'xfail':
1789              False}
1790
1791              get_content_handler(content_type)
1792                     Determine the content handler for this media type.
1793
1794              load_data_file(filename)
1795                     Read a file from the current test directory.
1796
1797              replace_template(message, escape_regex=False)
1798                     Replace magic strings in message.
1799
1800              run(result=None)
1801                     Store the current result handler on this test.
1802
1803              setUp()
1804                     Hook method for setting up the test fixture before  exer‐
1805                     cising it.
1806
1807              shortDescription()
1808                     Returns a one-line description of the test, or None if no
1809                     description has been provided.
1810
1811                     The default implementation of  this  method  returns  the
1812                     first line of the specified test method's docstring.
1813
1814              tearDown()
1815                     Hook  method  for  deconstructing  the test fixture after
1816                     testing it.
1817
1818              test_request()
1819                     Run this request if it has not yet run.
1820
1821                     If there is a prior test in the sequence, run it first.
1822
1823       gabbi.case.potentialFailure(func)
1824              Decorate a test method that is expected to fail  if  'xfail'  is
1825              true.
1826

DRIVER MODULE

SUITEMAKER MODULE

FIXTURE MODULE

1830       Manage fixtures for gabbi at the test suite level.
1831
1832       class gabbi.fixture.GabbiFixture
1833              Bases: object
1834
1835              A context manager that operates as a fixture.
1836
1837              Subclasses  must  implement start_fixture and stop_fixture, each
1838              of which contain the logic for stopping  and  starting  whatever
1839              the fixture is. What a fixture is is left as an exercise for the
1840              implementor.
1841
1842              These context managers will be nested so any actual  work  needs
1843              to happen in start_fixture and stop_fixture and not in __init__.
1844              Otherwise exception handling will not work properly.
1845
1846              start_fixture()
1847                     Implement the actual workings  of  starting  the  fixture
1848                     here.
1849
1850              stop_fixture()
1851                     Implement  the  actual  workings  of stopping the fixture
1852                     here.
1853
1854       exception gabbi.fixture.GabbiFixtureError
1855              Bases: Exception
1856
1857              Generic exception for GabbiFixture.
1858
1859       class gabbi.fixture.SkipAllFixture
1860              Bases: GabbiFixture
1861
1862              A fixture that skips all the tests in the current suite.
1863
1864              start_fixture()
1865                     Implement the actual workings  of  starting  the  fixture
1866                     here.
1867
1868       gabbi.fixture.nest(fixtures)
1869              Nest a series of fixtures.
1870
1871              This  is  duplicated  from  nested in the stdlib, which has been
1872              deprecated because of issues with how exceptions  are  difficult
1873              to handle during __init__. Gabbi needs to nest an unknown number
1874              of fixtures dynamically, so the with syntax that replaces nested
1875              will not work.
1876

HANDLERS MODULE

1878       Package  for  response  and content handlers that process the body of a
1879       response in various ways.
1880
1881   handlers.base Module
1882       Base classes for response and content handlers.
1883
1884       class gabbi.handlers.base.ContentHandler
1885              Bases: ResponseHandler
1886
1887              A subclass of ResponseHandlers that adds content handling.
1888
1889              static accepts(content_type)
1890                     Return True if this handler can handler this type.
1891
1892              static dumps(data, pretty=False, test=None)
1893                     Return structured data as a string.
1894
1895                     If pretty is true, prettify.
1896
1897              static load_data_file(test, file_path)
1898                     Return the string content of the file  specified  by  the
1899                     file_path.
1900
1901              static loads(data)
1902                     Create structured (Python) data from a stream.
1903
1904                     If  there  is  a failure decoding then the handler should
1905                     repackage the error as a gabbi.exception.GabbiDataLoadEr‐
1906                     ror.
1907
1908              classmethod replacer(response_data, path)
1909                     Return the string that is replacing RESPONSE.
1910
1911       class gabbi.handlers.base.ResponseHandler
1912              Bases: object
1913
1914              Add functionality for making assertions about an HTTP response.
1915
1916              A subclass may implement two methods: action and preprocess.
1917
1918              preprocess  takes  one  argument, the TestCase. It is called ex‐
1919              actly once for each test before looping across  the  assertions.
1920              It  is  used, rarely, to copy the test.output into a useful form
1921              (such as a parsed DOM).
1922
1923              action takes two or three arguments. If test_key_value is a list
1924              action  is  called with the test case and a single list item. If
1925              test_key_value is a dict then action is  called  with  the  test
1926              case and a key and value pair.
1927
1928              action(test, item, value=None)
1929                     Test an individual entry for this response handler.
1930
1931                     If  the  entry is a key value pair the key is in item and
1932                     the value in value. Otherwise the entry is  considered  a
1933                     single item from a list.
1934
1935              preprocess(test)
1936                     Do any pre-single-test preprocessing.
1937
1938              test_key_suffix = ''
1939
1940              test_key_value = []
1941
1942   handlers.core Module
1943       Core response handlers.
1944
1945       class gabbi.handlers.core.ForbiddenHeadersResponseHandler
1946              Bases: ResponseHandler
1947
1948              Test that listed headers are not in the response.
1949
1950              action(test, forbidden, value=None)
1951                     Test an individual entry for this response handler.
1952
1953                     If  the  entry is a key value pair the key is in item and
1954                     the value in value. Otherwise the entry is  considered  a
1955                     single item from a list.
1956
1957              test_key_suffix = 'forbidden_headers'
1958
1959              test_key_value = []
1960
1961       class gabbi.handlers.core.HeadersResponseHandler
1962              Bases: ResponseHandler
1963
1964              Compare expected headers with actual headers.
1965
1966              If a header value is wrapped in / it is treated as a raw regular
1967              expression.
1968
1969              Headers values are always treated as strings.
1970
1971              action(test, header, value=None)
1972                     Test an individual entry for this response handler.
1973
1974                     If the entry is a key value pair the key is in  item  and
1975                     the  value  in value. Otherwise the entry is considered a
1976                     single item from a list.
1977
1978              test_key_suffix = 'headers'
1979
1980              test_key_value = {}
1981
1982       class gabbi.handlers.core.StringResponseHandler
1983              Bases: ResponseHandler
1984
1985              Test for matching strings in the the response body.
1986
1987              action(test, expected, value=None)
1988                     Test an individual entry for this response handler.
1989
1990                     If the entry is a key value pair the key is in  item  and
1991                     the  value  in value. Otherwise the entry is considered a
1992                     single item from a list.
1993
1994              test_key_suffix = 'strings'
1995
1996              test_key_value = []
1997
1998   handlers.jsonhandler Module
1999       JSON-related content handling.
2000
2001       class gabbi.handlers.jsonhandler.JSONHandler
2002              Bases: ContentHandler
2003
2004              A ContentHandler for JSON
2005
2006              • Structured test data is turned into  JSON  when  request  con‐
2007                tent-type is JSON.
2008
2009              • Response  bodies  that  are  JSON strings are made into Python
2010                data on the test response_data  attribute  when  the  response
2011                content-type is JSON.
2012
2013              • A response_json_paths response handler is added.
2014
2015              • JSONPaths in $RESPONSE substitutions are supported.
2016
2017              static accepts(content_type)
2018                     Return True if this handler can handler this type.
2019
2020              action(test, path, value=None)
2021                     Test json_paths against json data.
2022
2023              static dumps(data, pretty=False, test=None)
2024                     Return structured data as a string.
2025
2026                     If pretty is true, prettify.
2027
2028              static extract_json_path_value(data, path)
2029                     Extract the value at JSON Path path from the data.
2030
2031                     The  input  data  is  a  Python datastructure, not a JSON
2032                     string.
2033
2034              static load_data_file(test, file_path)
2035                     Return the string content of the file  specified  by  the
2036                     file_path.
2037
2038              static loads(data)
2039                     Create structured (Python) data from a stream.
2040
2041                     If  there  is  a failure decoding then the handler should
2042                     repackage the error as a gabbi.exception.GabbiDataLoadEr‐
2043                     ror.
2044
2045              classmethod replacer(response_data, match)
2046                     Return the string that is replacing RESPONSE.
2047
2048              test_key_suffix = 'json_paths'
2049
2050              test_key_value = {}
2051
2052   handlers.yaml_disk_loading_jsonhandler Module
2053       JSON-related content handling with YAML data disk loading.
2054
2055       class gabbi.handlers.yaml_disk_loading_jsonhandler.YAMLDiskLoadingJSON‐
2056       Handler
2057              Bases: JSONHandler
2058
2059              A ContentHandler for JSON responses that loads YAML from disk
2060
2061              • Structured test data is turned into  JSON  when  request  con‐
2062                tent-type is JSON.
2063
2064              • Response  bodies  that  are  JSON strings are made into Python
2065                data on the test response_data  attribute  when  the  response
2066                content-type is JSON.
2067
2068              • A  response_json_paths  response  handler  is added. Data read
2069                from  disk  during  this  handle  will  be  loaded  with   the
2070                yaml.safe_load  method  to  support  both  JSON  and YAML data
2071                sources from disk.
2072
2073              • JSONPaths in $RESPONSE substitutions are supported.
2074
2075              static load_data_file(test, file_path)
2076                     Return the string content of the file  specified  by  the
2077                     file_path.
2078

SUITE MODULE

2080       A TestSuite for containing gabbi tests.
2081
2082       This  suite has two features: the contained tests are ordered and there
2083       are suite-level fixtures that operate as context managers.
2084
2085       class gabbi.suite.GabbiSuite(tests=())
2086              Bases: TestSuite
2087
2088              A TestSuite with fixtures.
2089
2090              The suite wraps the tests with a set of nested context  managers
2091              that operate as fixtures.
2092
2093              If a fixture raises unittest.case.SkipTest during setup, all the
2094              tests in this suite will be skipped.
2095
2096              run(result, debug=False)
2097                     Override TestSuite run to start suite-level fixtures.
2098
2099                     To avoid exception confusion, use  a  null  Fixture  when
2100                     there are no fixtures.
2101
2102              start(result, tests=None)
2103                     Start fixtures when using pytest.
2104
2105              stop() Stop fixtures when using pytest.
2106
2107       gabbi.suite.noop(*args)
2108              A noop method used to disable collected tests.
2109

RUNNER MODULE

REPORTER MODULE

2112       TestRunner and TestResult for gabbi-run.
2113
2114       class gabbi.reporter.ConciseTestResult(stream, descriptions, verbosity)
2115              Bases: TextTestResult
2116
2117              A TextTestResult with simple but useful output.
2118
2119              If  the output is a tty or GABBI_FORCE_COLOR is set in the envi‐
2120              ronment, output will be colorized.
2121
2122              addError(test, err)
2123                     Called when an error has occurred. 'err' is  a  tuple  of
2124                     values as returned by sys.exc_info().
2125
2126              addExpectedFailure(test, err)
2127                     Called when an expected failure/error occurred.
2128
2129              addFailure(test, err)
2130                     Called  when  an  error has occurred. 'err' is a tuple of
2131                     values as returned by sys.exc_info().
2132
2133              addSkip(test, reason)
2134                     Called when a test is skipped.
2135
2136              addSuccess(test)
2137                     Called when a test has completed successfully
2138
2139              addUnexpectedSuccess(test)
2140                     Called when a test was expected to fail, but succeed.
2141
2142              getDescription(test)
2143
2144              printErrorList(flavor, errors)
2145
2146              startTest(test)
2147                     Called when the given test is about to be run
2148
2149       class gabbi.reporter.ConciseTestRunner(stream=None,  descriptions=True,
2150       verbosity=1,   failfast=False,  buffer=False,  resultclass=None,  warn‐
2151       ings=None, *, tb_locals=False, durations=None)
2152              Bases: TextTestRunner
2153
2154              A TextTestRunner that uses ConciseTestResult for  reporting  re‐
2155              sults.
2156
2157              resultclass
2158                     alias of ConciseTestResult
2159
2160       class  gabbi.reporter.PyTestResult(stream=None, descriptions=None, ver‐
2161       bosity=None)
2162              Bases: TestResult
2163
2164              Wrap a test result to allow it to work with pytest.
2165
2166              The main behaviors here are:
2167
2168              • to turn what had been exceptions back into exceptions
2169
2170              • use pytest's skip and xfail methods
2171
2172              addError(test, err)
2173                     Called when an error has occurred. 'err' is  a  tuple  of
2174                     values as returned by sys.exc_info().
2175
2176              addExpectedFailure(test, err)
2177                     Called when an expected failure/error occurred.
2178
2179              addFailure(test, err)
2180                     Called  when  an  error has occurred. 'err' is a tuple of
2181                     values as returned by sys.exc_info().
2182
2183              addSkip(test, reason)
2184                     Called when a test is skipped.
2185

UTILS MODULE

2187       Utility functions grab bag.
2188
2189       gabbi.utils.create_url(base_url, host, port=None, prefix='', ssl=False)
2190              Given pieces of a path-based url, return a fully qualified url.
2191
2192       gabbi.utils.decode_response_content(header_dict, content)
2193              Decode content to a proper string.
2194
2195       gabbi.utils.extract_content_type(header_dict,  default='application/bi‐
2196       nary')
2197              Extract parsed content-type from headers.
2198
2199       gabbi.utils.get_colorizer(stream)
2200              Return a function to colorize a string.
2201
2202              Only if stream is a tty .
2203
2204       gabbi.utils.host_info_from_target(target, prefix=None)
2205              Turn url or host:port and target into test destination.
2206
2207       gabbi.utils.load_yaml(handle=None, yaml_file=None, safe=True)
2208              Read and parse any YAML file or filehandle.
2209
2210              Let exceptions flow where they may.
2211
2212              If no file or handle is provided, read from STDIN.
2213
2214       gabbi.utils.not_binary(content_type)
2215              Decide if something is content we'd like to treat as a string.
2216
2217       gabbi.utils.parse_content_type(content_type, default_charset='utf-8')
2218              Parse content type value for media type and charset.
2219

EXCEPTION MODULE

2221       Gabbi specific exceptions.
2222
2223       exception gabbi.exception.GabbiDataLoadError
2224              Bases: ValueError
2225
2226              An exception to alert when data streams cannot be loaded.
2227
2228       exception gabbi.exception.GabbiFormatError
2229              Bases: ValueError
2230
2231              An exception to encapsulate poorly formed test data.
2232
2233       exception gabbi.exception.GabbiSyntaxWarning
2234              Bases: SyntaxWarning
2235
2236              A warning about syntax that is not desirable.
2237

HTTPCLIENT MODULE

JSON_PARSER MODULE

2240       Keep one single global jsonpath parser.
2241
2242       gabbi.json_parser.parse(path)
2243              Parse a JSONPath expression use the global parser.
2244
2245       Gabbi is a tool for running HTTP tests where requests and responses are
2246       expressed as declarations in YAML files:
2247
2248          tests:
2249          - name: retrieve items
2250            GET: /items
2251
2252       See the rest of these docs for more details on the  many  features  and
2253       formats  for  setting  request  headers  and  bodies and evaluating re‐
2254       sponses.
2255
2256       Tests can be run from the command line with gabbi-run  or  programmati‐
2257       cally   using  either  py.test  or  unittest-style  test  runners.  See
2258       installation instructions below.
2259
2260       The name is derived from "gabby": excessively talkative. In a test  en‐
2261       vironment  having visibility of what a test is actually doing is a good
2262       thing. This is especially true when the goal of a test is to  test  the
2263       HTTP,  not  the testing infrastructure. Gabbi tries to put the HTTP in‐
2264       teraction in the foreground of testing.
2265
2266       If you want to get straight to creating tests look  at  Example  Tests,
2267       the test files in the source distribution and Test Format. A gabbi-demo
2268       repository provides a tutorial of using gabbi to build an API, via  the
2269       commit history of the repo.
2270

PURPOSE

2272       Gabbi  works  to  bridge the gap between human readable YAML files (see
2273       Test Format for details) that represent HTTP requests and expected  re‐
2274       sponses and the rather complex world of automated testing.
2275
2276       Each  YAML  file represents an ordered list of HTTP requests along with
2277       the expected responses. This  allows  a  single  file  to  represent  a
2278       process in the API being tested. For example:
2279
2280       • Create a resource.
2281
2282       • Retrieve a resource.
2283
2284       • Delete a resource.
2285
2286       • Retrieve a resource again to confirm it is gone.
2287
2288       At  the same time it is still possible to ask gabbi to run just one re‐
2289       quest. If it is in a sequence of tests, those tests prior to it in  the
2290       YAML  file  will be run (in order). In any single process any test will
2291       only be run once. Concurrency is handled such that one file runs in one
2292       process.
2293
2294       These features mean that it is possible to create tests that are useful
2295       for both humans (as tools for learning, improving and developing  APIs)
2296       and automated CI systems.
2297
2298       Significant  flexibility  and  power is available in the Test Format to
2299       make it relatively straightforward to test existing complex APIs.  This
2300       extended  functionality  includes the use of JSONPath to query response
2301       bodies and templating of test data to allow access to  the  prior  HTTP
2302       response  in  the current request. For APIs which do not use JSON addi‐
2303       tional Content Handlers can be created.
2304
2305       Care should be taken when using this functionality when you are  creat‐
2306       ing  a  new  API.  If your API is so complex that it needs complex test
2307       files then you may wish to take that as a sign that your API itself too
2308       complex. One goal of gabbi is to encourage transparent and comprehensi‐
2309       ble APIs.
2310
2311       Though gabbi is written in Python and under the  covers  uses  unittest
2312       data  structures and processes, there is no requirement that the Target
2313       Host be a Python-based service. Anything talking HTTP can be tested.  A
2314       YAML  Runner  makes  it  possible to simply create YAML files and point
2315       them at a running server.
2316

INSTALLATION

2318       As a Python package, gabbi is typically installed via pip:
2319
2320          pip install gabbi
2321
2322       You might want to create a virtual environment; an isolated context for
2323       Python  packages,  keeping gabbi cleany separated from the rest of your
2324       system.
2325
2326       Python 3 comes with a built-in tool to create virtual environments:
2327
2328          python3 -m venv venv
2329          . venv/bin/activate
2330
2331          pip install gabbi
2332
2333       This way we can later use deactivate and safely remove the venv  direc‐
2334       tory, thus erasing any trace of gabbi from the system.
2335
2336       If  you prefer to not install gabbi, or perhaps want to use it in a dy‐
2337       namic fashion in a CI setting, there is  an  official  container  image
2338       hosted  at  docker hub as cdent/gabbi. It allows running gabbi-run with
2339       any arguments you might need, providing tests on STDIN of via a mounted
2340       volume.
2341

AUTHOR

2343       Chris Dent
2344
2345
2346
2347
2348                                 Jul 21, 2023                         GABBI(1)
Impressum