1PYTHON-SWIFTCLIENT(1) python-swiftclient PYTHON-SWIFTCLIENT(1)
2
3
4
6 python-swiftclient - python-swiftclient 3.8.1
7
9 Introduction
10 Where to Start?
11 The python-swiftclient project comprises a command line tool and two
12 separate APIs for accessing swift programmatically. Choosing the most
13 appropriate method for a given use case is the first problem a user
14 needs to solve.
15
16 Use Cases
17 Alongside the command line tool, the python-swiftclient includes two
18 levels of API:
19
20 · A low level client API that provides simple Python wrappers around
21 the various authentication mechanisms and the individual HTTP
22 requests.
23
24 · A high level service API that provides methods for performing common
25 operations in parallel on a thread pool.
26
27 Example use cases:
28
29 ·
30
31 Uploading and retrieving data
32 Use the command line tool if you are simply uploading and
33 downloading files and directories to and from your filesystem.
34 The command line tool can be integrated into a shell script to
35 automate tasks.
36
37 ·
38
39 Integrating into an automated Python workflow
40 Use the SwiftService API to perform operations offered by the
41 CLI if your use case requires integration with a Python-based
42 workflow. This method offers greater control and flexibility
43 over individual object operations, such as the metadata set on
44 each object. The SwiftService class provides methods to per‐
45 form multiple sets of operations against a swift object store
46 using a configurable shared thread pool. A single instance of
47 the SwiftService class can be shared between multiple threads
48 in your own code.
49
50 ·
51
52 Developing an application in Python to access a swift object store
53 Use the SwiftService API to develop Python applications that
54 use swift to store and retrieve objects. A SwiftService
55 instance provides a configurable thread pool for performing
56 all operations supported by the CLI.
57
58 ·
59
60 Fine-grained control over threading or the requests being performed
61 Use the Connection API if your use case requires fine grained
62 control over advanced features or you wish to use your own
63 existing threading model. Examples of advanced features
64 requiring the use of the Connection API include creating an
65 SLO manifest that references already existing objects, or fine
66 grained control over the query strings supplied with each HTTP
67 request.
68
69 Important considerations
70 This section covers some important considerations, helpful hints, and
71 things to avoid when integrating an object store into your workflow.
72
73 An object store is not a filesystem
74 It cannot be stressed enough that your usage of the object store should
75 reflect the proper use case, and not treat the storage like a tradi‐
76 tional filesystem. There are two main restrictions to bear in mind
77 when designing an application that uses an object store:
78
79 · You cannot rename objects. Due to fact that the name of an object is
80 one of the factors that determines where the object and its replicas
81 are stored, renaming would require multiple copies of the data to be
82 moved between physical storage devices. If you want to rename an
83 object you must upload to the new location, or make a server side
84 copy request to the new location, and then delete the original.
85
86 · You cannot modify objects. Objects are stored in multiple locations
87 and are checked for integrity based on the MD5 sum calculated during
88 upload. In order to modify the contents of an object, the entire
89 desired contents must be re-uploaded. In certain special cases it is
90 possible to work around this restriction using large objects, but no
91 general file-like access is available to modify a stored object.
92
93 Objects cannot be locked
94 There is no mechanism to perform a combination of reading the
95 data/metadata from an object and writing an update to that data/meta‐
96 data in an atomic way. Any user with access to a container could update
97 the contents or metadata associated with an object at any time.
98
99 Workflows that assume that no updates have been made since the last
100 read of an object should be discouraged. Enabling a workflow of this
101 type requires an external object locking mechanism and/or cooperation
102 between all clients accessing the data.
103
105 CLI
106 The swift tool is a command line utility for communicating with an
107 OpenStack Object Storage (swift) environment. It allows one to perform
108 several types of operations.
109
110 For help on a specific swift command, enter:
111
112 $ swift COMMAND --help
113
114 swift usage
115 Usage: swift [--version] [--help] [--os-help] [--snet] [--verbose]
116 [--debug] [--info] [--quiet] [--auth <auth_url>]
117 [--auth-version <auth_version> |
118 --os-identity-api-version <auth_version> ]
119 [--user <username>]
120 [--key <api_key>] [--retries <num_retries>]
121 [--os-username <auth-user-name>] [--os-password <auth-password>]
122 [--os-user-id <auth-user-id>]
123 [--os-user-domain-id <auth-user-domain-id>]
124 [--os-user-domain-name <auth-user-domain-name>]
125 [--os-tenant-id <auth-tenant-id>]
126 [--os-tenant-name <auth-tenant-name>]
127 [--os-project-id <auth-project-id>]
128 [--os-project-name <auth-project-name>]
129 [--os-project-domain-id <auth-project-domain-id>]
130 [--os-project-domain-name <auth-project-domain-name>]
131 [--os-auth-url <auth-url>] [--os-auth-token <auth-token>]
132 [--os-storage-url <storage-url>] [--os-region-name <region-name>]
133 [--os-service-type <service-type>]
134 [--os-endpoint-type <endpoint-type>]
135 [--os-cacert <ca-certificate>] [--insecure]
136 [--os-cert <client-certificate-file>]
137 [--os-key <client-certificate-key-file>]
138 [--no-ssl-compression]
139 <subcommand> [--help] [<subcommand options>]
140
141 Subcommands:
142
143 delete Delete a container or objects within a container.
144
145 download
146 Download objects from containers.
147
148 list Lists the containers for the account or the objects for a con‐
149 tainer.
150
151 post Updates meta information for the account, container, or object;
152 creates containers if not present.
153
154 copy Copies object, optionally adds meta
155
156 stat Displays information for the account, container, or object.
157
158 upload Uploads files or directories to the given container.
159
160 capabilities
161 List cluster capabilities.
162
163 tempurl
164 Create a temporary URL.
165
166 auth Display auth related environment variables.
167
168 swift optional arguments
169 --version
170 show program’s version number and exit
171
172 -h, --help
173 show this help message and exit
174
175 --os-help
176 Show OpenStack authentication options.
177
178 -s, --snet
179 Use SERVICENET internal network.
180
181 -v, --verbose
182 Print more info.
183
184 --debug
185 Show the curl commands and results of all http queries regard‐
186 less of result status.
187
188 --info Show the curl commands and results of all http queries which
189 return an error.
190
191 -q, --quiet
192 Suppress status output.
193
194 -A AUTH, --auth=AUTH
195 URL for obtaining an auth token.
196
197 -V AUTH_VERSION, --auth-version=AUTH_VERSION, --os-identity-api-ver‐
198 sion=AUTH_VERSION
199 Specify a version for authentication. Defaults to
200 env[ST_AUTH_VERSION], env[OS_AUTH_VERSION], env[OS_IDEN‐
201 TITY_API_VERSION] or 1.0.
202
203 -U USER, --user=USER
204 User name for obtaining an auth token.
205
206 -K KEY, --key=KEY
207 Key for obtaining an auth token.
208
209 -R RETRIES, --retries=RETRIES
210 The number of times to retry a failed connection.
211
212 --insecure
213 Allow swiftclient to access servers without having to verify the
214 SSL certificate. Defaults to env[SWIFTCLIENT_INSECURE] (set to
215 ‘true’ to enable).
216
217 --no-ssl-compression
218 This option is deprecated and not used anymore. SSL compression
219 should be disabled by default by the system SSL library.
220
221 --prompt
222 Prompt user to enter a password which overrides any password
223 supplied via --key, --os-password or environment variables.
224
225 Authentication
226 This section covers the options for authenticating with a swift object
227 store. The combinations of options required for each authentication
228 version are detailed below, but are just a subset of those that can be
229 used to successfully authenticate. These are the most common and recom‐
230 mended combinations.
231
232 You should obtain the details of your authentication version and cre‐
233 dentials from your storage provider. These details should make it
234 clearer which of the authentication sections below are most likely to
235 allow you to connect to your storage account.
236
237 Keystone v3
238 swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \
239 --os-project-name project1 --os-project-domain-name domain1 \
240 --os-username user --os-user-domain-name domain1 \
241 --os-password password list
242
243 swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \
244 --os-project-id 0123456789abcdef0123456789abcdef \
245 --os-user-id abcdef0123456789abcdef0123456789 \
246 --os-password password list
247
248 Manually specifying the options above on the command line can be
249 avoided by setting the following combinations of environment variables:
250
251 ST_AUTH_VERSION=3
252 OS_USERNAME=user
253 OS_USER_DOMAIN_NAME=domain1
254 OS_PASSWORD=password
255 OS_PROJECT_NAME=project1
256 OS_PROJECT_DOMAIN_NAME=domain1
257 OS_AUTH_URL=https://api.example.com:5000/v3
258
259 ST_AUTH_VERSION=3
260 OS_USER_ID=abcdef0123456789abcdef0123456789
261 OS_PASSWORD=password
262 OS_PROJECT_ID=0123456789abcdef0123456789abcdef
263 OS_AUTH_URL=https://api.example.com:5000/v3
264
265 Keystone v2
266 swift --os-auth-url https://api.example.com:5000/v2.0 \
267 --os-tenant-name tenant \
268 --os-username user --os-password password list
269
270 Manually specifying the options above on the command line can be
271 avoided by setting the following environment variables:
272
273 ST_AUTH_VERSION=2.0
274 OS_USERNAME=user
275 OS_PASSWORD=password
276 OS_TENANT_NAME=tenant
277 OS_AUTH_URL=https://api.example.com:5000/v2.0
278
279 Legacy auth systems
280 You can configure swift to work with any number of other authentication
281 systems that we will not cover in this document. If your storage
282 provider is not using Keystone to provide access tokens, please contact
283 them for instructions on the required options. It is likely that the
284 options will need to be specified as below:
285
286 swift -A https://api.example.com/v1.0 -U user -K api_key list
287
288 Specifying the options above manually on the command line can be
289 avoided by setting the following environment variables:
290
291 ST_AUTH_VERSION=1.0
292 ST_AUTH=https://api.example.com/v1.0
293 ST_USER=user
294 ST_KEY=key
295
296 It is also possible that you need to use a completely separate auth
297 system, in which case swiftclient cannot request a token for you. In
298 this case you should make the authentication request separately and
299 access your storage using the token and storage URL options shown
300 below:
301
302 swift --os-auth-token 6ee5eb33efad4e45ab46806eac010566 \
303 --os-storage-url https://10.1.5.2:8080/v1/AUTH_ced809b6a4baea7aeab61a \
304 list
305
306 NOTE:
307 Leftover environment variables are a common source of confusion when
308 authorization fails.
309
310 CLI commands
311 Auth
312 Usage: swift auth
313
314 Display authentication variables in shell friendly format. Command to
315 run to export storage URL and auth token into OS_STORAGE_URL and
316 OS_AUTH_TOKEN: swift auth. Command to append to a runcom file (e.g.
317 ~/.bashrc, /etc/profile) for automatic authentication: swift auth -v -U
318 test:tester -K testing.
319
320 swift stat
321 Usage: swift stat [--lh] [--header <header:value>]
322 [<container> [<object>]]
323
324 Displays information for the account, container, or object depending on
325 the arguments given (if any). In verbose mode, the storage URL and the
326 authentication token are displayed as well.
327
328 Positional arguments:
329
330 [container]
331 Name of container to stat from.
332
333 [object]
334 Name of object to stat.
335
336 Optional arguments:
337
338 --lh Report sizes in human readable format similar to ls -lh.
339
340 -H, --header <header:value>
341 Adds a custom request header to use for stat.
342
343 swift list
344 Usage: swift list [--long] [--lh] [--totals] [--prefix <prefix>]
345 [--delimiter <delimiter>] [--header <header:value>]
346 [<container>]
347
348 Lists the containers for the account or the objects for a container.
349 The -p <prefix> or --prefix <prefix> is an option that will only list
350 items beginning with that prefix. The -d <delimiter> or --delimiter
351 <delimiter> is an option (for container listings only) that will roll
352 up items with the given delimiter (see OpenStack Swift general documen‐
353 tation <https://docs.openstack.org/swift/latest/> for what this means).
354
355 The -l and --lh options provide more detail, similar to ls -l and ls
356 -lh, the latter providing sizes in human readable format (For example:
357 3K, 12M, etc). The latter two switches use more overhead to retrieve
358 the displayed details, which is directly proportional to the number of
359 container or objects listed.
360
361 Positional arguments:
362
363 [container]
364 Name of container to list object in.
365
366 Optional arguments:
367
368 -l, --long
369 Long listing format, similar to ls -l.
370
371 --lh Report sizes in human readable format similar to ls -lh.
372
373 -t, --totals
374 Used with -l or –lh, only report totals.
375
376 -p <prefix>, --prefix <prefix>
377 Only list items beginning with the prefix.
378
379 -d <delim>, --delimiter <delim>
380 Roll up items with the given delimiter. For containers only. See
381 OpenStack Swift API documentation for what this means.
382
383 -H, --header <header:value>
384 Adds a custom request header to use for listing.
385
386 swift upload
387 Usage: swift upload [--changed] [--skip-identical] [--segment-size <size>]
388 [--segment-container <container>] [--leave-segments]
389 [--object-threads <thread>] [--segment-threads <threads>]
390 [--header <header>] [--use-slo] [--ignore-checksum]
391 [--object-name <object-name>]
392 <container> <file_or_directory> [<file_or_directory>] [...]
393
394 Uploads the files and directories specified by the remaining arguments
395 to the given container. The -c or --changed is an option that will only
396 upload files that have changed since the last upload. The --object-name
397 <object-name> is an option that will upload a file and name object to
398 <object-name> or upload a directory and use <object-name> as object
399 prefix. If the file name is “-“, client reads content from standard
400 input. In this case --object-name is required to set the name of the
401 object and no other files may be given. The -S <size> or --seg‐
402 ment-size <size> and --leave-segments are options as well (see --help
403 for more).
404
405 Positional arguments:
406
407 <container>
408 Name of container to upload to.
409
410 <file_or_directory>
411 Name of file or directory to upload. Specify multiple times for
412 multiple uploads.
413
414 Optional arguments:
415
416 -c, --changed
417 Only upload files that have changed since the last upload.
418
419 --skip-identical
420 Skip uploading files that are identical on both sides.
421
422 -S, --segment-size <size>
423 Upload files in segments no larger than <size> (in Bytes) and
424 then create a “manifest” file that will download all the seg‐
425 ments as if it were the original file.
426
427 --segment-container <container>
428 Upload the segments into the specified container. If not speci‐
429 fied, the segments will be uploaded to a <container>_segments
430 container to not pollute the main <container> listings.
431
432 --leave-segments
433 Indicates that you want the older segments of manifest objects
434 left alone (in the case of overwrites).
435
436 --object-threads <threads>
437 Number of threads to use for uploading full objects. Default is
438 10.
439
440 --segment-threads <threads>
441 Number of threads to use for uploading object segments. Default
442 is 10.
443
444 -H, --header <header:value>
445 Adds a customized request header. This option may be repeated.
446 Example: -H “content-type:text/plain” -H “Content-Length: 4000”.
447
448 --use-slo
449 When used in conjunction with –segment-size it will create a
450 Static Large Object instead of the default Dynamic Large Object.
451
452 --object-name <object-name>
453 Upload file and name object to <object-name> or upload dir and
454 use <object-name> as object prefix instead of folder name.
455
456 --ignore-checksum
457 Turn off checksum validation for uploads.
458
459 swift post
460 Usage: swift post [--read-acl <acl>] [--write-acl <acl>] [--sync-to <sync-to>]
461 [--sync-key <sync-key>] [--meta <name:value>]
462 [--header <header>]
463 [<container> [<object>]]
464
465 Updates meta information for the account, container, or object depend‐
466 ing on the arguments given. If the container is not found, the swift‐
467 client will create it automatically, but this is not true for accounts
468 and objects. Containers also allow the -r <read-acl> (or --read-acl
469 <read-acl>) and -w <write-acl> (or --write-acl <write-acl>) options.
470 The -m or --meta option is allowed on accounts, containers and objects,
471 and is used to define the user metadata items to set in the form
472 Name:Value. You can repeat this option. For example: post -m
473 Color:Blue -m Size:Large
474
475 For more information about ACL formats see the documentation: ACLs.
476
477 Positional arguments:
478
479 [container]
480 Name of container to post to.
481
482 [object]
483 Name of object to post.
484
485 Optional arguments:
486
487 -r, --read-acl <acl>
488 Read ACL for containers. Quick summary of ACL syntax: .r:*,
489 .r:-.example.com, .r:www.example.com, account1 (v1.0 identity
490 API only), account1:*, account2:user2 (v2.0+ identity API).
491
492 -w, --write-acl <acl>
493 Write ACL for containers. Quick summary of ACL syntax: account1
494 (v1.0 identity API only), account1:*, account2:user2 (v2.0+
495 identity API).
496
497 -t, --sync-to <sync-to>
498 Sync To for containers, for multi-cluster replication.
499
500 -k, --sync-key <sync-key>
501 Sync Key for containers, for multi-cluster replication.
502
503 -m, --meta <name:value>
504 Sets a meta data item. This option may be repeated.
505
506 Example: -m Color:Blue -m Size:Large
507
508 -H, --header <header:value>
509 Adds a customized request header. This option may be repeated.
510
511 Example: -H “content-type:text/plain” -H “Content-Length: 4000”
512
513 swift download
514 Usage: swift download [--all] [--marker <marker>] [--prefix <prefix>]
515 [--output <out_file>] [--output-dir <out_directory>]
516 [--object-threads <threads>] [--ignore-checksum]
517 [--container-threads <threads>] [--no-download]
518 [--skip-identical] [--remove-prefix]
519 [--header <header:value>] [--no-shuffle]
520 [<container> [<object>] [...]]
521
522 Downloads everything in the account (with --all), or everything in a
523 container, or a list of objects depending on the arguments given. For a
524 single object download, you may use the -o <filename> or --output
525 <filename> option to redirect the output to a specific file or - to re‐
526 direct to stdout. The --ignore-checksum is an option that turn off
527 checksum validation. You can specify optional headers with the repeat‐
528 able cURL-like option -H [--header <name:value>]. --ignore-mtime
529 ignores the x-object-meta-mtime metadata entry on the object (if
530 present) and instead creates the downloaded files with fresh atime and
531 mtime values.
532
533 Positional arguments:
534
535 <container>
536 Name of container to download from. To download a whole account,
537 omit this and specify –all.
538
539 <object>
540 Name of object to download. Specify multiple times for multiple
541 objects. Omit this to download all objects from the container.
542
543 Optional arguments:
544
545 -a, --all
546 Indicates that you really want to download everything in the
547 account.
548
549 -m, --marker <marker>
550 Marker to use when starting a container or account download.
551
552 -p, --prefix <prefix>
553 Only download items beginning with <prefix>
554
555 -r, --remove-prefix
556 An optional flag for –prefix <prefix>, use this option to down‐
557 load items without <prefix>
558
559 -o, --output <out_file>
560 For a single file download, stream the output to <out_file>.
561 Specifying “-” as <out_file> will redirect to stdout.
562
563 -D, --output-dir <out_directory>
564 An optional directory to which to store objects. By default,
565 all objects are recreated in the current directory.
566
567 --object-threads <threads>
568 Number of threads to use for downloading objects. Default is
569 10.
570
571 --container-threads <threads>
572 Number of threads to use for downloading containers. Default is
573 10.
574
575 --no-download
576 Perform download(s), but don’t actually write anything to disk.
577
578 -H, --header <header:value>
579 Adds a customized request header to the query, like “Range” or
580 “If-Match”. This option may be repeated.
581
582 Example: –header “content-type:text/plain”
583
584 --skip-identical
585 Skip downloading files that are identical on both sides.
586
587 --ignore-checksum
588 Turn off checksum validation for downloads.
589
590 --no-shuffle
591 By default, when downloading a complete account or container,
592 download order is randomised in order to reduce the load on
593 individual drives when multiple clients are executed simultane‐
594 ously to download the same set of objects (e.g. a nightly auto‐
595 mated download script to multiple servers). Enable this option
596 to submit download jobs to the thread pool in the order they are
597 listed in the object store.
598
599 swift delete
600 Usage: swift delete [--all] [--leave-segments]
601 [--object-threads <threads>]
602 [--container-threads <threads>]
603 [--header <header:value>]
604 [<container> [<object>] [...]]
605
606 Deletes everything in the account (with --all), or everything in a con‐
607 tainer, or a list of objects depending on the arguments given. Segments
608 of manifest objects will be deleted as well, unless you specify the
609 --leave-segments option.
610
611 Positional arguments:
612
613 [<container>]
614 Name of container to delete from.
615
616 [<object>]
617 Name of object to delete. Specify multiple times for multiple
618 objects.
619
620 Optional arguments:
621
622 -a, --all
623 Delete all containers and objects.
624
625 --leave-segments
626 Do not delete segments of manifest objects.
627
628 -H, --header <header:value>
629 Adds a custom request header to use for deleting objects or an
630 entire container.
631
632 --object-threads <threads>
633 Number of threads to use for deleting objects. Default is 10.
634
635 --container-threads <threads>
636 Number of threads to use for deleting containers. Default is
637 10.
638
639 swift copy
640 Usage: swift copy [--destination </container/object>] [--fresh-metadata]
641 [--meta <name:value>] [--header <header>] <container>
642 <object> [<object>] [...]
643
644 Copies an object to a new destination or adds user metadata to an
645 object. Depending on the options supplied, you can preserve existing
646 metadata in contrast to the post command. The --destination option sets
647 the copy target destination in the form /container/object. If not set,
648 the object will be copied onto itself which is useful for adding meta‐
649 data. You can use the -M or --fresh-metadata option to copy an object
650 without existing user meta data, and the -m or --meta option to define
651 user meta data items to set in the form Name:Value. You can repeat this
652 option. For example: copy -m Color:Blue -m Size:Large.
653
654 Positional arguments:
655
656 <container>
657 Name of container to copy from.
658
659 <object>
660 Name of object to copy. Specify multiple times for multiple
661 objects
662
663 Optional arguments:
664
665 -d, --destination </container[/object]>
666 The container and name of the destination object. Name of desti‐
667 nation object can be omitted, then will be same as name of
668 source object. Supplying multiple objects and destination with
669 object name is invalid.
670
671 -M, --fresh-metadata
672 Copy the object without any existing metadata, If not set, meta‐
673 data will be preserved or appended
674
675 -m, --meta <name:value>
676 Sets a meta data item. This option may be repeated.
677
678 Example: -m Color:Blue -m Size:Large
679
680 -H, --header <header:value>
681 Adds a customized request header. This option may be repeated.
682
683 Example: -H “content-type:text/plain” -H “Content-Length: 4000”
684
685 swift capabilities
686 Usage: swift capabilities [--json] [<proxy_url>]
687
688 Displays cluster capabilities. The output includes the list of the
689 activated Swift middlewares as well as relevant options for each ones.
690 Additionally the command displays relevant options for the Swift core.
691 If the proxy-url option is not provided, the storage URL retrieved
692 after authentication is used as proxy-url.
693
694 Optional positional arguments:
695
696 <proxy_url>
697 Proxy URL of the cluster to retrieve capabilities.
698
699 --json Print the cluster capabilities in JSON format.
700
701 swift tempurl
702 Usage: swift tempurl [--absolute] [--prefix-based]
703 <method> <seconds> <path> <key>
704
705 Generates a temporary URL for a Swift object. method option sets an
706 HTTP method to allow for this temporary URL that is usually GET or PUT.
707 time option sets the amount of time the temporary URL will be valid
708 for. time can be specified as an integer, denoting the number of sec‐
709 onds from now on until the URL shall be valid; or, if --absolute is
710 passed, the Unix timestamp when the temporary URL will expire. But
711 beyond that, time can also be specified as an ISO 8601 timestamp in one
712 of following formats:
713
714 i. Complete date: YYYY-MM-DD (e.g. 1997-07-16)
715
716 ii. Complete date plus hours, minutes and seconds: YYYY-MM-DDThh:mm:ss
717 (e.g. 1997-07-16T19:20:30)
718
719 iii. Complete date plus hours, minutes and seconds with UTC designator:
720 YYYY-MM-DDThh:mm:ssZ (e.g. 1997-07-16T19:20:30Z)
721
722 Please be aware that if you don’t provide the UTC designator (i.e., Z)
723 the timestamp is generated using your local timezone. If only a date is
724 specified, the time part used will equal to 00:00:00.
725
726 path option sets the full path to the Swift object. Example:
727 /v1/AUTH_account/c/o. key option is the secret temporary URL key set on
728 the Swift cluster. To set a key, run swift post -m "Temp-URL-Key: <your
729 secret key>". To generate a prefix-based temporary URL use the --pre‐
730 fix-based option. This URL will contain the path to the prefix. Do not
731 forget to append the desired objectname at the end of the path portion
732 (and before the query portion) before sharing the URL. It is possible
733 to use ISO 8601 UTC timestamps within the URL by using the --iso8601
734 option.
735
736 Positional arguments:
737
738 <method>
739 An HTTP method to allow for this temporary URL. Usually ‘GET’
740 or ‘PUT’.
741
742 <seconds>
743 The amount of time in seconds the temporary URL will be valid
744 for; or, if –absolute is passed, the Unix timestamp when the
745 temporary URL will expire.
746
747 <path> The full path to the Swift object.
748
749 Example: /v1/AUTH_account/c/o or:
750 http://saio:8080/v1/AUTH_account/c/o
751
752 <key> The secret temporary URL key set on the Swift cluster. To set a
753 key, run ‘swift post -m
754 “Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4”’
755
756 Optional arguments:
757
758 --absolute
759 Interpret the <seconds> positional argument as a Unix timestamp
760 rather than a number of seconds in the future.
761
762 --prefix-based
763 If present, a prefix-based tempURL will be generated.
764
765 Examples
766 In this section we present some example usage of the swift CLI. To keep
767 the examples as short as possible, these examples assume that the rele‐
768 vant authentication options have been set using environment variables.
769 You can obtain the full list of commands and options available in the
770 swift CLI by executing the following:
771
772 > swift --help
773 > swift <command> --help
774
775 Simple examples
776 List the existing swift containers:
777
778 > swift list
779
780 container_1
781
782 Create a new container:
783
784 > swift post TestContainer
785
786 Upload an object into a container:
787
788 > swift upload TestContainer testSwift.txt
789
790 testSwift.txt
791
792 List the contents of a container:
793
794 > swift list TestContainer
795
796 testSwift.txt
797
798 Copy an object to new destination:
799
800 > swift copy -d /DestContainer/testSwift.txt SourceContainer testSwift.txt
801
802 SourceContainer/testSwift.txt copied to /DestContainer/testSwift.txt
803
804 Delete an object from a container:
805
806 > swift delete TestContainer testSwift.txt
807
808 testSwift.txt
809
810 Delete a container:
811
812 > swift delete TestContainer
813
814 TestContainer
815
816 Display auth related authentication variables in shell friendly format:
817
818 > swift auth
819
820 export OS_STORAGE_URL=http://127.0.0.1:8080/v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7
821 export OS_AUTH_TOKEN=c597015ae19943a18438b52ef3762e79
822
823 Download an object from a container:
824
825 > swift download TestContainer testSwift.txt
826
827 testSwift.txt [auth 0.028s, headers 0.045s, total 0.045s, 0.002 MB/s]
828
829 NOTE:
830 To upload an object to a container, your current working directory
831 must be where the file is located or you must provide the complete
832 path to the file. In other words, the –object-name <object-name> is
833 an option that will upload file and name object to <object-name> or
834 upload directory and use <object-name> as object prefix. In the case
835 that you provide the complete path of the file, that complete path
836 will be the name of the uploaded object.
837
838 For example:
839
840 > swift upload TestContainer /home/swift/testSwift/testSwift.txt
841
842 home/swift/testSwift/testSwift.txt
843
844 > swift list TestContainer
845
846 home/swift/testSwift/testSwift.txt
847
848 More complex examples
849 Swift has a single object size limit of 5GiB. In order to upload files
850 larger than this, we must create a large object that consists of
851 smaller segments. The example below shows how to upload a large video
852 file as a static large object in 1GiB segments:
853
854 > swift upload videos --use-slo --segment-size 1G myvideo.mp4
855
856 myvideo.mp4 segment 8
857 myvideo.mp4 segment 4
858 myvideo.mp4 segment 2
859 myvideo.mp4 segment 7
860 myvideo.mp4 segment 0
861 myvideo.mp4 segment 1
862 myvideo.mp4 segment 3
863 myvideo.mp4 segment 6
864 myvideo.mp4 segment 5
865 myvideo.mp4
866
867 This command will upload segments to a container named videos_segments,
868 and create a manifest file describing the entire object in the videos
869 container. For more information on large objects, see the documenta‐
870 tion here.
871
872 > swift list videos
873
874 myvideo.mp4
875
876 > swift list videos_segments
877
878 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000000
879 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000001
880 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000002
881 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000003
882 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000004
883 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000005
884 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000006
885 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000007
886 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000008
887
888 Firstly, the key should be set, then generate a temporary URL for a
889 Swift object:
890
891 > swift post -m "Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4"
892
893 > swift tempurl GET 6000 /v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7\
894 /firstcontainer/clean.sh b3968d0207b54ece87cccc06515a89d4
895
896 /v1/AUTH_/firstcontainer/clean.sh?temp_url_sig=\
897 9218fc288cc09e5edd857b6a3d43cf2122b906dc&temp_url_expires=1472203614
898
899 The swiftclient.SwiftService API
900 A higher-level API aimed at allowing developers an easy way to perform
901 multiple operations asynchronously using a configurable thread pool.
902 Documentation for each service method call can be found here: swift‐
903 client.service.
904
905 Authentication
906 This section covers the various options for authenticating with a swift
907 object store. The combinations of options required for each authentica‐
908 tion version are detailed below. Once again, these are just a subset of
909 those that can be used to successfully authenticate, but they are the
910 most common and recommended.
911
912 The relevant authentication options are presented as python dictionar‐
913 ies that should be added to any other options you are supplying to your
914 SwiftService instance. As indicated in the python code, you can also
915 set these options as environment variables that will be loaded automat‐
916 ically if the relevant option is not specified.
917
918 The SwiftService authentication attempts to automatically select the
919 auth version based on the combination of options specified, but supply‐
920 ing options from multiple different auth versions can cause unexpected
921 behaviour.
922
923 NOTE:
924 Leftover environment variables are a common source of confusion when
925 authorization fails.
926
927 Keystone V3
928 {
929 ...
930 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '3'
931 "os_username": environ.get('OS_USERNAME'),
932 "os_password": environ.get('OS_PASSWORD'),
933 "os_project_name": environ.get('OS_PROJECT_NAME'),
934 "os_project_domain_name": environ.get('OS_PROJECT_DOMAIN_NAME'),
935 "os_auth_url": environ.get('OS_AUTH_URL'),
936 ...
937 }
938
939 {
940 ...
941 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '3'
942 "os_username": environ.get('OS_USERNAME'),
943 "os_password": environ.get('OS_PASSWORD'),
944 "os_project_id": environ.get('OS_PROJECT_ID'),
945 "os_project_domain_id": environ.get('OS_PROJECT_DOMAIN_ID'),
946 "os_auth_url": environ.get('OS_AUTH_URL'),
947 ...
948 }
949
950 Keystone V2
951 {
952 ...
953 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '2.0'
954 "os_username": environ.get('OS_USERNAME'),
955 "os_password": environ.get('OS_PASSWORD'),
956 "os_tenant_name": environ.get('OS_TENANT_NAME'),
957 "os_auth_url": environ.get('OS_AUTH_URL'),
958 ...
959 }
960
961 Legacy Auth
962 {
963 ...
964 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '1.0'
965 "auth": environ.get('ST_AUTH'),
966 "user": environ.get('ST_USER'),
967 "key": environ.get('ST_KEY'),
968 ...
969 }
970
971 Configuration
972 When you create an instance of a SwiftService, you can override a col‐
973 lection of default options to suit your use case. Typically, the
974 defaults are sensible to get us started, but depending on your needs
975 you might want to tweak them to improve performance (options affecting
976 large objects and thread counts can significantly alter performance in
977 the right situation).
978
979 Service level defaults and some extra options can also be overridden on
980 a per-operation (or even in some cases per-object) basis, and you will
981 call out which options affect which operations later in the document.
982
983 The configuration of the service API is performed using an options dic‐
984 tionary passed to the SwiftService during initialisation. The options
985 available in this dictionary are described below, along with their
986 defaults:
987
988 Options
989 retries: 5
990 The number of times that the library should attempt to retry
991 HTTP actions before giving up and reporting a failure.
992
993 container_threads: 10
994
995 object_dd_threads: 10
996
997 object_uu_threads: 10
998
999 segment_threads: 10
1000 The above options determine the size of the available thread
1001 pools for performing swift operations. Container operations
1002 (such as listing a container) operate in the container threads,
1003 and a similar pattern applies to object and segment threads.
1004
1005 NOTE:
1006 Object threads are separated into two separate thread pools:
1007 uu and dd. This stands for “upload/update” and “down‐
1008 load/delete”, and the corresponding actions will be run on
1009 separate threads pools.
1010
1011 segment_size: None
1012 If specified, this option enables uploading of large objects.
1013 Should the object being uploaded be larger than 5G in size, this
1014 option is mandatory otherwise the upload will fail. This option
1015 should be specified as a size in bytes.
1016
1017 use_slo: False
1018 Used in combination with the above option, use_slo will upload
1019 large objects as static rather than dynamic. Only static large
1020 objects provide error checking for the downloaded object, so we
1021 recommend this option.
1022
1023 segment_container: None
1024 Allows the user to select the container into which large object
1025 segments will be uploaded. We do not recommend changing this
1026 value as it could make locating orphaned segments more difficult
1027 in the case of errors.
1028
1029 leave_segments: False
1030 Setting this option to true means that when deleting or over‐
1031 writing a large object, its segments will be left in the object
1032 store and must be cleaned up manually. This option can be useful
1033 when sharing large object segments between multiple objects in
1034 more advanced scenarios, but must be treated with care, as it
1035 could lead to ever increasing storage usage.
1036
1037 changed: None
1038 This option affects uploads and simply means that those objects
1039 which already exist in the object store will not be overwritten
1040 if the mtime and size of the source is the same as the existing
1041 object.
1042
1043 skip_identical: False
1044 A slightly more thorough case of the above, but rather than
1045 mtime and size uses an object’s MD5 sum.
1046
1047 yes_all: False
1048 This options affects only download and delete, and in each case
1049 must be specified in order to download/delete the entire con‐
1050 tents of an account. This option has no effect on any other
1051 calls.
1052
1053 no_download: False
1054 This option only affects download and means that all operations
1055 proceed as normal with the exception that no data is written to
1056 disk.
1057
1058 header: []
1059 Used with upload and post operations to set headers on objects.
1060 Headers are specified as colon separated strings, e.g. “con‐
1061 tent-type:text/plain”.
1062
1063 meta: []
1064 Used to set metadata on an object similarly to headers.
1065
1066 NOTE:
1067 Setting metadata is a destructive operation, so when updating
1068 one of many metadata values all desired metadata for an
1069 object must be re-applied.
1070
1071 long: False
1072 Affects only list operations, and results in more metrics being
1073 made available in the results at the expense of lower perfor‐
1074 mance.
1075
1076 fail_fast: False
1077 Applies to delete and upload operations, and attempts to abort
1078 queued tasks in the event of errors.
1079
1080 prefix: None
1081 Affects list operations; only objects with the given prefix will
1082 be returned/affected. It is not advisable to set at the service
1083 level, as those operations that call list to discover objects on
1084 which they should operate will also be affected.
1085
1086 delimiter: None
1087 Affects list operations, and means that listings only contain
1088 results up to the first instance of the delimiter in the object
1089 name. This is useful for working with objects containing ‘/’ in
1090 their names to simulate folder structures.
1091
1092 dir_marker: False
1093 Affects uploads, and allows empty ‘pseudofolder’ objects to be
1094 created when the source of an upload is None.
1095
1096 checksum: True
1097 Affects uploads and downloads. If set check md5 sum for the
1098 transfer.
1099
1100 shuffle: False
1101 When downloading objects, the default behaviour of the CLI is to
1102 shuffle lists of objects in order to spread the load on storage
1103 drives when multiple clients are downloading the same files to
1104 multiple locations (e.g. in the event of distributing an
1105 update). When using the SwiftService directly, object downloads
1106 are scheduled in the same order as they appear in the container
1107 listing. When combined with a single download thread this means
1108 that objects are downloaded in lexically-sorted order. Setting
1109 this option to True gives the same shuffling behaviour as the
1110 CLI.
1111
1112 destination: None
1113 When copying objects, this specifies the destination where the
1114 object will be copied to. The default of None means copy will
1115 be the same as source.
1116
1117 fresh_metadata: None
1118 When copying objects, this specifies that the object metadata on
1119 the source will not be applied to the destination object - the
1120 destination object will have a new fresh set of metadata that
1121 includes only the metadata specified in the meta option if any
1122 at all.
1123
1124 Other available options can be found in swiftclient/service.py in the
1125 source code for python-swiftclient. Each SwiftService method also
1126 allows for an optional dictionary to override those specified at init
1127 time, and the appropriate docstrings show which options modify each
1128 method’s behaviour.
1129
1130 Available Operations
1131 Each operation provided by the service API may raise a SwiftError or
1132 ClientException for any call that fails completely (or a call which
1133 performs only one operation at an account or container level). In the
1134 case of a successful call an operation returns one of the following:
1135
1136 · A dictionary detailing the results of a single operation.
1137
1138 · An iterator that produces result dictionaries (for calls that perform
1139 multiple sub-operations).
1140
1141 A result dictionary can indicate either the success or failure of an
1142 individual operation (detailed in the success key), and will either
1143 contain the successful result, or an error key detailing the error
1144 encountered (usually an instance of Exception).
1145
1146 An example result dictionary is given below:
1147
1148 result = {
1149 'action': 'download_object',
1150 'success': True,
1151 'container': container,
1152 'object': obj,
1153 'path': path,
1154 'start_time': start_time,
1155 'finish_time': finish_time,
1156 'headers_receipt': headers_receipt,
1157 'auth_end_time': conn.auth_end_time,
1158 'read_length': bytes_read,
1159 'attempts': conn.attempts
1160 }
1161
1162 All the possible action values are detailed below:
1163
1164 [
1165 'stat_account',
1166 'stat_container',
1167 'stat_object',
1168 'post_account',
1169 'post_container',
1170 'post_object',
1171 'list_part', # list yields zero or more 'list_part' results
1172 'download_object',
1173 'create_container', # from upload
1174 'create_dir_marker', # from upload
1175 'upload_object',
1176 'upload_segment',
1177 'delete_container',
1178 'delete_object',
1179 'delete_segment', # from delete_object operations
1180 'capabilities',
1181 ]
1182
1183 Stat
1184 Stat can be called against an account, a container, or a list of
1185 objects to get account stats, container stats or information about the
1186 given objects. In the first two cases a dictionary is returned contain‐
1187 ing the results of the operation, and in the case of a list of object
1188 names being supplied, an iterator over the results generated for each
1189 object is returned.
1190
1191 Information returned includes the amount of data used by the given
1192 object/container/account and any headers or metadata set (this includes
1193 user set data as well as content-type and modification times).
1194
1195 See swiftclient.service.SwiftService.stat for docs generated from the
1196 method docstring.
1197
1198 Valid calls for this method are as follows:
1199
1200 · stat([options]): Returns stats for the configured account.
1201
1202 · stat(<container>, [options]): Returns stats for the given container.
1203
1204 · stat(<container>, <object_list>, [options]): Returns stats for each
1205 of the given objects in the given container (through the returned
1206 iterator).
1207
1208 Results from stat are dictionaries indicating the success or failure of
1209 each operation. In the case of a successful stat against an account or
1210 container, the method returns immediately with one of the following
1211 results:
1212
1213 {
1214 'action': 'stat_account',
1215 'success': True,
1216 'items': items,
1217 'headers': headers
1218 }
1219
1220 {
1221 'action': 'stat_container',
1222 'container': <container>,
1223 'success': True,
1224 'items': items,
1225 'headers': headers
1226 }
1227
1228 In the case of stat called against a list of objects, the method
1229 returns a generator that returns the results of individual object stat
1230 operations as they are performed on the thread pool:
1231
1232 {
1233 'action': 'stat_object',
1234 'object': <object_name>,
1235 'container': <container>,
1236 'success': True,
1237 'items': items,
1238 'headers': headers
1239 }
1240
1241 In the case of a failure the dictionary returned will indicate that the
1242 operation was not successful, and will include the keys below:
1243
1244 {
1245 'action': <'stat_object'|'stat_container'|'stat_account'>,
1246 'object': <'object_name'>, # Only for stat with objects list
1247 'container': <container>, # Only for stat with objects list or container
1248 'success': False,
1249 'error': <error>,
1250 'traceback': <trace>,
1251 'error_timestamp': <timestamp>
1252 }
1253
1254 Example
1255 The code below demonstrates the use of stat to retrieve the headers for
1256 a given list of objects in a container using 20 threads. The code cre‐
1257 ates a mapping from object name to headers which is then pretty printed
1258 to the log.
1259
1260 import logging
1261 import pprint
1262
1263 from swiftclient.service import SwiftService
1264 from sys import argv
1265
1266 logging.basicConfig(level=logging.ERROR)
1267 logging.getLogger("requests").setLevel(logging.CRITICAL)
1268 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1269 logger = logging.getLogger(__name__)
1270
1271 _opts = {'object_dd_threads': 20}
1272 with SwiftService(options=_opts) as swift:
1273 container = argv[1]
1274 objects = argv[2:]
1275 header_data = {}
1276 stats_it = swift.stat(container=container, objects=objects)
1277 for stat_res in stats_it:
1278 if stat_res['success']:
1279 header_data[stat_res['object']] = stat_res['headers']
1280 else:
1281 logger.error(
1282 'Failed to retrieve stats for %s' % stat_res['object']
1283 )
1284 pprint.pprint(header_data)
1285
1286
1287 List
1288 List can be called against an account or a container to retrieve the
1289 containers or objects contained within them. Each call returns an iter‐
1290 ator that returns pages of results (by default, up to 10000 results in
1291 each page).
1292
1293 See swiftclient.service.SwiftService.list for docs generated from the
1294 method docstring.
1295
1296 If the given container or account does not exist, the list method will
1297 raise a SwiftError, but for all other success/failures a dictionary is
1298 returned. Each successfully listed page returns a dictionary as
1299 described below:
1300
1301 {
1302 'action': <'list_account_part'|'list_container_part'>,
1303 'container': <container>, # Only for listing a container
1304 'prefix': <prefix>, # The prefix of returned objects/containers
1305 'success': True,
1306 'listing': [Item], # A list of results
1307 # (only in the event of success)
1308 'marker': <marker> # The last item name in the list
1309 # (only in the event of success)
1310 }
1311
1312 Where an item contains the following keys:
1313
1314 {
1315 'name': <name>,
1316 'bytes': 10485760,
1317 'last_modified': '2014-12-11T12:02:38.774540',
1318 'hash': 'fb938269cbeabe4c234e1127bbd3b74a',
1319 'content_type': 'application/octet-stream',
1320 'meta': <metadata> # Full metadata listing from stat'ing each object
1321 # this key only exists if 'long' is specified in options
1322 }
1323
1324 Any failure listing an account or container that exists will return a
1325 failure dictionary as described below:
1326
1327 {
1328 'action': <'list_account_part'|'list_container_part'>,,
1329 'container': container, # Only for listing a container
1330 'prefix': options['prefix'],
1331 'success': success,
1332 'marker': marker,
1333 'error': error,
1334 'traceback': <trace>,
1335 'error_timestamp': <timestamp>
1336 }
1337
1338 Example
1339 The code below demonstrates the use of list to list all items in a con‐
1340 tainer that are over 10MiB in size:
1341
1342 import logging
1343
1344 from swiftclient.service import SwiftService, SwiftError
1345 from sys import argv
1346
1347 logging.basicConfig(level=logging.ERROR)
1348 logging.getLogger("requests").setLevel(logging.CRITICAL)
1349 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1350 logger = logging.getLogger(__name__)
1351
1352 container = argv[1]
1353 minimum_size = 10*1024**2
1354 with SwiftService() as swift:
1355 try:
1356 list_parts_gen = swift.list(container=container)
1357 for page in list_parts_gen:
1358 if page["success"]:
1359 for item in page["listing"]:
1360
1361 i_size = int(item["bytes"])
1362 if i_size > minimum_size:
1363 i_name = item["name"]
1364 i_etag = item["hash"]
1365 print(
1366 "%s [size: %s] [etag: %s]" %
1367 (i_name, i_size, i_etag)
1368 )
1369 else:
1370 raise page["error"]
1371
1372 except SwiftError as e:
1373 logger.error(e.value)
1374
1375
1376 Post
1377 Post can be called against an account, container or list of objects in
1378 order to update the metadata attached to the given items. In the first
1379 two cases a single dictionary is returned containing the results of the
1380 operation, and in the case of a list of objects being supplied, an
1381 iterator over the results generated for each object post is returned.
1382
1383 Each element of the object list may be a plain string of the object
1384 name, or a SwiftPostObject that allows finer control over the options
1385 and metadata applied to each of the individual post operations. When a
1386 string is given for the object name, the options and metadata applied
1387 are a combination of those supplied to the call to post() and the
1388 defaults of the SwiftService object.
1389
1390 If the given container or account does not exist, the post method will
1391 raise a SwiftError. Successful metadata update results are dictionaries
1392 as described below:
1393
1394 {
1395 'action': <'post_account'|'post_container'|'post_object'>,
1396 'success': True,
1397 'container': <container>,
1398 'object': <object>,
1399 'headers': {},
1400 'response_dict': <HTTP response details>
1401 }
1402
1403 NOTE:
1404 Updating user metadata keys will not only add any specified keys,
1405 but will also remove user metadata that has previously been set.
1406 This means that each time user metadata is updated, the complete set
1407 of desired key-value pairs must be specified.
1408
1409 Example
1410 The code below demonstrates the use of post to set an archive folder in
1411 a given container to expire after a 24 hour delay:
1412
1413 import logging
1414
1415 from swiftclient.service import SwiftService, SwiftError
1416 from sys import argv
1417
1418 logging.basicConfig(level=logging.ERROR)
1419 logging.getLogger("requests").setLevel(logging.CRITICAL)
1420 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1421 logger = logging.getLogger(__name__)
1422
1423 container = argv[1]
1424 with SwiftService() as swift:
1425 try:
1426 list_options = {"prefix": "archive_2016-01-01/"}
1427 list_parts_gen = swift.list(container=container)
1428 for page in list_parts_gen:
1429 if page["success"]:
1430 objects = [obj["name"] for obj in page["listing"]]
1431 post_options = {"header": "X-Delete-After:86400"}
1432 for post_res in swift.post(
1433 container=container,
1434 objects=objects,
1435 options=post_options):
1436 if post_res['success']:
1437 print("Object '%s' POST success" % post_res['object'])
1438 else:
1439 print("Object '%s' POST failed" % post_res['object'])
1440 else:
1441 raise page["error"]
1442 except SwiftError as e:
1443 logger.error(e.value)
1444
1445
1446 Download
1447 Download can be called against an entire account, a single container,
1448 or a list of objects in a given container. Each element of the object
1449 list is a string detailing the full name of an object to download.
1450
1451 In order to download the full contents of an entire account, you must
1452 set the value of yes_all to True in the options dictionary supplied to
1453 either the SwiftService instance or the call to download.
1454
1455 If the given container or account does not exist, the download method
1456 will raise a SwiftError, otherwise an iterator over the results gener‐
1457 ated for each object download is returned.
1458
1459 See swiftclient.service.SwiftService.download for docs generated from
1460 the method docstring.
1461
1462 For each successfully downloaded object, the results returned by the
1463 iterator will be a dictionary as described below (results are not
1464 returned for completed container or object segment downloads):
1465
1466 {
1467 'action': 'download_object',
1468 'container': <container>,
1469 'object': <object name>,
1470 'success': True,
1471 'path': <local path to downloaded object>,
1472 'pseudodir': <if true, the download created an empty directory>,
1473 'start_time': <time download started>,
1474 'end_time': <time download completed>,
1475 'headers_receipt': <time the headers from the object were retrieved>,
1476 'auth_end_time': <time authentication completed>,
1477 'read_length': <bytes_read>,
1478 'attempts': <attempt count>,
1479 'response_dict': <HTTP response details>
1480 }
1481
1482 Any failure uploading an object will return a failure dictionary as
1483 described below:
1484
1485 {
1486 'action': 'download_object',
1487 'container': <container>,
1488 'object': <object name>,
1489 'success': False,
1490 'path': <local path of the failed download>,
1491 'pseudodir': <if true, the failed download was an empty directory>,
1492 'attempts': <attempt count>,
1493 'error': <error>,
1494 'traceback': <trace>,
1495 'error_timestamp': <timestamp>,
1496 'response_dict': <HTTP response details>
1497 }
1498
1499 Example
1500 The code below demonstrates the use of download to download all PNG
1501 images from a dated archive folder in a given container:
1502
1503 import logging
1504
1505 from swiftclient.service import SwiftService, SwiftError
1506 from sys import argv
1507
1508 logging.basicConfig(level=logging.ERROR)
1509 logging.getLogger("requests").setLevel(logging.CRITICAL)
1510 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1511 logger = logging.getLogger(__name__)
1512
1513 def is_png(obj):
1514 return (
1515 obj["name"].lower().endswith('.png') or
1516 obj["content_type"] == 'image/png'
1517 )
1518
1519 container = argv[1]
1520 with SwiftService() as swift:
1521 try:
1522 list_options = {"prefix": "archive_2016-01-01/"}
1523 list_parts_gen = swift.list(container=container)
1524 for page in list_parts_gen:
1525 if page["success"]:
1526 objects = [
1527 obj["name"] for obj in page["listing"] if is_png(obj)
1528 ]
1529 for down_res in swift.download(
1530 container=container,
1531 objects=objects):
1532 if down_res['success']:
1533 print("'%s' downloaded" % down_res['object'])
1534 else:
1535 print("'%s' download failed" % down_res['object'])
1536 else:
1537 raise page["error"]
1538 except SwiftError as e:
1539 logger.error(e.value)
1540
1541
1542 Upload
1543 Upload is always called against an account and container and with a
1544 list of objects to upload. Each element of the object list may be a
1545 plain string detailing the path of the object to upload, or a SwiftU‐
1546 ploadObject that allows finer control over some aspects of the individ‐
1547 ual operations.
1548
1549 When a simple string is supplied to specify a file to upload, the name
1550 of the object uploaded is the full path of the specified file and the
1551 options used for the upload are those supplied to the call to upload.
1552
1553 Constructing a SwiftUploadObject allows the user to supply an object
1554 name for the uploaded file, and modify the options used by upload at
1555 the granularity of individual files.
1556
1557 If the given container or account does not exist, the upload method
1558 will raise a SwiftError, otherwise an iterator over the results gener‐
1559 ated for each object upload is returned.
1560
1561 See swiftclient.service.SwiftService.upload for docs generated from the
1562 method docstring.
1563
1564 For each successfully uploaded object (or object segment), the results
1565 returned by the iterator will be a dictionary as described below:
1566
1567 {
1568 'action': 'upload_object',
1569 'container': <container>,
1570 'object': <object name>,
1571 'success': True,
1572 'status': <'uploaded'|'skipped-identical'|'skipped-changed'>,
1573 'attempts': <attempt count>,
1574 'response_dict': <HTTP response details>
1575 }
1576
1577 {
1578 'action': 'upload_segment',
1579 'for_container': <container>,
1580 'for_object': <object name>,
1581 'segment_index': <segment_index>,
1582 'segment_size': <segment_size>,
1583 'segment_location': <segment_path>
1584 'segment_etag': <etag>,
1585 'log_line': <object segment n>
1586 'success': True,
1587 'response_dict': <HTTP response details>,
1588 'attempts': <attempt count>
1589 }
1590
1591 Any failure uploading an object will return a failure dictionary as
1592 described below:
1593
1594 {
1595 'action': 'upload_object',
1596 'container': <container>,
1597 'object': <object name>,
1598 'success': False,
1599 'attempts': <attempt count>,
1600 'error': <error>,
1601 'traceback': <trace>,
1602 'error_timestamp': <timestamp>,
1603 'response_dict': <HTTP response details>
1604 }
1605
1606 {
1607 'action': 'upload_segment',
1608 'for_container': <container>,
1609 'for_object': <object name>,
1610 'segment_index': <segment_index>,
1611 'segment_size': <segment_size>,
1612 'segment_location': <segment_path>,
1613 'log_line': <object segment n>,
1614 'success': False,
1615 'error': <error>,
1616 'traceback': <trace>,
1617 'error_timestamp': <timestamp>,
1618 'response_dict': <HTTP response details>,
1619 'attempts': <attempt count>
1620 }
1621
1622 Example
1623 The code below demonstrates the use of upload to upload all files and
1624 folders in a given directory, and rename each object by replacing the
1625 root directory name with ‘my-<d>-objects’, where <d> is the name of the
1626 uploaded directory:
1627
1628 import logging
1629
1630 from os import walk
1631 from os.path import join
1632 from swiftclient.multithreading import OutputManager
1633 from swiftclient.service import SwiftError, SwiftService, SwiftUploadObject
1634 from sys import argv
1635
1636 logging.basicConfig(level=logging.ERROR)
1637 logging.getLogger("requests").setLevel(logging.CRITICAL)
1638 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1639 logger = logging.getLogger(__name__)
1640
1641 _opts = {'object_uu_threads': 20}
1642 dir = argv[1]
1643 container = argv[2]
1644 with SwiftService(options=_opts) as swift, OutputManager() as out_manager:
1645 try:
1646 # Collect all the files and folders in the given directory
1647 objs = []
1648 dir_markers = []
1649 for (_dir, _ds, _fs) in walk(dir):
1650 if not (_ds + _fs):
1651 dir_markers.append(_dir)
1652 else:
1653 objs.extend([join(_dir, _f) for _f in _fs])
1654
1655 # Now that we've collected all the required files and dir markers
1656 # build the ``SwiftUploadObject``s for the call to upload
1657 objs = [
1658 SwiftUploadObject(
1659 o, object_name=o.replace(
1660 dir, 'my-%s-objects' % dir, 1
1661 )
1662 ) for o in objs
1663 ]
1664 dir_markers = [
1665 SwiftUploadObject(
1666 None, object_name=d.replace(
1667 dir, 'my-%s-objects' % dir, 1
1668 ), options={'dir_marker': True}
1669 ) for d in dir_markers
1670 ]
1671
1672 # Schedule uploads on the SwiftService thread pool and iterate
1673 # over the results
1674 for r in swift.upload(container, objs + dir_markers):
1675 if r['success']:
1676 if 'object' in r:
1677 print(r['object'])
1678 elif 'for_object' in r:
1679 print(
1680 '%s segment %s' % (r['for_object'],
1681 r['segment_index'])
1682 )
1683 else:
1684 error = r['error']
1685 if r['action'] == "create_container":
1686 logger.warning(
1687 'Warning: failed to create container '
1688 "'%s'%s", container, error
1689 )
1690 elif r['action'] == "upload_object":
1691 logger.error(
1692 "Failed to upload object %s to container %s: %s" %
1693 (container, r['object'], error)
1694 )
1695 else:
1696 logger.error("%s" % error)
1697
1698 except SwiftError as e:
1699 logger.error(e.value)
1700
1701
1702 Delete
1703 Delete can be called against an account or a container to remove the
1704 containers or objects contained within them. Each call to delete
1705 returns an iterator over results of each resulting sub-request.
1706
1707 If the number of requested delete operations is large and the target
1708 swift cluster is running the bulk middleware, the call to SwiftSer‐
1709 vice.delete will make use of bulk operations and the returned result
1710 iterator will return bulk_delete results rather than individual
1711 delete_object, delete_container or delete_segment results.
1712
1713 See swiftclient.service.SwiftService.delete for docs generated from the
1714 method docstring.
1715
1716 For each successfully deleted container, object or segment, the results
1717 returned by the iterator will be a dictionary as described below:
1718
1719 {
1720 'action': <'delete_object'|'delete_segment'>,
1721 'container': <container>,
1722 'object': <object name>,
1723 'success': True,
1724 'attempts': <attempt count>,
1725 'response_dict': <HTTP response details>
1726 }
1727
1728 {
1729 'action': 'delete_container',
1730 'container': <container>,
1731 'success': True,
1732 'response_dict': <HTTP response details>,
1733 'attempts': <attempt count>
1734 }
1735
1736 {
1737 'action': 'bulk_delete',
1738 'container': <container>,
1739 'objects': <[objects]>,
1740 'success': True,
1741 'attempts': <attempt count>,
1742 'response_dict': <HTTP response details>
1743 }
1744
1745 Any failure in a delete operation will return a failure dictionary as
1746 described below:
1747
1748 {
1749 'action': ('delete_object'|'delete_segment'),
1750 'container': <container>,
1751 'object': <object name>,
1752 'success': False,
1753 'attempts': <attempt count>,
1754 'error': <error>,
1755 'traceback': <trace>,
1756 'error_timestamp': <timestamp>,
1757 'response_dict': <HTTP response details>
1758 }
1759
1760 {
1761 'action': 'delete_container',
1762 'container': <container>,
1763 'success': False,
1764 'error': <error>,
1765 'traceback': <trace>,
1766 'error_timestamp': <timestamp>,
1767 'response_dict': <HTTP response details>,
1768 'attempts': <attempt count>
1769 }
1770
1771 {
1772 'action': 'bulk_delete',
1773 'container': <container>,
1774 'objects': <[objects]>,
1775 'success': False,
1776 'attempts': <attempt count>,
1777 'error': <error>,
1778 'traceback': <trace>,
1779 'error_timestamp': <timestamp>,
1780 'response_dict': <HTTP response details>
1781 }
1782
1783 Example
1784 The code below demonstrates the use of delete to remove a given list of
1785 objects from a specified container. As the objects are deleted the
1786 transaction ID of the relevant request is printed along with the object
1787 name and number of attempts required. By printing the transaction ID,
1788 the printed operations can be easily linked to events in the swift
1789 server logs:
1790
1791 import logging
1792
1793 from swiftclient.service import SwiftService
1794 from sys import argv
1795
1796
1797 logging.basicConfig(level=logging.ERROR)
1798 logging.getLogger("requests").setLevel(logging.CRITICAL)
1799 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1800 logger = logging.getLogger(__name__)
1801
1802 _opts = {'object_dd_threads': 20}
1803 container = argv[1]
1804 objects = argv[2:]
1805 with SwiftService(options=_opts) as swift:
1806 del_iter = swift.delete(container=container, objects=objects)
1807 for del_res in del_iter:
1808 c = del_res.get('container', '')
1809 o = del_res.get('object', '')
1810 a = del_res.get('attempts')
1811 if del_res['success'] and not del_res['action'] == 'bulk_delete':
1812 rd = del_res.get('response_dict')
1813 if rd is not None:
1814 t = dict(rd.get('headers', {}))
1815 if t:
1816 print(
1817 'Successfully deleted {0}/{1} in {2} attempts '
1818 '(transaction id: {3})'.format(c, o, a, t)
1819 )
1820 else:
1821 print(
1822 'Successfully deleted {0}/{1} in {2} '
1823 'attempts'.format(c, o, a)
1824 )
1825
1826
1827 Copy
1828 Copy can be called to copy an object or update the metadata on the
1829 given items.
1830
1831 Each element of the object list may be a plain string of the object
1832 name, or a SwiftCopyObject that allows finer control over the options
1833 applied to each of the individual copy operations (destination,
1834 fresh_metadata, options).
1835
1836 Destination should be in format /container/object; if not set, the
1837 object will be copied onto itself. Fresh_metadata sets mode of opera‐
1838 tion on metadata. If not set, current object user metadata will be
1839 copied/preserved; if set, all current user metadata will be removed.
1840
1841 Returns an iterator over the results generated for each object copy
1842 (and may also include the results of creating destination containers).
1843
1844 When a string is given for the object name, destination and fresh meta‐
1845 data will default to None and None, which result in adding metadata to
1846 existing objects.
1847
1848 Successful copy results are dictionaries as described below:
1849
1850 {
1851 'action': 'copy_object',
1852 'success': True,
1853 'container': <container>,
1854 'object': <object>,
1855 'destination': <destination>,
1856 'headers': {},
1857 'fresh_metadata': <boolean>,
1858 'response_dict': <HTTP response details>
1859 }
1860
1861 Any failure in a copy operation will return a failure dictionary as
1862 described below:
1863
1864 {
1865 'action': 'copy_object',
1866 'success': False,
1867 'container': <container>,
1868 'object': <object>,
1869 'destination': <destination>,
1870 'headers': {},
1871 'fresh_metadata': <boolean>,
1872 'response_dict': <HTTP response details>,
1873 'error': <error>,
1874 'traceback': <traceback>,
1875 'error_timestamp': <timestamp>
1876 }
1877
1878 Example
1879 The code below demonstrates the use of copy to add new user metadata
1880 for objects a and b, and to copy object c to d (with added metadata).
1881
1882 import logging
1883
1884 from swiftclient.service import SwiftService, SwiftCopyObject, SwiftError
1885
1886 logging.basicConfig(level=logging.ERROR)
1887 logging.getLogger("requests").setLevel(logging.CRITICAL)
1888 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1889 logger = logging.getLogger(__name__)
1890
1891 with SwiftService() as swift:
1892 try:
1893 obj = SwiftCopyObject("c", {"Destination": "/cont/d"})
1894 for i in swift.copy(
1895 "cont", ["a", "b", obj],
1896 {"meta": ["foo:bar"], "Destination": "/cc"}):
1897 if i["success"]:
1898 if i["action"] == "copy_object":
1899 print(
1900 "object %s copied from /%s/%s" %
1901 (i["destination"], i["container"], i["object"])
1902 )
1903 if i["action"] == "create_container":
1904 print(
1905 "container %s created" % i["container"]
1906 )
1907 else:
1908 if "error" in i and isinstance(i["error"], Exception):
1909 raise i["error"]
1910 except SwiftError as e:
1911 logger.error(e.value)
1912
1913
1914 Capabilities
1915 Capabilities can be called against an account or a particular proxy URL
1916 in order to determine the capabilities of the swift cluster. These
1917 capabilities include details about configuration options and the mid‐
1918 dlewares that are installed in the proxy pipeline.
1919
1920 See swiftclient.service.SwiftService.capabilities for docs generated
1921 from the method docstring.
1922
1923 For each successful call to list capabilities, a result dictionary will
1924 be returned with the contents described below:
1925
1926 {
1927 'action': 'capabilities',
1928 'timestamp': <time of the call>,
1929 'success': True,
1930 'capabilities': <dictionary containing capability details>
1931 }
1932
1933 The contents of the capabilities dictionary contain the core swift
1934 capabilities under the key swift; all other keys show the configuration
1935 options for additional middlewares deployed in the proxy pipeline. An
1936 example capabilities dictionary is given below:
1937
1938 {
1939 'account_quotas': {},
1940 'bulk_delete': {
1941 'max_deletes_per_request': 10000,
1942 'max_failed_deletes': 1000
1943 },
1944 'bulk_upload': {
1945 'max_containers_per_extraction': 10000,
1946 'max_failed_extractions': 1000
1947 },
1948 'container_quotas': {},
1949 'container_sync': {'realms': {}},
1950 'formpost': {},
1951 'keystoneauth': {},
1952 'slo': {
1953 'max_manifest_segments': 1000,
1954 'max_manifest_size': 2097152,
1955 'min_segment_size': 1048576
1956 },
1957 'swift': {
1958 'account_autocreate': True,
1959 'account_listing_limit': 10000,
1960 'allow_account_management': True,
1961 'container_listing_limit': 10000,
1962 'extra_header_count': 0,
1963 'max_account_name_length': 256,
1964 'max_container_name_length': 256,
1965 'max_file_size': 5368709122,
1966 'max_header_size': 8192,
1967 'max_meta_count': 90,
1968 'max_meta_name_length': 128,
1969 'max_meta_overall_size': 4096,
1970 'max_meta_value_length': 256,
1971 'max_object_name_length': 1024,
1972 'policies': [
1973 {'default': True, 'name': 'Policy-0'}
1974 ],
1975 'strict_cors_mode': False,
1976 'version': '2.2.2'
1977 },
1978 'tempurl': {
1979 'methods': ['GET', 'HEAD', 'PUT']
1980 }
1981 }
1982
1983 Example
1984 The code below demonstrates the use of capabilities to determine if the
1985 Swift cluster supports static large objects, and if so, the maximum
1986 number of segments that can be described in a single manifest file,
1987 along with the size restrictions on those objects:
1988
1989 import logging
1990
1991 from swiftclient.exceptions import ClientException
1992 from swiftclient.service import SwiftService
1993
1994 logging.basicConfig(level=logging.ERROR)
1995 logging.getLogger("requests").setLevel(logging.CRITICAL)
1996 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1997 logger = logging.getLogger(__name__)
1998
1999 with SwiftService() as swift:
2000 try:
2001 capabilities_result = swift.capabilities()
2002 capabilities = capabilities_result['capabilities']
2003 if 'slo' in capabilities:
2004 print('SLO is supported')
2005 else:
2006 print('SLO is not supported')
2007 except ClientException as e:
2008 logger.error(e.value)
2009
2010
2011 The swiftclient.Connection API
2012 A low level API that provides methods for authentication and methods
2013 that correspond to the individual REST API calls described in the swift
2014 documentation.
2015
2016 For usage details see the client docs: swiftclient.client.
2017
2018 Authentication
2019 This section covers the various combinations of kwargs required when
2020 creating an instance of the Connection object for communicating with a
2021 swift object store. The combinations of options required for each
2022 authentication version are detailed below, but are just a subset of
2023 those that can be used to successfully authenticate. These are the most
2024 common and recommended combinations.
2025
2026 Keystone Session
2027 from keystoneauth1 import session
2028 from keystoneauth1.identity import v3
2029
2030 # Create a password auth plugin
2031 auth = v3.Password(auth_url='http://127.0.0.1:5000/v3/',
2032 username='tester',
2033 password='testing',
2034 user_domain_name='Default',
2035 project_name='Default',
2036 project_domain_name='Default')
2037
2038 # Create session
2039 keystone_session = session.Session(auth=auth)
2040
2041 # Create swiftclient Connection
2042 swift_conn = Connection(session=keystone_session)
2043
2044 Keystone v3
2045 _authurl = 'http://127.0.0.1:5000/v3/'
2046 _auth_version = '3'
2047 _user = 'tester'
2048 _key = 'testing'
2049 _os_options = {
2050 'user_domain_name': 'Default',
2051 'project_domain_name': 'Default',
2052 'project_name': 'Default'
2053 }
2054
2055 conn = Connection(
2056 authurl=_authurl,
2057 user=_user,
2058 key=_key,
2059 os_options=_os_options,
2060 auth_version=_auth_version
2061 )
2062
2063 Keystone v2
2064 _authurl = 'http://127.0.0.1:5000/v2.0/'
2065 _auth_version = '2'
2066 _user = 'tester'
2067 _key = 'testing'
2068 _tenant_name = 'test'
2069
2070 conn = Connection(
2071 authurl=_authurl,
2072 user=_user,
2073 key=_key,
2074 tenant_name=_tenant_name,
2075 auth_version=_auth_version
2076 )
2077
2078 Legacy Auth
2079 _authurl = 'http://127.0.0.1:8080/'
2080 _auth_version = '1'
2081 _user = 'tester'
2082 _key = 'testing'
2083 _tenant_name = 'test'
2084
2085 conn = Connection(
2086 authurl=_authurl,
2087 user=_user,
2088 key=_key,
2089 tenant_name=_tenant_name,
2090 auth_version=_auth_version
2091 )
2092
2093 Examples
2094 In this section we present some simple code examples that demonstrate
2095 the usage of the Connection API. You can find full details of the
2096 options and methods available to the Connection API in the docstring
2097 generated documentation: swiftclient.client.
2098
2099 List the available containers:
2100
2101 resp_headers, containers = conn.get_account()
2102 print("Response headers: %s" % resp_headers)
2103 for container in containers:
2104 print(container)
2105
2106 Create a new container:
2107
2108 container = 'new-container'
2109 conn.put_container(container)
2110 resp_headers, containers = conn.get_account()
2111 if container in containers:
2112 print("The container was created")
2113
2114 Create a new object with the contents of a local text file:
2115
2116 container = 'new-container'
2117 with open('local.txt', 'r') as local:
2118 conn.put_object(
2119 container,
2120 'local_object.txt',
2121 contents=local,
2122 content_type='text/plain'
2123 )
2124
2125 Confirm presence of the object:
2126
2127 obj = 'local_object.txt'
2128 container = 'new-container'
2129 try:
2130 resp_headers = conn.head_object(container, obj)
2131 print('The object was successfully created')
2132 except ClientException as e:
2133 if e.http_status = '404':
2134 print('The object was not found')
2135 else:
2136 print('An error occurred checking for the existence of the object')
2137
2138 Download the created object:
2139
2140 obj = 'local_object.txt'
2141 container = 'new-container'
2142 resp_headers, obj_contents = conn.get_object(container, obj)
2143 with open('local_copy.txt', 'w') as local:
2144 local.write(obj_contents)
2145
2146 Delete the created object:
2147
2148 obj = 'local_object.txt'
2149 container = 'new-container'
2150 try:
2151 conn.delete_object(container, obj)
2152 print("Successfully deleted the object")
2153 except ClientException as e:
2154 print("Failed to delete the object with error: %s" % e)
2155
2157 swiftclient
2158 OpenStack Swift Python client binding.
2159
2160 swiftclient.authv1
2161 swiftclient.client
2162 OpenStack Swift client library used internally
2163
2164 class swiftclient.client.Connection(authurl=None, user=None, key=None,
2165 retries=5, preauthurl=None, preauthtoken=None, snet=False, start‐
2166 ing_backoff=1, max_backoff=64, tenant_name=None, os_options=None,
2167 auth_version='1', cacert=None, insecure=False, cert=None,
2168 cert_key=None, ssl_compression=True, retry_on_ratelimit=False, time‐
2169 out=None, session=None, force_auth_retry=False)
2170 Bases: object
2171
2172 Convenience class to make requests that will also retry the
2173 request
2174
2175 Requests will have an X-Auth-Token header whose value is either
2176 the preauthtoken or a token obtained from the auth service using
2177 the user credentials provided as args to the constructor. If
2178 os_options includes a service_username then requests will also
2179 have an X-Service-Token header whose value is a token obtained
2180 from the auth service using the service credentials. In this
2181 case the request url will be set to the storage_url obtained
2182 from the auth service for the service user, unless this is over‐
2183 ridden by a preauthurl.
2184
2185 Parameters
2186
2187 · authurl -- authentication URL
2188
2189 · user -- user name to authenticate as
2190
2191 · key -- key/password to authenticate with
2192
2193 · retries -- Number of times to retry the request before
2194 failing
2195
2196 · preauthurl -- storage URL (if you have already authen‐
2197 ticated)
2198
2199 · preauthtoken -- authentication token (if you have
2200 already authenticated) note authurl/user/key/ten‐
2201 ant_name are not required when specifying preauthtoken
2202
2203 · snet -- use SERVICENET internal network default is
2204 False
2205
2206 · starting_backoff -- initial delay between retries (sec‐
2207 onds)
2208
2209 · max_backoff -- maximum delay between retries (seconds)
2210
2211 · auth_version -- OpenStack auth version, default is 1.0
2212
2213 · tenant_name -- The tenant/account name, required when
2214 connecting to an auth 2.0 system.
2215
2216 · os_options -- The OpenStack options which can have ten‐
2217 ant_id, auth_token, service_type, endpoint_type, ten‐
2218 ant_name, object_storage_url, region_name, ser‐
2219 vice_username, service_project_name, service_key
2220
2221 · insecure -- Allow to access servers without checking
2222 SSL certs. The server's certificate will not be veri‐
2223 fied.
2224
2225 · cert -- Client certificate file to connect on SSL
2226 server requiring SSL client certificate.
2227
2228 · cert_key -- Client certificate private key file.
2229
2230 · ssl_compression -- Whether to enable compression at the
2231 SSL layer. If set to 'False' and the pyOpenSSL library
2232 is present an attempt to disable SSL compression will
2233 be made. This may provide a performance increase for
2234 https upload/download operations.
2235
2236 · retry_on_ratelimit -- by default, a ratelimited connec‐
2237 tion will raise an exception to the caller. Setting
2238 this parameter to True will cause a retry after a back‐
2239 off.
2240
2241 · timeout -- The connect timeout for the HTTP connection.
2242
2243 · session -- A keystoneauth session object.
2244
2245 · force_auth_retry -- reset auth info even if client got
2246 unexpected error except 401 Unauthorized.
2247
2248 close()
2249
2250 copy_object(container, obj, destination=None, headers=None,
2251 fresh_metadata=None, response_dict=None)
2252 Wrapper for copy_object()
2253
2254 delete_container(container, response_dict=None,
2255 query_string=None, headers={})
2256 Wrapper for delete_container()
2257
2258 delete_object(container, obj, query_string=None,
2259 response_dict=None, headers=None)
2260 Wrapper for delete_object()
2261
2262 get_account(marker=None, limit=None, prefix=None,
2263 end_marker=None, full_listing=False, headers=None, delim‐
2264 iter=None)
2265 Wrapper for get_account()
2266
2267 get_auth()
2268
2269 get_capabilities(url=None)
2270
2271 get_container(container, marker=None, limit=None, prefix=None,
2272 delimiter=None, end_marker=None, path=None, full_listing=False,
2273 headers=None, query_string=None)
2274 Wrapper for get_container()
2275
2276 get_object(container, obj, resp_chunk_size=None,
2277 query_string=None, response_dict=None, headers=None)
2278 Wrapper for get_object()
2279
2280 get_service_auth()
2281
2282 head_account(headers=None)
2283 Wrapper for head_account()
2284
2285 head_container(container, headers=None)
2286 Wrapper for head_container()
2287
2288 head_object(container, obj, headers=None, query_string=None)
2289 Wrapper for head_object()
2290
2291 http_connection(url=None)
2292
2293 post_account(headers, response_dict=None, query_string=None,
2294 data=None)
2295 Wrapper for post_account()
2296
2297 post_container(container, headers, response_dict=None)
2298 Wrapper for post_container()
2299
2300 post_object(container, obj, headers, response_dict=None)
2301 Wrapper for post_object()
2302
2303 put_container(container, headers=None, response_dict=None,
2304 query_string=None)
2305 Wrapper for put_container()
2306
2307 put_object(container, obj, contents, content_length=None,
2308 etag=None, chunk_size=None, content_type=None, headers=None,
2309 query_string=None, response_dict=None)
2310 Wrapper for put_object()
2311
2312 class swiftclient.client.HTTPConnection(url, proxy=None, cacert=None,
2313 insecure=False, cert=None, cert_key=None, ssl_compression=False,
2314 default_user_agent=None, timeout=None)
2315 Bases: object
2316
2317 Make an HTTPConnection or HTTPSConnection
2318
2319 Parameters
2320
2321 · url -- url to connect to
2322
2323 · proxy -- proxy to connect through, if any; None by
2324 default; str of the format 'http://127.0.0.1:8888' to
2325 set one
2326
2327 · cacert -- A CA bundle file to use in verifying a TLS
2328 server certificate.
2329
2330 · insecure -- Allow to access servers without checking
2331 SSL certs. The server's certificate will not be veri‐
2332 fied.
2333
2334 · cert -- Client certificate file to connect on SSL
2335 server requiring SSL client certificate.
2336
2337 · cert_key -- Client certificate private key file.
2338
2339 · ssl_compression -- SSL compression should be disabled
2340 by default and this setting is not usable as of now.
2341 The parameter is kept for backward compatibility.
2342
2343 · default_user_agent -- Set the User-Agent header on
2344 every request. If set to None (default), the user
2345 agent will be "python-swiftclient-<version>". This may
2346 be overridden on a per-request basis by explicitly set‐
2347 ting the user-agent header on a call to request().
2348
2349 · timeout -- socket read timeout value, passed directly
2350 to the requests library.
2351
2352 Raises ClientException -- Unable to handle protocol scheme
2353
2354 close()
2355
2356 getresponse()
2357 Adapt requests response to httplib interface
2358
2359 putrequest(full_path, data=None, headers=None, files=None)
2360 Use python-requests files upload
2361
2362 Parameters
2363
2364 · data -- Use data generator for chunked-transfer
2365
2366 · files -- Use files for default transfer
2367
2368 request(method, full_path, data=None, headers=None, files=None)
2369 Encode url and header, then call requests.request
2370
2371 swiftclient.client.LOGGER_SENSITIVE_HEADERS = ['x-auth-token',
2372 'x-auth-key', 'x-service-token', 'x-storage-token',
2373 'x-account-meta-temp-url-key', 'x-account-meta-temp-url-key-2', 'x-con‐
2374 tainer-meta-temp-url-key', 'x-container-meta-temp-url-key-2',
2375 'set-cookie']
2376 A list of sensitive headers to redact in logs. Note that when
2377 extending this list, the header names must be added in all lower
2378 case.
2379
2380 swiftclient.client.copy_object(url, token, container, name, destina‐
2381 tion=None, headers=None, fresh_metadata=None, http_conn=None,
2382 response_dict=None, service_token=None)
2383 Copy object
2384
2385 Parameters
2386
2387 · url -- storage URL
2388
2389 · token -- auth token; if None, no token will be sent
2390
2391 · container -- container name that the source object is
2392 in
2393
2394 · name -- source object name
2395
2396 · destination -- The container and object name of the
2397 destination object in the form of /container/object; if
2398 None, the copy will use the source as the destination.
2399
2400 · headers -- additional headers to include in the request
2401
2402 · fresh_metadata -- Enables object creation that omits
2403 existing user metadata, default None
2404
2405 · http_conn -- HTTP connection object (If None, it will
2406 create the conn object)
2407
2408 · response_dict -- an optional dictionary into which to
2409 place the response - status, reason and headers
2410
2411 · service_token -- service auth token
2412
2413 Raises ClientException -- HTTP COPY request failed
2414
2415 swiftclient.client.delete_container(url, token, container,
2416 http_conn=None, response_dict=None, service_token=None,
2417 query_string=None, headers=None)
2418 Delete a container
2419
2420 Parameters
2421
2422 · url -- storage URL
2423
2424 · token -- auth token
2425
2426 · container -- container name to delete
2427
2428 · http_conn -- a tuple of (parsed url, HTTPConnection
2429 object), (If None, it will create the conn object)
2430
2431 · response_dict -- an optional dictionary into which to
2432 place the response - status, reason and headers
2433
2434 · service_token -- service auth token
2435
2436 · query_string -- if set will be appended with '?' to
2437 generated path
2438
2439 · headers -- additional headers to include in the request
2440
2441 Raises ClientException -- HTTP DELETE request failed
2442
2443 swiftclient.client.delete_object(url, token=None, container=None,
2444 name=None, http_conn=None, headers=None, proxy=None, query_string=None,
2445 response_dict=None, service_token=None)
2446 Delete object
2447
2448 Parameters
2449
2450 · url -- storage URL
2451
2452 · token -- auth token; if None, no token will be sent
2453
2454 · container -- container name that the object is in; if
2455 None, the container name is expected to be part of the
2456 url
2457
2458 · name -- object name to delete; if None, the object name
2459 is expected to be part of the url
2460
2461 · http_conn -- a tuple of (parsed url, HTTPConnection
2462 object), (If None, it will create the conn object)
2463
2464 · headers -- additional headers to include in the request
2465
2466 · proxy -- proxy to connect through, if any; None by
2467 default; str of the format 'http://127.0.0.1:8888' to
2468 set one
2469
2470 · query_string -- if set will be appended with '?' to
2471 generated path
2472
2473 · response_dict -- an optional dictionary into which to
2474 place the response - status, reason and headers
2475
2476 · service_token -- service auth token
2477
2478 Raises ClientException -- HTTP DELETE request failed
2479
2480 swiftclient.client.encode_meta_headers(headers)
2481 Only encode metadata headers keys
2482
2483 swiftclient.client.encode_utf8(value)
2484
2485 swiftclient.client.get_account(url, token, marker=None, limit=None,
2486 prefix=None, end_marker=None, http_conn=None, full_listing=False, ser‐
2487 vice_token=None, headers=None, delimiter=None)
2488 Get a listing of containers for the account.
2489
2490 Parameters
2491
2492 · url -- storage URL
2493
2494 · token -- auth token
2495
2496 · marker -- marker query
2497
2498 · limit -- limit query
2499
2500 · prefix -- prefix query
2501
2502 · end_marker -- end_marker query
2503
2504 · http_conn -- a tuple of (parsed url, HTTPConnection
2505 object), (If None, it will create the conn object)
2506
2507 · full_listing -- if True, return a full listing, else
2508 returns a max of 10000 listings
2509
2510 · service_token -- service auth token
2511
2512 · headers -- additional headers to include in the request
2513
2514 · delimiter -- delimiter query
2515
2516 Returns
2517 a tuple of (response headers, a list of containers) The
2518 response headers will be a dict and all header names will
2519 be lowercase.
2520
2521 Raises ClientException -- HTTP GET request failed
2522
2523 swiftclient.client.get_auth(auth_url, user, key, **kwargs)
2524 Get authentication/authorization credentials.
2525
2526 Parameters
2527
2528 · auth_version -- the api version of the supplied auth
2529 params
2530
2531 · os_options -- a dict, the openstack identity service
2532 options
2533
2534 Returns
2535 a tuple, (storage_url, token)
2536
2537 N.B. if the optional os_options parameter includes a non-empty
2538 'object_storage_url' key it will override the default storage
2539 url returned by the auth service.
2540
2541 The snet parameter is used for Rackspace's ServiceNet internal
2542 network implementation. In this function, it simply adds snet-
2543 to the beginning of the host name for the returned storage URL.
2544 With Rackspace Cloud Files, use of this network path causes no
2545 bandwidth charges but requires the client to be running on
2546 Rackspace's ServiceNet network.
2547
2548 swiftclient.client.get_auth_1_0(url, user, key, snet, **kwargs)
2549
2550 swiftclient.client.get_auth_keystone(auth_url, user, key, os_options,
2551 **kwargs)
2552 Authenticate against a keystone server.
2553
2554 We are using the keystoneclient library for authentication.
2555
2556 swiftclient.client.get_capabilities(http_conn)
2557 Get cluster capability infos.
2558
2559 Parameters
2560 http_conn -- a tuple of (parsed url, HTTPConnection
2561 object)
2562
2563 Returns
2564 a dict containing the cluster capabilities
2565
2566 Raises ClientException -- HTTP Capabilities GET failed
2567
2568 swiftclient.client.get_container(url, token, container, marker=None,
2569 limit=None, prefix=None, delimiter=None, end_marker=None, path=None,
2570 http_conn=None, full_listing=False, service_token=None, headers=None,
2571 query_string=None)
2572 Get a listing of objects for the container.
2573
2574 Parameters
2575
2576 · url -- storage URL
2577
2578 · token -- auth token
2579
2580 · container -- container name to get a listing for
2581
2582 · marker -- marker query
2583
2584 · limit -- limit query
2585
2586 · prefix -- prefix query
2587
2588 · delimiter -- string to delimit the queries on
2589
2590 · end_marker -- marker query
2591
2592 · path -- path query (equivalent: "delimiter=/" and "pre‐
2593 fix=path/")
2594
2595 · http_conn -- a tuple of (parsed url, HTTPConnection
2596 object), (If None, it will create the conn object)
2597
2598 · full_listing -- if True, return a full listing, else
2599 returns a max of 10000 listings
2600
2601 · service_token -- service auth token
2602
2603 · headers -- additional headers to include in the request
2604
2605 · query_string -- if set will be appended with '?' to
2606 generated path
2607
2608 Returns
2609 a tuple of (response headers, a list of objects) The
2610 response headers will be a dict and all header names will
2611 be lowercase.
2612
2613 Raises ClientException -- HTTP GET request failed
2614
2615 swiftclient.client.get_keystoneclient_2_0(auth_url, user, key,
2616 os_options, **kwargs)
2617
2618 swiftclient.client.get_object(url, token, container, name,
2619 http_conn=None, resp_chunk_size=None, query_string=None,
2620 response_dict=None, headers=None, service_token=None)
2621 Get an object
2622
2623 Parameters
2624
2625 · url -- storage URL
2626
2627 · token -- auth token
2628
2629 · container -- container name that the object is in
2630
2631 · name -- object name to get
2632
2633 · http_conn -- a tuple of (parsed url, HTTPConnection
2634 object), (If None, it will create the conn object)
2635
2636 · resp_chunk_size -- if defined, chunk size of data to
2637 read. NOTE: If you specify a resp_chunk_size you must
2638 fully read the object's contents before making another
2639 request.
2640
2641 · query_string -- if set will be appended with '?' to
2642 generated path
2643
2644 · response_dict -- an optional dictionary into which to
2645 place the response - status, reason and headers
2646
2647 · headers -- an optional dictionary with additional head‐
2648 ers to include in the request
2649
2650 · service_token -- service auth token
2651
2652 Returns
2653 a tuple of (response headers, the object's contents) The
2654 response headers will be a dict and all header names will
2655 be lowercase.
2656
2657 Raises ClientException -- HTTP GET request failed
2658
2659 swiftclient.client.head_account(url, token, http_conn=None, head‐
2660 ers=None, service_token=None)
2661 Get account stats.
2662
2663 Parameters
2664
2665 · url -- storage URL
2666
2667 · token -- auth token
2668
2669 · http_conn -- a tuple of (parsed url, HTTPConnection
2670 object), (If None, it will create the conn object)
2671
2672 · headers -- additional headers to include in the request
2673
2674 · service_token -- service auth token
2675
2676 Returns
2677 a dict containing the response's headers (all header
2678 names will be lowercase)
2679
2680 Raises ClientException -- HTTP HEAD request failed
2681
2682 swiftclient.client.head_container(url, token, container,
2683 http_conn=None, headers=None, service_token=None)
2684 Get container stats.
2685
2686 Parameters
2687
2688 · url -- storage URL
2689
2690 · token -- auth token
2691
2692 · container -- container name to get stats for
2693
2694 · http_conn -- a tuple of (parsed url, HTTPConnection
2695 object), (If None, it will create the conn object)
2696
2697 · headers -- additional headers to include in the request
2698
2699 · service_token -- service auth token
2700
2701 Returns
2702 a dict containing the response's headers (all header
2703 names will be lowercase)
2704
2705 Raises ClientException -- HTTP HEAD request failed
2706
2707 swiftclient.client.head_object(url, token, container, name,
2708 http_conn=None, service_token=None, headers=None, query_string=None)
2709 Get object info
2710
2711 Parameters
2712
2713 · url -- storage URL
2714
2715 · token -- auth token
2716
2717 · container -- container name that the object is in
2718
2719 · name -- object name to get info for
2720
2721 · http_conn -- a tuple of (parsed url, HTTPConnection
2722 object), (If None, it will create the conn object)
2723
2724 · service_token -- service auth token
2725
2726 · headers -- additional headers to include in the request
2727
2728 Returns
2729 a dict containing the response's headers (all header
2730 names will be lowercase)
2731
2732 Raises ClientException -- HTTP HEAD request failed
2733
2734 swiftclient.client.http_connection(*arg, **kwarg)
2735
2736 Returns
2737 tuple of (parsed url, connection object)
2738
2739 swiftclient.client.http_log(args, kwargs, resp, body)
2740
2741 swiftclient.client.logger_settings = {'redact_sensitive_headers': True,
2742 'reveal_sensitive_prefix': 16}
2743 Default behaviour is to redact header values known to contain
2744 secrets, such as X-Auth-Key and X-Auth-Token. Up to the first 16
2745 chars may be revealed.
2746
2747 To disable, set the value of redact_sensitive_headers to False.
2748
2749 When header redaction is enabled, reveal_sensitive_prefix con‐
2750 figures the maximum length of any sensitive header data sent to
2751 the logs. If the header is less than twice this length, only
2752 int(len(value)/2) chars will be logged; if it is less than 15
2753 chars long, even less will be logged.
2754
2755 swiftclient.client.parse_header_string(data)
2756
2757 swiftclient.client.post_account(url, token, headers, http_conn=None,
2758 response_dict=None, service_token=None, query_string=None, data=None)
2759 Update an account's metadata.
2760
2761 Parameters
2762
2763 · url -- storage URL
2764
2765 · token -- auth token
2766
2767 · headers -- additional headers to include in the request
2768
2769 · http_conn -- a tuple of (parsed url, HTTPConnection
2770 object), (If None, it will create the conn object)
2771
2772 · response_dict -- an optional dictionary into which to
2773 place the response - status, reason and headers
2774
2775 · service_token -- service auth token
2776
2777 · query_string -- if set will be appended with '?' to
2778 generated path
2779
2780 · data -- an optional message body for the request
2781
2782 Raises ClientException -- HTTP POST request failed
2783
2784 Returns
2785 resp_headers, body
2786
2787 swiftclient.client.post_container(url, token, container, headers,
2788 http_conn=None, response_dict=None, service_token=None)
2789 Update a container's metadata.
2790
2791 Parameters
2792
2793 · url -- storage URL
2794
2795 · token -- auth token
2796
2797 · container -- container name to update
2798
2799 · headers -- additional headers to include in the request
2800
2801 · http_conn -- a tuple of (parsed url, HTTPConnection
2802 object), (If None, it will create the conn object)
2803
2804 · response_dict -- an optional dictionary into which to
2805 place the response - status, reason and headers
2806
2807 · service_token -- service auth token
2808
2809 Raises ClientException -- HTTP POST request failed
2810
2811 swiftclient.client.post_object(url, token, container, name, headers,
2812 http_conn=None, response_dict=None, service_token=None)
2813 Update object metadata
2814
2815 Parameters
2816
2817 · url -- storage URL
2818
2819 · token -- auth token
2820
2821 · container -- container name that the object is in
2822
2823 · name -- name of the object to update
2824
2825 · headers -- additional headers to include in the request
2826
2827 · http_conn -- a tuple of (parsed url, HTTPConnection
2828 object), (If None, it will create the conn object)
2829
2830 · response_dict -- an optional dictionary into which to
2831 place the response - status, reason and headers
2832
2833 · service_token -- service auth token
2834
2835 Raises ClientException -- HTTP POST request failed
2836
2837 swiftclient.client.prepare_unicode_headers(self, headers)
2838
2839 swiftclient.client.put_container(url, token, container, headers=None,
2840 http_conn=None, response_dict=None, service_token=None,
2841 query_string=None)
2842 Create a container
2843
2844 Parameters
2845
2846 · url -- storage URL
2847
2848 · token -- auth token
2849
2850 · container -- container name to create
2851
2852 · headers -- additional headers to include in the request
2853
2854 · http_conn -- a tuple of (parsed url, HTTPConnection
2855 object), (If None, it will create the conn object)
2856
2857 · response_dict -- an optional dictionary into which to
2858 place the response - status, reason and headers
2859
2860 · service_token -- service auth token
2861
2862 · query_string -- if set will be appended with '?' to
2863 generated path
2864
2865 Raises ClientException -- HTTP PUT request failed
2866
2867 swiftclient.client.put_object(url, token=None, container=None,
2868 name=None, contents=None, content_length=None, etag=None,
2869 chunk_size=None, content_type=None, headers=None, http_conn=None,
2870 proxy=None, query_string=None, response_dict=None, service_token=None)
2871 Put an object
2872
2873 Parameters
2874
2875 · url -- storage URL
2876
2877 · token -- auth token; if None, no token will be sent
2878
2879 · container -- container name that the object is in; if
2880 None, the container name is expected to be part of the
2881 url
2882
2883 · name -- object name to put; if None, the object name is
2884 expected to be part of the url
2885
2886 · contents -- a string, a file-like object or an iterable
2887 to read object data from; if None, a zero-byte put will
2888 be done
2889
2890 · content_length -- value to send as content-length
2891 header; also limits the amount read from contents; if
2892 None, it will be computed via the contents or chunked
2893 transfer encoding will be used
2894
2895 · etag -- etag of contents; if None, no etag will be sent
2896
2897 · chunk_size -- chunk size of data to write; it defaults
2898 to 65536; used only if the contents object has a 'read'
2899 method, e.g. file-like objects, ignored otherwise
2900
2901 · content_type -- value to send as content-type header,
2902 overriding any value included in the headers param; if
2903 None and no value is found in the headers param, an
2904 empty string value will be sent
2905
2906 · headers -- additional headers to include in the
2907 request, if any
2908
2909 · http_conn -- a tuple of (parsed url, HTTPConnection
2910 object), (If None, it will create the conn object)
2911
2912 · proxy -- proxy to connect through, if any; None by
2913 default; str of the format 'http://127.0.0.1:8888' to
2914 set one
2915
2916 · query_string -- if set will be appended with '?' to
2917 generated path
2918
2919 · response_dict -- an optional dictionary into which to
2920 place the response - status, reason and headers
2921
2922 · service_token -- service auth token
2923
2924 Returns
2925 etag
2926
2927 Raises ClientException -- HTTP PUT request failed
2928
2929 swiftclient.client.quote(value, safe='/')
2930 Patched version of urllib.quote that encodes utf8 strings before
2931 quoting. On Python 3, call directly urllib.parse.quote().
2932
2933 swiftclient.client.resp_header_dict(resp)
2934
2935 swiftclient.client.safe_value(name, value)
2936 Only show up to logger_settings['reveal_sensitive_prefix'] char‐
2937 acters from a sensitive header.
2938
2939 Parameters
2940
2941 · name -- Header name
2942
2943 · value -- Header value
2944
2945 Returns
2946 Safe header value
2947
2948 swiftclient.client.scrub_headers(headers)
2949 Redact header values that can contain sensitive information that
2950 should not be logged.
2951
2952 Parameters
2953 headers -- Either a dict or an iterable of two-element
2954 tuples
2955
2956 Returns
2957 Safe dictionary of headers with sensitive information
2958 removed
2959
2960 swiftclient.client.store_response(resp, response_dict)
2961 store information about an operation into a dict
2962
2963 Parameters
2964
2965 · resp -- an http response object containing the response
2966 headers
2967
2968 · response_dict -- a dict into which are placed the sta‐
2969 tus, reason and a dict of lower-cased headers
2970
2971 swiftclient.service
2972 class swiftclient.service.ResultsIterator(futures)
2973 Bases: object
2974
2975 class swiftclient.service.SwiftCopyObject(object_name, options=None)
2976 Bases: object
2977
2978 Class for specifying an object copy, allowing the destina‐
2979 tion/headers/metadata/fresh_metadata to be specified separately
2980 for each individual object. destination and fresh_metadata
2981 should be set in options
2982
2983 exception swiftclient.service.SwiftError(value, container=None,
2984 obj=None, segment=None, exc=None)
2985 Bases: Exception
2986
2987 class swiftclient.service.SwiftPostObject(object_name, options=None)
2988 Bases: object
2989
2990 Class for specifying an object post, allowing the headers/meta‐
2991 data to be specified separately for each individual object.
2992
2993 class swiftclient.service.SwiftService(options=None)
2994 Bases: object
2995
2996 Service for performing swift operations
2997
2998 capabilities(url=None, refresh_cache=False)
2999 List the cluster capabilities.
3000
3001 Parameters
3002 url -- Proxy URL of the cluster to retrieve capa‐
3003 bilities.
3004
3005 Returns
3006 A dictionary containing the capabilities of the
3007 cluster.
3008
3009 Raises ClientException --
3010
3011 copy(container, objects, options=None)
3012 Copy operations on a list of objects in a container. Des‐
3013 tination containers will be created.
3014
3015 Parameters
3016
3017 · container -- The container from which to copy
3018 the objects.
3019
3020 · objects --
3021
3022 A list of object names (strings) or SwiftCopyOb‐
3023 ject instances containing an object name and an
3024 options dict (can be None) to override the
3025 options for that individual copy operation:
3026
3027 [
3028 'object_name',
3029 SwiftCopyObject(
3030 'object_name',
3031 options={
3032 'destination': '/container/object',
3033 'fresh_metadata': False,
3034 ...
3035 }),
3036 ...
3037 ]
3038
3039 The options dict is described below.
3040
3041
3042 · options --
3043
3044 A dictionary containing options to override the
3045 global options specified during the service
3046 object creation. These options are applied to
3047 all copy operations performed by this call,
3048 unless overridden on a per object basis. The
3049 options "destination" and "fresh_metadata" do
3050 not need to be set, in this case objects will be
3051 copied onto themselves and metadata will not be
3052 refreshed. The option "destination" can also be
3053 specified in the format '/container', in which
3054 case objects without an explicit destination
3055 will be copied to the destination /con‐
3056 tainer/original_object_name. Combinations of
3057 multiple objects and a destination in the format
3058 '/container/object' is invalid. Possible options
3059 are given below:
3060
3061 {
3062 'meta': [],
3063 'header': [],
3064 'destination': '/container/object',
3065 'fresh_metadata': False,
3066 }
3067
3068
3069 Returns
3070 A generator returning the results of copying the
3071 given list of objects.
3072
3073 Raises SwiftError --
3074
3075 delete(container=None, objects=None, options=None)
3076 Delete operations on an account, optional container and
3077 optional list of objects.
3078
3079 Parameters
3080
3081 · container -- The container to delete or delete
3082 from.
3083
3084 · objects -- The list of objects to delete.
3085
3086 · options --
3087
3088 A dictionary containing options to override the
3089 global options specified during the service
3090 object creation:
3091
3092 {
3093 'yes_all': False,
3094 'leave_segments': False,
3095 'prefix': None,
3096 'header': [],
3097 }
3098
3099
3100 Returns
3101 A generator for returning the results of the
3102 delete operations. Each result yielded from the
3103 generator is either a 'delete_container',
3104 'delete_object', 'delete_segment', or
3105 'bulk_delete' dictionary containing the results of
3106 an individual delete operation.
3107
3108 Raises
3109
3110 · ClientException --
3111
3112 · SwiftError --
3113
3114 download(container=None, objects=None, options=None)
3115 Download operations on an account, optional container and
3116 optional list of objects.
3117
3118 Parameters
3119
3120 · container -- The container to download from.
3121
3122 · objects -- A list of object names to download (a
3123 list of strings).
3124
3125 · options --
3126
3127 A dictionary containing options to override the
3128 global options specified during the service
3129 object creation:
3130
3131 {
3132 'yes_all': False,
3133 'marker': '',
3134 'prefix': None,
3135 'no_download': False,
3136 'header': [],
3137 'skip_identical': False,
3138 'out_directory': None,
3139 'checksum': True,
3140 'out_file': None,
3141 'remove_prefix': False,
3142 'shuffle' : False
3143 }
3144
3145
3146 Returns
3147 A generator for returning the results of the down‐
3148 load operations. Each result yielded from the gen‐
3149 erator is a 'download_object' dictionary contain‐
3150 ing the results of an individual file download.
3151
3152 Raises
3153
3154 · ClientException --
3155
3156 · SwiftError --
3157
3158 list(container=None, options=None)
3159 List operations on an account, container.
3160
3161 Parameters
3162
3163 · container -- The container to make the list
3164 operation against.
3165
3166 · options --
3167
3168 A dictionary containing options to override the
3169 global options specified during the service
3170 object creation:
3171
3172 {
3173 'long': False,
3174 'prefix': None,
3175 'delimiter': None,
3176 'header': []
3177 }
3178
3179
3180 Returns
3181 A generator for returning the results of the list
3182 operation on an account or container. Each result
3183 yielded from the generator is either a
3184 'list_account_part' or 'list_container_part', con‐
3185 taining part of the listing.
3186
3187 post(container=None, objects=None, options=None)
3188 Post operations on an account, container or list of
3189 objects
3190
3191 Parameters
3192
3193 · container -- The container to make the post
3194 operation against.
3195
3196 · objects --
3197
3198 A list of object names (strings) or SwiftPostOb‐
3199 ject instances containing an object name, and an
3200 options dict (can be None) to override the
3201 options for that individual post operation:
3202
3203 [
3204 'object_name',
3205 SwiftPostObject('object_name', options={...}),
3206 ...
3207 ]
3208
3209 The options dict is described below.
3210
3211
3212 · options --
3213
3214 A dictionary containing options to override the
3215 global options specified during the service
3216 object creation. These options are applied to
3217 all post operations performed by this call,
3218 unless overridden on a per object basis. Possi‐
3219 ble options are given below:
3220
3221 {
3222 'meta': [],
3223 'header': [],
3224 'read_acl': None, # For containers only
3225 'write_acl': None, # For containers only
3226 'sync_to': None, # For containers only
3227 'sync_key': None # For containers only
3228 }
3229
3230
3231 Returns
3232 Either a single result dictionary in the case of a
3233 post to a container/account, or an iterator for
3234 returning the results of posts to a list of
3235 objects.
3236
3237 Raises SwiftError --
3238
3239 stat(container=None, objects=None, options=None)
3240 Get account stats, container stats or information about a
3241 list of objects in a container.
3242
3243 Parameters
3244
3245 · container -- The container to query.
3246
3247 · objects -- A list of object paths about which to
3248 return information (a list of strings).
3249
3250 · options --
3251
3252 A dictionary containing options to override the
3253 global options specified during the service
3254 object creation. These options are applied to
3255 all stat operations performed by this call:
3256
3257 {
3258 'human': False,
3259 'header': []
3260 }
3261
3262
3263 Returns
3264 Either a single dictionary containing stats about
3265 an account or container, or an iterator for
3266 returning the results of the stat operations on a
3267 list of objects.
3268
3269 Raises SwiftError --
3270
3271 upload(container, objects, options=None)
3272 Upload a list of objects to a given container.
3273
3274 Parameters
3275
3276 · container -- The container (or pseudo-folder
3277 path) to put the uploads into.
3278
3279 · objects --
3280
3281 A list of file/directory names (strings) or
3282 SwiftUploadObject instances containing a source
3283 for the created object, an object name, and an
3284 options dict (can be None) to override the
3285 options for that individual upload operation:
3286
3287 [
3288 '/path/to/file',
3289 SwiftUploadObject('/path', object_name='obj1'),
3290 ...
3291 ]
3292
3293 The options dict is as described below.
3294
3295 The SwiftUploadObject source may be one of:
3296
3297 · A file-like object (with a read method)
3298
3299 · A string containing the path to a local
3300 file or directory
3301
3302 · None, to indicate that we want an empty
3303 object
3304
3305
3306 · options --
3307
3308 A dictionary containing options to override the
3309 global options specified during the service
3310 object creation. These options are applied to
3311 all upload operations performed by this call,
3312 unless overridden on a per object basis. Possi‐
3313 ble options are given below:
3314
3315 {
3316 'meta': [],
3317 'header': [],
3318 'segment_size': None,
3319 'use_slo': False,
3320 'segment_container': None,
3321 'leave_segments': False,
3322 'changed': None,
3323 'skip_identical': False,
3324 'fail_fast': False,
3325 'dir_marker': False # Only for None sources
3326 }
3327
3328
3329 Returns
3330 A generator for returning the results of the
3331 uploads.
3332
3333 Raises
3334
3335 · SwiftError --
3336
3337 · ClientException --
3338
3339 class swiftclient.service.SwiftUploadObject(source, object_name=None,
3340 options=None)
3341 Bases: object
3342
3343 Class for specifying an object upload, allowing the object
3344 source, name and options to be specified separately for each
3345 individual object.
3346
3347 swiftclient.service.get_conn(options)
3348 Return a connection building it from the options.
3349
3350 swiftclient.service.get_from_queue(q, timeout=864000)
3351
3352 swiftclient.service.get_future_result(f, timeout=86400)
3353
3354 swiftclient.service.interruptable_as_completed(fs, timeout=86400)
3355
3356 swiftclient.service.mkdirs(path)
3357
3358 swiftclient.service.process_options(options)
3359
3360 swiftclient.service.split_headers(options, prefix='')
3361 Splits 'Key: Value' strings and returns them as a dictionary.
3362
3363 Parameters
3364
3365 · options -- Must be one of: * an iterable of 'Key:
3366 Value' strings * an iterable of ('Key', 'Value') pairs
3367 * a dict of {'Key': 'Value'} pairs
3368
3369 · prefix -- String to prepend to all of the keys in the
3370 dictionary. reporting.
3371
3372 swiftclient.exceptions
3373 exception swiftclient.exceptions.ClientException(msg, http_scheme='',
3374 http_host='', http_port='', http_path='', http_query='', http_sta‐
3375 tus=None, http_reason='', http_device='', http_response_content='',
3376 http_response_headers=None)
3377 Bases: Exception
3378
3379 classmethod from_response(resp, msg=None, body=None)
3380
3381 swiftclient.multithreading
3382 class swiftclient.multithreading.ConnectionThreadPoolExecutor(cre‐
3383 ate_connection, max_workers)
3384 Bases: concurrent.futures.thread.ThreadPoolExecutor
3385
3386 A wrapper class to maintain a pool of connections alongside the
3387 thread pool. We start by creating a priority queue of connec‐
3388 tions, and each job submitted takes one of those connections
3389 (initialising if necessary) and passes it as the first arg to
3390 the executed function.
3391
3392 At the end of execution that connection is returned to the
3393 queue.
3394
3395 By using a PriorityQueue we avoid creating more connections than
3396 required. We will only create as many connections as are
3397 required concurrently.
3398
3399 Initializes a new ThreadPoolExecutor instance.
3400
3401 Parameters
3402
3403 · create_connection -- callable to use to create new con‐
3404 nections
3405
3406 · max_workers -- the maximum number of threads that can
3407 be used
3408
3409 submit(fn, *args, **kwargs)
3410 Schedules the callable, fn, to be executed
3411
3412 Parameters
3413
3414 · fn -- the callable to be invoked
3415
3416 · args -- the positional arguments for the
3417 callable
3418
3419 · kwargs -- the keyword arguments for the callable
3420
3421 Returns
3422 a Future object representing the execution of the
3423 callable
3424
3425 class swiftclient.multithreading.MultiThreadingManager(create_connec‐
3426 tion, segment_threads=10, object_dd_threads=10, object_uu_threads=10,
3427 container_threads=10)
3428 Bases: object
3429
3430 One object to manage context for multi-threading. This should
3431 make bin/swift less error-prone and allow us to test this code.
3432
3433 Parameters
3434
3435 · segment_threads -- The number of threads allocated to
3436 segment uploads
3437
3438 · object_dd_threads -- The number of threads allocated to
3439 object download/delete jobs
3440
3441 · object_uu_threads -- The number of threads allocated to
3442 object upload/update based jobs
3443
3444 · container_threads -- The number of threads allocated to
3445 container/account level jobs
3446
3447 class swiftclient.multithreading.OutputManager(print_stream=None,
3448 error_stream=None)
3449 Bases: object
3450
3451 One object to manage and provide helper functions for output.
3452
3453 This object is a context manager and returns itself into the
3454 context. When entering the context, two printing threads are
3455 created (see below) and they are waited on and cleaned up when
3456 exiting the context.
3457
3458 Also, thread-safe printing to two streams is provided. The
3459 print_msg() method will print to the supplied print_stream
3460 (defaults to sys.stdout) and the error() method will print to
3461 the supplied error_stream (defaults to sys.stderr). Both of
3462 these printing methods will format the given string with any
3463 supplied *args (a la printf). On Python 2, Unicode messages are
3464 encoded to utf8.
3465
3466 The attribute self.error_count is incremented once per error
3467 message printed, so an application can tell if any worker
3468 threads encountered exceptions or otherwise called error() on
3469 this instance. The swift command-line tool uses this to exit
3470 non-zero if any error strings were printed.
3471
3472 Parameters
3473
3474 · print_stream -- The stream to which print_msg() sends
3475 formatted messages.
3476
3477 · error_stream -- The stream to which error() sends for‐
3478 matted messages.
3479
3480 On Python 2, Unicode messages are encoded to utf8.
3481
3482 DEFAULT_OFFSET = 14
3483
3484 error(msg, *fmt_args)
3485
3486 get_error_count()
3487
3488 print_items(items, offset=14, skip_missing=False)
3489
3490 print_msg(msg, *fmt_args)
3491
3492 print_raw(data)
3493
3494 warning(msg, *fmt_args)
3495
3496 swiftclient.utils
3497 Miscellaneous utility functions for use with Swift.
3498
3499 class swiftclient.utils.JSONableIterable(iterable)
3500 Bases: list
3501
3502 class swiftclient.utils.LengthWrapper(readable, length, md5=False)
3503 Bases: object
3504
3505 Wrap a filelike object with a maximum length.
3506
3507 Fix for https://github.com/kennethreitz/requests/issues/1648.
3508 It is recommended to use this class only on files opened in
3509 binary mode.
3510
3511 Parameters
3512
3513 · readable -- The filelike object to read from.
3514
3515 · length -- The maximum amount of content that can be
3516 read from the filelike object before it is simulated to
3517 be empty.
3518
3519 · md5 -- Flag to enable calculating the MD5 of the con‐
3520 tent as it is read.
3521
3522 get_md5sum()
3523
3524 read(size=-1)
3525
3526 property reset
3527
3528 class swiftclient.utils.NoopMD5(*a, **kw)
3529 Bases: object
3530
3531 hexdigest(*a, **kw)
3532
3533 update(*a, **kw)
3534
3535 class swiftclient.utils.ReadableToIterable(content, chunk_size=65536,
3536 md5=False)
3537 Bases: object
3538
3539 Wrap a filelike object and act as an iterator.
3540
3541 It is recommended to use this class only on files opened in
3542 binary mode. Due to the Unicode changes in Python 3, files are
3543 now opened using an encoding not suitable for use with the md5
3544 class and because of this hit the exception on every call to
3545 next. This could cause problems, especially with large files and
3546 small chunk sizes.
3547
3548 Parameters
3549
3550 · content -- The filelike object that is yielded from.
3551
3552 · chunk_size -- The max size of each yielded item.
3553
3554 · md5 -- Flag to enable calculating the MD5 of the con‐
3555 tent as it is yielded.
3556
3557 get_md5sum()
3558
3559 next()
3560
3561 swiftclient.utils.config_true_value(value)
3562 Returns True if the value is either True or a string in
3563 TRUE_VALUES. Returns False otherwise. This function comes from
3564 swift.common.utils.config_true_value()
3565
3566 swiftclient.utils.generate_temp_url(path, seconds, key, method, abso‐
3567 lute=False, prefix=False, iso8601=False, ip_range=None)
3568 Generates a temporary URL that gives unauthenticated access to
3569 the Swift object.
3570
3571 Parameters
3572
3573 · path -- The full path to the Swift object or prefix if
3574 a prefix-based temporary URL should be generated. Exam‐
3575 ple: /v1/AUTH_account/c/o or /v1/AUTH_account/c/prefix.
3576
3577 · seconds -- time in seconds or ISO 8601 timestamp. If
3578 absolute is False and this is the string representation
3579 of an integer, then this specifies the amount of time
3580 in seconds for which the temporary URL will be valid.
3581 If absolute is True then this specifies an absolute
3582 time at which the temporary URL will expire.
3583
3584 · key -- The secret temporary URL key set on the Swift
3585 cluster. To set a key, run 'swift post -m
3586 "Temp-URL-Key: <substitute tempurl key here>"'
3587
3588 · method -- A HTTP method, typically either GET or PUT,
3589 to allow for this temporary URL.
3590
3591 · absolute -- if True then the seconds parameter is
3592 interpreted as a Unix timestamp, if seconds represents
3593 an integer.
3594
3595 · prefix -- if True then a prefix-based temporary URL
3596 will be generated.
3597
3598 · iso8601 -- if True, a URL containing an ISO 8601 UTC
3599 timestamp instead of a UNIX timestamp will be created.
3600
3601 · ip_range -- if a valid ip range, restricts the tempo‐
3602 rary URL to the range of ips.
3603
3604 Raises ValueError -- if timestamp or path is not in valid for‐
3605 mat.
3606
3607 Returns
3608 the path portion of a temporary URL
3609
3610 swiftclient.utils.get_body(headers, body)
3611
3612 swiftclient.utils.iter_wrapper(iterable)
3613
3614 swiftclient.utils.n_at_a_time(seq, n)
3615
3616 swiftclient.utils.n_groups(seq, n)
3617
3618 swiftclient.utils.normalize_manifest_path(path)
3619
3620 swiftclient.utils.parse_api_response(headers, body)
3621
3622 swiftclient.utils.prt_bytes(num_bytes, human_flag)
3623 convert a number > 1024 to printable format, either in 4 char -h
3624 format as with ls -lh or return as 12 char right justified
3625 string
3626
3627 swiftclient.utils.report_traceback()
3628 Reports a timestamp and full traceback for a given exception.
3629
3630 Returns
3631 Full traceback and timestamp.
3632
3633 swiftclient.utils.split_request_headers(options, prefix='')
3634
3636 · genindex
3637
3638 · modindex
3639
3640 · search
3641
3643 Copyright 2013 OpenStack, LLC.
3644
3645 Licensed under the Apache License, Version 2.0 (the “License”); you may
3646 not use this file except in compliance with the License. You may
3647 obtain a copy of the License at
3648
3649 · http://www.apache.org/licenses/LICENSE-2.0
3650
3651 Unless required by applicable law or agreed to in writing, software
3652 distributed under the License is distributed on an “AS IS” BASIS, WITH‐
3653 OUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
3654 See the License for the specific language governing permissions and
3655 limitations under the License.
3656
3658 unknown
3659
3661 2013-2020 OpenStack, LLC.
3662
3663
3664
3665
36663.8.1 Jan 30, 2020 PYTHON-SWIFTCLIENT(1)