1PYTHON-SWIFTCLIENT(1) python-swiftclient PYTHON-SWIFTCLIENT(1)
2
6 python-swiftclient - python-swiftclient 4.1.0
7
9 Introduction
10 Where to Start?
11 The python-swiftclient project comprises a command line tool and two
12 separate APIs for accessing swift programmatically. Choosing the most
13 appropriate method for a given use case is the first problem a user
14 needs to solve.
15 Use Cases
17 Alongside the command line tool, the python-swiftclient includes two
18 levels of API:
19 • A low level client API that provides simple Python wrappers around
21 the various authentication mechanisms and the individual HTTP re‐
22 quests.
23 • A high level service API that provides methods for performing common
25 operations in parallel on a thread pool.
26 Example use cases:
28 •
30 Uploading and retrieving data
32 Use the command line tool if you are simply uploading and
33 downloading files and directories to and from your filesystem.
34 The command line tool can be integrated into a shell script to
35 automate tasks.
36 •
38 Integrating into an automated Python workflow
40 Use the SwiftService API to perform operations offered by the
41 CLI if your use case requires integration with a Python-based
42 workflow. This method offers greater control and flexibility
43 over individual object operations, such as the metadata set on
44 each object. The SwiftService class provides methods to per‐
45 form multiple sets of operations against a swift object store
46 using a configurable shared thread pool. A single instance of
47 the SwiftService class can be shared between multiple threads
48 in your own code.
49 •
51 Developing an application in Python to access a swift object store
53 Use the SwiftService API to develop Python applications that
54 use swift to store and retrieve objects. A SwiftService in‐
55 stance provides a configurable thread pool for performing all
56 operations supported by the CLI.
57 •
59 Fine-grained control over threading or the requests being performed
61 Use the Connection API if your use case requires fine grained
62 control over advanced features or you wish to use your own ex‐
63 isting threading model. Examples of advanced features requir‐
64 ing the use of the Connection API include creating an SLO man‐
65 ifest that references already existing objects, or fine
66 grained control over the query strings supplied with each HTTP
67 request.
68 Important considerations
70 This section covers some important considerations, helpful hints, and
71 things to avoid when integrating an object store into your workflow.
72 An object store is not a filesystem
74 It cannot be stressed enough that your usage of the object store should
75 reflect the proper use case, and not treat the storage like a tradi‐
76 tional filesystem. There are two main restrictions to bear in mind
77 when designing an application that uses an object store:
78 • You cannot rename objects. Due to fact that the name of an object is
80 one of the factors that determines where the object and its replicas
81 are stored, renaming would require multiple copies of the data to be
82 moved between physical storage devices. If you want to rename an ob‐
83 ject you must upload to the new location, or make a server side copy
84 request to the new location, and then delete the original.
85 • You cannot modify objects. Objects are stored in multiple locations
87 and are checked for integrity based on the MD5 sum calculated during
88 upload. In order to modify the contents of an object, the entire de‐
89 sired contents must be re-uploaded. In certain special cases it is
90 possible to work around this restriction using large objects, but no
91 general file-like access is available to modify a stored object.
92 Objects cannot be locked
94 There is no mechanism to perform a combination of reading the
95 data/metadata from an object and writing an update to that data/meta‐
96 data in an atomic way. Any user with access to a container could update
97 the contents or metadata associated with an object at any time.
98 Workflows that assume that no updates have been made since the last
100 read of an object should be discouraged. Enabling a workflow of this
101 type requires an external object locking mechanism and/or cooperation
102 between all clients accessing the data.
103
105 So You Want to Contribute...
106 For general information on contributing to OpenStack, please check out
107 the contributor guide to get started. It covers all the basics that
108 are common to all OpenStack projects: the accounts you need, the basics
109 of interacting with our Gerrit review system, how we communicate as a
110 community, etc.
111 The python-swiftclient is maintained by the OpenStack Swift project.
113 To understand our development process and how you can contribute to it,
114 please look at the Swift project's general contributor's page:
115 http://docs.openstack.org/swift/latest/contributor/contributing.html
116 CLI
118 The swift tool is a command line utility for communicating with an
119 OpenStack Object Storage (swift) environment. It allows one to perform
120 several types of operations.
121 For help on a specific swift command, enter:
123 $ swift COMMAND --help
125 swift usage
127 Usage: swift [--version] [--help] [--os-help] [--snet] [--verbose]
128 [--debug] [--info] [--quiet] [--auth <auth_url>]
129 [--auth-version <auth_version> |
130 --os-identity-api-version <auth_version> ]
131 [--user <username>]
132 [--key <api_key>] [--retries <num_retries>]
133 [--os-username <auth-user-name>] [--os-password <auth-password>]
134 [--os-user-id <auth-user-id>]
135 [--os-user-domain-id <auth-user-domain-id>]
136 [--os-user-domain-name <auth-user-domain-name>]
137 [--os-tenant-id <auth-tenant-id>]
138 [--os-tenant-name <auth-tenant-name>]
139 [--os-project-id <auth-project-id>]
140 [--os-project-name <auth-project-name>]
141 [--os-project-domain-id <auth-project-domain-id>]
142 [--os-project-domain-name <auth-project-domain-name>]
143 [--os-auth-url <auth-url>] [--os-auth-token <auth-token>]
144 [--os-storage-url <storage-url>] [--os-region-name <region-name>]
145 [--os-service-type <service-type>]
146 [--os-endpoint-type <endpoint-type>]
147 [--os-cacert <ca-certificate>] [--insecure]
148 [--os-cert <client-certificate-file>]
149 [--os-key <client-certificate-key-file>]
150 [--no-ssl-compression]
151 <subcommand> [--help] [<subcommand options>]
152 Subcommands:
154 delete Delete a container or objects within a container.
156 download
158 Download objects from containers.
159 list Lists the containers for the account or the objects for a con‐
161 tainer.
162 post Updates meta information for the account, container, or object;
164 creates containers if not present.
165 copy Copies object, optionally adds meta
167 stat Displays information for the account, container, or object.
169 upload Uploads files or directories to the given container.
171 capabilities
173 List cluster capabilities.
174 tempurl
176 Create a temporary URL.
177 auth Display auth related environment variables.
179 swift optional arguments
181 --version
182 show program's version number and exit
183 -h, --help
185 show this help message and exit
186 --os-help
188 Show OpenStack authentication options.
189 -s, --snet
191 Use SERVICENET internal network.
192 -v, --verbose
194 Print more info.
195 --debug
197 Show the curl commands and results of all http queries regard‐
198 less of result status.
199 --info Show the curl commands and results of all http queries which re‐
201 turn an error.
202 -q, --quiet
204 Suppress status output.
205 -A AUTH, --auth=AUTH
207 URL for obtaining an auth token.
208 -V AUTH_VERSION, --auth-version=AUTH_VERSION, --os-identity-api-ver‐
210 sion=AUTH_VERSION
211 Specify a version for authentication. Defaults to
212 env[ST_AUTH_VERSION], env[OS_AUTH_VERSION], env[OS_IDEN‐
213 TITY_API_VERSION] or 1.0.
214 -U USER, --user=USER
216 User name for obtaining an auth token.
217 -K KEY, --key=KEY
219 Key for obtaining an auth token.
220 -R RETRIES, --retries=RETRIES
222 The number of times to retry a failed connection.
223 --insecure
225 Allow swiftclient to access servers without having to verify the
226 SSL certificate. Defaults to env[SWIFTCLIENT_INSECURE] (set to
227 'true' to enable).
228 --no-ssl-compression
230 This option is deprecated and not used anymore. SSL compression
231 should be disabled by default by the system SSL library.
232 --prompt
234 Prompt user to enter a password which overrides any password
235 supplied via --key, --os-password or environment variables.
236 Authentication
238 This section covers the options for authenticating with a swift object
239 store. The combinations of options required for each authentication
240 version are detailed below, but are just a subset of those that can be
241 used to successfully authenticate. These are the most common and recom‐
242 mended combinations.
243 You should obtain the details of your authentication version and cre‐
245 dentials from your storage provider. These details should make it
246 clearer which of the authentication sections below are most likely to
247 allow you to connect to your storage account.
248 Keystone v3
250 swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \
251 --os-project-name project1 --os-project-domain-name domain1 \
252 --os-username user --os-user-domain-name domain1 \
253 --os-password password list
254 swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \
256 --os-project-id 0123456789abcdef0123456789abcdef \
257 --os-user-id abcdef0123456789abcdef0123456789 \
258 --os-password password list
259 Manually specifying the options above on the command line can be
261 avoided by setting the following combinations of environment variables:
262 ST_AUTH_VERSION=3
264 OS_USERNAME=user
265 OS_USER_DOMAIN_NAME=domain1
266 OS_PASSWORD=password
267 OS_PROJECT_NAME=project1
268 OS_PROJECT_DOMAIN_NAME=domain1
269 OS_AUTH_URL=https://api.example.com:5000/v3
270 ST_AUTH_VERSION=3
272 OS_USER_ID=abcdef0123456789abcdef0123456789
273 OS_PASSWORD=password
274 OS_PROJECT_ID=0123456789abcdef0123456789abcdef
275 OS_AUTH_URL=https://api.example.com:5000/v3
276 Keystone v2
278 swift --os-auth-url https://api.example.com:5000/v2.0 \
279 --os-tenant-name tenant \
280 --os-username user --os-password password list
281 Manually specifying the options above on the command line can be
283 avoided by setting the following environment variables:
284 ST_AUTH_VERSION=2.0
286 OS_USERNAME=user
287 OS_PASSWORD=password
288 OS_TENANT_NAME=tenant
289 OS_AUTH_URL=https://api.example.com:5000/v2.0
290 Legacy auth systems
292 You can configure swift to work with any number of other authentication
293 systems that we will not cover in this document. If your storage
294 provider is not using Keystone to provide access tokens, please contact
295 them for instructions on the required options. It is likely that the
296 options will need to be specified as below:
297 swift -A https://api.example.com/v1.0 -U user -K api_key list
299 Specifying the options above manually on the command line can be
301 avoided by setting the following environment variables:
302 ST_AUTH_VERSION=1.0
304 ST_AUTH=https://api.example.com/v1.0
305 ST_USER=user
306 ST_KEY=key
307 It is also possible that you need to use a completely separate auth
309 system, in which case swiftclient cannot request a token for you. In
310 this case you should make the authentication request separately and ac‐
311 cess your storage using the token and storage URL options shown below:
312 swift --os-auth-token 6ee5eb33efad4e45ab46806eac010566 \
314 --os-storage-url https://10.1.5.2:8080/v1/AUTH_ced809b6a4baea7aeab61a \
315 list
316 NOTE:
318 Leftover environment variables are a common source of confusion when
319 authorization fails.
320 CLI commands
322 Auth
323 Usage: swift auth
324 Display authentication variables in shell friendly format. Command to
326 run to export storage URL and auth token into OS_STORAGE_URL and
327 OS_AUTH_TOKEN: swift auth. Command to append to a runcom file (e.g.
328 ~/.bashrc, /etc/profile) for automatic authentication: swift auth -v -U
329 test:tester -K testing.
330 swift stat
332 Usage: swift stat [--lh] [--header <header:value>]
333 [<container> [<object>]]
334 Displays information for the account, container, or object depending on
336 the arguments given (if any). In verbose mode, the storage URL and the
337 authentication token are displayed as well.
338 Positional arguments:
340 [container]
342 Name of container to stat from.
343 [object]
345 Name of object to stat.
346 Optional arguments:
348 --lh Report sizes in human readable format similar to ls -lh.
350 -H, --header <header:value>
352 Adds a custom request header to use for stat.
353 swift list
355 Usage: swift list [--long] [--lh] [--totals] [--prefix <prefix>]
356 [--delimiter <delimiter>] [--header <header:value>]
357 [<container>]
358 Lists the containers for the account or the objects for a container.
360 The -p <prefix> or --prefix <prefix> is an option that will only list
361 items beginning with that prefix. The -d <delimiter> or --delimiter
362 <delimiter> is an option (for container listings only) that will roll
363 up items with the given delimiter (see OpenStack Swift general documen‐
364 tation <https://docs.openstack.org/swift/latest/> for what this means).
365 The -l and --lh options provide more detail, similar to ls -l and ls
367 -lh, the latter providing sizes in human readable format (For example:
368 3K, 12M, etc). The latter two switches use more overhead to retrieve
369 the displayed details, which is directly proportional to the number of
370 container or objects listed.
371 Positional arguments:
373 [container]
375 Name of container to list object in.
376 Optional arguments:
378 -l, --long
380 Long listing format, similar to ls -l.
381 --lh Report sizes in human readable format similar to ls -lh.
383 -t, --totals
385 Used with -l or --lh, only report totals.
386 -p <prefix>, --prefix <prefix>
388 Only list items beginning with the prefix.
389 -d <delim>, --delimiter <delim>
391 Roll up items with the given delimiter. For containers only. See
392 OpenStack Swift API documentation for what this means.
393 -H, --header <header:value>
395 Adds a custom request header to use for listing.
396 swift upload
398 Usage: swift upload [--changed] [--skip-identical] [--segment-size <size>]
399 [--segment-container <container>] [--leave-segments]
400 [--object-threads <thread>] [--segment-threads <threads>]
401 [--header <header>] [--use-slo] [--ignore-checksum]
402 [--object-name <object-name>]
403 <container> <file_or_directory> [<file_or_directory>] [...]
404 Uploads the files and directories specified by the remaining arguments
406 to the given container. The -c or --changed is an option that will only
407 upload files that have changed since the last upload. The --object-name
408 <object-name> is an option that will upload a file and name object to
409 <object-name> or upload a directory and use <object-name> as object
410 prefix. If the file name is "-", client reads content from standard in‐
411 put. In this case --object-name is required to set the name of the ob‐
412 ject and no other files may be given. The -S <size> or --segment-size
413 <size> and --leave-segments are options as well (see --help for more).
414 Positional arguments:
416 <container>
418 Name of container to upload to.
419 <file_or_directory>
421 Name of file or directory to upload. Specify multiple times for
422 multiple uploads.
423 Optional arguments:
425 -c, --changed
427 Only upload files that have changed since the last upload.
428 --skip-identical
430 Skip uploading files that are identical on both sides.
431 -S, --segment-size <size>
433 Upload files in segments no larger than <size> (in Bytes) and
434 then create a "manifest" file that will download all the seg‐
435 ments as if it were the original file.
436 --segment-container <container>
438 Upload the segments into the specified container. If not speci‐
439 fied, the segments will be uploaded to a <container>_segments
440 container to not pollute the main <container> listings.
441 --leave-segments
443 Indicates that you want the older segments of manifest objects
444 left alone (in the case of overwrites).
445 --object-threads <threads>
447 Number of threads to use for uploading full objects. Default is
448 10.
449 --segment-threads <threads>
451 Number of threads to use for uploading object segments. Default
452 is 10.
453 -H, --header <header:value>
455 Adds a customized request header. This option may be repeated.
456 Example: -H "content-type:text/plain" -H "Content-Length: 4000".
457 --use-slo
459 When used in conjunction with --segment-size it will create a
460 Static Large Object instead of the default Dynamic Large Object.
461 --object-name <object-name>
463 Upload file and name object to <object-name> or upload dir and
464 use <object-name> as object prefix instead of folder name.
465 --ignore-checksum
467 Turn off checksum validation for uploads.
468 swift post
470 Usage: swift post [--read-acl <acl>] [--write-acl <acl>] [--sync-to <sync-to>]
471 [--sync-key <sync-key>] [--meta <name:value>]
472 [--header <header>]
473 [<container> [<object>]]
474 Updates meta information for the account, container, or object depend‐
476 ing on the arguments given. If the container is not found, the swift‐
477 client will create it automatically, but this is not true for accounts
478 and objects. Containers also allow the -r <read-acl> (or --read-acl
479 <read-acl>) and -w <write-acl> (or --write-acl <write-acl>) options.
480 The -m or --meta option is allowed on accounts, containers and objects,
481 and is used to define the user metadata items to set in the form
482 Name:Value. You can repeat this option. For example: post -m
483 Color:Blue -m Size:Large
484 For more information about ACL formats see the documentation: ACLs.
486 Positional arguments:
488 [container]
490 Name of container to post to.
491 [object]
493 Name of object to post.
494 Optional arguments:
496 -r, --read-acl <acl>
498 Read ACL for containers. Quick summary of ACL syntax: .r:*,
499 .r:-.example.com, .r:www.example.com, account1 (v1.0 identity
500 API only), account1:*, account2:user2 (v2.0+ identity API).
501 -w, --write-acl <acl>
503 Write ACL for containers. Quick summary of ACL syntax: account1
504 (v1.0 identity API only), account1:*, account2:user2 (v2.0+
505 identity API).
506 -t, --sync-to <sync-to>
508 Sync To for containers, for multi-cluster replication.
509 -k, --sync-key <sync-key>
511 Sync Key for containers, for multi-cluster replication.
512 -m, --meta <name:value>
514 Sets a meta data item. This option may be repeated.
515 Example: -m Color:Blue -m Size:Large
517 -H, --header <header:value>
519 Adds a customized request header. This option may be repeated.
520 Example: -H "content-type:text/plain" -H "Content-Length: 4000"
522 swift download
524 Usage: swift download [--all] [--marker <marker>] [--prefix <prefix>]
525 [--output <out_file>] [--output-dir <out_directory>]
526 [--object-threads <threads>] [--ignore-checksum]
527 [--container-threads <threads>] [--no-download]
528 [--skip-identical] [--remove-prefix]
529 [--header <header:value>] [--no-shuffle]
530 [<container> [<object>] [...]]
531 Downloads everything in the account (with --all), or everything in a
533 container, or a list of objects depending on the arguments given. For a
534 single object download, you may use the -o <filename> or --output
535 <filename> option to redirect the output to a specific file or - to re‐
536 direct to stdout. The --ignore-checksum is an option that turn off
537 checksum validation. You can specify optional headers with the repeat‐
538 able cURL-like option -H [--header <name:value>]. --ignore-mtime ig‐
539 nores the x-object-meta-mtime metadata entry on the object (if present)
540 and instead creates the downloaded files with fresh atime and mtime
541 values.
542 Positional arguments:
544 <container>
546 Name of container to download from. To download a whole account,
547 omit this and specify --all.
548 <object>
550 Name of object to download. Specify multiple times for multiple
551 objects. Omit this to download all objects from the container.
552 Optional arguments:
554 -a, --all
556 Indicates that you really want to download everything in the ac‐
557 count.
558 -m, --marker <marker>
560 Marker to use when starting a container or account download.
561 -p, --prefix <prefix>
563 Only download items beginning with <prefix>
564 -r, --remove-prefix
566 An optional flag for --prefix <prefix>, use this option to down‐
567 load items without <prefix>
568 -o, --output <out_file>
570 For a single file download, stream the output to <out_file>.
571 Specifying "-" as <out_file> will redirect to stdout.
572 -D, --output-dir <out_directory>
574 An optional directory to which to store objects. By default,
575 all objects are recreated in the current directory.
576 --object-threads <threads>
578 Number of threads to use for downloading objects. Default is
579 10.
580 --container-threads <threads>
582 Number of threads to use for downloading containers. Default is
583 10.
584 --no-download
586 Perform download(s), but don't actually write anything to disk.
587 -H, --header <header:value>
589 Adds a customized request header to the query, like "Range" or
590 "If-Match". This option may be repeated.
591 Example: --header "content-type:text/plain"
593 --skip-identical
595 Skip downloading files that are identical on both sides.
596 --ignore-checksum
598 Turn off checksum validation for downloads.
599 --no-shuffle
601 By default, when downloading a complete account or container,
602 download order is randomised in order to reduce the load on in‐
603 dividual drives when multiple clients are executed simultane‐
604 ously to download the same set of objects (e.g. a nightly auto‐
605 mated download script to multiple servers). Enable this option
606 to submit download jobs to the thread pool in the order they are
607 listed in the object store.
608 swift delete
610 Usage: swift delete [--all] [--leave-segments]
611 [--object-threads <threads>]
612 [--container-threads <threads>]
613 [--header <header:value>]
614 [<container> [<object>] [...]]
615 Deletes everything in the account (with --all), or everything in a con‐
617 tainer, or a list of objects depending on the arguments given. Segments
618 of manifest objects will be deleted as well, unless you specify the
619 --leave-segments option.
620 Positional arguments:
622 [<container>]
624 Name of container to delete from.
625 [<object>]
627 Name of object to delete. Specify multiple times for multiple
628 objects.
629 Optional arguments:
631 -a, --all
633 Delete all containers and objects.
634 --leave-segments
636 Do not delete segments of manifest objects.
637 -H, --header <header:value>
639 Adds a custom request header to use for deleting objects or an
640 entire container.
641 --object-threads <threads>
643 Number of threads to use for deleting objects. Default is 10.
644 --container-threads <threads>
646 Number of threads to use for deleting containers. Default is
647 10.
648 swift copy
650 Usage: swift copy [--destination </container/object>] [--fresh-metadata]
651 [--meta <name:value>] [--header <header>] <container>
652 <object> [<object>] [...]
653 Copies an object to a new destination or adds user metadata to an ob‐
655 ject. Depending on the options supplied, you can preserve existing
656 metadata in contrast to the post command. The --destination option sets
657 the copy target destination in the form /container/object. If not set,
658 the object will be copied onto itself which is useful for adding meta‐
659 data. You can use the -M or --fresh-metadata option to copy an object
660 without existing user meta data, and the -m or --meta option to define
661 user meta data items to set in the form Name:Value. You can repeat this
662 option. For example: copy -m Color:Blue -m Size:Large.
663 Positional arguments:
665 <container>
667 Name of container to copy from.
668 <object>
670 Name of object to copy. Specify multiple times for multiple ob‐
671 jects
672 Optional arguments:
674 -d, --destination </container[/object]>
676 The container and name of the destination object. Name of desti‐
677 nation object can be omitted, then will be same as name of
678 source object. Supplying multiple objects and destination with
679 object name is invalid.
680 -M, --fresh-metadata
682 Copy the object without any existing metadata, If not set, meta‐
683 data will be preserved or appended
684 -m, --meta <name:value>
686 Sets a meta data item. This option may be repeated.
687 Example: -m Color:Blue -m Size:Large
689 -H, --header <header:value>
691 Adds a customized request header. This option may be repeated.
692 Example: -H "content-type:text/plain" -H "Content-Length: 4000"
694 swift capabilities
696 Usage: swift capabilities [--json] [<proxy_url>]
697 Displays cluster capabilities. The output includes the list of the ac‐
699 tivated Swift middlewares as well as relevant options for each ones.
700 Additionally the command displays relevant options for the Swift core.
701 If the proxy-url option is not provided, the storage URL retrieved af‐
702 ter authentication is used as proxy-url.
703 Optional positional arguments:
705 <proxy_url>
707 Proxy URL of the cluster to retrieve capabilities.
708 --json Print the cluster capabilities in JSON format.
710 swift tempurl
712 Usage: swift tempurl [--absolute] [--prefix-based]
713 <method> <seconds> <path> <key>
714 Generates a temporary URL for a Swift object. method option sets an
716 HTTP method to allow for this temporary URL that is usually GET or PUT.
717 time option sets the amount of time the temporary URL will be valid
718 for. time can be specified as an integer, denoting the number of sec‐
719 onds from now on until the URL shall be valid; or, if --absolute is
720 passed, the Unix timestamp when the temporary URL will expire. But be‐
721 yond that, time can also be specified as an ISO 8601 timestamp in one
722 of following formats:
723 i. Complete date: YYYY-MM-DD (e.g. 1997-07-16)
725 ii. Complete date plus hours, minutes and seconds: YYYY-MM-DDThh:mm:ss
727 (e.g. 1997-07-16T19:20:30)
728 iii. Complete date plus hours, minutes and seconds with UTC designator:
730 YYYY-MM-DDThh:mm:ssZ (e.g. 1997-07-16T19:20:30Z)
731 Please be aware that if you don't provide the UTC designator (i.e., Z)
733 the timestamp is generated using your local timezone. If only a date is
734 specified, the time part used will equal to 00:00:00.
735 path option sets the full path to the Swift object. Example:
737 /v1/AUTH_account/c/o. key option is the secret temporary URL key set on
738 the Swift cluster. To set a key, run swift post -m "Temp-URL-Key: <your
739 secret key>". To generate a prefix-based temporary URL use the --pre‐
740 fix-based option. This URL will contain the path to the prefix. Do not
741 forget to append the desired objectname at the end of the path portion
742 (and before the query portion) before sharing the URL. It is possible
743 to use ISO 8601 UTC timestamps within the URL by using the --iso8601
744 option.
745 Positional arguments:
747 <method>
749 An HTTP method to allow for this temporary URL. Usually 'GET'
750 or 'PUT'.
751 <seconds>
753 The amount of time in seconds the temporary URL will be valid
754 for; or, if --absolute is passed, the Unix timestamp when the
755 temporary URL will expire.
756 <path> The full path to the Swift object.
758 Example: /v1/AUTH_account/c/o or:
760 http://saio:8080/v1/AUTH_account/c/o
761 <key> The secret temporary URL key set on the Swift cluster. To set a
763 key, run 'swift post -m
764 "Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4"'
765 Optional arguments:
767 --absolute
769 Interpret the <seconds> positional argument as a Unix timestamp
770 rather than a number of seconds in the future.
771 --prefix-based
773 If present, a prefix-based tempURL will be generated.
774 Examples
776 In this section we present some example usage of the swift CLI. To keep
777 the examples as short as possible, these examples assume that the rele‐
778 vant authentication options have been set using environment variables.
779 You can obtain the full list of commands and options available in the
780 swift CLI by executing the following:
781 > swift --help
783 > swift <command> --help
784 Simple examples
786 List the existing swift containers:
787 > swift list
789 container_1
791 Create a new container:
793 > swift post TestContainer
795 Upload an object into a container:
797 > swift upload TestContainer testSwift.txt
799 testSwift.txt
801 List the contents of a container:
803 > swift list TestContainer
805 testSwift.txt
807 Copy an object to new destination:
809 > swift copy -d /DestContainer/testSwift.txt SourceContainer testSwift.txt
811 SourceContainer/testSwift.txt copied to /DestContainer/testSwift.txt
813 Delete an object from a container:
815 > swift delete TestContainer testSwift.txt
817 testSwift.txt
819 Delete a container:
821 > swift delete TestContainer
823 TestContainer
825 Display auth related authentication variables in shell friendly format:
827 > swift auth
829 export OS_STORAGE_URL=http://127.0.0.1:8080/v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7
831 export OS_AUTH_TOKEN=c597015ae19943a18438b52ef3762e79
832 Download an object from a container:
834 > swift download TestContainer testSwift.txt
836 testSwift.txt [auth 0.028s, headers 0.045s, total 0.045s, 0.002 MB/s]
838 NOTE:
840 To upload an object to a container, your current working directory
841 must be where the file is located or you must provide the complete
842 path to the file. In other words, the --object-name <object-name>
843 is an option that will upload file and name object to <object-name>
844 or upload directory and use <object-name> as object prefix. In the
845 case that you provide the complete path of the file, that complete
846 path will be the name of the uploaded object.
847 For example:
849 > swift upload TestContainer /home/swift/testSwift/testSwift.txt
851 home/swift/testSwift/testSwift.txt
853 > swift list TestContainer
855 home/swift/testSwift/testSwift.txt
857 More complex examples
859 Swift has a single object size limit of 5GiB. In order to upload files
860 larger than this, we must create a large object that consists of
861 smaller segments. The example below shows how to upload a large video
862 file as a static large object in 1GiB segments:
863 > swift upload videos --use-slo --segment-size 1G myvideo.mp4
865 myvideo.mp4 segment 8
867 myvideo.mp4 segment 4
868 myvideo.mp4 segment 2
869 myvideo.mp4 segment 7
870 myvideo.mp4 segment 0
871 myvideo.mp4 segment 1
872 myvideo.mp4 segment 3
873 myvideo.mp4 segment 6
874 myvideo.mp4 segment 5
875 myvideo.mp4
876 This command will upload segments to a container named videos_segments,
878 and create a manifest file describing the entire object in the videos
879 container. For more information on large objects, see the documenta‐
880 tion here.
881 > swift list videos
883 myvideo.mp4
885 > swift list videos_segments
887 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000000
889 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000001
890 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000002
891 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000003
892 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000004
893 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000005
894 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000006
895 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000007
896 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000008
897 Firstly, the key should be set, then generate a temporary URL for a
899 Swift object:
900 > swift post -m "Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4"
902 > swift tempurl GET 6000 /v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7\
904 /firstcontainer/clean.sh b3968d0207b54ece87cccc06515a89d4
905 /v1/AUTH_/firstcontainer/clean.sh?temp_url_sig=\
907 9218fc288cc09e5edd857b6a3d43cf2122b906dc&temp_url_expires=1472203614
908 The swiftclient.SwiftService API
910 A higher-level API aimed at allowing developers an easy way to perform
911 multiple operations asynchronously using a configurable thread pool.
912 Documentation for each service method call can be found here:
913 swiftclient.service.
914 Authentication
916 This section covers the various options for authenticating with a swift
917 object store. The combinations of options required for each authentica‐
918 tion version are detailed below. Once again, these are just a subset of
919 those that can be used to successfully authenticate, but they are the
920 most common and recommended.
921 The relevant authentication options are presented as python dictionar‐
923 ies that should be added to any other options you are supplying to your
924 SwiftService instance. As indicated in the python code, you can also
925 set these options as environment variables that will be loaded automat‐
926 ically if the relevant option is not specified.
927 The SwiftService authentication attempts to automatically select the
929 auth version based on the combination of options specified, but supply‐
930 ing options from multiple different auth versions can cause unexpected
931 behaviour.
932 NOTE:
934 Leftover environment variables are a common source of confusion when
935 authorization fails.
936 Keystone V3
938 {
939 ...
940 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '3'
941 "os_username": environ.get('OS_USERNAME'),
942 "os_password": environ.get('OS_PASSWORD'),
943 "os_project_name": environ.get('OS_PROJECT_NAME'),
944 "os_project_domain_name": environ.get('OS_PROJECT_DOMAIN_NAME'),
945 "os_auth_url": environ.get('OS_AUTH_URL'),
946 ...
947 }
948 {
950 ...
951 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '3'
952 "os_username": environ.get('OS_USERNAME'),
953 "os_password": environ.get('OS_PASSWORD'),
954 "os_project_id": environ.get('OS_PROJECT_ID'),
955 "os_project_domain_id": environ.get('OS_PROJECT_DOMAIN_ID'),
956 "os_auth_url": environ.get('OS_AUTH_URL'),
957 ...
958 }
959 Keystone V2
961 {
962 ...
963 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '2.0'
964 "os_username": environ.get('OS_USERNAME'),
965 "os_password": environ.get('OS_PASSWORD'),
966 "os_tenant_name": environ.get('OS_TENANT_NAME'),
967 "os_auth_url": environ.get('OS_AUTH_URL'),
968 ...
969 }
970 Legacy Auth
972 {
973 ...
974 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '1.0'
975 "auth": environ.get('ST_AUTH'),
976 "user": environ.get('ST_USER'),
977 "key": environ.get('ST_KEY'),
978 ...
979 }
980 Configuration
982 When you create an instance of a SwiftService, you can override a col‐
983 lection of default options to suit your use case. Typically, the de‐
984 faults are sensible to get us started, but depending on your needs you
985 might want to tweak them to improve performance (options affecting
986 large objects and thread counts can significantly alter performance in
987 the right situation).
988 Service level defaults and some extra options can also be overridden on
990 a per-operation (or even in some cases per-object) basis, and you will
991 call out which options affect which operations later in the document.
992 The configuration of the service API is performed using an options dic‐
994 tionary passed to the SwiftService during initialisation. The options
995 available in this dictionary are described below, along with their de‐
996 faults:
997 Options
999 retries: 5
1000 The number of times that the library should attempt to retry
1001 HTTP actions before giving up and reporting a failure.
1002 container_threads: 10
1004 object_dd_threads: 10
1006 object_uu_threads: 10
1008 segment_threads: 10
1010 The above options determine the size of the available thread
1011 pools for performing swift operations. Container operations
1012 (such as listing a container) operate in the container threads,
1013 and a similar pattern applies to object and segment threads.
1014 NOTE:
1016 Object threads are separated into two separate thread pools:
1017 uu and dd. This stands for "upload/update" and "down‐
1018 load/delete", and the corresponding actions will be run on
1019 separate threads pools.
1020 segment_size: None
1022 If specified, this option enables uploading of large objects.
1023 Should the object being uploaded be larger than 5G in size, this
1024 option is mandatory otherwise the upload will fail. This option
1025 should be specified as a size in bytes.
1026 use_slo: False
1028 Used in combination with the above option, use_slo will upload
1029 large objects as static rather than dynamic. Only static large
1030 objects provide error checking for the downloaded object, so we
1031 recommend this option.
1032 segment_container: None
1034 Allows the user to select the container into which large object
1035 segments will be uploaded. We do not recommend changing this
1036 value as it could make locating orphaned segments more difficult
1037 in the case of errors.
1038 leave_segments: False
1040 Setting this option to true means that when deleting or over‐
1041 writing a large object, its segments will be left in the object
1042 store and must be cleaned up manually. This option can be useful
1043 when sharing large object segments between multiple objects in
1044 more advanced scenarios, but must be treated with care, as it
1045 could lead to ever increasing storage usage.
1046 changed: None
1048 This option affects uploads and simply means that those objects
1049 which already exist in the object store will not be overwritten
1050 if the mtime and size of the source is the same as the existing
1051 object.
1052 skip_identical: False
1054 A slightly more thorough case of the above, but rather than
1055 mtime and size uses an object's MD5 sum.
1056 yes_all: False
1058 This options affects only download and delete, and in each case
1059 must be specified in order to download/delete the entire con‐
1060 tents of an account. This option has no effect on any other
1061 calls.
1062 no_download: False
1064 This option only affects download and means that all operations
1065 proceed as normal with the exception that no data is written to
1066 disk.
1067 header: []
1069 Used with upload and post operations to set headers on objects.
1070 Headers are specified as colon separated strings, e.g. "con‐
1071 tent-type:text/plain".
1072 meta: []
1074 Used to set metadata on an object similarly to headers.
1075 NOTE:
1077 Setting metadata is a destructive operation, so when updating
1078 one of many metadata values all desired metadata for an ob‐
1079 ject must be re-applied.
1080 long: False
1082 Affects only list operations, and results in more metrics being
1083 made available in the results at the expense of lower perfor‐
1084 mance.
1085 fail_fast: False
1087 Applies to delete and upload operations, and attempts to abort
1088 queued tasks in the event of errors.
1089 prefix: None
1091 Affects list operations; only objects with the given prefix will
1092 be returned/affected. It is not advisable to set at the service
1093 level, as those operations that call list to discover objects on
1094 which they should operate will also be affected.
1095 delimiter: None
1097 Affects list operations, and means that listings only contain
1098 results up to the first instance of the delimiter in the object
1099 name. This is useful for working with objects containing '/' in
1100 their names to simulate folder structures.
1101 dir_marker: False
1103 Affects uploads, and allows empty 'pseudofolder' objects to be
1104 created when the source of an upload is None.
1105 checksum: True
1107 Affects uploads and downloads. If set check md5 sum for the
1108 transfer.
1109 shuffle: False
1111 When downloading objects, the default behaviour of the CLI is to
1112 shuffle lists of objects in order to spread the load on storage
1113 drives when multiple clients are downloading the same files to
1114 multiple locations (e.g. in the event of distributing an up‐
1115 date). When using the SwiftService directly, object downloads
1116 are scheduled in the same order as they appear in the container
1117 listing. When combined with a single download thread this means
1118 that objects are downloaded in lexically-sorted order. Setting
1119 this option to True gives the same shuffling behaviour as the
1120 CLI.
1121 destination: None
1123 When copying objects, this specifies the destination where the
1124 object will be copied to. The default of None means copy will
1125 be the same as source.
1126 fresh_metadata: None
1128 When copying objects, this specifies that the object metadata on
1129 the source will not be applied to the destination object - the
1130 destination object will have a new fresh set of metadata that
1131 includes only the metadata specified in the meta option if any
1132 at all.
1133 Other available options can be found in swiftclient/service.py in the
1135 source code for python-swiftclient. Each SwiftService method also al‐
1136 lows for an optional dictionary to override those specified at init
1137 time, and the appropriate docstrings show which options modify each
1138 method's behaviour.
1139 Available Operations
1141 Each operation provided by the service API may raise a SwiftError or
1142 ClientException for any call that fails completely (or a call which
1143 performs only one operation at an account or container level). In the
1144 case of a successful call an operation returns one of the following:
1145 • A dictionary detailing the results of a single operation.
1147 • An iterator that produces result dictionaries (for calls that perform
1149 multiple sub-operations).
1150 A result dictionary can indicate either the success or failure of an
1152 individual operation (detailed in the success key), and will either
1153 contain the successful result, or an error key detailing the error en‐
1154 countered (usually an instance of Exception).
1155 An example result dictionary is given below:
1157 result = {
1159 'action': 'download_object',
1160 'success': True,
1161 'container': container,
1162 'object': obj,
1163 'path': path,
1164 'start_time': start_time,
1165 'finish_time': finish_time,
1166 'headers_receipt': headers_receipt,
1167 'auth_end_time': conn.auth_end_time,
1168 'read_length': bytes_read,
1169 'attempts': conn.attempts
1170 }
1171 All the possible action values are detailed below:
1173 [
1175 'stat_account',
1176 'stat_container',
1177 'stat_object',
1178 'post_account',
1179 'post_container',
1180 'post_object',
1181 'list_part', # list yields zero or more 'list_part' results
1182 'download_object',
1183 'create_container', # from upload
1184 'create_dir_marker', # from upload
1185 'upload_object',
1186 'upload_segment',
1187 'delete_container',
1188 'delete_object',
1189 'delete_segment', # from delete_object operations
1190 'capabilities',
1191 ]
1192 Stat
1194 Stat can be called against an account, a container, or a list of ob‐
1195 jects to get account stats, container stats or information about the
1196 given objects. In the first two cases a dictionary is returned contain‐
1197 ing the results of the operation, and in the case of a list of object
1198 names being supplied, an iterator over the results generated for each
1199 object is returned.
1200 Information returned includes the amount of data used by the given ob‐
1202 ject/container/account and any headers or metadata set (this includes
1203 user set data as well as content-type and modification times).
1204 See swiftclient.service.SwiftService.stat for docs generated from the
1206 method docstring.
1207 Valid calls for this method are as follows:
1209 • stat([options]): Returns stats for the configured account.
1211 • stat(<container>, [options]): Returns stats for the given container.
1213 • stat(<container>, <object_list>, [options]): Returns stats for each
1215 of the given objects in the given container (through the returned it‐
1216 erator).
1217 Results from stat are dictionaries indicating the success or failure of
1219 each operation. In the case of a successful stat against an account or
1220 container, the method returns immediately with one of the following re‐
1221 sults:
1222 {
1224 'action': 'stat_account',
1225 'success': True,
1226 'items': items,
1227 'headers': headers
1228 }
1229 {
1231 'action': 'stat_container',
1232 'container': <container>,
1233 'success': True,
1234 'items': items,
1235 'headers': headers
1236 }
1237 In the case of stat called against a list of objects, the method re‐
1239 turns a generator that returns the results of individual object stat
1240 operations as they are performed on the thread pool:
1241 {
1243 'action': 'stat_object',
1244 'object': <object_name>,
1245 'container': <container>,
1246 'success': True,
1247 'items': items,
1248 'headers': headers
1249 }
1250 In the case of a failure the dictionary returned will indicate that the
1252 operation was not successful, and will include the keys below:
1253 {
1255 'action': <'stat_object'|'stat_container'|'stat_account'>,
1256 'object': <'object_name'>, # Only for stat with objects list
1257 'container': <container>, # Only for stat with objects list or container
1258 'success': False,
1259 'error': <error>,
1260 'traceback': <trace>,
1261 'error_timestamp': <timestamp>
1262 }
1263 Example
1265 The code below demonstrates the use of stat to retrieve the headers for
1266 a given list of objects in a container using 20 threads. The code cre‐
1267 ates a mapping from object name to headers which is then pretty printed
1268 to the log.
1269 import logging
1271 import pprint
1272 from swiftclient.service import SwiftService
1274 from sys import argv
1275 logging.basicConfig(level=logging.ERROR)
1277 logging.getLogger("requests").setLevel(logging.CRITICAL)
1278 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1279 logger = logging.getLogger(__name__)
1280 _opts = {'object_dd_threads': 20}
1282 with SwiftService(options=_opts) as swift:
1283 container = argv[1]
1284 objects = argv[2:]
1285 header_data = {}
1286 stats_it = swift.stat(container=container, objects=objects)
1287 for stat_res in stats_it:
1288 if stat_res['success']:
1289 header_data[stat_res['object']] = stat_res['headers']
1290 else:
1291 logger.error(
1292 'Failed to retrieve stats for %s' % stat_res['object']
1293 )
1294 pprint.pprint(header_data)
1295 List
1298 List can be called against an account or a container to retrieve the
1299 containers or objects contained within them. Each call returns an iter‐
1300 ator that returns pages of results (by default, up to 10000 results in
1301 each page).
1302 See swiftclient.service.SwiftService.list for docs generated from the
1304 method docstring.
1305 If the given container or account does not exist, the list method will
1307 raise a SwiftError, but for all other success/failures a dictionary is
1308 returned. Each successfully listed page returns a dictionary as de‐
1309 scribed below:
1310 {
1312 'action': <'list_account_part'|'list_container_part'>,
1313 'container': <container>, # Only for listing a container
1314 'prefix': <prefix>, # The prefix of returned objects/containers
1315 'success': True,
1316 'listing': [Item], # A list of results
1317 # (only in the event of success)
1318 'marker': <marker> # The last item name in the list
1319 # (only in the event of success)
1320 }
1321 Where an item contains the following keys:
1323 {
1325 'name': <name>,
1326 'bytes': 10485760,
1327 'last_modified': '2014-12-11T12:02:38.774540',
1328 'hash': 'fb938269cbeabe4c234e1127bbd3b74a',
1329 'content_type': 'application/octet-stream',
1330 'meta': <metadata> # Full metadata listing from stat'ing each object
1331 # this key only exists if 'long' is specified in options
1332 }
1333 Any failure listing an account or container that exists will return a
1335 failure dictionary as described below:
1336 {
1338 'action': <'list_account_part'|'list_container_part'>,,
1339 'container': container, # Only for listing a container
1340 'prefix': options['prefix'],
1341 'success': success,
1342 'marker': marker,
1343 'error': error,
1344 'traceback': <trace>,
1345 'error_timestamp': <timestamp>
1346 }
1347 Example
1349 The code below demonstrates the use of list to list all items in a con‐
1350 tainer that are over 10MiB in size:
1351 import logging
1353 from swiftclient.service import SwiftService, SwiftError
1355 from sys import argv
1356 logging.basicConfig(level=logging.ERROR)
1358 logging.getLogger("requests").setLevel(logging.CRITICAL)
1359 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1360 logger = logging.getLogger(__name__)
1361 container = argv[1]
1363 minimum_size = 10*1024**2
1364 with SwiftService() as swift:
1365 try:
1366 list_parts_gen = swift.list(container=container)
1367 for page in list_parts_gen:
1368 if page["success"]:
1369 for item in page["listing"]:
1370 i_size = int(item["bytes"])
1372 if i_size > minimum_size:
1373 i_name = item["name"]
1374 i_etag = item["hash"]
1375 print(
1376 "%s [size: %s] [etag: %s]" %
1377 (i_name, i_size, i_etag)
1378 )
1379 else:
1380 raise page["error"]
1381 except SwiftError as e:
1383 logger.error(e.value)
1384 Post
1387 Post can be called against an account, container or list of objects in
1388 order to update the metadata attached to the given items. In the first
1389 two cases a single dictionary is returned containing the results of the
1390 operation, and in the case of a list of objects being supplied, an it‐
1391 erator over the results generated for each object post is returned.
1392 Each element of the object list may be a plain string of the object
1394 name, or a SwiftPostObject that allows finer control over the options
1395 and metadata applied to each of the individual post operations. When a
1396 string is given for the object name, the options and metadata applied
1397 are a combination of those supplied to the call to post() and the de‐
1398 faults of the SwiftService object.
1399 If the given container or account does not exist, the post method will
1401 raise a SwiftError. Successful metadata update results are dictionaries
1402 as described below:
1403 {
1405 'action': <'post_account'|'post_container'|'post_object'>,
1406 'success': True,
1407 'container': <container>,
1408 'object': <object>,
1409 'headers': {},
1410 'response_dict': <HTTP response details>
1411 }
1412 NOTE:
1414 Updating user metadata keys will not only add any specified keys,
1415 but will also remove user metadata that has previously been set.
1416 This means that each time user metadata is updated, the complete set
1417 of desired key-value pairs must be specified.
1418 Example
1420 The code below demonstrates the use of post to set an archive folder in
1421 a given container to expire after a 24 hour delay:
1422 import logging
1424 from swiftclient.service import SwiftService, SwiftError
1426 from sys import argv
1427 logging.basicConfig(level=logging.ERROR)
1429 logging.getLogger("requests").setLevel(logging.CRITICAL)
1430 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1431 logger = logging.getLogger(__name__)
1432 container = argv[1]
1434 with SwiftService() as swift:
1435 try:
1436 list_options = {"prefix": "archive_2016-01-01/"}
1437 list_parts_gen = swift.list(container=container)
1438 for page in list_parts_gen:
1439 if page["success"]:
1440 objects = [obj["name"] for obj in page["listing"]]
1441 post_options = {"header": "X-Delete-After:86400"}
1442 for post_res in swift.post(
1443 container=container,
1444 objects=objects,
1445 options=post_options):
1446 if post_res['success']:
1447 print("Object '%s' POST success" % post_res['object'])
1448 else:
1449 print("Object '%s' POST failed" % post_res['object'])
1450 else:
1451 raise page["error"]
1452 except SwiftError as e:
1453 logger.error(e.value)
1454 Download
1457 Download can be called against an entire account, a single container,
1458 or a list of objects in a given container. Each element of the object
1459 list is a string detailing the full name of an object to download.
1460 In order to download the full contents of an entire account, you must
1462 set the value of yes_all to True in the options dictionary supplied to
1463 either the SwiftService instance or the call to download.
1464 If the given container or account does not exist, the download method
1466 will raise a SwiftError, otherwise an iterator over the results gener‐
1467 ated for each object download is returned.
1468 See swiftclient.service.SwiftService.download for docs generated from
1470 the method docstring.
1471 For each successfully downloaded object, the results returned by the
1473 iterator will be a dictionary as described below (results are not re‐
1474 turned for completed container or object segment downloads):
1475 {
1477 'action': 'download_object',
1478 'container': <container>,
1479 'object': <object name>,
1480 'success': True,
1481 'path': <local path to downloaded object>,
1482 'pseudodir': <if true, the download created an empty directory>,
1483 'start_time': <time download started>,
1484 'end_time': <time download completed>,
1485 'headers_receipt': <time the headers from the object were retrieved>,
1486 'auth_end_time': <time authentication completed>,
1487 'read_length': <bytes_read>,
1488 'attempts': <attempt count>,
1489 'response_dict': <HTTP response details>
1490 }
1491 Any failure uploading an object will return a failure dictionary as de‐
1493 scribed below:
1494 {
1496 'action': 'download_object',
1497 'container': <container>,
1498 'object': <object name>,
1499 'success': False,
1500 'path': <local path of the failed download>,
1501 'pseudodir': <if true, the failed download was an empty directory>,
1502 'attempts': <attempt count>,
1503 'error': <error>,
1504 'traceback': <trace>,
1505 'error_timestamp': <timestamp>,
1506 'response_dict': <HTTP response details>
1507 }
1508 Example
1510 The code below demonstrates the use of download to download all PNG im‐
1511 ages from a dated archive folder in a given container:
1512 import logging
1514 from swiftclient.service import SwiftService, SwiftError
1516 from sys import argv
1517 logging.basicConfig(level=logging.ERROR)
1519 logging.getLogger("requests").setLevel(logging.CRITICAL)
1520 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1521 logger = logging.getLogger(__name__)
1522 def is_png(obj):
1524 return (
1525 obj["name"].lower().endswith('.png') or
1526 obj["content_type"] == 'image/png'
1527 )
1528 container = argv[1]
1530 with SwiftService() as swift:
1531 try:
1532 list_options = {"prefix": "archive_2016-01-01/"}
1533 list_parts_gen = swift.list(container=container)
1534 for page in list_parts_gen:
1535 if page["success"]:
1536 objects = [
1537 obj["name"] for obj in page["listing"] if is_png(obj)
1538 ]
1539 for down_res in swift.download(
1540 container=container,
1541 objects=objects):
1542 if down_res['success']:
1543 print("'%s' downloaded" % down_res['object'])
1544 else:
1545 print("'%s' download failed" % down_res['object'])
1546 else:
1547 raise page["error"]
1548 except SwiftError as e:
1549 logger.error(e.value)
1550 Upload
1553 Upload is always called against an account and container and with a
1554 list of objects to upload. Each element of the object list may be a
1555 plain string detailing the path of the object to upload, or a SwiftU‐
1556 ploadObject that allows finer control over some aspects of the individ‐
1557 ual operations.
1558 When a simple string is supplied to specify a file to upload, the name
1560 of the object uploaded is the full path of the specified file and the
1561 options used for the upload are those supplied to the call to upload.
1562 Constructing a SwiftUploadObject allows the user to supply an object
1564 name for the uploaded file, and modify the options used by upload at
1565 the granularity of individual files.
1566 If the given container or account does not exist, the upload method
1568 will raise a SwiftError, otherwise an iterator over the results gener‐
1569 ated for each object upload is returned.
1570 See swiftclient.service.SwiftService.upload for docs generated from the
1572 method docstring.
1573 For each successfully uploaded object (or object segment), the results
1575 returned by the iterator will be a dictionary as described below:
1576 {
1578 'action': 'upload_object',
1579 'container': <container>,
1580 'object': <object name>,
1581 'success': True,
1582 'status': <'uploaded'|'skipped-identical'|'skipped-changed'>,
1583 'attempts': <attempt count>,
1584 'response_dict': <HTTP response details>
1585 }
1586 {
1588 'action': 'upload_segment',
1589 'for_container': <container>,
1590 'for_object': <object name>,
1591 'segment_index': <segment_index>,
1592 'segment_size': <segment_size>,
1593 'segment_location': <segment_path>
1594 'segment_etag': <etag>,
1595 'log_line': <object segment n>
1596 'success': True,
1597 'response_dict': <HTTP response details>,
1598 'attempts': <attempt count>
1599 }
1600 Any failure uploading an object will return a failure dictionary as de‐
1602 scribed below:
1603 {
1605 'action': 'upload_object',
1606 'container': <container>,
1607 'object': <object name>,
1608 'success': False,
1609 'attempts': <attempt count>,
1610 'error': <error>,
1611 'traceback': <trace>,
1612 'error_timestamp': <timestamp>,
1613 'response_dict': <HTTP response details>
1614 }
1615 {
1617 'action': 'upload_segment',
1618 'for_container': <container>,
1619 'for_object': <object name>,
1620 'segment_index': <segment_index>,
1621 'segment_size': <segment_size>,
1622 'segment_location': <segment_path>,
1623 'log_line': <object segment n>,
1624 'success': False,
1625 'error': <error>,
1626 'traceback': <trace>,
1627 'error_timestamp': <timestamp>,
1628 'response_dict': <HTTP response details>,
1629 'attempts': <attempt count>
1630 }
1631 Example
1633 The code below demonstrates the use of upload to upload all files and
1634 folders in a given directory, and rename each object by replacing the
1635 root directory name with 'my-<d>-objects', where <d> is the name of the
1636 uploaded directory:
1637 import logging
1639 from os import walk
1641 from os.path import join
1642 from swiftclient.multithreading import OutputManager
1643 from swiftclient.service import SwiftError, SwiftService, SwiftUploadObject
1644 from sys import argv
1645 logging.basicConfig(level=logging.ERROR)
1647 logging.getLogger("requests").setLevel(logging.CRITICAL)
1648 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1649 logger = logging.getLogger(__name__)
1650 _opts = {'object_uu_threads': 20}
1652 dir = argv[1]
1653 container = argv[2]
1654 with SwiftService(options=_opts) as swift, OutputManager() as out_manager:
1655 try:
1656 # Collect all the files and folders in the given directory
1657 objs = []
1658 dir_markers = []
1659 for (_dir, _ds, _fs) in walk(dir):
1660 if not (_ds + _fs):
1661 dir_markers.append(_dir)
1662 else:
1663 objs.extend([join(_dir, _f) for _f in _fs])
1664 # Now that we've collected all the required files and dir markers
1666 # build the ``SwiftUploadObject``s for the call to upload
1667 objs = [
1668 SwiftUploadObject(
1669 o, object_name=o.replace(
1670 dir, 'my-%s-objects' % dir, 1
1671 )
1672 ) for o in objs
1673 ]
1674 dir_markers = [
1675 SwiftUploadObject(
1676 None, object_name=d.replace(
1677 dir, 'my-%s-objects' % dir, 1
1678 ), options={'dir_marker': True}
1679 ) for d in dir_markers
1680 ]
1681 # Schedule uploads on the SwiftService thread pool and iterate
1683 # over the results
1684 for r in swift.upload(container, objs + dir_markers):
1685 if r['success']:
1686 if 'object' in r:
1687 print(r['object'])
1688 elif 'for_object' in r:
1689 print(
1690 '%s segment %s' % (r['for_object'],
1691 r['segment_index'])
1692 )
1693 else:
1694 error = r['error']
1695 if r['action'] == "create_container":
1696 logger.warning(
1697 'Warning: failed to create container '
1698 "'%s'%s", container, error
1699 )
1700 elif r['action'] == "upload_object":
1701 logger.error(
1702 "Failed to upload object %s to container %s: %s" %
1703 (container, r['object'], error)
1704 )
1705 else:
1706 logger.error("%s" % error)
1707 except SwiftError as e:
1709 logger.error(e.value)
1710 Delete
1713 Delete can be called against an account or a container to remove the
1714 containers or objects contained within them. Each call to delete re‐
1715 turns an iterator over results of each resulting sub-request.
1716 If the number of requested delete operations is large and the target
1718 swift cluster is running the bulk middleware, the call to SwiftSer‐
1719 vice.delete will make use of bulk operations and the returned result
1720 iterator will return bulk_delete results rather than individual
1721 delete_object, delete_container or delete_segment results.
1722 See swiftclient.service.SwiftService.delete for docs generated from the
1724 method docstring.
1725 For each successfully deleted container, object or segment, the results
1727 returned by the iterator will be a dictionary as described below:
1728 {
1730 'action': <'delete_object'|'delete_segment'>,
1731 'container': <container>,
1732 'object': <object name>,
1733 'success': True,
1734 'attempts': <attempt count>,
1735 'response_dict': <HTTP response details>
1736 }
1737 {
1739 'action': 'delete_container',
1740 'container': <container>,
1741 'success': True,
1742 'response_dict': <HTTP response details>,
1743 'attempts': <attempt count>
1744 }
1745 {
1747 'action': 'bulk_delete',
1748 'container': <container>,
1749 'objects': <[objects]>,
1750 'success': True,
1751 'attempts': <attempt count>,
1752 'response_dict': <HTTP response details>
1753 }
1754 Any failure in a delete operation will return a failure dictionary as
1756 described below:
1757 {
1759 'action': ('delete_object'|'delete_segment'),
1760 'container': <container>,
1761 'object': <object name>,
1762 'success': False,
1763 'attempts': <attempt count>,
1764 'error': <error>,
1765 'traceback': <trace>,
1766 'error_timestamp': <timestamp>,
1767 'response_dict': <HTTP response details>
1768 }
1769 {
1771 'action': 'delete_container',
1772 'container': <container>,
1773 'success': False,
1774 'error': <error>,
1775 'traceback': <trace>,
1776 'error_timestamp': <timestamp>,
1777 'response_dict': <HTTP response details>,
1778 'attempts': <attempt count>
1779 }
1780 {
1782 'action': 'bulk_delete',
1783 'container': <container>,
1784 'objects': <[objects]>,
1785 'success': False,
1786 'attempts': <attempt count>,
1787 'error': <error>,
1788 'traceback': <trace>,
1789 'error_timestamp': <timestamp>,
1790 'response_dict': <HTTP response details>
1791 }
1792 Example
1794 The code below demonstrates the use of delete to remove a given list of
1795 objects from a specified container. As the objects are deleted the
1796 transaction ID of the relevant request is printed along with the object
1797 name and number of attempts required. By printing the transaction ID,
1798 the printed operations can be easily linked to events in the swift
1799 server logs:
1800 import logging
1802 from swiftclient.service import SwiftService
1804 from sys import argv
1805 logging.basicConfig(level=logging.ERROR)
1808 logging.getLogger("requests").setLevel(logging.CRITICAL)
1809 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1810 logger = logging.getLogger(__name__)
1811 _opts = {'object_dd_threads': 20}
1813 container = argv[1]
1814 objects = argv[2:]
1815 with SwiftService(options=_opts) as swift:
1816 del_iter = swift.delete(container=container, objects=objects)
1817 for del_res in del_iter:
1818 c = del_res.get('container', '')
1819 o = del_res.get('object', '')
1820 a = del_res.get('attempts')
1821 if del_res['success'] and not del_res['action'] == 'bulk_delete':
1822 rd = del_res.get('response_dict')
1823 if rd is not None:
1824 t = dict(rd.get('headers', {}))
1825 if t:
1826 print(
1827 'Successfully deleted {0}/{1} in {2} attempts '
1828 '(transaction id: {3})'.format(c, o, a, t)
1829 )
1830 else:
1831 print(
1832 'Successfully deleted {0}/{1} in {2} '
1833 'attempts'.format(c, o, a)
1834 )
1835 Copy
1838 Copy can be called to copy an object or update the metadata on the
1839 given items.
1840 Each element of the object list may be a plain string of the object
1842 name, or a SwiftCopyObject that allows finer control over the options
1843 applied to each of the individual copy operations (destination,
1844 fresh_metadata, options).
1845 Destination should be in format /container/object; if not set, the ob‐
1847 ject will be copied onto itself. Fresh_metadata sets mode of operation
1848 on metadata. If not set, current object user metadata will be
1849 copied/preserved; if set, all current user metadata will be removed.
1850 Returns an iterator over the results generated for each object copy
1852 (and may also include the results of creating destination containers).
1853 When a string is given for the object name, destination and fresh meta‐
1855 data will default to None and None, which result in adding metadata to
1856 existing objects.
1857 Successful copy results are dictionaries as described below:
1859 {
1861 'action': 'copy_object',
1862 'success': True,
1863 'container': <container>,
1864 'object': <object>,
1865 'destination': <destination>,
1866 'headers': {},
1867 'fresh_metadata': <boolean>,
1868 'response_dict': <HTTP response details>
1869 }
1870 Any failure in a copy operation will return a failure dictionary as de‐
1872 scribed below:
1873 {
1875 'action': 'copy_object',
1876 'success': False,
1877 'container': <container>,
1878 'object': <object>,
1879 'destination': <destination>,
1880 'headers': {},
1881 'fresh_metadata': <boolean>,
1882 'response_dict': <HTTP response details>,
1883 'error': <error>,
1884 'traceback': <traceback>,
1885 'error_timestamp': <timestamp>
1886 }
1887 Example
1889 The code below demonstrates the use of copy to add new user metadata
1890 for objects a and b, and to copy object c to d (with added metadata).
1891 import logging
1893 from swiftclient.service import SwiftService, SwiftCopyObject, SwiftError
1895 logging.basicConfig(level=logging.ERROR)
1897 logging.getLogger("requests").setLevel(logging.CRITICAL)
1898 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1899 logger = logging.getLogger(__name__)
1900 with SwiftService() as swift:
1902 try:
1903 obj = SwiftCopyObject("c", {"destination": "/cont/d"})
1904 for i in swift.copy(
1905 "cont", ["a", "b", obj],
1906 {"meta": ["foo:bar"], "destination": "/cc"}):
1907 if i["success"]:
1908 if i["action"] == "copy_object":
1909 print(
1910 "object %s copied from /%s/%s" %
1911 (i["destination"], i["container"], i["object"])
1912 )
1913 elif i["action"] == "create_container":
1914 print(
1915 "container %s created" % i["container"]
1916 )
1917 else:
1918 if "error" in i and isinstance(i["error"], Exception):
1919 raise i["error"]
1920 except SwiftError as e:
1921 logger.error(e.value)
1922 Capabilities
1925 Capabilities can be called against an account or a particular proxy URL
1926 in order to determine the capabilities of the swift cluster. These ca‐
1927 pabilities include details about configuration options and the middle‐
1928 wares that are installed in the proxy pipeline.
1929 See swiftclient.service.SwiftService.capabilities for docs generated
1931 from the method docstring.
1932 For each successful call to list capabilities, a result dictionary will
1934 be returned with the contents described below:
1935 {
1937 'action': 'capabilities',
1938 'timestamp': <time of the call>,
1939 'success': True,
1940 'capabilities': <dictionary containing capability details>
1941 }
1942 The contents of the capabilities dictionary contain the core swift ca‐
1944 pabilities under the key swift; all other keys show the configuration
1945 options for additional middlewares deployed in the proxy pipeline. An
1946 example capabilities dictionary is given below:
1947 {
1949 'account_quotas': {},
1950 'bulk_delete': {
1951 'max_deletes_per_request': 10000,
1952 'max_failed_deletes': 1000
1953 },
1954 'bulk_upload': {
1955 'max_containers_per_extraction': 10000,
1956 'max_failed_extractions': 1000
1957 },
1958 'container_quotas': {},
1959 'container_sync': {'realms': {}},
1960 'formpost': {},
1961 'keystoneauth': {},
1962 'slo': {
1963 'max_manifest_segments': 1000,
1964 'max_manifest_size': 2097152,
1965 'min_segment_size': 1048576
1966 },
1967 'swift': {
1968 'account_autocreate': True,
1969 'account_listing_limit': 10000,
1970 'allow_account_management': True,
1971 'container_listing_limit': 10000,
1972 'extra_header_count': 0,
1973 'max_account_name_length': 256,
1974 'max_container_name_length': 256,
1975 'max_file_size': 5368709122,
1976 'max_header_size': 8192,
1977 'max_meta_count': 90,
1978 'max_meta_name_length': 128,
1979 'max_meta_overall_size': 4096,
1980 'max_meta_value_length': 256,
1981 'max_object_name_length': 1024,
1982 'policies': [
1983 {'default': True, 'name': 'Policy-0'}
1984 ],
1985 'strict_cors_mode': False,
1986 'version': '2.2.2'
1987 },
1988 'tempurl': {
1989 'methods': ['GET', 'HEAD', 'PUT']
1990 }
1991 }
1992 Example
1994 The code below demonstrates the use of capabilities to determine if the
1995 Swift cluster supports static large objects, and if so, the maximum
1996 number of segments that can be described in a single manifest file,
1997 along with the size restrictions on those objects:
1998 import logging
2000 from swiftclient.exceptions import ClientException
2002 from swiftclient.service import SwiftService
2003 logging.basicConfig(level=logging.ERROR)
2005 logging.getLogger("requests").setLevel(logging.CRITICAL)
2006 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
2007 logger = logging.getLogger(__name__)
2008 with SwiftService() as swift:
2010 try:
2011 capabilities_result = swift.capabilities()
2012 capabilities = capabilities_result['capabilities']
2013 if 'slo' in capabilities:
2014 print('SLO is supported')
2015 else:
2016 print('SLO is not supported')
2017 except ClientException as e:
2018 logger.error(e.value)
2019 The swiftclient.Connection API
2022 A low level API that provides methods for authentication and methods
2023 that correspond to the individual REST API calls described in the swift
2024 documentation.
2025 For usage details see the client docs: swiftclient.client.
2027 Authentication
2029 This section covers the various combinations of kwargs required when
2030 creating an instance of the Connection object for communicating with a
2031 swift object store. The combinations of options required for each au‐
2032 thentication version are detailed below, but are just a subset of those
2033 that can be used to successfully authenticate. These are the most com‐
2034 mon and recommended combinations.
2035 Keystone Session
2037 from keystoneauth1 import session
2038 from keystoneauth1.identity import v3
2039 # Create a password auth plugin
2041 auth = v3.Password(auth_url='http://127.0.0.1:5000/v3/',
2042 username='tester',
2043 password='testing',
2044 user_domain_name='Default',
2045 project_name='Default',
2046 project_domain_name='Default')
2047 # Create session
2049 keystone_session = session.Session(auth=auth)
2050 # Create swiftclient Connection
2052 swift_conn = Connection(session=keystone_session)
2053 Keystone v3
2055 _authurl = 'http://127.0.0.1:5000/v3/'
2056 _auth_version = '3'
2057 _user = 'tester'
2058 _key = 'testing'
2059 _os_options = {
2060 'user_domain_name': 'Default',
2061 'project_domain_name': 'Default',
2062 'project_name': 'Default'
2063 }
2064 conn = Connection(
2066 authurl=_authurl,
2067 user=_user,
2068 key=_key,
2069 os_options=_os_options,
2070 auth_version=_auth_version
2071 )
2072 Keystone v2
2074 _authurl = 'http://127.0.0.1:5000/v2.0/'
2075 _auth_version = '2'
2076 _user = 'tester'
2077 _key = 'testing'
2078 _tenant_name = 'test'
2079 conn = Connection(
2081 authurl=_authurl,
2082 user=_user,
2083 key=_key,
2084 tenant_name=_tenant_name,
2085 auth_version=_auth_version
2086 )
2087 Legacy Auth
2089 _authurl = 'http://127.0.0.1:8080/'
2090 _auth_version = '1'
2091 _user = 'tester'
2092 _key = 'testing'
2093 _tenant_name = 'test'
2094 conn = Connection(
2096 authurl=_authurl,
2097 user=_user,
2098 key=_key,
2099 tenant_name=_tenant_name,
2100 auth_version=_auth_version
2101 )
2102 Examples
2104 In this section we present some simple code examples that demonstrate
2105 the usage of the Connection API. You can find full details of the op‐
2106 tions and methods available to the Connection API in the docstring gen‐
2107 erated documentation: swiftclient.client.
2108 List the available containers:
2110 resp_headers, containers = conn.get_account()
2112 print("Response headers: %s" % resp_headers)
2113 for container in containers:
2114 print(container)
2115 Create a new container:
2117 container = 'new-container'
2119 conn.put_container(container)
2120 resp_headers, containers = conn.get_account()
2121 if container in containers:
2122 print("The container was created")
2123 Create a new object with the contents of a local text file:
2125 container = 'new-container'
2127 with open('local.txt', 'r') as local:
2128 conn.put_object(
2129 container,
2130 'local_object.txt',
2131 contents=local,
2132 content_type='text/plain'
2133 )
2134 Confirm presence of the object:
2136 obj = 'local_object.txt'
2138 container = 'new-container'
2139 try:
2140 resp_headers = conn.head_object(container, obj)
2141 print('The object was successfully created')
2142 except ClientException as e:
2143 if e.http_status = '404':
2144 print('The object was not found')
2145 else:
2146 print('An error occurred checking for the existence of the object')
2147 Download the created object:
2149 obj = 'local_object.txt'
2151 container = 'new-container'
2152 resp_headers, obj_contents = conn.get_object(container, obj)
2153 with open('local_copy.txt', 'w') as local:
2154 local.write(obj_contents)
2155 Delete the created object:
2157 obj = 'local_object.txt'
2159 container = 'new-container'
2160 try:
2161 conn.delete_object(container, obj)
2162 print("Successfully deleted the object")
2163 except ClientException as e:
2164 print("Failed to delete the object with error: %s" % e)
2165
2167 swiftclient
2168 OpenStack Swift Python client binding.
2169 swiftclient.authv1
2171 Authentication plugin for keystoneauth to support v1 endpoints.
2172 Way back in the long-long ago, there was no Keystone. Swift used an
2174 auth mechanism now known as "v1", which used only HTTP headers. Auth
2175 requests and responses would look something like:
2176 > GET /auth/v1.0 HTTP/1.1
2178 > Host: <swift server>
2179 > X-Auth-User: <tenant>:<user>
2180 > X-Auth-Key: <password>
2181 >
2182 < HTTP/1.1 200 OK
2183 < X-Storage-Url: http://<swift server>/v1/<tenant account>
2184 < X-Auth-Token: <token>
2185 < X-Storage-Token: <token>
2186 <
2187 This plugin provides a way for Keystone sessions (and clients that use
2189 them, like python-openstackclient) to communicate with old auth end‐
2190 points that still use this mechanism, such as tempauth, swauth, or
2191 https://identity.api.rackspacecloud.com/v1.0
2192 class swiftclient.authv1.AccessInfoV1(auth_url, storage_url, account,
2194 username, auth_token, token_life)
2195 An object for encapsulating a raw v1 auth token.
2196 classmethod from_state(data)
2198 Deserialize the given state.
2199 Returns
2201 a new AccessInfoV1 object with the given state
2202 get_state()
2204 Serialize the current state.
2205 will_expire_soon(stale_duration)
2207 Determines if expiration is about to occur.
2208 Returns
2210 true if expiration is within the given duration
2211 class swiftclient.authv1.PasswordLoader
2213 Option handling for the v1password plugin.
2214 property available
2216 Return if the plugin is available for loading.
2217 If a plugin is missing dependencies or for some other
2219 reason should not be available to the current system it
2220 should override this property and return False to exclude
2221 itself from the plugin list.
2222 Return type
2224 bool
2225 create_plugin(**kwargs)
2227 Create a plugin from the options available for the
2228 loader.
2229 Given the options that were specified by the loader cre‐
2231 ate an appropriate plugin. You can override this function
2232 in your loader.
2233 This used to be specified by providing the plugin_class
2235 property and this is still supported, however specifying
2236 a property didn't let you choose a plugin type based upon
2237 the options that were presented.
2238 Override this function if you wish to return different
2240 plugins based on the options presented, otherwise you can
2241 simply provide the plugin_class property.
2242 Added 2.9
2244 get_options()
2246 Return the list of parameters associated with the auth
2247 plugin.
2248 This list may be used to generate CLI or config argu‐
2250 ments.
2251 load_from_options(**kwargs)
2253 Create a plugin from the arguments retrieved from get_op‐
2254 tions.
2255 A client can override this function to do argument vali‐
2257 dation or to handle differences between the registered
2258 options and what is required to create the plugin.
2259 load_from_options_getter(getter, **kwargs)
2261 Load a plugin from getter function that returns appropri‐
2262 ate values.
2263 To handle cases other than the provided CONF and CLI
2265 loading you can specify a custom loader function that
2266 will be queried for the option value. The getter is a
2267 function that takes a keystoneauth1.loading.Opt and re‐
2268 turns a value to load with.
2269 Parameters
2271 getter (callable) -- A function that returns a
2272 value for the given opt.
2273 Returns
2275 An authentication Plugin.
2276 Return type
2278 keystoneauth1.plugin.BaseAuthPlugin
2279 plugin_class
2281 alias of PasswordPlugin
2282 class swiftclient.authv1.PasswordPlugin(auth_url, username, password,
2284 project_name=None, reauthenticate=True)
2285 A plugin for authenticating with a username and password.
2286 Subclassing from BaseIdentityPlugin gets us a few niceties, like
2288 handling token invalidation and locking during authentication.
2289 Parameters
2291 • auth_url (string) -- Identity v1 endpoint for autho‐
2293 rization.
2294 • username (string) -- Username for authentication.
2296 • password (string) -- Password for authentication.
2298 • project_name (string) -- Swift account to use after au‐
2300 thentication. We use 'project_name' to be consistent
2301 with other auth plugins.
2302 • reauthenticate (string) -- Whether to allow re-authen‐
2304 tication.
2305 access_class
2307 alias of AccessInfoV1
2308 get_access(session, **kwargs)
2310 Fetch or return a current AccessInfo object.
2311 If a valid AccessInfo is present then it is returned oth‐
2313 erwise a new one will be fetched.
2314 Parameters
2316 session (keystoneauth1.session.Session) -- A ses‐
2317 sion object that can be used for communication.
2318 Raises keystoneauth1.exceptions.http.HttpError -- An er‐
2320 ror from an invalid HTTP response.
2321 Returns
2323 Valid AccessInfo
2324 Return type
2326 keystoneauth1.access.AccessInfo
2327 get_all_version_data(session, interface='public', re‐
2329 gion_name=None, service_type=None, **kwargs)
2330 Get version data for all services in the catalog.
2331 Parameters
2333 • session (keystoneauth1.session.Session) -- A
2335 session object that can be used for communica‐
2336 tion.
2337 • interface -- Type of endpoint to get version
2339 data for. Can be a single value or a list of
2340 values. A value of None indicates that all in‐
2341 terfaces should be queried. (optional, defaults
2342 to public)
2343 • region_name (string) -- Region of endpoints to
2345 get version data for. A valueof None indicates
2346 that all regions should be queried. (optional,
2347 defaults to None)
2348 • service_type (string) -- Limit the version data
2350 to a single service. (optional, defaults to
2351 None)
2352 Returns
2354 A dictionary keyed by region_name with values con‐
2355 taining dictionaries keyed by interface with val‐
2356 ues being a list of VersionData.
2357 get_api_major_version(session, service_type=None, inter‐
2359 face=None, region_name=None, service_name=None, version=None,
2360 allow=None, allow_version_hack=True, skip_discovery=False, dis‐
2361 cover_versions=False, min_version=None, max_version=None,
2362 **kwargs)
2363 Return the major API version for a service.
2364 If a valid token is not present then a new one will be
2366 fetched using the session and kwargs.
2367 version, min_version and max_version can all be given ei‐
2369 ther as a string or a tuple.
2370 Valid interface types: public or publicURL,
2372 internal or internalURL, admin or 'adminURL`
2373 Parameters
2375 • session (keystoneauth1.session.Session) -- A
2377 session object that can be used for communica‐
2378 tion.
2379 • service_type (string) -- The type of service to
2381 lookup the endpoint for. This plugin will return
2382 None (failure) if service_type is not provided.
2383 • interface -- Type of endpoint. Can be a single
2385 value or a list of values. If it's a list of
2386 values, they will be looked for in order of
2387 preference. Can also be key‐
2388 stoneauth1.plugin.AUTH_INTERFACE to indicate
2389 that the auth_url should be used instead of the
2390 value in the catalog. (optional, defaults to
2391 public)
2392 • region_name (string) -- The region the endpoint
2394 should exist in. (optional)
2395 • service_name (string) -- The name of the service
2397 in the catalog. (optional)
2398 • version -- The minimum version number required
2400 for this endpoint. (optional)
2401 • allow (dict) -- Extra filters to pass when dis‐
2403 covering API versions. (optional)
2404 • allow_version_hack (bool) -- Allow keystoneauth
2406 to hack up catalog URLS to support older
2407 schemes. (optional, default True)
2408 • skip_discovery (bool) -- Whether to skip version
2410 discovery even if a version has been given. This
2411 is useful if endpoint_override or similar has
2412 been given and grabbing additional information
2413 about the endpoint is not useful.
2414 • discover_versions (bool) -- Whether to get ver‐
2416 sion metadata from the version discovery docu‐
2417 ment even if it's not neccessary to fulfill the
2418 major version request. Defaults to False because
2419 get_endpoint doesn't need metadata. (optional,
2420 defaults to False)
2421 • min_version -- The minimum version that is ac‐
2423 ceptable. Mutually exclusive with version. If
2424 min_version is given with no max_version it is
2425 as if max version is 'latest'. (optional)
2426 • max_version -- The maximum version that is ac‐
2428 ceptable. Mutually exclusive with version. If
2429 min_version is given with no max_version it is
2430 as if max version is 'latest'. (optional)
2431 Raises keystoneauth1.exceptions.http.HttpError -- An er‐
2433 ror from an invalid HTTP response.
2434 Returns
2436 The major version of the API of the service dis‐
2437 covered.
2438 Return type
2440 tuple or None
2441 NOTE:
2443 Implementation notes follow. Users should not need to
2444 wrap their head around these implementation notes.
2445 get_api_major_version should do what is expected with
2446 the least possible cost while still consistently re‐
2447 turning a value if possible.
2448 There are many cases when major version can be satisfied
2450 without actually calling the discovery endpoint (like
2451 when the version is in the url). If the user has a cloud
2452 with the versioned endpoint https://volume.example.com/v3
2453 in the catalog for the block-storage service and they do:
2454 client = adapter.Adapter(
2456 session, service_type='block-storage', min_version=2,
2457 max_version=3)
2458 volume_version = client.get_api_major_version()
2459 The version actually be returned with no api calls other
2461 than getting the token. For that reason,
2462 get_api_major_version() first calls get_endpoint_data()
2463 with discover_versions=False.
2464 If their catalog has an unversioned endpoint https://vol‐
2466 ume.example.com for the block-storage service and they do
2467 this:
2468 client = adapter.Adapter(session, service_type='block-storage')
2470 client is now set up to "use whatever is in the catalog".
2472 Since the url doesn't have a version, get_endpoint_data()
2473 with discover_versions=False will result in api_ver‐
2474 sion=None. (No version was requested so it didn't need
2475 to do the round trip)
2476 In order to find out what version the endpoint actually
2478 is, we must make a round trip. Therefore, if api_version
2479 is None after the first call, get_api_major_version()
2480 will make a second call to get_endpoint_data() with dis‐
2481 cover_versions=True.
2482 get_auth_ref(session, **kwargs)
2484 Obtain a token from a v1 endpoint.
2485 This function should not be called independently and is
2487 expected to be invoked via the do_authenticate function.
2488 This function will be invoked if the AcessInfo object
2490 cached by the plugin is not valid. Thus plugins should
2491 always fetch a new AccessInfo when invoked. If you are
2492 looking to just retrieve the current auth data then you
2493 should use get_access.
2494 Parameters
2496 session -- A session object that can be used for
2497 communication.
2498 Returns
2500 Token access information.
2501 get_auth_state()
2503 Retrieve the current authentication state for the plugin.
2504 Returns
2506 raw python data (which can be JSON serialized)
2507 that can be moved into another plugin (of the same
2508 type) to have the same authenticated state.
2509 get_cache_id()
2511 Fetch an identifier that uniquely identifies the auth op‐
2512 tions.
2513 The returned identifier need not be decomposable or oth‐
2515 erwise provide any way to recreate the plugin.
2516 This string MUST change if any of the parameters that are
2518 used to uniquely identity this plugin change. It should
2519 not change upon a reauthentication of the plugin.
2520 Returns
2522 A unique string for the set of options
2523 Return type
2525 str or None if this is unsupported or unavailable.
2526 get_cache_id_elements()
2528 Get the elements for this auth plugin that make it
2529 unique.
2530 get_connection_params(session, **kwargs)
2532 Return any additional connection parameters required for
2533 the plugin.
2534 Parameters
2536 session (keystoneauth1.session.Session) -- The
2537 session object that the auth_plugin belongs to.
2538 Returns
2540 Headers that are set to authenticate a message or
2541 None for failure. Note that when checking this
2542 value that the empty dict is a valid, non-failure
2543 response.
2544 Return type
2546 dict
2547 get_discovery(*args, **kwargs)
2549 Return the discovery object for a URL.
2550 Check the session and the plugin cache to see if we have
2552 already performed discovery on the URL and if so return
2553 it, otherwise create a new discovery object, cache it and
2554 return it.
2555 This function is expected to be used by subclasses and
2557 should not be needed by users.
2558 Parameters
2560 • session (keystoneauth1.session.Session) -- A
2562 session object to discover with.
2563 • url (str) -- The url to lookup.
2565 • authenticated (bool) -- Include a token in the
2567 discovery call. (optional) Defaults to None
2568 (use a token if a plugin is installed).
2569 Raises
2571 • keystoneauth1.exceptions.discovery.Discovery‐
2573 Failure -- if for some reason the lookup fails.
2574 • keystoneauth1.exceptions.http.HttpError -- An
2576 error from an invalid HTTP response.
2577 Returns
2579 A discovery object with the results of looking up
2580 that URL.
2581 get_endpoint(session, interface='public', **kwargs)
2583 Return an endpoint for the client.
2584 get_endpoint_data(session, service_type=None, interface=None,
2586 region_name=None, service_name=None, allow=None, allow_ver‐
2587 sion_hack=True, discover_versions=True, skip_discovery=False,
2588 min_version=None, max_version=None, endpoint_override=None,
2589 **kwargs)
2590 Return a valid endpoint data for a service.
2591 If a valid token is not present then a new one will be
2593 fetched using the session and kwargs.
2594 version, min_version and max_version can all be given ei‐
2596 ther as a string or a tuple.
2597 Valid interface types: public or publicURL,
2599 internal or internalURL, admin or 'adminURL`
2600 Parameters
2602 • session (keystoneauth1.session.Session) -- A
2604 session object that can be used for communica‐
2605 tion.
2606 • service_type (string) -- The type of service to
2608 lookup the endpoint for. This plugin will return
2609 None (failure) if service_type is not provided.
2610 • interface -- Type of endpoint. Can be a single
2612 value or a list of values. If it's a list of
2613 values, they will be looked for in order of
2614 preference. Can also be key‐
2615 stoneauth1.plugin.AUTH_INTERFACE to indicate
2616 that the auth_url should be used instead of the
2617 value in the catalog. (optional, defaults to
2618 public)
2619 • region_name (string) -- The region the endpoint
2621 should exist in. (optional)
2622 • service_name (string) -- The name of the service
2624 in the catalog. (optional)
2625 • allow (dict) -- Extra filters to pass when dis‐
2627 covering API versions. (optional)
2628 • allow_version_hack (bool) -- Allow keystoneauth
2630 to hack up catalog URLS to support older
2631 schemes. (optional, default True)
2632 • discover_versions (bool) -- Whether to get ver‐
2634 sion metadata from the version discovery docu‐
2635 ment even if it's not neccessary to fulfill the
2636 major version request. (optional, defaults to
2637 True)
2638 • skip_discovery (bool) -- Whether to skip version
2640 discovery even if a version has been given. This
2641 is useful if endpoint_override or similar has
2642 been given and grabbing additional information
2643 about the endpoint is not useful.
2644 • min_version -- The minimum version that is ac‐
2646 ceptable. Mutually exclusive with version. If
2647 min_version is given with no max_version it is
2648 as if max version is 'latest'. (optional)
2649 • max_version -- The maximum version that is ac‐
2651 ceptable. Mutually exclusive with version. If
2652 min_version is given with no max_version it is
2653 as if max version is 'latest'. (optional)
2654 • endpoint_override (str) -- URL to use instead of
2656 looking in the catalog. Catalog lookup will be
2657 skipped, but version discovery will be run.
2658 Sets allow_version_hack to False (optional)
2659 • kwargs -- Ignored.
2661 Raises keystoneauth1.exceptions.http.HttpError -- An er‐
2663 ror from an invalid HTTP response.
2664 Returns
2666 Valid EndpointData or None if not available.
2667 Return type
2669 keystoneauth1.discover.EndpointData or None
2670 get_headers(session, **kwargs)
2672 Fetch authentication headers for message.
2673 This is a more generalized replacement of the older
2675 get_token to allow plugins to specify different or addi‐
2676 tional authentication headers to the OpenStack standard
2677 'X-Auth-Token' header.
2678 How the authentication headers are obtained is up to the
2680 plugin. If the headers are still valid they may be
2681 re-used, retrieved from cache or the plugin may invoke an
2682 authentication request against a server.
2683 The default implementation of get_headers calls the
2685 get_token method to enable older style plugins to con‐
2686 tinue functioning unchanged. Subclasses should feel free
2687 to completely override this function to provide the head‐
2688 ers that they want.
2689 There are no required kwargs. They are passed directly to
2691 the auth plugin and they are implementation specific.
2692 Returning None will indicate that no token was able to be
2694 retrieved and that authorization was a failure. Adding no
2695 authentication data can be achieved by returning an empty
2696 dictionary.
2697 Parameters
2699 session (keystoneauth1.session.Session) -- The
2700 session object that the auth_plugin belongs to.
2701 Returns
2703 Headers that are set to authenticate a message or
2704 None for failure. Note that when checking this
2705 value that the empty dict is a valid, non-failure
2706 response.
2707 Return type
2709 dict
2710 get_project_id(session, **kwargs)
2712 Return the project id that we are authenticated to.
2713 Wherever possible the project id should be inferred from
2715 the token however there are certain URLs and other places
2716 that require access to the currently authenticated
2717 project id.
2718 Parameters
2720 session (keystoneauth1.session.Session) -- A ses‐
2721 sion object so the plugin can make HTTP calls.
2722 Returns
2724 A project identifier or None if one is not avail‐
2725 able.
2726 Return type
2728 str
2729 get_sp_auth_url(*args, **kwargs)
2731 Return auth_url from the Service Provider object.
2732 This url is used for obtaining unscoped federated token
2734 from remote cloud.
2735 Parameters
2737 sp_id (string) -- ID of the Service Provider to be
2738 queried.
2739 Returns
2741 A Service Provider auth_url or None if one is not
2742 available.
2743 Return type
2745 str
2746 get_sp_url(*args, **kwargs)
2748 Return sp_url from the Service Provider object.
2749 This url is used for passing SAML2 assertion to the re‐
2751 mote cloud.
2752 Parameters
2754 sp_id (str) -- ID of the Service Provider to be
2755 queried.
2756 Returns
2758 A Service Provider sp_url or None if one is not
2759 available.
2760 Return type
2762 str
2763 get_token(session, **kwargs)
2765 Return a valid auth token.
2766 If a valid token is not present then a new one will be
2768 fetched.
2769 Parameters
2771 session (keystoneauth1.session.Session) -- A ses‐
2772 sion object that can be used for communication.
2773 Raises keystoneauth1.exceptions.http.HttpError -- An er‐
2775 ror from an invalid HTTP response.
2776 Returns
2778 A valid token.
2779 Return type
2781 string
2782 get_user_id(session, **kwargs)
2784 Return a unique user identifier of the plugin.
2785 Wherever possible the user id should be inferred from the
2787 token however there are certain URLs and other places
2788 that require access to the currently authenticated user
2789 id.
2790 Parameters
2792 session (keystoneauth1.session.Session) -- A ses‐
2793 sion object so the plugin can make HTTP calls.
2794 Returns
2796 A user identifier or None if one is not available.
2797 Return type
2799 str
2800 invalidate()
2802 Invalidate the current authentication data.
2803 This should result in fetching a new token on next call.
2805 A plugin may be invalidated if an Unauthorized HTTP re‐
2807 sponse is returned to indicate that the token may have
2808 been revoked or is otherwise now invalid.
2809 Returns
2811 True if there was something that the plugin did to
2812 invalidate. This means that it makes sense to try
2813 again. If nothing happens returns False to indi‐
2814 cate give up.
2815 Return type
2817 bool
2818 set_auth_state(data)
2820 Install existing authentication state for a plugin.
2821 Take the output of get_auth_state and install that au‐
2823 thentication state into the current authentication
2824 plugin.
2825 swiftclient.client
2827 OpenStack Swift client library used internally
2828 swiftclient.service
2830 swiftclient.exceptions
2831 swiftclient.multithreading
2832 swiftclient.utils
2833 Miscellaneous utility functions for use with Swift.
2834
2836 • Index
2837 • Module Index
2839 • Search Page
2841
2843 Copyright 2013 OpenStack, LLC.
2844 Licensed under the Apache License, Version 2.0 (the "License"); you may
2846 not use this file except in compliance with the License. You may ob‐
2847 tain a copy of the License at
2848 • http://www.apache.org/licenses/LICENSE-2.0
2850 Unless required by applicable law or agreed to in writing, software
2852 distributed under the License is distributed on an "AS IS" BASIS, WITH‐
2853 OUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
2854 See the License for the specific language governing permissions and
2855 limitations under the License.
2856
2858 unknown
2859
2861 2013-2023 OpenStack, LLC.
28624.1.0 Jan 20, 2023 PYTHON-SWIFTCLIENT(1)