1PYTHON-SWIFTCLIENT(1) python-swiftclient PYTHON-SWIFTCLIENT(1)
2
6 python-swiftclient - python-swiftclient 3.5.0
7
9 Introduction
10 Where to Start?
11 The python-swiftclient project comprises a command line tool and two
12 separate APIs for accessing swift programmatically. Choosing the most
13 appropriate method for a given use case is the first problem a user
14 needs to solve.
15 Use Cases
17 Alongside the command line tool, the python-swiftclient includes two
18 levels of API:
19 · A low level client API that provides simple Python wrappers around
21 the various authentication mechanisms and the individual HTTP
22 requests.
23 · A high level service API that provides methods for performing com‐
25 mon operations in parallel on a thread pool.
26 Example use cases:
28 ·
30 Uploading and retrieving data
32 Use the command line tool if you are simply uploading and
33 downloading files and directories to and from your filesys‐
34 tem. The command line tool can be integrated into a shell
35 script to automate tasks.
36 ·
38 Integrating into an automated Python workflow
40 Use the SwiftService API to perform operations offered by
41 the CLI if your use case requires integration with a
42 Python-based workflow. This method offers greater control
43 and flexibility over individual object operations, such as
44 the metadata set on each object. The SwiftService class
45 provides methods to perform multiple sets of operations
46 against a swift object store using a configurable shared
47 thread pool. A single instance of the SwiftService class
48 can be shared between multiple threads in your own code.
49 ·
51 Developing an application in Python to access a swift object store
53 Use the SwiftService API to develop Python applications
54 that use swift to store and retrieve objects. A SwiftSer‐
55 vice instance provides a configurable thread pool for per‐
56 forming all operations supported by the CLI.
57 ·
59 Fine-grained control over threading or the requests being per‐
61 formed
62 Use the Connection API if your use case requires fine
63 grained control over advanced features or you wish to use
64 your own existing threading model. Examples of advanced
65 features requiring the use of the Connection API include
66 creating an SLO manifest that references already existing
67 objects, or fine grained control over the query strings
68 supplied with each HTTP request.
69 Important considerations
71 This section covers some important considerations, helpful hints, and
72 things to avoid when integrating an object store into your workflow.
73 An object store is not a filesystem
75 It cannot be stressed enough that your usage of the object store should
76 reflect the proper use case, and not treat the storage like a tradi‐
77 tional filesystem. There are two main restrictions to bear in mind
78 when designing an application that uses an object store:
79 · You cannot rename objects. Due to fact that the name of an object
81 is one of the factors that determines where the object and its
82 replicas are stored, renaming would require multiple copies of the
83 data to be moved between physical storage devices. If you want to
84 rename an object you must upload to the new location, or make a
85 server side copy request to the new location, and then delete the
86 original.
87 · You cannot modify objects. Objects are stored in multiple loca‐
89 tions and are checked for integrity based on the MD5 sum calcu‐
90 lated during upload. In order to modify the contents of an object,
91 the entire desired contents must be re-uploaded. In certain spe‐
92 cial cases it is possible to work around this restriction using
93 large objects, but no general file-like access is available to
94 modify a stored object.
95 Objects cannot be locked
97 There is no mechanism to perform a combination of reading the
98 data/metadata from an object and writing an update to that data/meta‐
99 data in an atomic way. Any user with access to a container could update
100 the contents or metadata associated with an object at any time.
101 Workflows that assume that no updates have been made since the last
103 read of an object should be discouraged. Enabling a workflow of this
104 type requires an external object locking mechanism and/or cooperation
105 between all clients accessing the data.
106
108 CLI
109 The swift tool is a command line utility for communicating with an
110 OpenStack Object Storage (swift) environment. It allows one to perform
111 several types of operations.
112 For help on a specific swift command, enter:
114 $ swift COMMAND --help
116 swift usage
118 Usage: swift [--version] [--help] [--os-help] [--snet] [--verbose]
119 [--debug] [--info] [--quiet] [--auth <auth_url>]
120 [--auth-version <auth_version> |
121 --os-identity-api-version <auth_version> ]
122 [--user <username>]
123 [--key <api_key>] [--retries <num_retries>]
124 [--os-username <auth-user-name>] [--os-password <auth-password>]
125 [--os-user-id <auth-user-id>]
126 [--os-user-domain-id <auth-user-domain-id>]
127 [--os-user-domain-name <auth-user-domain-name>]
128 [--os-tenant-id <auth-tenant-id>]
129 [--os-tenant-name <auth-tenant-name>]
130 [--os-project-id <auth-project-id>]
131 [--os-project-name <auth-project-name>]
132 [--os-project-domain-id <auth-project-domain-id>]
133 [--os-project-domain-name <auth-project-domain-name>]
134 [--os-auth-url <auth-url>] [--os-auth-token <auth-token>]
135 [--os-storage-url <storage-url>] [--os-region-name <region-name>]
136 [--os-service-type <service-type>]
137 [--os-endpoint-type <endpoint-type>]
138 [--os-cacert <ca-certificate>] [--insecure]
139 [--os-cert <client-certificate-file>]
140 [--os-key <client-certificate-key-file>]
141 [--no-ssl-compression]
142 <subcommand> [--help] [<subcommand options>]
143 Subcommands:
145 delete Delete a container or objects within a container.
147 download
149 Download objects from containers.
150 list Lists the containers for the account or the objects for a con‐
152 tainer.
153 post Updates meta information for the account, container, or object;
155 creates containers if not present.
156 copy Copies object, optionally adds meta
158 stat Displays information for the account, container, or object.
160 upload Uploads files or directories to the given container.
162 capabilities
164 List cluster capabilities.
165 tempurl
167 Create a temporary URL.
168 auth Display auth related environment variables.
170 swift optional arguments
172 --version
173 show program’s version number and exit
174 -h, --help
176 show this help message and exit
177 --os-help
179 Show OpenStack authentication options.
180 -s, --snet
182 Use SERVICENET internal network.
183 -v, --verbose
185 Print more info.
186 --debug
188 Show the curl commands and results of all http queries regard‐
189 less of result status.
190 --info Show the curl commands and results of all http queries which
192 return an error.
193 -q, --quiet
195 Suppress status output.
196 -A AUTH, --auth=AUTH
198 URL for obtaining an auth token.
199 -V AUTH_VERSION, --auth-version=AUTH_VERSION, --os-identity-api-ver‐
201 sion=AUTH_VERSION
202 Specify a version for authentication. Defaults to
203 env[ST_AUTH_VERSION], env[OS_AUTH_VERSION], env[OS_IDEN‐
204 TITY_API_VERSION] or 1.0.
205 -U USER, --user=USER
207 User name for obtaining an auth token.
208 -K KEY, --key=KEY
210 Key for obtaining an auth token.
211 -R RETRIES, --retries=RETRIES
213 The number of times to retry a failed connection.
214 --insecure
216 Allow swiftclient to access servers without having to verify the
217 SSL certificate. Defaults to env[SWIFTCLIENT_INSECURE] (set to
218 ‘true’ to enable).
219 --no-ssl-compression
221 This option is deprecated and not used anymore. SSL compression
222 should be disabled by default by the system SSL library.
223 Authentication
225 This section covers the options for authenticating with a swift object
226 store. The combinations of options required for each authentication
227 version are detailed below, but are just a subset of those that can be
228 used to successfully authenticate. These are the most common and recom‐
229 mended combinations.
230 You should obtain the details of your authentication version and cre‐
232 dentials from your storage provider. These details should make it
233 clearer which of the authentication sections below are most likely to
234 allow you to connect to your storage account.
235 Keystone v3
237 swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \
238 --os-project-name project1 --os-project-domain-name domain1 \
239 --os-username user --os-user-domain-name domain1 \
240 --os-password password list
241 swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \
243 --os-project-id 0123456789abcdef0123456789abcdef \
244 --os-user-id abcdef0123456789abcdef0123456789 \
245 --os-password password list
246 Manually specifying the options above on the command line can be
248 avoided by setting the following combinations of environment variables:
249 ST_AUTH_VERSION=3
251 OS_USERNAME=user
252 OS_USER_DOMAIN_NAME=domain1
253 OS_PASSWORD=password
254 OS_PROJECT_NAME=project1
255 OS_PROJECT_DOMAIN_NAME=domain1
256 OS_AUTH_URL=https://api.example.com:5000/v3
257 ST_AUTH_VERSION=3
259 OS_USER_ID=abcdef0123456789abcdef0123456789
260 OS_PASSWORD=password
261 OS_PROJECT_ID=0123456789abcdef0123456789abcdef
262 OS_AUTH_URL=https://api.example.com:5000/v3
263 Keystone v2
265 swift --os-auth-url https://api.example.com:5000/v2.0 \
266 --os-tenant-name tenant \
267 --os-username user --os-password password list
268 Manually specifying the options above on the command line can be
270 avoided by setting the following environment variables:
271 ST_AUTH_VERSION=2.0
273 OS_USERNAME=user
274 OS_PASSWORD=password
275 OS_TENANT_NAME=tenant
276 OS_AUTH_URL=https://api.example.com:5000/v2.0
277 Legacy auth systems
279 You can configure swift to work with any number of other authentication
280 systems that we will not cover in this document. If your storage
281 provider is not using Keystone to provide access tokens, please contact
282 them for instructions on the required options. It is likely that the
283 options will need to be specified as below:
284 swift -A https://api.example.com/v1.0 -U user -K api_key list
286 Specifying the options above manually on the command line can be
288 avoided by setting the following environment variables:
289 ST_AUTH_VERSION=1.0
291 ST_AUTH=https://api.example.com/v1.0
292 ST_USER=user
293 ST_KEY=key
294 It is also possible that you need to use a completely separate auth
296 system, in which case swiftclient cannot request a token for you. In
297 this case you should make the authentication request separately and
298 access your storage using the token and storage URL options shown
299 below:
300 swift --os-auth-token 6ee5eb33efad4e45ab46806eac010566 \
302 --os-storage-url https://10.1.5.2:8080/v1/AUTH_ced809b6a4baea7aeab61a \
303 list
304 NOTE:
308 Leftover environment variables are a common source of confusion
309 when authorization fails.
310 CLI commands
312 Auth
313 Usage: swift auth
314 Display authentication variables in shell friendly format. Command to
316 run to export storage URL and auth token into OS_STORAGE_URL and
317 OS_AUTH_TOKEN: swift auth. Command to append to a runcom file (e.g.
318 ~/.bashrc, /etc/profile) for automatic authentication: swift auth -v -U
319 test:tester -K testing.
320 swift stat
322 Usage: swift stat [--lh] [--header <header:value>]
323 [<container> [<object>]]
324 Displays information for the account, container, or object depending on
326 the arguments given (if any). In verbose mode, the storage URL and the
327 authentication token are displayed as well.
328 Positional arguments:
330 [container]
332 Name of container to stat from.
333 [object]
335 Name of object to stat.
336 Optional arguments:
338 --lh Report sizes in human readable format similar to ls -lh.
340 -H, --header <header:value>
342 Adds a custom request header to use for stat.
343 swift list
345 Usage: swift list [--long] [--lh] [--totals] [--prefix <prefix>]
346 [--delimiter <delimiter>] [--header <header:value>]
347 [<container>]
348 Lists the containers for the account or the objects for a container.
350 The -p <prefix> or --prefix <prefix> is an option that will only list
351 items beginning with that prefix. The -d <delimiter> or --delimiter
352 <delimiter> is an option (for container listings only) that will roll
353 up items with the given delimiter (see OpenStack Swift general documen‐
354 tation <http://docs.openstack.org/swift/latest/> for what this means).
355 The -l and --lh options provide more detail, similar to ls -l and ls
357 -lh, the latter providing sizes in human readable format (For example:
358 3K, 12M, etc). The latter two switches use more overhead to retrieve
359 the displayed details, which is directly proportional to the number of
360 container or objects listed.
361 Positional arguments:
363 [container]
365 Name of container to list object in.
366 Optional arguments:
368 -l, --long
370 Long listing format, similar to ls -l.
371 --lh Report sizes in human readable format similar to ls -lh.
373 -t, --totals
375 Used with -l or –lh, only report totals.
376 -p <prefix>, --prefix <prefix>
378 Only list items beginning with the prefix.
379 -d <delim>, --delimiter <delim>
381 Roll up items with the given delimiter. For containers only. See
382 OpenStack Swift API documentation for what this means.
383 -H, --header <header:value>
385 Adds a custom request header to use for listing.
386 swift upload
388 Usage: swift upload [--changed] [--skip-identical] [--segment-size <size>]
389 [--segment-container <container>] [--leave-segments]
390 [--object-threads <thread>] [--segment-threads <threads>]
391 [--header <header>] [--use-slo] [--ignore-checksum]
392 [--object-name <object-name>]
393 <container> <file_or_directory> [<file_or_directory>] [...]
394 Uploads the files and directories specified by the remaining arguments
396 to the given container. The -c or --changed is an option that will only
397 upload files that have changed since the last upload. The --object-name
398 <object-name> is an option that will upload a file and name object to
399 <object-name> or upload a directory and use <object-name> as object
400 prefix. If the file name is “-“, client reads content from standard
401 input. In this case --object-name is required to set the name of the
402 object and no other files may be given. The -S <size> or --seg‐
403 ment-size <size> and --leave-segments are options as well (see --help
404 for more).
405 Positional arguments:
407 <container>
409 Name of container to upload to.
410 <file_or_directory>
412 Name of file or directory to upload. Specify multiple times for
413 multiple uploads.
414 Optional arguments:
416 -c, --changed
418 Only upload files that have changed since the last upload.
419 --skip-identical
421 Skip uploading files that are identical on both sides.
422 -S, --segment-size <size>
424 Upload files in segments no larger than <size> (in Bytes) and
425 then create a “manifest” file that will download all the seg‐
426 ments as if it were the original file.
427 --segment-container <container>
429 Upload the segments into the specified container. If not speci‐
430 fied, the segments will be uploaded to a <container>_segments
431 container to not pollute the main <container> listings.
432 --leave-segments
434 Indicates that you want the older segments of manifest objects
435 left alone (in the case of overwrites).
436 --object-threads <threads>
438 Number of threads to use for uploading full objects. Default is
439 10.
440 --segment-threads <threads>
442 Number of threads to use for uploading object segments. Default
443 is 10.
444 -H, --header <header:value>
446 Adds a customized request header. This option may be repeated.
447 Example: -H “content-type:text/plain” -H “Content-Length: 4000”.
448 --use-slo
450 When used in conjunction with –segment-size it will create a
451 Static Large Object instead of the default Dynamic Large Object.
452 --object-name <object-name>
454 Upload file and name object to <object-name> or upload dir and
455 use <object-name> as object prefix instead of folder name.
456 --ignore-checksum
458 Turn off checksum validation for uploads.
459 swift post
461 Usage: swift post [--read-acl <acl>] [--write-acl <acl>] [--sync-to]
462 [--sync-key <sync-key>] [--meta <name:value>]
463 [--header <header>]
464 [<container> [<object>]]
465 Updates meta information for the account, container, or object depend‐
467 ing on the arguments given. If the container is not found, the swift‐
468 client will create it automatically, but this is not true for accounts
469 and objects. Containers also allow the -r <read-acl> (or --read-acl
470 <read-acl>) and -w <write-acl> (or --write-acl <write-acl>) options.
471 The -m or --meta option is allowed on accounts, containers and objects,
472 and is used to define the user metadata items to set in the form
473 Name:Value. You can repeat this option. For example: post -m
474 Color:Blue -m Size:Large
475 For more information about ACL formats see the documentation: ACLs.
477 Positional arguments:
479 [container]
481 Name of container to post to.
482 [object]
484 Name of object to post.
485 Optional arguments:
487 -r, --read-acl <acl>
489 Read ACL for containers. Quick summary of ACL syntax: .r:*,
490 .r:-.example.com, .r:www.example.com, account1 (v1.0 identity
491 API only), account1:*, account2:user2 (v2.0+ identity API).
492 -w, --write-acl <acl>
494 Write ACL for containers. Quick summary of ACL syntax: account1
495 (v1.0 identity API only), account1:*, account2:user2 (v2.0+
496 identity API).
497 -t, --sync-to <sync-to>
499 Sync To for containers, for multi-cluster replication.
500 -k, --sync-key <sync-key>
502 Sync Key for containers, for multi-cluster replication.
503 -m, --meta <name:value>
505 Sets a meta data item. This option may be repeated.
506 Example: -m Color:Blue -m Size:Large
508 -H, --header <header:value>
510 Adds a customized request header. This option may be repeated.
511 Example: -H “content-type:text/plain” -H “Content-Length: 4000”
513 swift download
515 Usage: swift download [--all] [--marker <marker>] [--prefix <prefix>]
516 [--output <out_file>] [--output-dir <out_directory>]
517 [--object-threads <threads>] [--ignore-checksum]
518 [--container-threads <threads>] [--no-download]
519 [--skip-identical] [--remove-prefix]
520 [--header <header:value>] [--no-shuffle]
521 [<container> [<object>] [...]]
522 Downloads everything in the account (with --all), or everything in a
524 container, or a list of objects depending on the arguments given. For a
525 single object download, you may use the -o <filename> or --output
526 <filename> option to redirect the output to a specific file or - to re‐
527 direct to stdout. The --ignore-checksum is an option that turn off
528 checksum validation. You can specify optional headers with the repeat‐
529 able cURL-like option -H [--header <name:value>]. --ignore-mtime
530 ignores the x-object-meta-mtime metadata entry on the object (if
531 present) and instead creates the downloaded files with fresh atime and
532 mtime values.
533 Positional arguments:
535 <container>
537 Name of container to download from. To download a whole account,
538 omit this and specify –all.
539 <object>
541 Name of object to download. Specify multiple times for multiple
542 objects. Omit this to download all objects from the container.
543 Optional arguments:
545 -a, --all
547 Indicates that you really want to download everything in the
548 account.
549 -m, --marker <marker>
551 Marker to use when starting a container or account download.
552 -p, --prefix <prefix>
554 Only download items beginning with <prefix>
555 -r, --remove-prefix
557 An optional flag for –prefix <prefix>, use this option to down‐
558 load items without <prefix>
559 -o, --output <out_file>
561 For a single file download, stream the output to <out_file>.
562 Specifying “-” as <out_file> will redirect to stdout.
563 -D, --output-dir <out_directory>
565 An optional directory to which to store objects. By default,
566 all objects are recreated in the current directory.
567 --object-threads <threads>
569 Number of threads to use for downloading objects. Default is
570 10.
571 --container-threads <threads>
573 Number of threads to use for downloading containers. Default is
574 10.
575 --no-download
577 Perform download(s), but don’t actually write anything to disk.
578 -H, --header <header:value>
580 Adds a customized request header to the query, like “Range” or
581 “If-Match”. This option may be repeated.
582 Example: –header “content-type:text/plain”
584 --skip-identical
586 Skip downloading files that are identical on both sides.
587 --ignore-checksum
589 Turn off checksum validation for downloads.
590 --no-shuffle
592 By default, when downloading a complete account or container,
593 download order is randomised in order to reduce the load on
594 individual drives when multiple clients are executed simultane‐
595 ously to download the same set of objects (e.g. a nightly auto‐
596 mated download script to multiple servers). Enable this option
597 to submit download jobs to the thread pool in the order they are
598 listed in the object store.
599 swift delete
601 Usage: swift delete [--all] [--leave-segments]
602 [--object-threads <threads>]
603 [--container-threads <threads>]
604 [--header <header:value>]
605 [<container> [<object>] [...]]
606 Deletes everything in the account (with --all), or everything in a con‐
608 tainer, or a list of objects depending on the arguments given. Segments
609 of manifest objects will be deleted as well, unless you specify the
610 --leave-segments option.
611 Positional arguments:
613 [<container>]
615 Name of container to delete from.
616 [<object>]
618 Name of object to delete. Specify multiple times for multiple
619 objects.
620 Optional arguments:
622 -a, --all
624 Delete all containers and objects.
625 --leave-segments
627 Do not delete segments of manifest objects.
628 -H, --header <header:value>
630 Adds a custom request header to use for deleting objects or an
631 entire container.
632 --object-threads <threads>
634 Number of threads to use for deleting objects. Default is 10.
635 --container-threads <threads>
637 Number of threads to use for deleting containers. Default is
638 10.
639 swift copy
641 Usage: swift copy [--destination </container/object>] [--fresh-metadata]
642 [--meta <name:value>] [--header <header>] <container>
643 <object> [<object>] [...]
644 Copies an object to a new destination or adds user metadata to an
646 object. Depending on the options supplied, you can preserve existing
647 metadata in contrast to the post command. The --destination option sets
648 the copy target destination in the form /container/object. If not set,
649 the object will be copied onto itself which is useful for adding meta‐
650 data. You can use the -M or --fresh-metadata option to copy an object
651 without existing user meta data, and the -m or --meta option to define
652 user meta data items to set in the form Name:Value. You can repeat this
653 option. For example: copy -m Color:Blue -m Size:Large.
654 Positional arguments:
656 <container>
658 Name of container to copy from.
659 <object>
661 Name of object to copy. Specify multiple times for multiple
662 objects
663 Optional arguments:
665 -d, --destination </container[/object]>
667 The container and name of the destination object. Name of desti‐
668 nation object can be omitted, then will be same as name of
669 source object. Supplying multiple objects and destination with
670 object name is invalid.
671 -M, --fresh-metadata
673 Copy the object without any existing metadata, If not set, meta‐
674 data will be preserved or appended
675 -m, --meta <name:value>
677 Sets a meta data item. This option may be repeated.
678 Example: -m Color:Blue -m Size:Large
680 -H, --header <header:value>
682 Adds a customized request header. This option may be repeated.
683 Example: -H “content-type:text/plain” -H “Content-Length: 4000”
685 swift capabilities
687 Usage: swift capabilities [--json] [<proxy_url>]
688 Displays cluster capabilities. The output includes the list of the
690 activated Swift middlewares as well as relevant options for each ones.
691 Additionally the command displays relevant options for the Swift core.
692 If the proxy-url option is not provided, the storage URL retrieved
693 after authentication is used as proxy-url.
694 Optional positional arguments:
696 <proxy_url>
698 Proxy URL of the cluster to retrieve capabilities.
699 --json Print the cluster capabilities in JSON format.
701 swift tempurl
703 Usage: swift tempurl [--absolute] [--prefix-based]
704 <method> <seconds> <path> <key>
705 Generates a temporary URL for a Swift object. method option sets an
707 HTTP method to allow for this temporary URL that is usually GET or PUT.
708 time option sets the amount of time the temporary URL will be valid
709 for. time can be specified as an integer, denoting the number of sec‐
710 onds from now on until the URL shall be valid; or, if --absolute is
711 passed, the Unix timestamp when the temporary URL will expire. But
712 beyond that, time can also be specified as an ISO 8601 timestamp in one
713 of following formats:
714 i. Complete date: YYYY-MM-DD (eg 1997-07-16)
716 ii. Complete date plus hours, minutes and seconds:
718 YYYY-MM-DDThh:mm:ss (eg 1997-07-16T19:20:30)
719 iii. Complete date plus hours, minutes and seconds with UTC designa‐
721 tor:
722 YYYY-MM-DDThh:mm:ssZ (eg 1997-07-16T19:20:30Z)
723 Please be aware that if you don’t provide the UTC designator (i.e., Z)
725 the timestamp is generated using your local timezone. If only a date is
726 specified, the time part used will equal to 00:00:00.
727 path option sets the full path to the Swift object. Example:
729 /v1/AUTH_account/c/o. key option is the secret temporary URL key set on
730 the Swift cluster. To set a key, run swift post -m "Temp-URL-Key: <your
731 secret key>". To generate a prefix-based temporary URL use the --pre‐
732 fix-based option. This URL will contain the path to the prefix. Do not
733 forget to append the desired objectname at the end of the path portion
734 (and before the query portion) before sharing the URL. It is possible
735 to use ISO 8601 UTC timestamps within the URL by using the --iso8601
736 option.
737 Positional arguments:
739 <method>
741 An HTTP method to allow for this temporary URL. Usually ‘GET’
742 or ‘PUT’.
743 <seconds>
745 The amount of time in seconds the temporary URL will be valid
746 for; or, if –absolute is passed, the Unix timestamp when the
747 temporary URL will expire.
748 <path> The full path to the Swift object.
750 Example: /v1/AUTH_account/c/o or:
752 http://saio:8080/v1/AUTH_account/c/o
753 <key> The secret temporary URL key set on the Swift cluster. To set a
755 key, run ‘swift post -m
756 “Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4”’
757 Optional arguments:
759 --absolute
761 Interpret the <seconds> positional argument as a Unix timestamp
762 rather than a number of seconds in the future.
763 --prefix-based
765 If present, a prefix-based tempURL will be generated.
766 Examples
768 In this section we present some example usage of the swift CLI. To keep
769 the examples as short as possible, these examples assume that the rele‐
770 vant authentication options have been set using environment variables.
771 You can obtain the full list of commands and options available in the
772 swift CLI by executing the following:
773 > swift --help
775 > swift <command> --help
776 Simple examples
778 List the existing swift containers:
779 > swift list
781 container_1
783 Create a new container:
785 > swift post TestContainer
787 Upload an object into a container:
789 > swift upload TestContainer testSwift.txt
791 testSwift.txt
793 List the contents of a container:
795 > swift list TestContainer
797 testSwift.txt
799 Copy an object to new destination:
801 > swift copy -d /DestContainer/testSwift.txt SourceContainer testSwift.txt
803 SourceContainer/testSwift.txt copied to /DestContainer/testSwift.txt
805 Delete an object from a container:
807 > swift delete TestContainer testSwift.txt
809 testSwift.txt
811 Delete a container:
813 > swift delete TestContainer
815 TestContainer
817 Display auth related authentication variables in shell friendly format:
819 > swift auth
821 export OS_STORAGE_URL=http://127.0.0.1:8080/v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7
823 export OS_AUTH_TOKEN=c597015ae19943a18438b52ef3762e79
824 Download an object from a container:
826 > swift download TestContainer testSwift.txt
828 testSwift.txt [auth 0.028s, headers 0.045s, total 0.045s, 0.002 MB/s]
830 NOTE:
834 To upload an object to a container, your current working direc‐
835 tory must be where the file is located or you must provide the
836 complete path to the file. In other words, the –object-name
837 <object-name> is an option that will upload file and name object
838 to <object-name> or upload directory and use <object-name> as
839 object prefix. In the case that you provide the complete path of
840 the file, that complete path will be the name of the uploaded
841 object.
842 For example:
844 > swift upload TestContainer /home/swift/testSwift/testSwift.txt
846 home/swift/testSwift/testSwift.txt
848 > swift list TestContainer
850 home/swift/testSwift/testSwift.txt
852 More complex examples
854 Swift has a single object size limit of 5GiB. In order to upload files
855 larger than this, we must create a large object that consists of
856 smaller segments. The example below shows how to upload a large video
857 file as a static large object in 1GiB segments:
858 > swift upload videos --use-slo --segment-size 1G myvideo.mp4
860 myvideo.mp4 segment 8
862 myvideo.mp4 segment 4
863 myvideo.mp4 segment 2
864 myvideo.mp4 segment 7
865 myvideo.mp4 segment 0
866 myvideo.mp4 segment 1
867 myvideo.mp4 segment 3
868 myvideo.mp4 segment 6
869 myvideo.mp4 segment 5
870 myvideo.mp4
871 This command will upload segments to a container named videos_segments,
873 and create a manifest file describing the entire object in the videos
874 container. For more information on large objects, see the documenta‐
875 tion here.
876 > swift list videos
878 myvideo.mp4
880 > swift list videos_segments
882 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000000
884 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000001
885 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000002
886 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000003
887 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000004
888 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000005
889 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000006
890 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000007
891 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000008
892 Firstly, the key should be set, then generate a temporary URL for a
894 Swift object:
895 > swift post -m "Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4"
897 > swift tempurl GET 6000 /v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7\
899 /firstcontainer/clean.sh b3968d0207b54ece87cccc06515a89d4
900 /v1/AUTH_/firstcontainer/clean.sh?temp_url_sig=\
902 9218fc288cc09e5edd857b6a3d43cf2122b906dc&temp_url_expires=1472203614
903 The swiftclient.SwiftService API
905 A higher-level API aimed at allowing developers an easy way to perform
906 multiple operations asynchronously using a configurable thread pool.
907 Documentation for each service method call can be found here: swift‐
908 client.service.
909 Authentication
911 This section covers the various options for authenticating with a swift
912 object store. The combinations of options required for each authentica‐
913 tion version are detailed below. Once again, these are just a subset of
914 those that can be used to successfully authenticate, but they are the
915 most common and recommended.
916 The relevant authentication options are presented as python dictionar‐
918 ies that should be added to any other options you are supplying to your
919 SwiftService instance. As indicated in the python code, you can also
920 set these options as environment variables that will be loaded automat‐
921 ically if the relevant option is not specified.
922 The SwiftService authentication attempts to automatically select the
924 auth version based on the combination of options specified, but supply‐
925 ing options from multiple different auth versions can cause unexpected
926 behaviour.
927 NOTE:
929 Leftover environment variables are a common source of confusion
930 when authorization fails.
931 Keystone V3
933 {
934 ...
935 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '3'
936 "os_username": environ.get('OS_USERNAME'),
937 "os_password": environ.get('OS_PASSWORD'),
938 "os_project_name": environ.get('OS_PROJECT_NAME'),
939 "os_project_domain_name": environ.get('OS_PROJECT_DOMAIN_NAME'),
940 "os_auth_url": environ.get('OS_AUTH_URL'),
941 ...
942 }
943 {
945 ...
946 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '3'
947 "os_username": environ.get('OS_USERNAME'),
948 "os_password": environ.get('OS_PASSWORD'),
949 "os_project_id": environ.get('OS_PROJECT_ID'),
950 "os_project_domain_id": environ.get('OS_PROJECT_DOMAIN_ID'),
951 "os_auth_url": environ.get('OS_AUTH_URL'),
952 ...
953 }
954 Keystone V2
956 {
957 ...
958 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '2.0'
959 "os_username": environ.get('OS_USERNAME'),
960 "os_password": environ.get('OS_PASSWORD'),
961 "os_tenant_name": environ.get('OS_TENANT_NAME'),
962 "os_auth_url": environ.get('OS_AUTH_URL'),
963 ...
964 }
965 Legacy Auth
967 {
968 ...
969 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '1.0'
970 "auth": environ.get('ST_AUTH'),
971 "user": environ.get('ST_USER'),
972 "key": environ.get('ST_KEY'),
973 ...
974 }
975 Configuration
977 When you create an instance of a SwiftService, you can override a col‐
978 lection of default options to suit your use case. Typically, the
979 defaults are sensible to get us started, but depending on your needs
980 you might want to tweak them to improve performance (options affecting
981 large objects and thread counts can significantly alter performance in
982 the right situation).
983 Service level defaults and some extra options can also be overridden on
985 a per-operation (or even in some cases per-object) basis, and you will
986 call out which options affect which operations later in the document.
987 The configuration of the service API is performed using an options dic‐
989 tionary passed to the SwiftService during initialisation. The options
990 available in this dictionary are described below, along with their
991 defaults:
992 Options
994 retries: 5
995 The number of times that the library should attempt to retry
996 HTTP actions before giving up and reporting a failure.
997 container_threads: 10
999 object_dd_threads: 10
1001 object_uu_threads: 10
1003 segment_threads: 10
1005 The above options determine the size of the available thread
1006 pools for performing swift operations. Container operations
1007 (such as listing a container) operate in the container
1008 threads, and a similar pattern applies to object and segment
1009 threads.
1010 NOTE:
1012 Object threads are separated into two separate thread
1013 pools: uu and dd. This stands for “upload/update” and
1014 “download/delete”, and the corresponding actions will be
1015 run on separate threads pools.
1016 segment_size: None
1018 If specified, this option enables uploading of large objects.
1019 Should the object being uploaded be larger than 5G in size,
1020 this option is mandatory otherwise the upload will fail. This
1021 option should be specified as a size in bytes.
1022 use_slo: False
1024 Used in combination with the above option, use_slo will
1025 upload large objects as static rather than dynamic. Only
1026 static large objects provide error checking for the down‐
1027 loaded object, so we recommend this option.
1028 segment_container: None
1030 Allows the user to select the container into which large
1031 object segments will be uploaded. We do not recommend chang‐
1032 ing this value as it could make locating orphaned segments
1033 more difficult in the case of errors.
1034 leave_segments: False
1036 Setting this option to true means that when deleting or over‐
1037 writing a large object, its segments will be left in the
1038 object store and must be cleaned up manually. This option can
1039 be useful when sharing large object segments between multiple
1040 objects in more advanced scenarios, but must be treated with
1041 care, as it could lead to ever increasing storage usage.
1042 changed: None
1044 This option affects uploads and simply means that those
1045 objects which already exist in the object store will not be
1046 overwritten if the mtime and size of the source is the same
1047 as the existing object.
1048 skip_identical: False
1050 A slightly more thorough case of the above, but rather than
1051 mtime and size uses an object’s MD5 sum.
1052 yes_all: False
1054 This options affects only download and delete, and in each
1055 case must be specified in order to download/delete the entire
1056 contents of an account. This option has no effect on any
1057 other calls.
1058 no_download: False
1060 This option only affects download and means that all opera‐
1061 tions proceed as normal with the exception that no data is
1062 written to disk.
1063 header: []
1065 Used with upload and post operations to set headers on
1066 objects. Headers are specified as colon separated strings,
1067 e.g. “content-type:text/plain”.
1068 meta: []
1070 Used to set metadata on an object similarly to headers.
1071 NOTE:
1073 Setting metadata is a destructive operation, so when
1074 updating one of many metadata values all desired metadata
1075 for an object must be re-applied.
1076 long: False
1078 Affects only list operations, and results in more metrics
1079 being made available in the results at the expense of lower
1080 performance.
1081 fail_fast: False
1083 Applies to delete and upload operations, and attempts to
1084 abort queued tasks in the event of errors.
1085 prefix: None
1087 Affects list operations; only objects with the given prefix
1088 will be returned/affected. It is not advisable to set at the
1089 service level, as those operations that call list to discover
1090 objects on which they should operate will also be affected.
1091 delimiter: None
1093 Affects list operations, and means that listings only contain
1094 results up to the first instance of the delimiter in the
1095 object name. This is useful for working with objects contain‐
1096 ing ‘/’ in their names to simulate folder structures.
1097 dir_marker: False
1099 Affects uploads, and allows empty ‘pseudofolder’ objects to
1100 be created when the source of an upload is None.
1101 checksum: True
1103 Affects uploads and downloads. If set check md5 sum for the
1104 transfer.
1105 shuffle: False
1107 When downloading objects, the default behaviour of the CLI is
1108 to shuffle lists of objects in order to spread the load on
1109 storage drives when multiple clients are downloading the same
1110 files to multiple locations (e.g. in the event of distribut‐
1111 ing an update). When using the SwiftService directly, object
1112 downloads are scheduled in the same order as they appear in
1113 the container listing. When combined with a single download
1114 thread this means that objects are downloaded in lexi‐
1115 cally-sorted order. Setting this option to True gives the
1116 same shuffling behaviour as the CLI.
1117 destination: None
1119 When copying objects, this specifies the destination where
1120 the object will be copied to. The default of None means copy
1121 will be the same as source.
1122 fresh_metadata: None
1124 When copying objects, this specifies that the object metadata
1125 on the source will not be applied to the destination object -
1126 the destination object will have a new fresh set of metadata
1127 that includes only the metadata specified in the meta option
1128 if any at all.
1129 Other available options can be found in swiftclient/service.py in the
1131 source code for python-swiftclient. Each SwiftService method also
1132 allows for an optional dictionary to override those specified at init
1133 time, and the appropriate docstrings show which options modify each
1134 method’s behaviour.
1135 Available Operations
1137 Each operation provided by the service API may raise a SwiftError or
1138 ClientException for any call that fails completely (or a call which
1139 performs only one operation at an account or container level). In the
1140 case of a successful call an operation returns one of the following:
1141 · A dictionary detailing the results of a single operation.
1143 · An iterator that produces result dictionaries (for calls that perform
1145 multiple sub-operations).
1146 A result dictionary can indicate either the success or failure of an
1148 individual operation (detailed in the success key), and will either
1149 contain the successful result, or an error key detailing the error
1150 encountered (usually an instance of Exception).
1151 An example result dictionary is given below:
1153 result = {
1155 'action': 'download_object',
1156 'success': True,
1157 'container': container,
1158 'object': obj,
1159 'path': path,
1160 'start_time': start_time,
1161 'finish_time': finish_time,
1162 'headers_receipt': headers_receipt,
1163 'auth_end_time': conn.auth_end_time,
1164 'read_length': bytes_read,
1165 'attempts': conn.attempts
1166 }
1167 All the possible action values are detailed below:
1169 [
1171 'stat_account',
1172 'stat_container',
1173 'stat_object',
1174 'post_account',
1175 'post_container',
1176 'post_object',
1177 'list_part', # list yields zero or more 'list_part' results
1178 'download_object',
1179 'create_container', # from upload
1180 'create_dir_marker', # from upload
1181 'upload_object',
1182 'upload_segment',
1183 'delete_container',
1184 'delete_object',
1185 'delete_segment', # from delete_object operations
1186 'capabilities',
1187 ]
1188 Stat
1190 Stat can be called against an account, a container, or a list of
1191 objects to get account stats, container stats or information about the
1192 given objects. In the first two cases a dictionary is returned contain‐
1193 ing the results of the operation, and in the case of a list of object
1194 names being supplied, an iterator over the results generated for each
1195 object is returned.
1196 Information returned includes the amount of data used by the given
1198 object/container/account and any headers or metadata set (this includes
1199 user set data as well as content-type and modification times).
1200 See swiftclient.service.SwiftService.stat for docs generated from the
1202 method docstring.
1203 Valid calls for this method are as follows:
1205 · stat([options]): Returns stats for the configured account.
1207 · stat(<container>, [options]): Returns stats for the given container.
1209 · stat(<container>, <object_list>, [options]): Returns stats for each
1211 of the given objects in the given container (through the returned
1212 iterator).
1213 Results from stat are dictionaries indicating the success or failure of
1215 each operation. In the case of a successful stat against an account or
1216 container, the method returns immediately with one of the following
1217 results:
1218 {
1220 'action': 'stat_account',
1221 'success': True,
1222 'items': items,
1223 'headers': headers
1224 }
1225 {
1227 'action': 'stat_container',
1228 'container': <container>,
1229 'success': True,
1230 'items': items,
1231 'headers': headers
1232 }
1233 In the case of stat called against a list of objects, the method
1235 returns a generator that returns the results of individual object stat
1236 operations as they are performed on the thread pool:
1237 {
1239 'action': 'stat_object',
1240 'object': <object_name>,
1241 'container': <container>,
1242 'success': True,
1243 'items': items,
1244 'headers': headers
1245 }
1246 In the case of a failure the dictionary returned will indicate that the
1248 operation was not successful, and will include the keys below:
1249 {
1251 'action': <'stat_object'|'stat_container'|'stat_account'>,
1252 'object': <'object_name'>, # Only for stat with objects list
1253 'container': <container>, # Only for stat with objects list or container
1254 'success': False,
1255 'error': <error>,
1256 'traceback': <trace>,
1257 'error_timestamp': <timestamp>
1258 }
1259 Example
1261 The code below demonstrates the use of stat to retrieve the headers for
1262 a given list of objects in a container using 20 threads. The code cre‐
1263 ates a mapping from object name to headers which is then pretty printed
1264 to the log.
1265 import logging
1267 import pprint
1268 from swiftclient.service import SwiftService
1270 from sys import argv
1271 logging.basicConfig(level=logging.ERROR)
1273 logging.getLogger("requests").setLevel(logging.CRITICAL)
1274 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1275 logger = logging.getLogger(__name__)
1276 _opts = {'object_dd_threads': 20}
1278 with SwiftService(options=_opts) as swift:
1279 container = argv[1]
1280 objects = argv[2:]
1281 header_data = {}
1282 stats_it = swift.stat(container=container, objects=objects)
1283 for stat_res in stats_it:
1284 if stat_res['success']:
1285 header_data[stat_res['object']] = stat_res['headers']
1286 else:
1287 logger.error(
1288 'Failed to retrieve stats for %s' % stat_res['object']
1289 )
1290 pprint.pprint(header_data)
1291 List
1294 List can be called against an account or a container to retrieve the
1295 containers or objects contained within them. Each call returns an iter‐
1296 ator that returns pages of results (by default, up to 10000 results in
1297 each page).
1298 See swiftclient.service.SwiftService.list for docs generated from the
1300 method docstring.
1301 If the given container or account does not exist, the list method will
1303 raise a SwiftError, but for all other success/failures a dictionary is
1304 returned. Each successfully listed page returns a dictionary as
1305 described below:
1306 {
1308 'action': <'list_account_part'|'list_container_part'>,
1309 'container': <container>, # Only for listing a container
1310 'prefix': <prefix>, # The prefix of returned objects/containers
1311 'success': True,
1312 'listing': [Item], # A list of results
1313 # (only in the event of success)
1314 'marker': <marker> # The last item name in the list
1315 # (only in the event of success)
1316 }
1317 Where an item contains the following keys:
1319 {
1321 'name': <name>,
1322 'bytes': 10485760,
1323 'last_modified': '2014-12-11T12:02:38.774540',
1324 'hash': 'fb938269cbeabe4c234e1127bbd3b74a',
1325 'content_type': 'application/octet-stream',
1326 'meta': <metadata> # Full metadata listing from stat'ing each object
1327 # this key only exists if 'long' is specified in options
1328 }
1329 Any failure listing an account or container that exists will return a
1331 failure dictionary as described below:
1332 {
1334 'action': <'list_account_part'|'list_container_part'>,,
1335 'container': container, # Only for listing a container
1336 'prefix': options['prefix'],
1337 'success': success,
1338 'marker': marker,
1339 'error': error,
1340 'traceback': <trace>,
1341 'error_timestamp': <timestamp>
1342 }
1343 Example
1345 The code below demonstrates the use of list to list all items in a con‐
1346 tainer that are over 10MiB in size:
1347 import logging
1349 from swiftclient.service import SwiftService, SwiftError
1351 from sys import argv
1352 logging.basicConfig(level=logging.ERROR)
1354 logging.getLogger("requests").setLevel(logging.CRITICAL)
1355 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1356 logger = logging.getLogger(__name__)
1357 container = argv[1]
1359 minimum_size = 10*1024**2
1360 with SwiftService() as swift:
1361 try:
1362 list_parts_gen = swift.list(container=container)
1363 for page in list_parts_gen:
1364 if page["success"]:
1365 for item in page["listing"]:
1366 i_size = int(item["bytes"])
1368 if i_size > minimum_size:
1369 i_name = item["name"]
1370 i_etag = item["hash"]
1371 print(
1372 "%s [size: %s] [etag: %s]" %
1373 (i_name, i_size, i_etag)
1374 )
1375 else:
1376 raise page["error"]
1377 except SwiftError as e:
1379 logger.error(e.value)
1380 Post
1383 Post can be called against an account, container or list of objects in
1384 order to update the metadata attached to the given items. In the first
1385 two cases a single dictionary is returned containing the results of the
1386 operation, and in the case of a list of objects being supplied, an
1387 iterator over the results generated for each object post is returned.
1388 Each element of the object list may be a plain string of the object
1390 name, or a SwiftPostObject that allows finer control over the options
1391 and metadata applied to each of the individual post operations. When a
1392 string is given for the object name, the options and metadata applied
1393 are a combination of those supplied to the call to post() and the
1394 defaults of the SwiftService object.
1395 If the given container or account does not exist, the post method will
1397 raise a SwiftError. Successful metadata update results are dictionaries
1398 as described below:
1399 {
1401 'action': <'post_account'|'post_container'|'post_object'>,
1402 'success': True,
1403 'container': <container>,
1404 'object': <object>,
1405 'headers': {},
1406 'response_dict': <HTTP response details>
1407 }
1408 NOTE:
1410 Updating user metadata keys will not only add any specified keys,
1411 but will also remove user metadata that has previously been set.
1412 This means that each time user metadata is updated, the complete set
1413 of desired key-value pairs must be specified.
1414 Example
1416 The code below demonstrates the use of post to set an archive folder in
1417 a given container to expire after a 24 hour delay:
1418 import logging
1420 from swiftclient.service import SwiftService, SwiftError
1422 from sys import argv
1423 logging.basicConfig(level=logging.ERROR)
1425 logging.getLogger("requests").setLevel(logging.CRITICAL)
1426 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1427 logger = logging.getLogger(__name__)
1428 container = argv[1]
1430 with SwiftService() as swift:
1431 try:
1432 list_options = {"prefix": "archive_2016-01-01/"}
1433 list_parts_gen = swift.list(container=container)
1434 for page in list_parts_gen:
1435 if page["success"]:
1436 objects = [obj["name"] for obj in page["listing"]]
1437 post_options = {"header": "X-Delete-After:86400"}
1438 for post_res in swift.post(
1439 container=container,
1440 objects=objects,
1441 options=post_options):
1442 if post_res['success']:
1443 print("Object '%s' POST success" % post_res['object'])
1444 else:
1445 print("Object '%s' POST failed" % post_res['object'])
1446 else:
1447 raise page["error"]
1448 except SwiftError as e:
1449 logger.error(e.value)
1450 Download
1453 Download can be called against an entire account, a single container,
1454 or a list of objects in a given container. Each element of the object
1455 list is a string detailing the full name of an object to download.
1456 In order to download the full contents of an entire account, you must
1458 set the value of yes_all to True in the options dictionary supplied to
1459 either the SwiftService instance or the call to download.
1460 If the given container or account does not exist, the download method
1462 will raise a SwiftError, otherwise an iterator over the results gener‐
1463 ated for each object download is returned.
1464 See swiftclient.service.SwiftService.download for docs generated from
1466 the method docstring.
1467 For each successfully downloaded object, the results returned by the
1469 iterator will be a dictionary as described below (results are not
1470 returned for completed container or object segment downloads):
1471 {
1473 'action': 'download_object',
1474 'container': <container>,
1475 'object': <object name>,
1476 'success': True,
1477 'path': <local path to downloaded object>,
1478 'pseudodir': <if true, the download created an empty directory>,
1479 'start_time': <time download started>,
1480 'end_time': <time download completed>,
1481 'headers_receipt': <time the headers from the object were retrieved>,
1482 'auth_end_time': <time authentication completed>,
1483 'read_length': <bytes_read>,
1484 'attempts': <attempt count>,
1485 'response_dict': <HTTP response details>
1486 }
1487 Any failure uploading an object will return a failure dictionary as
1489 described below:
1490 {
1492 'action': 'download_object',
1493 'container': <container>,
1494 'object': <object name>,
1495 'success': False,
1496 'path': <local path of the failed download>,
1497 'pseudodir': <if true, the failed download was an empty directory>,
1498 'attempts': <attempt count>,
1499 'error': <error>,
1500 'traceback': <trace>,
1501 'error_timestamp': <timestamp>,
1502 'response_dict': <HTTP response details>
1503 }
1504 Example
1506 The code below demonstrates the use of download to download all PNG
1507 images from a dated archive folder in a given container:
1508 import logging
1510 from swiftclient.service import SwiftService, SwiftError
1512 from sys import argv
1513 logging.basicConfig(level=logging.ERROR)
1515 logging.getLogger("requests").setLevel(logging.CRITICAL)
1516 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1517 logger = logging.getLogger(__name__)
1518 def is_png(obj):
1520 return (
1521 obj["name"].lower().endswith('.png') or
1522 obj["content_type"] == 'image/png'
1523 )
1524 container = argv[1]
1526 with SwiftService() as swift:
1527 try:
1528 list_options = {"prefix": "archive_2016-01-01/"}
1529 list_parts_gen = swift.list(container=container)
1530 for page in list_parts_gen:
1531 if page["success"]:
1532 objects = [
1533 obj["name"] for obj in page["listing"] if is_png(obj)
1534 ]
1535 for down_res in swift.download(
1536 container=container,
1537 objects=objects):
1538 if down_res['success']:
1539 print("'%s' downloaded" % down_res['object'])
1540 else:
1541 print("'%s' download failed" % down_res['object'])
1542 else:
1543 raise page["error"]
1544 except SwiftError as e:
1545 logger.error(e.value)
1546 Upload
1549 Upload is always called against an account and container and with a
1550 list of objects to upload. Each element of the object list may be a
1551 plain string detailing the path of the object to upload, or a SwiftU‐
1552 ploadObject that allows finer control over some aspects of the individ‐
1553 ual operations.
1554 When a simple string is supplied to specify a file to upload, the name
1556 of the object uploaded is the full path of the specified file and the
1557 options used for the upload are those supplied to the call to upload.
1558 Constructing a SwiftUploadObject allows the user to supply an object
1560 name for the uploaded file, and modify the options used by upload at
1561 the granularity of individual files.
1562 If the given container or account does not exist, the upload method
1564 will raise a SwiftError, otherwise an iterator over the results gener‐
1565 ated for each object upload is returned.
1566 See swiftclient.service.SwiftService.upload for docs generated from the
1568 method docstring.
1569 For each successfully uploaded object (or object segment), the results
1571 returned by the iterator will be a dictionary as described below:
1572 {
1574 'action': 'upload_object',
1575 'container': <container>,
1576 'object': <object name>,
1577 'success': True,
1578 'status': <'uploaded'|'skipped-identical'|'skipped-changed'>,
1579 'attempts': <attempt count>,
1580 'response_dict': <HTTP response details>
1581 }
1582 {
1584 'action': 'upload_segment',
1585 'for_container': <container>,
1586 'for_object': <object name>,
1587 'segment_index': <segment_index>,
1588 'segment_size': <segment_size>,
1589 'segment_location': <segment_path>
1590 'segment_etag': <etag>,
1591 'log_line': <object segment n>
1592 'success': True,
1593 'response_dict': <HTTP response details>,
1594 'attempts': <attempt count>
1595 }
1596 Any failure uploading an object will return a failure dictionary as
1598 described below:
1599 {
1601 'action': 'upload_object',
1602 'container': <container>,
1603 'object': <object name>,
1604 'success': False,
1605 'attempts': <attempt count>,
1606 'error': <error>,
1607 'traceback': <trace>,
1608 'error_timestamp': <timestamp>,
1609 'response_dict': <HTTP response details>
1610 }
1611 {
1613 'action': 'upload_segment',
1614 'for_container': <container>,
1615 'for_object': <object name>,
1616 'segment_index': <segment_index>,
1617 'segment_size': <segment_size>,
1618 'segment_location': <segment_path>,
1619 'log_line': <object segment n>,
1620 'success': False,
1621 'error': <error>,
1622 'traceback': <trace>,
1623 'error_timestamp': <timestamp>,
1624 'response_dict': <HTTP response details>,
1625 'attempts': <attempt count>
1626 }
1627 Example
1629 The code below demonstrates the use of upload to upload all files and
1630 folders in a given directory, and rename each object by replacing the
1631 root directory name with ‘my-<d>-objects’, where <d> is the name of the
1632 uploaded directory:
1633 import logging
1635 from os import walk
1637 from os.path import join
1638 from swiftclient.multithreading import OutputManager
1639 from swiftclient.service import SwiftError, SwiftService, SwiftUploadObject
1640 from sys import argv
1641 logging.basicConfig(level=logging.ERROR)
1643 logging.getLogger("requests").setLevel(logging.CRITICAL)
1644 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1645 logger = logging.getLogger(__name__)
1646 _opts = {'object_uu_threads': 20}
1648 dir = argv[1]
1649 container = argv[2]
1650 with SwiftService(options=_opts) as swift, OutputManager() as out_manager:
1651 try:
1652 # Collect all the files and folders in the given directory
1653 objs = []
1654 dir_markers = []
1655 for (_dir, _ds, _fs) in walk(dir):
1656 if not (_ds + _fs):
1657 dir_markers.append(_dir)
1658 else:
1659 objs.extend([join(_dir, _f) for _f in _fs])
1660 # Now that we've collected all the required files and dir markers
1662 # build the ``SwiftUploadObject``s for the call to upload
1663 objs = [
1664 SwiftUploadObject(
1665 o, object_name=o.replace(
1666 dir, 'my-%s-objects' % dir, 1
1667 )
1668 ) for o in objs
1669 ]
1670 dir_markers = [
1671 SwiftUploadObject(
1672 None, object_name=d.replace(
1673 dir, 'my-%s-objects' % dir, 1
1674 ), options={'dir_marker': True}
1675 ) for d in dir_markers
1676 ]
1677 # Schedule uploads on the SwiftService thread pool and iterate
1679 # over the results
1680 for r in swift.upload(container, objs + dir_markers):
1681 if r['success']:
1682 if 'object' in r:
1683 print(r['object'])
1684 elif 'for_object' in r:
1685 print(
1686 '%s segment %s' % (r['for_object'],
1687 r['segment_index'])
1688 )
1689 else:
1690 error = r['error']
1691 if r['action'] == "create_container":
1692 logger.warning(
1693 'Warning: failed to create container '
1694 "'%s'%s", container, error
1695 )
1696 elif r['action'] == "upload_object":
1697 logger.error(
1698 "Failed to upload object %s to container %s: %s" %
1699 (container, r['object'], error)
1700 )
1701 else:
1702 logger.error("%s" % error)
1703 except SwiftError as e:
1705 logger.error(e.value)
1706 Delete
1709 Delete can be called against an account or a container to remove the
1710 containers or objects contained within them. Each call to delete
1711 returns an iterator over results of each resulting sub-request.
1712 If the number of requested delete operations is large and the target
1714 swift cluster is running the bulk middleware, the call to SwiftSer‐
1715 vice.delete will make use of bulk operations and the returned result
1716 iterator will return bulk_delete results rather than individual
1717 delete_object, delete_container or delete_segment results.
1718 See swiftclient.service.SwiftService.delete for docs generated from the
1720 method docstring.
1721 For each successfully deleted container, object or segment, the results
1723 returned by the iterator will be a dictionary as described below:
1724 {
1726 'action': <'delete_object'|'delete_segment'>,
1727 'container': <container>,
1728 'object': <object name>,
1729 'success': True,
1730 'attempts': <attempt count>,
1731 'response_dict': <HTTP response details>
1732 }
1733 {
1735 'action': 'delete_container',
1736 'container': <container>,
1737 'success': True,
1738 'response_dict': <HTTP response details>,
1739 'attempts': <attempt count>
1740 }
1741 {
1743 'action': 'bulk_delete',
1744 'container': <container>,
1745 'objects': <[objects]>,
1746 'success': True,
1747 'attempts': <attempt count>,
1748 'response_dict': <HTTP response details>
1749 }
1750 Any failure in a delete operation will return a failure dictionary as
1752 described below:
1753 {
1755 'action': ('delete_object'|'delete_segment'),
1756 'container': <container>,
1757 'object': <object name>,
1758 'success': False,
1759 'attempts': <attempt count>,
1760 'error': <error>,
1761 'traceback': <trace>,
1762 'error_timestamp': <timestamp>,
1763 'response_dict': <HTTP response details>
1764 }
1765 {
1767 'action': 'delete_container',
1768 'container': <container>,
1769 'success': False,
1770 'error': <error>,
1771 'traceback': <trace>,
1772 'error_timestamp': <timestamp>,
1773 'response_dict': <HTTP response details>,
1774 'attempts': <attempt count>
1775 }
1776 {
1778 'action': 'bulk_delete',
1779 'container': <container>,
1780 'objects': <[objects]>,
1781 'success': False,
1782 'attempts': <attempt count>,
1783 'error': <error>,
1784 'traceback': <trace>,
1785 'error_timestamp': <timestamp>,
1786 'response_dict': <HTTP response details>
1787 }
1788 Example
1790 The code below demonstrates the use of delete to remove a given list of
1791 objects from a specified container. As the objects are deleted the
1792 transaction ID of the relevant request is printed along with the object
1793 name and number of attempts required. By printing the transaction ID,
1794 the printed operations can be easily linked to events in the swift
1795 server logs:
1796 import logging
1798 from swiftclient.service import SwiftService
1800 from sys import argv
1801 logging.basicConfig(level=logging.ERROR)
1804 logging.getLogger("requests").setLevel(logging.CRITICAL)
1805 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1806 logger = logging.getLogger(__name__)
1807 _opts = {'object_dd_threads': 20}
1809 container = argv[1]
1810 objects = argv[2:]
1811 with SwiftService(options=_opts) as swift:
1812 del_iter = swift.delete(container=container, objects=objects)
1813 for del_res in del_iter:
1814 c = del_res.get('container', '')
1815 o = del_res.get('object', '')
1816 a = del_res.get('attempts')
1817 if del_res['success'] and not del_res['action'] == 'bulk_delete':
1818 rd = del_res.get('response_dict')
1819 if rd is not None:
1820 t = dict(rd.get('headers', {}))
1821 if t:
1822 print(
1823 'Successfully deleted {0}/{1} in {2} attempts '
1824 '(transaction id: {3})'.format(c, o, a, t)
1825 )
1826 else:
1827 print(
1828 'Successfully deleted {0}/{1} in {2} '
1829 'attempts'.format(c, o, a)
1830 )
1831 Copy
1834 Copy can be called to copy an object or update the metadata on the
1835 given items.
1836 Each element of the object list may be a plain string of the object
1838 name, or a SwiftCopyObject that allows finer control over the options
1839 applied to each of the individual copy operations (destination,
1840 fresh_metadata, options).
1841 Destination should be in format /container/object; if not set, the
1843 object will be copied onto itself. Fresh_metadata sets mode of opera‐
1844 tion on metadata. If not set, current object user metadata will be
1845 copied/preserved; if set, all current user metadata will be removed.
1846 Returns an iterator over the results generated for each object copy
1848 (and may also include the results of creating destination containers).
1849 When a string is given for the object name, destination and fresh meta‐
1851 data will default to None and None, which result in adding metadata to
1852 existing objects.
1853 Successful copy results are dictionaries as described below:
1855 {
1857 'action': 'copy_object',
1858 'success': True,
1859 'container': <container>,
1860 'object': <object>,
1861 'destination': <destination>,
1862 'headers': {},
1863 'fresh_metadata': <boolean>,
1864 'response_dict': <HTTP response details>
1865 }
1866 Any failure in a copy operation will return a failure dictionary as
1868 described below:
1869 {
1871 'action': 'copy_object',
1872 'success': False,
1873 'container': <container>,
1874 'object': <object>,
1875 'destination': <destination>,
1876 'headers': {},
1877 'fresh_metadata': <boolean>,
1878 'response_dict': <HTTP response details>,
1879 'error': <error>,
1880 'traceback': <traceback>,
1881 'error_timestamp': <timestamp>
1882 }
1883 Example
1885 The code below demonstrates the use of copy to add new user metadata
1886 for objects a and b, and to copy object c to d (with added metadata).
1887 import logging
1889 from swiftclient.service import SwiftService, SwiftCopyObject, SwiftError
1891 logging.basicConfig(level=logging.ERROR)
1893 logging.getLogger("requests").setLevel(logging.CRITICAL)
1894 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1895 logger = logging.getLogger(__name__)
1896 with SwiftService() as swift:
1898 try:
1899 obj = SwiftCopyObject("c", {"Destination": "/cont/d"})
1900 for i in swift.copy(
1901 "cont", ["a", "b", obj],
1902 {"meta": ["foo:bar"], "Destination": "/cc"}):
1903 if i["success"]:
1904 if i["action"] == "copy_object":
1905 print(
1906 "object %s copied from /%s/%s" %
1907 (i["destination"], i["container"], i["object"])
1908 )
1909 if i["action"] == "create_container":
1910 print(
1911 "container %s created" % i["container"]
1912 )
1913 else:
1914 if "error" in i and isinstance(i["error"], Exception):
1915 raise i["error"]
1916 except SwiftError as e:
1917 logger.error(e.value)
1918 Capabilities
1921 Capabilities can be called against an account or a particular proxy URL
1922 in order to determine the capabilities of the swift cluster. These
1923 capabilities include details about configuration options and the mid‐
1924 dlewares that are installed in the proxy pipeline.
1925 See swiftclient.service.SwiftService.capabilities for docs generated
1927 from the method docstring.
1928 For each successful call to list capabilities, a result dictionary will
1930 be returned with the contents described below:
1931 {
1933 'action': 'capabilities',
1934 'timestamp': <time of the call>,
1935 'success': True,
1936 'capabilities': <dictionary containing capability details>
1937 }
1938 The contents of the capabilities dictionary contain the core swift
1940 capabilities under the key swift; all other keys show the configuration
1941 options for additional middlewares deployed in the proxy pipeline. An
1942 example capabilities dictionary is given below:
1943 {
1945 'account_quotas': {},
1946 'bulk_delete': {
1947 'max_deletes_per_request': 10000,
1948 'max_failed_deletes': 1000
1949 },
1950 'bulk_upload': {
1951 'max_containers_per_extraction': 10000,
1952 'max_failed_extractions': 1000
1953 },
1954 'container_quotas': {},
1955 'container_sync': {'realms': {}},
1956 'formpost': {},
1957 'keystoneauth': {},
1958 'slo': {
1959 'max_manifest_segments': 1000,
1960 'max_manifest_size': 2097152,
1961 'min_segment_size': 1048576
1962 },
1963 'swift': {
1964 'account_autocreate': True,
1965 'account_listing_limit': 10000,
1966 'allow_account_management': True,
1967 'container_listing_limit': 10000,
1968 'extra_header_count': 0,
1969 'max_account_name_length': 256,
1970 'max_container_name_length': 256,
1971 'max_file_size': 5368709122,
1972 'max_header_size': 8192,
1973 'max_meta_count': 90,
1974 'max_meta_name_length': 128,
1975 'max_meta_overall_size': 4096,
1976 'max_meta_value_length': 256,
1977 'max_object_name_length': 1024,
1978 'policies': [
1979 {'default': True, 'name': 'Policy-0'}
1980 ],
1981 'strict_cors_mode': False,
1982 'version': '2.2.2'
1983 },
1984 'tempurl': {
1985 'methods': ['GET', 'HEAD', 'PUT']
1986 }
1987 }
1988 Example
1990 The code below demonstrates the use of capabilities to determine if the
1991 Swift cluster supports static large objects, and if so, the maximum
1992 number of segments that can be described in a single manifest file,
1993 along with the size restrictions on those objects:
1994 import logging
1996 from swiftclient.exceptions import ClientException
1998 from swiftclient.service import SwiftService
1999 logging.basicConfig(level=logging.ERROR)
2001 logging.getLogger("requests").setLevel(logging.CRITICAL)
2002 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
2003 logger = logging.getLogger(__name__)
2004 with SwiftService() as swift:
2006 try:
2007 capabilities_result = swift.capabilities()
2008 capabilities = capabilities_result['capabilities']
2009 if 'slo' in capabilities:
2010 print('SLO is supported')
2011 else:
2012 print('SLO is not supported')
2013 except ClientException as e:
2014 logger.error(e.value)
2015 The swiftclient.Connection API
2018 A low level API that provides methods for authentication and methods
2019 that correspond to the individual REST API calls described in the swift
2020 documentation.
2021 For usage details see the client docs: swiftclient.client.
2023 Authentication
2025 This section covers the various combinations of kwargs required when
2026 creating an instance of the Connection object for communicating with a
2027 swift object store. The combinations of options required for each
2028 authentication version are detailed below, but are just a subset of
2029 those that can be used to successfully authenticate. These are the most
2030 common and recommended combinations.
2031 Keystone Session
2033 from keystoneauth1 import session
2034 from keystoneauth1.identity import v3
2035 # Create a password auth plugin
2037 auth = v3.Password(auth_url='http://127.0.0.1:5000/v3/',
2038 username='tester',
2039 password='testing',
2040 user_domain_name='Default',
2041 project_name='Default',
2042 project_domain_name='Default')
2043 # Create session
2045 keystone_session = session.Session(auth=auth)
2046 # Create swiftclient Connection
2048 swift_conn = Connection(session=keystone_session)
2049 Keystone v3
2051 _authurl = 'http://127.0.0.1:5000/v3/'
2052 _auth_version = '3'
2053 _user = 'tester'
2054 _key = 'testing'
2055 _os_options = {
2056 'user_domain_name': 'Default',
2057 'project_domain_name': 'Default',
2058 'project_name': 'Default'
2059 }
2060 conn = Connection(
2062 authurl=_authurl,
2063 user=_user,
2064 key=_key,
2065 os_options=_os_options,
2066 auth_version=_auth_version
2067 )
2068 Keystone v2
2070 _authurl = 'http://127.0.0.1:5000/v2.0/'
2071 _auth_version = '2'
2072 _user = 'tester'
2073 _key = 'testing'
2074 _tenant_name = 'test'
2075 conn = Connection(
2077 authurl=_authurl,
2078 user=_user,
2079 key=_key,
2080 tenant_name=_tenant_name,
2081 auth_version=_auth_version
2082 )
2083 Legacy Auth
2085 _authurl = 'http://127.0.0.1:8080/'
2086 _auth_version = '1'
2087 _user = 'tester'
2088 _key = 'testing'
2089 _tenant_name = 'test'
2090 conn = Connection(
2092 authurl=_authurl,
2093 user=_user,
2094 key=_key,
2095 tenant_name=_tenant_name,
2096 auth_version=_auth_version
2097 )
2098 Examples
2100 In this section we present some simple code examples that demonstrate
2101 the usage of the Connection API. You can find full details of the
2102 options and methods available to the Connection API in the docstring
2103 generated documentation: swiftclient.client.
2104 List the available containers:
2106 resp_headers, containers = conn.get_account()
2108 print("Response headers: %s" % resp_headers)
2109 for container in containers:
2110 print(container)
2111 Create a new container:
2113 container = 'new-container'
2115 conn.put_container(container)
2116 resp_headers, containers = conn.get_account()
2117 if container in containers:
2118 print("The container was created")
2119 Create a new object with the contents of a local text file:
2121 container = 'new-container'
2123 with open('local.txt', 'r') as local:
2124 conn.put_object(
2125 container,
2126 'local_object.txt',
2127 contents=local,
2128 content_type='text/plain'
2129 )
2130 Confirm presence of the object:
2132 obj = 'local_object.txt'
2134 container = 'new-container'
2135 try:
2136 resp_headers = conn.head_object(container, obj)
2137 print('The object was successfully created')
2138 except ClientException as e:
2139 if e.http_status = '404':
2140 print('The object was not found')
2141 else:
2142 print('An error occurred checking for the existence of the object')
2143 Download the created object:
2145 obj = 'local_object.txt'
2147 container = 'new-container'
2148 resp_headers, obj_contents = conn.get_object(container, obj)
2149 with open('local_copy.txt', 'w') as local:
2150 local.write(obj_contents)
2151 Delete the created object:
2153 obj = 'local_object.txt'
2155 container = 'new-container'
2156 try:
2157 conn.delete_object(container, obj)
2158 print("Successfully deleted the object")
2159 except ClientException as e:
2160 print("Failed to delete the object with error: %s" % e)
2161
2163 swiftclient
2164 OpenStack Swift Python client binding.
2165 swiftclient.authv1
2167 swiftclient.client
2168 OpenStack Swift client library used internally
2169 class swiftclient.client.Connection(authurl=None, user=None, key=None,
2171 retries=5, preauthurl=None, preauthtoken=None, snet=False, start‐
2172 ing_backoff=1, max_backoff=64, tenant_name=None, os_options=None,
2173 auth_version='1', cacert=None, insecure=False, cert=None,
2174 cert_key=None, ssl_compression=True, retry_on_ratelimit=False, time‐
2175 out=None, session=None)
2176 Bases: object
2177 Convenience class to make requests that will also retry the
2179 request
2180 Requests will have an X-Auth-Token header whose value is either
2182 the preauthtoken or a token obtained from the auth service using
2183 the user credentials provided as args to the constructor. If
2184 os_options includes a service_username then requests will also
2185 have an X-Service-Token header whose value is a token obtained
2186 from the auth service using the service credentials. In this
2187 case the request url will be set to the storage_url obtained
2188 from the auth service for the service user, unless this is over‐
2189 ridden by a preauthurl.
2190 Parameters
2192 · authurl -- authentication URL
2194 · user -- user name to authenticate as
2196 · key -- key/password to authenticate with
2198 · retries -- Number of times to retry the request before
2200 failing
2201 · preauthurl -- storage URL (if you have already authen‐
2203 ticated)
2204 · preauthtoken -- authentication token (if you have
2206 already authenticated) note authurl/user/key/ten‐
2207 ant_name are not required when specifying preauthtoken
2208 · snet -- use SERVICENET internal network default is
2210 False
2211 · starting_backoff -- initial delay between retries (sec‐
2213 onds)
2214 · max_backoff -- maximum delay between retries (seconds)
2216 · auth_version -- OpenStack auth version, default is 1.0
2218 · tenant_name -- The tenant/account name, required when
2220 connecting to an auth 2.0 system.
2221 · os_options -- The OpenStack options which can have ten‐
2223 ant_id, auth_token, service_type, endpoint_type, ten‐
2224 ant_name, object_storage_url, region_name, ser‐
2225 vice_username, service_project_name, service_key
2226 · insecure -- Allow to access servers without checking
2228 SSL certs. The server's certificate will not be veri‐
2229 fied.
2230 · cert -- Client certificate file to connect on SSL
2232 server requiring SSL client certificate.
2233 · cert_key -- Client certificate private key file.
2235 · ssl_compression -- Whether to enable compression at the
2237 SSL layer. If set to 'False' and the pyOpenSSL library
2238 is present an attempt to disable SSL compression will
2239 be made. This may provide a performance increase for
2240 https upload/download operations.
2241 · retry_on_ratelimit -- by default, a ratelimited connec‐
2243 tion will raise an exception to the caller. Setting
2244 this parameter to True will cause a retry after a back‐
2245 off.
2246 · timeout -- The connect timeout for the HTTP connection.
2248 · session -- A keystoneauth session object.
2250 close()
2252 copy_object(container, obj, destination=None, headers=None,
2254 fresh_metadata=None, response_dict=None)
2255 Wrapper for copy_object()
2256 delete_container(container, response_dict=None,
2258 query_string=None, headers={})
2259 Wrapper for delete_container()
2260 delete_object(container, obj, query_string=None,
2262 response_dict=None, headers=None)
2263 Wrapper for delete_object()
2264 get_account(marker=None, limit=None, prefix=None,
2266 end_marker=None, full_listing=False, headers=None)
2267 Wrapper for get_account()
2268 get_auth()
2270 get_capabilities(url=None)
2272 get_container(container, marker=None, limit=None, prefix=None,
2274 delimiter=None, end_marker=None, path=None, full_listing=False,
2275 headers=None, query_string=None)
2276 Wrapper for get_container()
2277 get_object(container, obj, resp_chunk_size=None,
2279 query_string=None, response_dict=None, headers=None)
2280 Wrapper for get_object()
2281 get_service_auth()
2283 head_account(headers=None)
2285 Wrapper for head_account()
2286 head_container(container, headers=None)
2288 Wrapper for head_container()
2289 head_object(container, obj, headers=None)
2291 Wrapper for head_object()
2292 http_connection(url=None)
2294 post_account(headers, response_dict=None, query_string=None,
2296 data=None)
2297 Wrapper for post_account()
2298 post_container(container, headers, response_dict=None)
2300 Wrapper for post_container()
2301 post_object(container, obj, headers, response_dict=None)
2303 Wrapper for post_object()
2304 put_container(container, headers=None, response_dict=None,
2306 query_string=None)
2307 Wrapper for put_container()
2308 put_object(container, obj, contents, content_length=None,
2310 etag=None, chunk_size=None, content_type=None, headers=None,
2311 query_string=None, response_dict=None)
2312 Wrapper for put_object()
2313 class swiftclient.client.HTTPConnection(url, proxy=None, cacert=None,
2315 insecure=False, cert=None, cert_key=None, ssl_compression=False,
2316 default_user_agent=None, timeout=None)
2317 Bases: object
2318 Make an HTTPConnection or HTTPSConnection
2320 Parameters
2322 · url -- url to connect to
2324 · proxy -- proxy to connect through, if any; None by
2326 default; str of the format 'http://127.0.0.1:8888' to
2327 set one
2328 · cacert -- A CA bundle file to use in verifying a TLS
2330 server certificate.
2331 · insecure -- Allow to access servers without checking
2333 SSL certs. The server's certificate will not be veri‐
2334 fied.
2335 · cert -- Client certificate file to connect on SSL
2337 server requiring SSL client certificate.
2338 · cert_key -- Client certificate private key file.
2340 · ssl_compression -- SSL compression should be disabled
2342 by default and this setting is not usable as of now.
2343 The parameter is kept for backward compatibility.
2344 · default_user_agent -- Set the User-Agent header on
2346 every request. If set to None (default), the user
2347 agent will be "python-swiftclient-<version>". This may
2348 be overridden on a per-request basis by explicitly set‐
2349 ting the user-agent header on a call to request().
2350 · timeout -- socket read timeout value, passed directly
2352 to the requests library.
2353 Raises ClientException -- Unable to handle protocol scheme
2355 getresponse()
2357 Adapt requests response to httplib interface
2358 putrequest(full_path, data=None, headers=None, files=None)
2360 Use python-requests files upload
2361 Parameters
2363 · data -- Use data generator for chunked-transfer
2365 · files -- Use files for default transfer
2367 request(method, full_path, data=None, headers=None, files=None)
2369 Encode url and header, then call requests.request
2370 swiftclient.client.LOGGER_SENSITIVE_HEADERS = ['x-auth-token',
2372 'x-auth-key', 'x-service-token', 'x-storage-token',
2373 'x-account-meta-temp-url-key', 'x-account-meta-temp-url-key-2', 'x-con‐
2374 tainer-meta-temp-url-key', 'x-container-meta-temp-url-key-2',
2375 'set-cookie']
2376 A list of sensitive headers to redact in logs. Note that when
2377 extending this list, the header names must be added in all lower
2378 case.
2379 swiftclient.client.copy_object(url, token, container, name, destina‐
2381 tion=None, headers=None, fresh_metadata=None, http_conn=None,
2382 response_dict=None, service_token=None)
2383 Copy object
2384 Parameters
2386 · url -- storage URL
2388 · token -- auth token; if None, no token will be sent
2390 · container -- container name that the source object is
2392 in
2393 · name -- source object name
2395 · destination -- The container and object name of the
2397 destination object in the form of /container/object; if
2398 None, the copy will use the source as the destination.
2399 · headers -- additional headers to include in the request
2401 · fresh_metadata -- Enables object creation that omits
2403 existing user metadata, default None
2404 · http_conn -- HTTP connection object (If None, it will
2406 create the conn object)
2407 · response_dict -- an optional dictionary into which to
2409 place the response - status, reason and headers
2410 · service_token -- service auth token
2412 Raises ClientException -- HTTP COPY request failed
2414 swiftclient.client.delete_container(url, token, container,
2416 http_conn=None, response_dict=None, service_token=None,
2417 query_string=None, headers=None)
2418 Delete a container
2419 Parameters
2421 · url -- storage URL
2423 · token -- auth token
2425 · container -- container name to delete
2427 · http_conn -- a tuple of (parsed url, HTTPConnection
2429 object), (If None, it will create the conn object)
2430 · response_dict -- an optional dictionary into which to
2432 place the response - status, reason and headers
2433 · service_token -- service auth token
2435 · query_string -- if set will be appended with '?' to
2437 generated path
2438 · headers -- additional headers to include in the request
2440 Raises ClientException -- HTTP DELETE request failed
2442 swiftclient.client.delete_object(url, token=None, container=None,
2444 name=None, http_conn=None, headers=None, proxy=None, query_string=None,
2445 response_dict=None, service_token=None)
2446 Delete object
2447 Parameters
2449 · url -- storage URL
2451 · token -- auth token; if None, no token will be sent
2453 · container -- container name that the object is in; if
2455 None, the container name is expected to be part of the
2456 url
2457 · name -- object name to delete; if None, the object name
2459 is expected to be part of the url
2460 · http_conn -- a tuple of (parsed url, HTTPConnection
2462 object), (If None, it will create the conn object)
2463 · headers -- additional headers to include in the request
2465 · proxy -- proxy to connect through, if any; None by
2467 default; str of the format 'http://127.0.0.1:8888' to
2468 set one
2469 · query_string -- if set will be appended with '?' to
2471 generated path
2472 · response_dict -- an optional dictionary into which to
2474 place the response - status, reason and headers
2475 · service_token -- service auth token
2477 Raises ClientException -- HTTP DELETE request failed
2479 swiftclient.client.encode_meta_headers(headers)
2481 Only encode metadata headers keys
2482 swiftclient.client.encode_utf8(value)
2484 swiftclient.client.get_account(url, token, marker=None, limit=None,
2486 prefix=None, end_marker=None, http_conn=None, full_listing=False, ser‐
2487 vice_token=None, headers=None)
2488 Get a listing of containers for the account.
2489 Parameters
2491 · url -- storage URL
2493 · token -- auth token
2495 · marker -- marker query
2497 · limit -- limit query
2499 · prefix -- prefix query
2501 · end_marker -- end_marker query
2503 · http_conn -- a tuple of (parsed url, HTTPConnection
2505 object), (If None, it will create the conn object)
2506 · full_listing -- if True, return a full listing, else
2508 returns a max of 10000 listings
2509 · service_token -- service auth token
2511 · headers -- additional headers to include in the request
2513 Returns
2515 a tuple of (response headers, a list of containers) The
2516 response headers will be a dict and all header names will
2517 be lowercase.
2518 Raises ClientException -- HTTP GET request failed
2520 swiftclient.client.get_auth(auth_url, user, key, **kwargs)
2522 Get authentication/authorization credentials.
2523 Parameters
2525 · auth_version -- the api version of the supplied auth
2527 params
2528 · os_options -- a dict, the openstack identity service
2530 options
2531 Returns
2533 a tuple, (storage_url, token)
2534 N.B. if the optional os_options parameter includes a non-empty
2536 'object_storage_url' key it will override the default storage
2537 url returned by the auth service.
2538 The snet parameter is used for Rackspace's ServiceNet internal
2540 network implementation. In this function, it simply adds snet-
2541 to the beginning of the host name for the returned storage URL.
2542 With Rackspace Cloud Files, use of this network path causes no
2543 bandwidth charges but requires the client to be running on
2544 Rackspace's ServiceNet network.
2545 swiftclient.client.get_auth_1_0(url, user, key, snet, **kwargs)
2547 swiftclient.client.get_auth_keystone(auth_url, user, key, os_options,
2549 **kwargs)
2550 Authenticate against a keystone server.
2551 We are using the keystoneclient library for authentication.
2553 swiftclient.client.get_capabilities(http_conn)
2555 Get cluster capability infos.
2556 Parameters
2558 http_conn -- a tuple of (parsed url, HTTPConnection
2559 object)
2560 Returns
2562 a dict containing the cluster capabilities
2563 Raises ClientException -- HTTP Capabilities GET failed
2565 swiftclient.client.get_container(url, token, container, marker=None,
2567 limit=None, prefix=None, delimiter=None, end_marker=None, path=None,
2568 http_conn=None, full_listing=False, service_token=None, headers=None,
2569 query_string=None)
2570 Get a listing of objects for the container.
2571 Parameters
2573 · url -- storage URL
2575 · token -- auth token
2577 · container -- container name to get a listing for
2579 · marker -- marker query
2581 · limit -- limit query
2583 · prefix -- prefix query
2585 · delimiter -- string to delimit the queries on
2587 · end_marker -- marker query
2589 · path -- path query (equivalent: "delimiter=/" and "pre‐
2591 fix=path/")
2592 · http_conn -- a tuple of (parsed url, HTTPConnection
2594 object), (If None, it will create the conn object)
2595 · full_listing -- if True, return a full listing, else
2597 returns a max of 10000 listings
2598 · service_token -- service auth token
2600 · headers -- additional headers to include in the request
2602 · query_string -- if set will be appended with '?' to
2604 generated path
2605 Returns
2607 a tuple of (response headers, a list of objects) The
2608 response headers will be a dict and all header names will
2609 be lowercase.
2610 Raises ClientException -- HTTP GET request failed
2612 swiftclient.client.get_keystoneclient_2_0(auth_url, user, key,
2614 os_options, **kwargs)
2615 swiftclient.client.get_object(url, token, container, name,
2617 http_conn=None, resp_chunk_size=None, query_string=None,
2618 response_dict=None, headers=None, service_token=None)
2619 Get an object
2620 Parameters
2622 · url -- storage URL
2624 · token -- auth token
2626 · container -- container name that the object is in
2628 · name -- object name to get
2630 · http_conn -- a tuple of (parsed url, HTTPConnection
2632 object), (If None, it will create the conn object)
2633 · resp_chunk_size -- if defined, chunk size of data to
2635 read. NOTE: If you specify a resp_chunk_size you must
2636 fully read the object's contents before making another
2637 request.
2638 · query_string -- if set will be appended with '?' to
2640 generated path
2641 · response_dict -- an optional dictionary into which to
2643 place the response - status, reason and headers
2644 · headers -- an optional dictionary with additional head‐
2646 ers to include in the request
2647 · service_token -- service auth token
2649 Returns
2651 a tuple of (response headers, the object's contents) The
2652 response headers will be a dict and all header names will
2653 be lowercase.
2654 Raises ClientException -- HTTP GET request failed
2656 swiftclient.client.head_account(url, token, http_conn=None, head‐
2658 ers=None, service_token=None)
2659 Get account stats.
2660 Parameters
2662 · url -- storage URL
2664 · token -- auth token
2666 · http_conn -- a tuple of (parsed url, HTTPConnection
2668 object), (If None, it will create the conn object)
2669 · headers -- additional headers to include in the request
2671 · service_token -- service auth token
2673 Returns
2675 a dict containing the response's headers (all header
2676 names will be lowercase)
2677 Raises ClientException -- HTTP HEAD request failed
2679 swiftclient.client.head_container(url, token, container,
2681 http_conn=None, headers=None, service_token=None)
2682 Get container stats.
2683 Parameters
2685 · url -- storage URL
2687 · token -- auth token
2689 · container -- container name to get stats for
2691 · http_conn -- a tuple of (parsed url, HTTPConnection
2693 object), (If None, it will create the conn object)
2694 · headers -- additional headers to include in the request
2696 · service_token -- service auth token
2698 Returns
2700 a dict containing the response's headers (all header
2701 names will be lowercase)
2702 Raises ClientException -- HTTP HEAD request failed
2704 swiftclient.client.head_object(url, token, container, name,
2706 http_conn=None, service_token=None, headers=None)
2707 Get object info
2708 Parameters
2710 · url -- storage URL
2712 · token -- auth token
2714 · container -- container name that the object is in
2716 · name -- object name to get info for
2718 · http_conn -- a tuple of (parsed url, HTTPConnection
2720 object), (If None, it will create the conn object)
2721 · service_token -- service auth token
2723 · headers -- additional headers to include in the request
2725 Returns
2727 a dict containing the response's headers (all header
2728 names will be lowercase)
2729 Raises ClientException -- HTTP HEAD request failed
2731 swiftclient.client.http_connection(*arg, **kwarg)
2733 Returns
2735 tuple of (parsed url, connection object)
2736 swiftclient.client.http_log(args, kwargs, resp, body)
2738 swiftclient.client.logger_settings = {'redact_sensitive_headers': True,
2740 'reveal_sensitive_prefix': 16}
2741 Default behaviour is to redact header values known to contain
2742 secrets, such as X-Auth-Key and X-Auth-Token. Up to the first 16
2743 chars may be revealed.
2744 To disable, set the value of redact_sensitive_headers to False.
2746 When header redaction is enabled, reveal_sensitive_prefix con‐
2748 figures the maximum length of any sensitive header data sent to
2749 the logs. If the header is less than twice this length, only
2750 int(len(value)/2) chars will be logged; if it is less than 15
2751 chars long, even less will be logged.
2752 swiftclient.client.parse_header_string(data)
2754 swiftclient.client.post_account(url, token, headers, http_conn=None,
2756 response_dict=None, service_token=None, query_string=None, data=None)
2757 Update an account's metadata.
2758 Parameters
2760 · url -- storage URL
2762 · token -- auth token
2764 · headers -- additional headers to include in the request
2766 · http_conn -- a tuple of (parsed url, HTTPConnection
2768 object), (If None, it will create the conn object)
2769 · response_dict -- an optional dictionary into which to
2771 place the response - status, reason and headers
2772 · service_token -- service auth token
2774 · query_string -- if set will be appended with '?' to
2776 generated path
2777 · data -- an optional message body for the request
2779 Raises ClientException -- HTTP POST request failed
2781 Returns
2783 resp_headers, body
2784 swiftclient.client.post_container(url, token, container, headers,
2786 http_conn=None, response_dict=None, service_token=None)
2787 Update a container's metadata.
2788 Parameters
2790 · url -- storage URL
2792 · token -- auth token
2794 · container -- container name to update
2796 · headers -- additional headers to include in the request
2798 · http_conn -- a tuple of (parsed url, HTTPConnection
2800 object), (If None, it will create the conn object)
2801 · response_dict -- an optional dictionary into which to
2803 place the response - status, reason and headers
2804 · service_token -- service auth token
2806 Raises ClientException -- HTTP POST request failed
2808 swiftclient.client.post_object(url, token, container, name, headers,
2810 http_conn=None, response_dict=None, service_token=None)
2811 Update object metadata
2812 Parameters
2814 · url -- storage URL
2816 · token -- auth token
2818 · container -- container name that the object is in
2820 · name -- name of the object to update
2822 · headers -- additional headers to include in the request
2824 · http_conn -- a tuple of (parsed url, HTTPConnection
2826 object), (If None, it will create the conn object)
2827 · response_dict -- an optional dictionary into which to
2829 place the response - status, reason and headers
2830 · service_token -- service auth token
2832 Raises ClientException -- HTTP POST request failed
2834 swiftclient.client.put_container(url, token, container, headers=None,
2836 http_conn=None, response_dict=None, service_token=None,
2837 query_string=None)
2838 Create a container
2839 Parameters
2841 · url -- storage URL
2843 · token -- auth token
2845 · container -- container name to create
2847 · headers -- additional headers to include in the request
2849 · http_conn -- a tuple of (parsed url, HTTPConnection
2851 object), (If None, it will create the conn object)
2852 · response_dict -- an optional dictionary into which to
2854 place the response - status, reason and headers
2855 · service_token -- service auth token
2857 · query_string -- if set will be appended with '?' to
2859 generated path
2860 Raises ClientException -- HTTP PUT request failed
2862 swiftclient.client.put_object(url, token=None, container=None,
2864 name=None, contents=None, content_length=None, etag=None,
2865 chunk_size=None, content_type=None, headers=None, http_conn=None,
2866 proxy=None, query_string=None, response_dict=None, service_token=None)
2867 Put an object
2868 Parameters
2870 · url -- storage URL
2872 · token -- auth token; if None, no token will be sent
2874 · container -- container name that the object is in; if
2876 None, the container name is expected to be part of the
2877 url
2878 · name -- object name to put; if None, the object name is
2880 expected to be part of the url
2881 · contents -- a string, a file-like object or an iterable
2883 to read object data from; if None, a zero-byte put will
2884 be done
2885 · content_length -- value to send as content-length
2887 header; also limits the amount read from contents; if
2888 None, it will be computed via the contents or chunked
2889 transfer encoding will be used
2890 · etag -- etag of contents; if None, no etag will be sent
2892 · chunk_size -- chunk size of data to write; it defaults
2894 to 65536; used only if the contents object has a 'read'
2895 method, e.g. file-like objects, ignored otherwise
2896 · content_type -- value to send as content-type header,
2898 overriding any value included in the headers param; if
2899 None and no value is found in the headers param, an
2900 empty string value will be sent
2901 · headers -- additional headers to include in the
2903 request, if any
2904 · http_conn -- a tuple of (parsed url, HTTPConnection
2906 object), (If None, it will create the conn object)
2907 · proxy -- proxy to connect through, if any; None by
2909 default; str of the format 'http://127.0.0.1:8888' to
2910 set one
2911 · query_string -- if set will be appended with '?' to
2913 generated path
2914 · response_dict -- an optional dictionary into which to
2916 place the response - status, reason and headers
2917 · service_token -- service auth token
2919 Returns
2921 etag
2922 Raises ClientException -- HTTP PUT request failed
2924 swiftclient.client.quote(value, safe='/')
2926 Patched version of urllib.quote that encodes utf8 strings before
2927 quoting. On Python 3, call directly urllib.parse.quote().
2928 swiftclient.client.resp_header_dict(resp)
2930 swiftclient.client.safe_value(name, value)
2932 Only show up to logger_settings['reveal_sensitive_prefix'] char‐
2933 acters from a sensitive header.
2934 Parameters
2936 · name -- Header name
2938 · value -- Header value
2940 Returns
2942 Safe header value
2943 swiftclient.client.scrub_headers(headers)
2945 Redact header values that can contain sensitive information that
2946 should not be logged.
2947 Parameters
2949 headers -- Either a dict or an iterable of two-element
2950 tuples
2951 Returns
2953 Safe dictionary of headers with sensitive information
2954 removed
2955 swiftclient.client.store_response(resp, response_dict)
2957 store information about an operation into a dict
2958 Parameters
2960 · resp -- an http response object containing the response
2962 headers
2963 · response_dict -- a dict into which are placed the sta‐
2965 tus, reason and a dict of lower-cased headers
2966 swiftclient.service
2968 class swiftclient.service.ResultsIterator(futures)
2969 Bases: six.Iterator
2970 class swiftclient.service.SwiftCopyObject(object_name, options=None)
2972 Bases: object
2973 Class for specifying an object copy, allowing the destina‐
2975 tion/headers/metadata/fresh_metadata to be specified separately
2976 for each individual object. destination and fresh_metadata
2977 should be set in options
2978 exception swiftclient.service.SwiftError(value, container=None,
2980 obj=None, segment=None, exc=None)
2981 Bases: exceptions.Exception
2982 class swiftclient.service.SwiftPostObject(object_name, options=None)
2984 Bases: object
2985 Class for specifying an object post, allowing the headers/meta‐
2987 data to be specified separately for each individual object.
2988 class swiftclient.service.SwiftService(options=None)
2990 Bases: object
2991 Service for performing swift operations
2993 capabilities(url=None, refresh_cache=False)
2995 List the cluster capabilities.
2996 Parameters
2998 url -- Proxy URL of the cluster to retrieve capa‐
2999 bilities.
3000 Returns
3002 A dictionary containing the capabilities of the
3003 cluster.
3004 Raises ClientException --
3006 copy(container, objects, options=None)
3008 Copy operations on a list of objects in a container. Des‐
3009 tination containers will be created.
3010 Parameters
3012 · container -- The container from which to copy
3014 the objects.
3015 · objects --
3017 A list of object names (strings) or SwiftCopyOb‐
3019 ject instances containing an object name and an
3020 options dict (can be None) to override the
3021 options for that individual copy operation:
3022 [
3024 'object_name',
3025 SwiftCopyObject(
3026 'object_name',
3027 options={
3028 'destination': '/container/object',
3029 'fresh_metadata': False,
3030 ...
3031 }),
3032 ...
3033 ]
3034 The options dict is described below.
3036 · options --
3039 A dictionary containing options to override the
3041 global options specified during the service
3042 object creation. These options are applied to
3043 all copy operations performed by this call,
3044 unless overridden on a per object basis. The
3045 options "destination" and "fresh_metadata" do
3046 not need to be set, in this case objects will be
3047 copied onto themselves and metadata will not be
3048 refreshed. The option "destination" can also be
3049 specified in the format '/container', in which
3050 case objects without an explicit destination
3051 will be copied to the destination /con‐
3052 tainer/original_object_name. Combinations of
3053 multiple objects and a destination in the format
3054 '/container/object' is invalid. Possible options
3055 are given below:
3056 {
3058 'meta': [],
3059 'header': [],
3060 'destination': '/container/object',
3061 'fresh_metadata': False,
3062 }
3063 Returns
3066 A generator returning the results of copying the
3067 given list of objects.
3068 Raises SwiftError --
3070 delete(container=None, objects=None, options=None)
3072 Delete operations on an account, optional container and
3073 optional list of objects.
3074 Parameters
3076 · container -- The container to delete or delete
3078 from.
3079 · objects -- The list of objects to delete.
3081 · options --
3083 A dictionary containing options to override the
3085 global options specified during the service
3086 object creation:
3087 {
3089 'yes_all': False,
3090 'leave_segments': False,
3091 'prefix': None,
3092 'header': [],
3093 }
3094 Returns
3097 A generator for returning the results of the
3098 delete operations. Each result yielded from the
3099 generator is either a 'delete_container',
3100 'delete_object', 'delete_segment', or
3101 'bulk_delete' dictionary containing the results of
3102 an individual delete operation.
3103 Raises
3105 · ClientException --
3107 · SwiftError --
3109 download(container=None, objects=None, options=None)
3111 Download operations on an account, optional container and
3112 optional list of objects.
3113 Parameters
3115 · container -- The container to download from.
3117 · objects -- A list of object names to download (a
3119 list of strings).
3120 · options --
3122 A dictionary containing options to override the
3124 global options specified during the service
3125 object creation:
3126 {
3128 'yes_all': False,
3129 'marker': '',
3130 'prefix': None,
3131 'no_download': False,
3132 'header': [],
3133 'skip_identical': False,
3134 'out_directory': None,
3135 'checksum': True,
3136 'out_file': None,
3137 'remove_prefix': False,
3138 'shuffle' : False
3139 }
3140 Returns
3143 A generator for returning the results of the down‐
3144 load operations. Each result yielded from the gen‐
3145 erator is a 'download_object' dictionary contain‐
3146 ing the results of an individual file download.
3147 Raises
3149 · ClientException --
3151 · SwiftError --
3153 list(container=None, options=None)
3155 List operations on an account, container.
3156 Parameters
3158 · container -- The container to make the list
3160 operation against.
3161 · options --
3163 A dictionary containing options to override the
3165 global options specified during the service
3166 object creation:
3167 {
3169 'long': False,
3170 'prefix': None,
3171 'delimiter': None,
3172 'header': []
3173 }
3174 Returns
3177 A generator for returning the results of the list
3178 operation on an account or container. Each result
3179 yielded from the generator is either a
3180 'list_account_part' or 'list_container_part', con‐
3181 taining part of the listing.
3182 post(container=None, objects=None, options=None)
3184 Post operations on an account, container or list of
3185 objects
3186 Parameters
3188 · container -- The container to make the post
3190 operation against.
3191 · objects --
3193 A list of object names (strings) or SwiftPostOb‐
3195 ject instances containing an object name, and an
3196 options dict (can be None) to override the
3197 options for that individual post operation:
3198 [
3200 'object_name',
3201 SwiftPostObject('object_name', options={...}),
3202 ...
3203 ]
3204 The options dict is described below.
3206 · options --
3209 A dictionary containing options to override the
3211 global options specified during the service
3212 object creation. These options are applied to
3213 all post operations performed by this call,
3214 unless overridden on a per object basis. Possi‐
3215 ble options are given below:
3216 {
3218 'meta': [],
3219 'header': [],
3220 'read_acl': None, # For containers only
3221 'write_acl': None, # For containers only
3222 'sync_to': None, # For containers only
3223 'sync_key': None # For containers only
3224 }
3225 Returns
3228 Either a single result dictionary in the case of a
3229 post to a container/account, or an iterator for
3230 returning the results of posts to a list of
3231 objects.
3232 Raises SwiftError --
3234 stat(container=None, objects=None, options=None)
3236 Get account stats, container stats or information about a
3237 list of objects in a container.
3238 Parameters
3240 · container -- The container to query.
3242 · objects -- A list of object paths about which to
3244 return information (a list of strings).
3245 · options --
3247 A dictionary containing options to override the
3249 global options specified during the service
3250 object creation. These options are applied to
3251 all stat operations performed by this call:
3252 {
3254 'human': False,
3255 'header': []
3256 }
3257 Returns
3260 Either a single dictionary containing stats about
3261 an account or container, or an iterator for
3262 returning the results of the stat operations on a
3263 list of objects.
3264 Raises SwiftError --
3266 upload(container, objects, options=None)
3268 Upload a list of objects to a given container.
3269 Parameters
3271 · container -- The container (or pseudo-folder
3273 path) to put the uploads into.
3274 · objects --
3276 A list of file/directory names (strings) or
3278 SwiftUploadObject instances containing a source
3279 for the created object, an object name, and an
3280 options dict (can be None) to override the
3281 options for that individual upload operation:
3282 [
3284 '/path/to/file',
3285 SwiftUploadObject('/path', object_name='obj1'),
3286 ...
3287 ]
3288 The options dict is as described below.
3290 The SwiftUploadObject source may be one of:
3292 · A file-like object (with a read method)
3294 · A string containing the path to a local
3296 file or directory
3297 · None, to indicate that we want an empty
3299 object
3300 · options --
3303 A dictionary containing options to override the
3305 global options specified during the service
3306 object creation. These options are applied to
3307 all upload operations performed by this call,
3308 unless overridden on a per object basis. Possi‐
3309 ble options are given below:
3310 {
3312 'meta': [],
3313 'header': [],
3314 'segment_size': None,
3315 'use_slo': False,
3316 'segment_container': None,
3317 'leave_segments': False,
3318 'changed': None,
3319 'skip_identical': False,
3320 'fail_fast': False,
3321 'dir_marker': False # Only for None sources
3322 }
3323 Returns
3326 A generator for returning the results of the
3327 uploads.
3328 Raises
3330 · SwiftError --
3332 · ClientException --
3334 class swiftclient.service.SwiftUploadObject(source, object_name=None,
3336 options=None)
3337 Bases: object
3338 Class for specifying an object upload, allowing the object
3340 source, name and options to be specified separately for each
3341 individual object.
3342 swiftclient.service.get_conn(options)
3344 Return a connection building it from the options.
3345 swiftclient.service.get_from_queue(q, timeout=864000)
3347 swiftclient.service.get_future_result(f, timeout=86400)
3349 swiftclient.service.interruptable_as_completed(fs, timeout=86400)
3351 swiftclient.service.mkdirs(path)
3353 swiftclient.service.process_options(options)
3355 swiftclient.service.split_headers(options, prefix=u'')
3357 Splits 'Key: Value' strings and returns them as a dictionary.
3358 Parameters
3360 · options -- Must be one of: * an iterable of 'Key:
3362 Value' strings * an iterable of ('Key', 'Value') pairs
3363 * a dict of {'Key': 'Value'} pairs
3364 · prefix -- String to prepend to all of the keys in the
3366 dictionary. reporting.
3367 swiftclient.exceptions
3369 exception swiftclient.exceptions.ClientException(msg, http_scheme='',
3370 http_host='', http_port='', http_path='', http_query='', http_sta‐
3371 tus=None, http_reason='', http_device='', http_response_content='',
3372 http_response_headers=None)
3373 Bases: exceptions.Exception
3374 classmethod from_response(resp, msg=None, body=None)
3376 swiftclient.multithreading
3378 class swiftclient.multithreading.ConnectionThreadPoolExecutor(cre‐
3379 ate_connection, max_workers)
3380 Bases: concurrent.futures.thread.ThreadPoolExecutor
3381 A wrapper class to maintain a pool of connections alongside the
3383 thread pool. We start by creating a priority queue of connec‐
3384 tions, and each job submitted takes one of those connections
3385 (initialising if necessary) and passes it as the first arg to
3386 the executed function.
3387 At the end of execution that connection is returned to the
3389 queue.
3390 By using a PriorityQueue we avoid creating more connections than
3392 required. We will only create as many connections as are
3393 required concurrently.
3394 submit(fn, *args, **kwargs)
3396 Submits a callable to be executed with the given argu‐
3397 ments.
3398 Schedules the callable to be executed as fn(
3400 *
3401 args,
3402 **
3403 kwargs) and returns a Future instance representing the
3404 execution of the callable.
3405 Returns:
3407 A Future representing the given call.
3408 class swiftclient.multithreading.MultiThreadingManager(create_connec‐
3410 tion, segment_threads=10, object_dd_threads=10, object_uu_threads=10,
3411 container_threads=10)
3412 Bases: object
3413 One object to manage context for multi-threading. This should
3415 make bin/swift less error-prone and allow us to test this code.
3416 Parameters
3418 · segment_threads -- The number of threads allocated to
3420 segment uploads
3421 · object_dd_threads -- The number of threads allocated to
3423 object download/delete jobs
3424 · object_uu_threads -- The number of threads allocated to
3426 object upload/update based jobs
3427 · container_threads -- The number of threads allocated to
3429 container/account level jobs
3430 class swiftclient.multithreading.OutputManager(print_stream=None,
3432 error_stream=None)
3433 Bases: object
3434 One object to manage and provide helper functions for output.
3436 This object is a context manager and returns itself into the
3438 context. When entering the context, two printing threads are
3439 created (see below) and they are waited on and cleaned up when
3440 exiting the context.
3441 Also, thread-safe printing to two streams is provided. The
3443 print_msg() method will print to the supplied print_stream
3444 (defaults to sys.stdout) and the error() method will print to
3445 the supplied error_stream (defaults to sys.stderr). Both of
3446 these printing methods will format the given string with any
3447 supplied *args (a la printf). On Python 2, Unicode messages are
3448 encoded to utf8.
3449 The attribute self.error_count is incremented once per error
3451 message printed, so an application can tell if any worker
3452 threads encountered exceptions or otherwise called error() on
3453 this instance. The swift command-line tool uses this to exit
3454 non-zero if any error strings were printed.
3455 Parameters
3457 · print_stream -- The stream to which print_msg() sends
3459 formatted messages.
3460 · error_stream -- The stream to which error() sends for‐
3462 matted messages.
3463 On Python 2, Unicode messages are encoded to utf8.
3465 DEFAULT_OFFSET = 14
3467 error(msg, *fmt_args)
3469 get_error_count()
3471 print_items(items, offset=14, skip_missing=False)
3473 print_msg(msg, *fmt_args)
3475 print_raw(data)
3477 warning(msg, *fmt_args)
3479 swiftclient.utils
3481 Miscellaneous utility functions for use with Swift.
3482 class swiftclient.utils.LengthWrapper(readable, length, md5=False)
3484 Bases: object
3485 Wrap a filelike object with a maximum length.
3487 Fix for https://github.com/kennethreitz/requests/issues/1648.
3489 It is recommended to use this class only on files opened in
3490 binary mode.
3491 Parameters
3493 · readable -- The filelike object to read from.
3495 · length -- The maximum amount of content that can be
3497 read from the filelike object before it is simulated to
3498 be empty.
3499 · md5 -- Flag to enable calculating the MD5 of the con‐
3501 tent as it is read.
3502 get_md5sum()
3504 read(size=-1)
3506 reset
3508 class swiftclient.utils.NoopMD5(*a, **kw)
3510 Bases: object
3511 hexdigest(*a, **kw)
3513 update(*a, **kw)
3515 class swiftclient.utils.ReadableToIterable(content, chunk_size=65536,
3517 md5=False)
3518 Bases: object
3519 Wrap a filelike object and act as an iterator.
3521 It is recommended to use this class only on files opened in
3523 binary mode. Due to the Unicode changes in Python 3, files are
3524 now opened using an encoding not suitable for use with the md5
3525 class and because of this hit the exception on every call to
3526 next. This could cause problems, especially with large files and
3527 small chunk sizes.
3528 Parameters
3530 · content -- The filelike object that is yielded from.
3532 · chunk_size -- The max size of each yielded item.
3534 · md5 -- Flag to enable calculating the MD5 of the con‐
3536 tent as it is yielded.
3537 get_md5sum()
3539 next()
3541 swiftclient.utils.config_true_value(value)
3543 Returns True if the value is either True or a string in
3544 TRUE_VALUES. Returns False otherwise. This function comes from
3545 swift.common.utils.config_true_value()
3546 swiftclient.utils.generate_temp_url(path, seconds, key, method, abso‐
3548 lute=False, prefix=False, iso8601=False)
3549 Generates a temporary URL that gives unauthenticated access to
3550 the Swift object.
3551 Parameters
3553 · path -- .INDENT 2.0
3555 The full path to the Swift object or prefix if
3557 a prefix-based temporary URL should be generated. Exam‐
3558 ple:
3559 /v1/AUTH_account/c/o or /v1/AUTH_account/c/prefix.
3561 · seconds -- time in seconds or ISO 8601 timestamp. If absolute
3564 is False and this is the string representation of an integer,
3565 then this specifies the amount of time in seconds for which
3566 the temporary URL will be valid. If absolute is True then
3567 this specifies an absolute time at which the temporary URL
3568 will expire.
3569 · key -- The secret temporary URL key set on the Swift cluster.
3571 To set a key, run 'swift post -m "Temp-URL-Key: <substitute
3572 tempurl key here>"'
3573 · method -- A HTTP method, typically either GET or PUT, to allow
3575 for this temporary URL.
3576 · absolute -- if True then the seconds parameter is interpreted
3578 as a Unix timestamp, if seconds represents an integer.
3579 · prefix -- if True then a prefix-based temporary URL will be
3581 generated.
3582 · iso8601 -- if True, a URL containing an ISO 8601 UTC timestamp
3584 instead of a UNIX timestamp will be created.
3585 Raises ValueError -- if timestamp or path is not in valid format.
3587 Returns
3589 the path portion of a temporary URL
3590 swiftclient.utils.get_body(headers, body)
3592 swiftclient.utils.iter_wrapper(iterable)
3594 swiftclient.utils.n_at_a_time(seq, n)
3596 swiftclient.utils.n_groups(seq, n)
3598 swiftclient.utils.parse_api_response(headers, body)
3600 swiftclient.utils.prt_bytes(num_bytes, human_flag)
3602 convert a number > 1024 to printable format, either in 4 char -h
3603 format as with ls -lh or return as 12 char right justified
3604 string
3605 swiftclient.utils.report_traceback()
3607 Reports a timestamp and full traceback for a given exception.
3608 Returns
3610 Full traceback and timestamp.
3611 swiftclient.utils.split_request_headers(options, prefix='')
3613
3615 · genindex
3616 · modindex
3618 · search
3620
3622 Copyright 2013 OpenStack, LLC.
3623 Licensed under the Apache License, Version 2.0 (the “License”); you
3625 may not use this file except in compliance with the License. You
3626 may obtain a copy of the License at
3627 http://www.apache.org/licenses/LICENSE-2.0
3628 Unless required by applicable law or agreed to in writing, software
3630 distributed under the License is distributed on an “AS IS” BASIS,
3631 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
3632 implied. See the License for the specific language governing per‐
3633 missions and limitations under the License.
3634
3636 2013-2016 OpenStack, LLC.
36373.5.0 Aug 25, 2018 PYTHON-SWIFTCLIENT(1)