1PYTHON-SWIFTCLIENT(1) python-swiftclient PYTHON-SWIFTCLIENT(1)
2
6 python-swiftclient - python-swiftclient 3.10.1
7
9 Introduction
10 Where to Start?
11 The python-swiftclient project comprises a command line tool and two
12 separate APIs for accessing swift programmatically. Choosing the most
13 appropriate method for a given use case is the first problem a user
14 needs to solve.
15 Use Cases
17 Alongside the command line tool, the python-swiftclient includes two
18 levels of API:
19 · A low level client API that provides simple Python wrappers around
21 the various authentication mechanisms and the individual HTTP
22 requests.
23 · A high level service API that provides methods for performing common
25 operations in parallel on a thread pool.
26 Example use cases:
28 ·
30 Uploading and retrieving data
32 Use the command line tool if you are simply uploading and
33 downloading files and directories to and from your filesystem.
34 The command line tool can be integrated into a shell script to
35 automate tasks.
36 ·
38 Integrating into an automated Python workflow
40 Use the SwiftService API to perform operations offered by the
41 CLI if your use case requires integration with a Python-based
42 workflow. This method offers greater control and flexibility
43 over individual object operations, such as the metadata set on
44 each object. The SwiftService class provides methods to per‐
45 form multiple sets of operations against a swift object store
46 using a configurable shared thread pool. A single instance of
47 the SwiftService class can be shared between multiple threads
48 in your own code.
49 ·
51 Developing an application in Python to access a swift object store
53 Use the SwiftService API to develop Python applications that
54 use swift to store and retrieve objects. A SwiftService
55 instance provides a configurable thread pool for performing
56 all operations supported by the CLI.
57 ·
59 Fine-grained control over threading or the requests being performed
61 Use the Connection API if your use case requires fine grained
62 control over advanced features or you wish to use your own
63 existing threading model. Examples of advanced features
64 requiring the use of the Connection API include creating an
65 SLO manifest that references already existing objects, or fine
66 grained control over the query strings supplied with each HTTP
67 request.
68 Important considerations
70 This section covers some important considerations, helpful hints, and
71 things to avoid when integrating an object store into your workflow.
72 An object store is not a filesystem
74 It cannot be stressed enough that your usage of the object store should
75 reflect the proper use case, and not treat the storage like a tradi‐
76 tional filesystem. There are two main restrictions to bear in mind
77 when designing an application that uses an object store:
78 · You cannot rename objects. Due to fact that the name of an object is
80 one of the factors that determines where the object and its replicas
81 are stored, renaming would require multiple copies of the data to be
82 moved between physical storage devices. If you want to rename an
83 object you must upload to the new location, or make a server side
84 copy request to the new location, and then delete the original.
85 · You cannot modify objects. Objects are stored in multiple locations
87 and are checked for integrity based on the MD5 sum calculated during
88 upload. In order to modify the contents of an object, the entire
89 desired contents must be re-uploaded. In certain special cases it is
90 possible to work around this restriction using large objects, but no
91 general file-like access is available to modify a stored object.
92 Objects cannot be locked
94 There is no mechanism to perform a combination of reading the
95 data/metadata from an object and writing an update to that data/meta‐
96 data in an atomic way. Any user with access to a container could update
97 the contents or metadata associated with an object at any time.
98 Workflows that assume that no updates have been made since the last
100 read of an object should be discouraged. Enabling a workflow of this
101 type requires an external object locking mechanism and/or cooperation
102 between all clients accessing the data.
103
105 So You Want to Contribute…
106 For general information on contributing to OpenStack, please check out
107 the contributor guide to get started. It covers all the basics that
108 are common to all OpenStack projects: the accounts you need, the basics
109 of interacting with our Gerrit review system, how we communicate as a
110 community, etc.
111 The python-swiftclient is maintained by the OpenStack Swift project.
113 To understand our development process and how you can contribute to it,
114 please look at the Swift project’s general contributor’s page:
115 http://docs.openstack.org/swift/latest/contributor/contributing.html
116 CLI
118 The swift tool is a command line utility for communicating with an
119 OpenStack Object Storage (swift) environment. It allows one to perform
120 several types of operations.
121 For help on a specific swift command, enter:
123 $ swift COMMAND --help
125 swift usage
127 Usage: swift [--version] [--help] [--os-help] [--snet] [--verbose]
128 [--debug] [--info] [--quiet] [--auth <auth_url>]
129 [--auth-version <auth_version> |
130 --os-identity-api-version <auth_version> ]
131 [--user <username>]
132 [--key <api_key>] [--retries <num_retries>]
133 [--os-username <auth-user-name>] [--os-password <auth-password>]
134 [--os-user-id <auth-user-id>]
135 [--os-user-domain-id <auth-user-domain-id>]
136 [--os-user-domain-name <auth-user-domain-name>]
137 [--os-tenant-id <auth-tenant-id>]
138 [--os-tenant-name <auth-tenant-name>]
139 [--os-project-id <auth-project-id>]
140 [--os-project-name <auth-project-name>]
141 [--os-project-domain-id <auth-project-domain-id>]
142 [--os-project-domain-name <auth-project-domain-name>]
143 [--os-auth-url <auth-url>] [--os-auth-token <auth-token>]
144 [--os-storage-url <storage-url>] [--os-region-name <region-name>]
145 [--os-service-type <service-type>]
146 [--os-endpoint-type <endpoint-type>]
147 [--os-cacert <ca-certificate>] [--insecure]
148 [--os-cert <client-certificate-file>]
149 [--os-key <client-certificate-key-file>]
150 [--no-ssl-compression]
151 <subcommand> [--help] [<subcommand options>]
152 Subcommands:
154 delete Delete a container or objects within a container.
156 download
158 Download objects from containers.
159 list Lists the containers for the account or the objects for a con‐
161 tainer.
162 post Updates meta information for the account, container, or object;
164 creates containers if not present.
165 copy Copies object, optionally adds meta
167 stat Displays information for the account, container, or object.
169 upload Uploads files or directories to the given container.
171 capabilities
173 List cluster capabilities.
174 tempurl
176 Create a temporary URL.
177 auth Display auth related environment variables.
179 swift optional arguments
181 --version
182 show program’s version number and exit
183 -h, --help
185 show this help message and exit
186 --os-help
188 Show OpenStack authentication options.
189 -s, --snet
191 Use SERVICENET internal network.
192 -v, --verbose
194 Print more info.
195 --debug
197 Show the curl commands and results of all http queries regard‐
198 less of result status.
199 --info Show the curl commands and results of all http queries which
201 return an error.
202 -q, --quiet
204 Suppress status output.
205 -A AUTH, --auth=AUTH
207 URL for obtaining an auth token.
208 -V AUTH_VERSION, --auth-version=AUTH_VERSION, --os-identity-api-ver‐
210 sion=AUTH_VERSION
211 Specify a version for authentication. Defaults to
212 env[ST_AUTH_VERSION], env[OS_AUTH_VERSION], env[OS_IDEN‐
213 TITY_API_VERSION] or 1.0.
214 -U USER, --user=USER
216 User name for obtaining an auth token.
217 -K KEY, --key=KEY
219 Key for obtaining an auth token.
220 -R RETRIES, --retries=RETRIES
222 The number of times to retry a failed connection.
223 --insecure
225 Allow swiftclient to access servers without having to verify the
226 SSL certificate. Defaults to env[SWIFTCLIENT_INSECURE] (set to
227 ‘true’ to enable).
228 --no-ssl-compression
230 This option is deprecated and not used anymore. SSL compression
231 should be disabled by default by the system SSL library.
232 --prompt
234 Prompt user to enter a password which overrides any password
235 supplied via --key, --os-password or environment variables.
236 Authentication
238 This section covers the options for authenticating with a swift object
239 store. The combinations of options required for each authentication
240 version are detailed below, but are just a subset of those that can be
241 used to successfully authenticate. These are the most common and recom‐
242 mended combinations.
243 You should obtain the details of your authentication version and cre‐
245 dentials from your storage provider. These details should make it
246 clearer which of the authentication sections below are most likely to
247 allow you to connect to your storage account.
248 Keystone v3
250 swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \
251 --os-project-name project1 --os-project-domain-name domain1 \
252 --os-username user --os-user-domain-name domain1 \
253 --os-password password list
254 swift --os-auth-url https://api.example.com:5000/v3 --auth-version 3 \
256 --os-project-id 0123456789abcdef0123456789abcdef \
257 --os-user-id abcdef0123456789abcdef0123456789 \
258 --os-password password list
259 Manually specifying the options above on the command line can be
261 avoided by setting the following combinations of environment variables:
262 ST_AUTH_VERSION=3
264 OS_USERNAME=user
265 OS_USER_DOMAIN_NAME=domain1
266 OS_PASSWORD=password
267 OS_PROJECT_NAME=project1
268 OS_PROJECT_DOMAIN_NAME=domain1
269 OS_AUTH_URL=https://api.example.com:5000/v3
270 ST_AUTH_VERSION=3
272 OS_USER_ID=abcdef0123456789abcdef0123456789
273 OS_PASSWORD=password
274 OS_PROJECT_ID=0123456789abcdef0123456789abcdef
275 OS_AUTH_URL=https://api.example.com:5000/v3
276 Keystone v2
278 swift --os-auth-url https://api.example.com:5000/v2.0 \
279 --os-tenant-name tenant \
280 --os-username user --os-password password list
281 Manually specifying the options above on the command line can be
283 avoided by setting the following environment variables:
284 ST_AUTH_VERSION=2.0
286 OS_USERNAME=user
287 OS_PASSWORD=password
288 OS_TENANT_NAME=tenant
289 OS_AUTH_URL=https://api.example.com:5000/v2.0
290 Legacy auth systems
292 You can configure swift to work with any number of other authentication
293 systems that we will not cover in this document. If your storage
294 provider is not using Keystone to provide access tokens, please contact
295 them for instructions on the required options. It is likely that the
296 options will need to be specified as below:
297 swift -A https://api.example.com/v1.0 -U user -K api_key list
299 Specifying the options above manually on the command line can be
301 avoided by setting the following environment variables:
302 ST_AUTH_VERSION=1.0
304 ST_AUTH=https://api.example.com/v1.0
305 ST_USER=user
306 ST_KEY=key
307 It is also possible that you need to use a completely separate auth
309 system, in which case swiftclient cannot request a token for you. In
310 this case you should make the authentication request separately and
311 access your storage using the token and storage URL options shown
312 below:
313 swift --os-auth-token 6ee5eb33efad4e45ab46806eac010566 \
315 --os-storage-url https://10.1.5.2:8080/v1/AUTH_ced809b6a4baea7aeab61a \
316 list
317 NOTE:
319 Leftover environment variables are a common source of confusion when
320 authorization fails.
321 CLI commands
323 Auth
324 Usage: swift auth
325 Display authentication variables in shell friendly format. Command to
327 run to export storage URL and auth token into OS_STORAGE_URL and
328 OS_AUTH_TOKEN: swift auth. Command to append to a runcom file (e.g.
329 ~/.bashrc, /etc/profile) for automatic authentication: swift auth -v -U
330 test:tester -K testing.
331 swift stat
333 Usage: swift stat [--lh] [--header <header:value>]
334 [<container> [<object>]]
335 Displays information for the account, container, or object depending on
337 the arguments given (if any). In verbose mode, the storage URL and the
338 authentication token are displayed as well.
339 Positional arguments:
341 [container]
343 Name of container to stat from.
344 [object]
346 Name of object to stat.
347 Optional arguments:
349 --lh Report sizes in human readable format similar to ls -lh.
351 -H, --header <header:value>
353 Adds a custom request header to use for stat.
354 swift list
356 Usage: swift list [--long] [--lh] [--totals] [--prefix <prefix>]
357 [--delimiter <delimiter>] [--header <header:value>]
358 [<container>]
359 Lists the containers for the account or the objects for a container.
361 The -p <prefix> or --prefix <prefix> is an option that will only list
362 items beginning with that prefix. The -d <delimiter> or --delimiter
363 <delimiter> is an option (for container listings only) that will roll
364 up items with the given delimiter (see OpenStack Swift general documen‐
365 tation <https://docs.openstack.org/swift/latest/> for what this means).
366 The -l and --lh options provide more detail, similar to ls -l and ls
368 -lh, the latter providing sizes in human readable format (For example:
369 3K, 12M, etc). The latter two switches use more overhead to retrieve
370 the displayed details, which is directly proportional to the number of
371 container or objects listed.
372 Positional arguments:
374 [container]
376 Name of container to list object in.
377 Optional arguments:
379 -l, --long
381 Long listing format, similar to ls -l.
382 --lh Report sizes in human readable format similar to ls -lh.
384 -t, --totals
386 Used with -l or –lh, only report totals.
387 -p <prefix>, --prefix <prefix>
389 Only list items beginning with the prefix.
390 -d <delim>, --delimiter <delim>
392 Roll up items with the given delimiter. For containers only. See
393 OpenStack Swift API documentation for what this means.
394 -H, --header <header:value>
396 Adds a custom request header to use for listing.
397 swift upload
399 Usage: swift upload [--changed] [--skip-identical] [--segment-size <size>]
400 [--segment-container <container>] [--leave-segments]
401 [--object-threads <thread>] [--segment-threads <threads>]
402 [--header <header>] [--use-slo] [--ignore-checksum]
403 [--object-name <object-name>]
404 <container> <file_or_directory> [<file_or_directory>] [...]
405 Uploads the files and directories specified by the remaining arguments
407 to the given container. The -c or --changed is an option that will only
408 upload files that have changed since the last upload. The --object-name
409 <object-name> is an option that will upload a file and name object to
410 <object-name> or upload a directory and use <object-name> as object
411 prefix. If the file name is “-“, client reads content from standard
412 input. In this case --object-name is required to set the name of the
413 object and no other files may be given. The -S <size> or --seg‐
414 ment-size <size> and --leave-segments are options as well (see --help
415 for more).
416 Positional arguments:
418 <container>
420 Name of container to upload to.
421 <file_or_directory>
423 Name of file or directory to upload. Specify multiple times for
424 multiple uploads.
425 Optional arguments:
427 -c, --changed
429 Only upload files that have changed since the last upload.
430 --skip-identical
432 Skip uploading files that are identical on both sides.
433 -S, --segment-size <size>
435 Upload files in segments no larger than <size> (in Bytes) and
436 then create a “manifest” file that will download all the seg‐
437 ments as if it were the original file.
438 --segment-container <container>
440 Upload the segments into the specified container. If not speci‐
441 fied, the segments will be uploaded to a <container>_segments
442 container to not pollute the main <container> listings.
443 --leave-segments
445 Indicates that you want the older segments of manifest objects
446 left alone (in the case of overwrites).
447 --object-threads <threads>
449 Number of threads to use for uploading full objects. Default is
450 10.
451 --segment-threads <threads>
453 Number of threads to use for uploading object segments. Default
454 is 10.
455 -H, --header <header:value>
457 Adds a customized request header. This option may be repeated.
458 Example: -H “content-type:text/plain” -H “Content-Length: 4000”.
459 --use-slo
461 When used in conjunction with –segment-size it will create a
462 Static Large Object instead of the default Dynamic Large Object.
463 --object-name <object-name>
465 Upload file and name object to <object-name> or upload dir and
466 use <object-name> as object prefix instead of folder name.
467 --ignore-checksum
469 Turn off checksum validation for uploads.
470 swift post
472 Usage: swift post [--read-acl <acl>] [--write-acl <acl>] [--sync-to <sync-to>]
473 [--sync-key <sync-key>] [--meta <name:value>]
474 [--header <header>]
475 [<container> [<object>]]
476 Updates meta information for the account, container, or object depend‐
478 ing on the arguments given. If the container is not found, the swift‐
479 client will create it automatically, but this is not true for accounts
480 and objects. Containers also allow the -r <read-acl> (or --read-acl
481 <read-acl>) and -w <write-acl> (or --write-acl <write-acl>) options.
482 The -m or --meta option is allowed on accounts, containers and objects,
483 and is used to define the user metadata items to set in the form
484 Name:Value. You can repeat this option. For example: post -m
485 Color:Blue -m Size:Large
486 For more information about ACL formats see the documentation: ACLs.
488 Positional arguments:
490 [container]
492 Name of container to post to.
493 [object]
495 Name of object to post.
496 Optional arguments:
498 -r, --read-acl <acl>
500 Read ACL for containers. Quick summary of ACL syntax: .r:*,
501 .r:-.example.com, .r:www.example.com, account1 (v1.0 identity
502 API only), account1:*, account2:user2 (v2.0+ identity API).
503 -w, --write-acl <acl>
505 Write ACL for containers. Quick summary of ACL syntax: account1
506 (v1.0 identity API only), account1:*, account2:user2 (v2.0+
507 identity API).
508 -t, --sync-to <sync-to>
510 Sync To for containers, for multi-cluster replication.
511 -k, --sync-key <sync-key>
513 Sync Key for containers, for multi-cluster replication.
514 -m, --meta <name:value>
516 Sets a meta data item. This option may be repeated.
517 Example: -m Color:Blue -m Size:Large
519 -H, --header <header:value>
521 Adds a customized request header. This option may be repeated.
522 Example: -H “content-type:text/plain” -H “Content-Length: 4000”
524 swift download
526 Usage: swift download [--all] [--marker <marker>] [--prefix <prefix>]
527 [--output <out_file>] [--output-dir <out_directory>]
528 [--object-threads <threads>] [--ignore-checksum]
529 [--container-threads <threads>] [--no-download]
530 [--skip-identical] [--remove-prefix]
531 [--header <header:value>] [--no-shuffle]
532 [<container> [<object>] [...]]
533 Downloads everything in the account (with --all), or everything in a
535 container, or a list of objects depending on the arguments given. For a
536 single object download, you may use the -o <filename> or --output
537 <filename> option to redirect the output to a specific file or - to re‐
538 direct to stdout. The --ignore-checksum is an option that turn off
539 checksum validation. You can specify optional headers with the repeat‐
540 able cURL-like option -H [--header <name:value>]. --ignore-mtime
541 ignores the x-object-meta-mtime metadata entry on the object (if
542 present) and instead creates the downloaded files with fresh atime and
543 mtime values.
544 Positional arguments:
546 <container>
548 Name of container to download from. To download a whole account,
549 omit this and specify –all.
550 <object>
552 Name of object to download. Specify multiple times for multiple
553 objects. Omit this to download all objects from the container.
554 Optional arguments:
556 -a, --all
558 Indicates that you really want to download everything in the
559 account.
560 -m, --marker <marker>
562 Marker to use when starting a container or account download.
563 -p, --prefix <prefix>
565 Only download items beginning with <prefix>
566 -r, --remove-prefix
568 An optional flag for –prefix <prefix>, use this option to down‐
569 load items without <prefix>
570 -o, --output <out_file>
572 For a single file download, stream the output to <out_file>.
573 Specifying “-” as <out_file> will redirect to stdout.
574 -D, --output-dir <out_directory>
576 An optional directory to which to store objects. By default,
577 all objects are recreated in the current directory.
578 --object-threads <threads>
580 Number of threads to use for downloading objects. Default is
581 10.
582 --container-threads <threads>
584 Number of threads to use for downloading containers. Default is
585 10.
586 --no-download
588 Perform download(s), but don’t actually write anything to disk.
589 -H, --header <header:value>
591 Adds a customized request header to the query, like “Range” or
592 “If-Match”. This option may be repeated.
593 Example: –header “content-type:text/plain”
595 --skip-identical
597 Skip downloading files that are identical on both sides.
598 --ignore-checksum
600 Turn off checksum validation for downloads.
601 --no-shuffle
603 By default, when downloading a complete account or container,
604 download order is randomised in order to reduce the load on
605 individual drives when multiple clients are executed simultane‐
606 ously to download the same set of objects (e.g. a nightly auto‐
607 mated download script to multiple servers). Enable this option
608 to submit download jobs to the thread pool in the order they are
609 listed in the object store.
610 swift delete
612 Usage: swift delete [--all] [--leave-segments]
613 [--object-threads <threads>]
614 [--container-threads <threads>]
615 [--header <header:value>]
616 [<container> [<object>] [...]]
617 Deletes everything in the account (with --all), or everything in a con‐
619 tainer, or a list of objects depending on the arguments given. Segments
620 of manifest objects will be deleted as well, unless you specify the
621 --leave-segments option.
622 Positional arguments:
624 [<container>]
626 Name of container to delete from.
627 [<object>]
629 Name of object to delete. Specify multiple times for multiple
630 objects.
631 Optional arguments:
633 -a, --all
635 Delete all containers and objects.
636 --leave-segments
638 Do not delete segments of manifest objects.
639 -H, --header <header:value>
641 Adds a custom request header to use for deleting objects or an
642 entire container.
643 --object-threads <threads>
645 Number of threads to use for deleting objects. Default is 10.
646 --container-threads <threads>
648 Number of threads to use for deleting containers. Default is
649 10.
650 swift copy
652 Usage: swift copy [--destination </container/object>] [--fresh-metadata]
653 [--meta <name:value>] [--header <header>] <container>
654 <object> [<object>] [...]
655 Copies an object to a new destination or adds user metadata to an
657 object. Depending on the options supplied, you can preserve existing
658 metadata in contrast to the post command. The --destination option sets
659 the copy target destination in the form /container/object. If not set,
660 the object will be copied onto itself which is useful for adding meta‐
661 data. You can use the -M or --fresh-metadata option to copy an object
662 without existing user meta data, and the -m or --meta option to define
663 user meta data items to set in the form Name:Value. You can repeat this
664 option. For example: copy -m Color:Blue -m Size:Large.
665 Positional arguments:
667 <container>
669 Name of container to copy from.
670 <object>
672 Name of object to copy. Specify multiple times for multiple
673 objects
674 Optional arguments:
676 -d, --destination </container[/object]>
678 The container and name of the destination object. Name of desti‐
679 nation object can be omitted, then will be same as name of
680 source object. Supplying multiple objects and destination with
681 object name is invalid.
682 -M, --fresh-metadata
684 Copy the object without any existing metadata, If not set, meta‐
685 data will be preserved or appended
686 -m, --meta <name:value>
688 Sets a meta data item. This option may be repeated.
689 Example: -m Color:Blue -m Size:Large
691 -H, --header <header:value>
693 Adds a customized request header. This option may be repeated.
694 Example: -H “content-type:text/plain” -H “Content-Length: 4000”
696 swift capabilities
698 Usage: swift capabilities [--json] [<proxy_url>]
699 Displays cluster capabilities. The output includes the list of the
701 activated Swift middlewares as well as relevant options for each ones.
702 Additionally the command displays relevant options for the Swift core.
703 If the proxy-url option is not provided, the storage URL retrieved
704 after authentication is used as proxy-url.
705 Optional positional arguments:
707 <proxy_url>
709 Proxy URL of the cluster to retrieve capabilities.
710 --json Print the cluster capabilities in JSON format.
712 swift tempurl
714 Usage: swift tempurl [--absolute] [--prefix-based]
715 <method> <seconds> <path> <key>
716 Generates a temporary URL for a Swift object. method option sets an
718 HTTP method to allow for this temporary URL that is usually GET or PUT.
719 time option sets the amount of time the temporary URL will be valid
720 for. time can be specified as an integer, denoting the number of sec‐
721 onds from now on until the URL shall be valid; or, if --absolute is
722 passed, the Unix timestamp when the temporary URL will expire. But
723 beyond that, time can also be specified as an ISO 8601 timestamp in one
724 of following formats:
725 i. Complete date: YYYY-MM-DD (e.g. 1997-07-16)
727 ii. Complete date plus hours, minutes and seconds: YYYY-MM-DDThh:mm:ss
729 (e.g. 1997-07-16T19:20:30)
730 iii. Complete date plus hours, minutes and seconds with UTC designator:
732 YYYY-MM-DDThh:mm:ssZ (e.g. 1997-07-16T19:20:30Z)
733 Please be aware that if you don’t provide the UTC designator (i.e., Z)
735 the timestamp is generated using your local timezone. If only a date is
736 specified, the time part used will equal to 00:00:00.
737 path option sets the full path to the Swift object. Example:
739 /v1/AUTH_account/c/o. key option is the secret temporary URL key set on
740 the Swift cluster. To set a key, run swift post -m "Temp-URL-Key: <your
741 secret key>". To generate a prefix-based temporary URL use the --pre‐
742 fix-based option. This URL will contain the path to the prefix. Do not
743 forget to append the desired objectname at the end of the path portion
744 (and before the query portion) before sharing the URL. It is possible
745 to use ISO 8601 UTC timestamps within the URL by using the --iso8601
746 option.
747 Positional arguments:
749 <method>
751 An HTTP method to allow for this temporary URL. Usually ‘GET’
752 or ‘PUT’.
753 <seconds>
755 The amount of time in seconds the temporary URL will be valid
756 for; or, if –absolute is passed, the Unix timestamp when the
757 temporary URL will expire.
758 <path> The full path to the Swift object.
760 Example: /v1/AUTH_account/c/o or:
762 http://saio:8080/v1/AUTH_account/c/o
763 <key> The secret temporary URL key set on the Swift cluster. To set a
765 key, run ‘swift post -m
766 “Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4”’
767 Optional arguments:
769 --absolute
771 Interpret the <seconds> positional argument as a Unix timestamp
772 rather than a number of seconds in the future.
773 --prefix-based
775 If present, a prefix-based tempURL will be generated.
776 Examples
778 In this section we present some example usage of the swift CLI. To keep
779 the examples as short as possible, these examples assume that the rele‐
780 vant authentication options have been set using environment variables.
781 You can obtain the full list of commands and options available in the
782 swift CLI by executing the following:
783 > swift --help
785 > swift <command> --help
786 Simple examples
788 List the existing swift containers:
789 > swift list
791 container_1
793 Create a new container:
795 > swift post TestContainer
797 Upload an object into a container:
799 > swift upload TestContainer testSwift.txt
801 testSwift.txt
803 List the contents of a container:
805 > swift list TestContainer
807 testSwift.txt
809 Copy an object to new destination:
811 > swift copy -d /DestContainer/testSwift.txt SourceContainer testSwift.txt
813 SourceContainer/testSwift.txt copied to /DestContainer/testSwift.txt
815 Delete an object from a container:
817 > swift delete TestContainer testSwift.txt
819 testSwift.txt
821 Delete a container:
823 > swift delete TestContainer
825 TestContainer
827 Display auth related authentication variables in shell friendly format:
829 > swift auth
831 export OS_STORAGE_URL=http://127.0.0.1:8080/v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7
833 export OS_AUTH_TOKEN=c597015ae19943a18438b52ef3762e79
834 Download an object from a container:
836 > swift download TestContainer testSwift.txt
838 testSwift.txt [auth 0.028s, headers 0.045s, total 0.045s, 0.002 MB/s]
840 NOTE:
842 To upload an object to a container, your current working directory
843 must be where the file is located or you must provide the complete
844 path to the file. In other words, the –object-name <object-name> is
845 an option that will upload file and name object to <object-name> or
846 upload directory and use <object-name> as object prefix. In the case
847 that you provide the complete path of the file, that complete path
848 will be the name of the uploaded object.
849 For example:
851 > swift upload TestContainer /home/swift/testSwift/testSwift.txt
853 home/swift/testSwift/testSwift.txt
855 > swift list TestContainer
857 home/swift/testSwift/testSwift.txt
859 More complex examples
861 Swift has a single object size limit of 5GiB. In order to upload files
862 larger than this, we must create a large object that consists of
863 smaller segments. The example below shows how to upload a large video
864 file as a static large object in 1GiB segments:
865 > swift upload videos --use-slo --segment-size 1G myvideo.mp4
867 myvideo.mp4 segment 8
869 myvideo.mp4 segment 4
870 myvideo.mp4 segment 2
871 myvideo.mp4 segment 7
872 myvideo.mp4 segment 0
873 myvideo.mp4 segment 1
874 myvideo.mp4 segment 3
875 myvideo.mp4 segment 6
876 myvideo.mp4 segment 5
877 myvideo.mp4
878 This command will upload segments to a container named videos_segments,
880 and create a manifest file describing the entire object in the videos
881 container. For more information on large objects, see the documenta‐
882 tion here.
883 > swift list videos
885 myvideo.mp4
887 > swift list videos_segments
889 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000000
891 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000001
892 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000002
893 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000003
894 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000004
895 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000005
896 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000006
897 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000007
898 myvideo.mp4/slo/1460229233.679546/9341553868/1073741824/00000008
899 Firstly, the key should be set, then generate a temporary URL for a
901 Swift object:
902 > swift post -m "Temp-URL-Key:b3968d0207b54ece87cccc06515a89d4"
904 > swift tempurl GET 6000 /v1/AUTH_bf5e63572f7a420a83fcf0aa8c72c2c7\
906 /firstcontainer/clean.sh b3968d0207b54ece87cccc06515a89d4
907 /v1/AUTH_/firstcontainer/clean.sh?temp_url_sig=\
909 9218fc288cc09e5edd857b6a3d43cf2122b906dc&temp_url_expires=1472203614
910 The swiftclient.SwiftService API
912 A higher-level API aimed at allowing developers an easy way to perform
913 multiple operations asynchronously using a configurable thread pool.
914 Documentation for each service method call can be found here: swift‐
915 client.service.
916 Authentication
918 This section covers the various options for authenticating with a swift
919 object store. The combinations of options required for each authentica‐
920 tion version are detailed below. Once again, these are just a subset of
921 those that can be used to successfully authenticate, but they are the
922 most common and recommended.
923 The relevant authentication options are presented as python dictionar‐
925 ies that should be added to any other options you are supplying to your
926 SwiftService instance. As indicated in the python code, you can also
927 set these options as environment variables that will be loaded automat‐
928 ically if the relevant option is not specified.
929 The SwiftService authentication attempts to automatically select the
931 auth version based on the combination of options specified, but supply‐
932 ing options from multiple different auth versions can cause unexpected
933 behaviour.
934 NOTE:
936 Leftover environment variables are a common source of confusion when
937 authorization fails.
938 Keystone V3
940 {
941 ...
942 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '3'
943 "os_username": environ.get('OS_USERNAME'),
944 "os_password": environ.get('OS_PASSWORD'),
945 "os_project_name": environ.get('OS_PROJECT_NAME'),
946 "os_project_domain_name": environ.get('OS_PROJECT_DOMAIN_NAME'),
947 "os_auth_url": environ.get('OS_AUTH_URL'),
948 ...
949 }
950 {
952 ...
953 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '3'
954 "os_username": environ.get('OS_USERNAME'),
955 "os_password": environ.get('OS_PASSWORD'),
956 "os_project_id": environ.get('OS_PROJECT_ID'),
957 "os_project_domain_id": environ.get('OS_PROJECT_DOMAIN_ID'),
958 "os_auth_url": environ.get('OS_AUTH_URL'),
959 ...
960 }
961 Keystone V2
963 {
964 ...
965 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '2.0'
966 "os_username": environ.get('OS_USERNAME'),
967 "os_password": environ.get('OS_PASSWORD'),
968 "os_tenant_name": environ.get('OS_TENANT_NAME'),
969 "os_auth_url": environ.get('OS_AUTH_URL'),
970 ...
971 }
972 Legacy Auth
974 {
975 ...
976 "auth_version": environ.get('ST_AUTH_VERSION'), # Should be '1.0'
977 "auth": environ.get('ST_AUTH'),
978 "user": environ.get('ST_USER'),
979 "key": environ.get('ST_KEY'),
980 ...
981 }
982 Configuration
984 When you create an instance of a SwiftService, you can override a col‐
985 lection of default options to suit your use case. Typically, the
986 defaults are sensible to get us started, but depending on your needs
987 you might want to tweak them to improve performance (options affecting
988 large objects and thread counts can significantly alter performance in
989 the right situation).
990 Service level defaults and some extra options can also be overridden on
992 a per-operation (or even in some cases per-object) basis, and you will
993 call out which options affect which operations later in the document.
994 The configuration of the service API is performed using an options dic‐
996 tionary passed to the SwiftService during initialisation. The options
997 available in this dictionary are described below, along with their
998 defaults:
999 Options
1001 retries: 5
1002 The number of times that the library should attempt to retry
1003 HTTP actions before giving up and reporting a failure.
1004 container_threads: 10
1006 object_dd_threads: 10
1008 object_uu_threads: 10
1010 segment_threads: 10
1012 The above options determine the size of the available thread
1013 pools for performing swift operations. Container operations
1014 (such as listing a container) operate in the container threads,
1015 and a similar pattern applies to object and segment threads.
1016 NOTE:
1018 Object threads are separated into two separate thread pools:
1019 uu and dd. This stands for “upload/update” and “down‐
1020 load/delete”, and the corresponding actions will be run on
1021 separate threads pools.
1022 segment_size: None
1024 If specified, this option enables uploading of large objects.
1025 Should the object being uploaded be larger than 5G in size, this
1026 option is mandatory otherwise the upload will fail. This option
1027 should be specified as a size in bytes.
1028 use_slo: False
1030 Used in combination with the above option, use_slo will upload
1031 large objects as static rather than dynamic. Only static large
1032 objects provide error checking for the downloaded object, so we
1033 recommend this option.
1034 segment_container: None
1036 Allows the user to select the container into which large object
1037 segments will be uploaded. We do not recommend changing this
1038 value as it could make locating orphaned segments more difficult
1039 in the case of errors.
1040 leave_segments: False
1042 Setting this option to true means that when deleting or over‐
1043 writing a large object, its segments will be left in the object
1044 store and must be cleaned up manually. This option can be useful
1045 when sharing large object segments between multiple objects in
1046 more advanced scenarios, but must be treated with care, as it
1047 could lead to ever increasing storage usage.
1048 changed: None
1050 This option affects uploads and simply means that those objects
1051 which already exist in the object store will not be overwritten
1052 if the mtime and size of the source is the same as the existing
1053 object.
1054 skip_identical: False
1056 A slightly more thorough case of the above, but rather than
1057 mtime and size uses an object’s MD5 sum.
1058 yes_all: False
1060 This options affects only download and delete, and in each case
1061 must be specified in order to download/delete the entire con‐
1062 tents of an account. This option has no effect on any other
1063 calls.
1064 no_download: False
1066 This option only affects download and means that all operations
1067 proceed as normal with the exception that no data is written to
1068 disk.
1069 header: []
1071 Used with upload and post operations to set headers on objects.
1072 Headers are specified as colon separated strings, e.g. “con‐
1073 tent-type:text/plain”.
1074 meta: []
1076 Used to set metadata on an object similarly to headers.
1077 NOTE:
1079 Setting metadata is a destructive operation, so when updating
1080 one of many metadata values all desired metadata for an
1081 object must be re-applied.
1082 long: False
1084 Affects only list operations, and results in more metrics being
1085 made available in the results at the expense of lower perfor‐
1086 mance.
1087 fail_fast: False
1089 Applies to delete and upload operations, and attempts to abort
1090 queued tasks in the event of errors.
1091 prefix: None
1093 Affects list operations; only objects with the given prefix will
1094 be returned/affected. It is not advisable to set at the service
1095 level, as those operations that call list to discover objects on
1096 which they should operate will also be affected.
1097 delimiter: None
1099 Affects list operations, and means that listings only contain
1100 results up to the first instance of the delimiter in the object
1101 name. This is useful for working with objects containing ‘/’ in
1102 their names to simulate folder structures.
1103 dir_marker: False
1105 Affects uploads, and allows empty ‘pseudofolder’ objects to be
1106 created when the source of an upload is None.
1107 checksum: True
1109 Affects uploads and downloads. If set check md5 sum for the
1110 transfer.
1111 shuffle: False
1113 When downloading objects, the default behaviour of the CLI is to
1114 shuffle lists of objects in order to spread the load on storage
1115 drives when multiple clients are downloading the same files to
1116 multiple locations (e.g. in the event of distributing an
1117 update). When using the SwiftService directly, object downloads
1118 are scheduled in the same order as they appear in the container
1119 listing. When combined with a single download thread this means
1120 that objects are downloaded in lexically-sorted order. Setting
1121 this option to True gives the same shuffling behaviour as the
1122 CLI.
1123 destination: None
1125 When copying objects, this specifies the destination where the
1126 object will be copied to. The default of None means copy will
1127 be the same as source.
1128 fresh_metadata: None
1130 When copying objects, this specifies that the object metadata on
1131 the source will not be applied to the destination object - the
1132 destination object will have a new fresh set of metadata that
1133 includes only the metadata specified in the meta option if any
1134 at all.
1135 Other available options can be found in swiftclient/service.py in the
1137 source code for python-swiftclient. Each SwiftService method also
1138 allows for an optional dictionary to override those specified at init
1139 time, and the appropriate docstrings show which options modify each
1140 method’s behaviour.
1141 Available Operations
1143 Each operation provided by the service API may raise a SwiftError or
1144 ClientException for any call that fails completely (or a call which
1145 performs only one operation at an account or container level). In the
1146 case of a successful call an operation returns one of the following:
1147 · A dictionary detailing the results of a single operation.
1149 · An iterator that produces result dictionaries (for calls that perform
1151 multiple sub-operations).
1152 A result dictionary can indicate either the success or failure of an
1154 individual operation (detailed in the success key), and will either
1155 contain the successful result, or an error key detailing the error
1156 encountered (usually an instance of Exception).
1157 An example result dictionary is given below:
1159 result = {
1161 'action': 'download_object',
1162 'success': True,
1163 'container': container,
1164 'object': obj,
1165 'path': path,
1166 'start_time': start_time,
1167 'finish_time': finish_time,
1168 'headers_receipt': headers_receipt,
1169 'auth_end_time': conn.auth_end_time,
1170 'read_length': bytes_read,
1171 'attempts': conn.attempts
1172 }
1173 All the possible action values are detailed below:
1175 [
1177 'stat_account',
1178 'stat_container',
1179 'stat_object',
1180 'post_account',
1181 'post_container',
1182 'post_object',
1183 'list_part', # list yields zero or more 'list_part' results
1184 'download_object',
1185 'create_container', # from upload
1186 'create_dir_marker', # from upload
1187 'upload_object',
1188 'upload_segment',
1189 'delete_container',
1190 'delete_object',
1191 'delete_segment', # from delete_object operations
1192 'capabilities',
1193 ]
1194 Stat
1196 Stat can be called against an account, a container, or a list of
1197 objects to get account stats, container stats or information about the
1198 given objects. In the first two cases a dictionary is returned contain‐
1199 ing the results of the operation, and in the case of a list of object
1200 names being supplied, an iterator over the results generated for each
1201 object is returned.
1202 Information returned includes the amount of data used by the given
1204 object/container/account and any headers or metadata set (this includes
1205 user set data as well as content-type and modification times).
1206 See swiftclient.service.SwiftService.stat for docs generated from the
1208 method docstring.
1209 Valid calls for this method are as follows:
1211 · stat([options]): Returns stats for the configured account.
1213 · stat(<container>, [options]): Returns stats for the given container.
1215 · stat(<container>, <object_list>, [options]): Returns stats for each
1217 of the given objects in the given container (through the returned
1218 iterator).
1219 Results from stat are dictionaries indicating the success or failure of
1221 each operation. In the case of a successful stat against an account or
1222 container, the method returns immediately with one of the following
1223 results:
1224 {
1226 'action': 'stat_account',
1227 'success': True,
1228 'items': items,
1229 'headers': headers
1230 }
1231 {
1233 'action': 'stat_container',
1234 'container': <container>,
1235 'success': True,
1236 'items': items,
1237 'headers': headers
1238 }
1239 In the case of stat called against a list of objects, the method
1241 returns a generator that returns the results of individual object stat
1242 operations as they are performed on the thread pool:
1243 {
1245 'action': 'stat_object',
1246 'object': <object_name>,
1247 'container': <container>,
1248 'success': True,
1249 'items': items,
1250 'headers': headers
1251 }
1252 In the case of a failure the dictionary returned will indicate that the
1254 operation was not successful, and will include the keys below:
1255 {
1257 'action': <'stat_object'|'stat_container'|'stat_account'>,
1258 'object': <'object_name'>, # Only for stat with objects list
1259 'container': <container>, # Only for stat with objects list or container
1260 'success': False,
1261 'error': <error>,
1262 'traceback': <trace>,
1263 'error_timestamp': <timestamp>
1264 }
1265 Example
1267 The code below demonstrates the use of stat to retrieve the headers for
1268 a given list of objects in a container using 20 threads. The code cre‐
1269 ates a mapping from object name to headers which is then pretty printed
1270 to the log.
1271 import logging
1273 import pprint
1274 from swiftclient.service import SwiftService
1276 from sys import argv
1277 logging.basicConfig(level=logging.ERROR)
1279 logging.getLogger("requests").setLevel(logging.CRITICAL)
1280 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1281 logger = logging.getLogger(__name__)
1282 _opts = {'object_dd_threads': 20}
1284 with SwiftService(options=_opts) as swift:
1285 container = argv[1]
1286 objects = argv[2:]
1287 header_data = {}
1288 stats_it = swift.stat(container=container, objects=objects)
1289 for stat_res in stats_it:
1290 if stat_res['success']:
1291 header_data[stat_res['object']] = stat_res['headers']
1292 else:
1293 logger.error(
1294 'Failed to retrieve stats for %s' % stat_res['object']
1295 )
1296 pprint.pprint(header_data)
1297 List
1300 List can be called against an account or a container to retrieve the
1301 containers or objects contained within them. Each call returns an iter‐
1302 ator that returns pages of results (by default, up to 10000 results in
1303 each page).
1304 See swiftclient.service.SwiftService.list for docs generated from the
1306 method docstring.
1307 If the given container or account does not exist, the list method will
1309 raise a SwiftError, but for all other success/failures a dictionary is
1310 returned. Each successfully listed page returns a dictionary as
1311 described below:
1312 {
1314 'action': <'list_account_part'|'list_container_part'>,
1315 'container': <container>, # Only for listing a container
1316 'prefix': <prefix>, # The prefix of returned objects/containers
1317 'success': True,
1318 'listing': [Item], # A list of results
1319 # (only in the event of success)
1320 'marker': <marker> # The last item name in the list
1321 # (only in the event of success)
1322 }
1323 Where an item contains the following keys:
1325 {
1327 'name': <name>,
1328 'bytes': 10485760,
1329 'last_modified': '2014-12-11T12:02:38.774540',
1330 'hash': 'fb938269cbeabe4c234e1127bbd3b74a',
1331 'content_type': 'application/octet-stream',
1332 'meta': <metadata> # Full metadata listing from stat'ing each object
1333 # this key only exists if 'long' is specified in options
1334 }
1335 Any failure listing an account or container that exists will return a
1337 failure dictionary as described below:
1338 {
1340 'action': <'list_account_part'|'list_container_part'>,,
1341 'container': container, # Only for listing a container
1342 'prefix': options['prefix'],
1343 'success': success,
1344 'marker': marker,
1345 'error': error,
1346 'traceback': <trace>,
1347 'error_timestamp': <timestamp>
1348 }
1349 Example
1351 The code below demonstrates the use of list to list all items in a con‐
1352 tainer that are over 10MiB in size:
1353 import logging
1355 from swiftclient.service import SwiftService, SwiftError
1357 from sys import argv
1358 logging.basicConfig(level=logging.ERROR)
1360 logging.getLogger("requests").setLevel(logging.CRITICAL)
1361 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1362 logger = logging.getLogger(__name__)
1363 container = argv[1]
1365 minimum_size = 10*1024**2
1366 with SwiftService() as swift:
1367 try:
1368 list_parts_gen = swift.list(container=container)
1369 for page in list_parts_gen:
1370 if page["success"]:
1371 for item in page["listing"]:
1372 i_size = int(item["bytes"])
1374 if i_size > minimum_size:
1375 i_name = item["name"]
1376 i_etag = item["hash"]
1377 print(
1378 "%s [size: %s] [etag: %s]" %
1379 (i_name, i_size, i_etag)
1380 )
1381 else:
1382 raise page["error"]
1383 except SwiftError as e:
1385 logger.error(e.value)
1386 Post
1389 Post can be called against an account, container or list of objects in
1390 order to update the metadata attached to the given items. In the first
1391 two cases a single dictionary is returned containing the results of the
1392 operation, and in the case of a list of objects being supplied, an
1393 iterator over the results generated for each object post is returned.
1394 Each element of the object list may be a plain string of the object
1396 name, or a SwiftPostObject that allows finer control over the options
1397 and metadata applied to each of the individual post operations. When a
1398 string is given for the object name, the options and metadata applied
1399 are a combination of those supplied to the call to post() and the
1400 defaults of the SwiftService object.
1401 If the given container or account does not exist, the post method will
1403 raise a SwiftError. Successful metadata update results are dictionaries
1404 as described below:
1405 {
1407 'action': <'post_account'|'post_container'|'post_object'>,
1408 'success': True,
1409 'container': <container>,
1410 'object': <object>,
1411 'headers': {},
1412 'response_dict': <HTTP response details>
1413 }
1414 NOTE:
1416 Updating user metadata keys will not only add any specified keys,
1417 but will also remove user metadata that has previously been set.
1418 This means that each time user metadata is updated, the complete set
1419 of desired key-value pairs must be specified.
1420 Example
1422 The code below demonstrates the use of post to set an archive folder in
1423 a given container to expire after a 24 hour delay:
1424 import logging
1426 from swiftclient.service import SwiftService, SwiftError
1428 from sys import argv
1429 logging.basicConfig(level=logging.ERROR)
1431 logging.getLogger("requests").setLevel(logging.CRITICAL)
1432 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1433 logger = logging.getLogger(__name__)
1434 container = argv[1]
1436 with SwiftService() as swift:
1437 try:
1438 list_options = {"prefix": "archive_2016-01-01/"}
1439 list_parts_gen = swift.list(container=container)
1440 for page in list_parts_gen:
1441 if page["success"]:
1442 objects = [obj["name"] for obj in page["listing"]]
1443 post_options = {"header": "X-Delete-After:86400"}
1444 for post_res in swift.post(
1445 container=container,
1446 objects=objects,
1447 options=post_options):
1448 if post_res['success']:
1449 print("Object '%s' POST success" % post_res['object'])
1450 else:
1451 print("Object '%s' POST failed" % post_res['object'])
1452 else:
1453 raise page["error"]
1454 except SwiftError as e:
1455 logger.error(e.value)
1456 Download
1459 Download can be called against an entire account, a single container,
1460 or a list of objects in a given container. Each element of the object
1461 list is a string detailing the full name of an object to download.
1462 In order to download the full contents of an entire account, you must
1464 set the value of yes_all to True in the options dictionary supplied to
1465 either the SwiftService instance or the call to download.
1466 If the given container or account does not exist, the download method
1468 will raise a SwiftError, otherwise an iterator over the results gener‐
1469 ated for each object download is returned.
1470 See swiftclient.service.SwiftService.download for docs generated from
1472 the method docstring.
1473 For each successfully downloaded object, the results returned by the
1475 iterator will be a dictionary as described below (results are not
1476 returned for completed container or object segment downloads):
1477 {
1479 'action': 'download_object',
1480 'container': <container>,
1481 'object': <object name>,
1482 'success': True,
1483 'path': <local path to downloaded object>,
1484 'pseudodir': <if true, the download created an empty directory>,
1485 'start_time': <time download started>,
1486 'end_time': <time download completed>,
1487 'headers_receipt': <time the headers from the object were retrieved>,
1488 'auth_end_time': <time authentication completed>,
1489 'read_length': <bytes_read>,
1490 'attempts': <attempt count>,
1491 'response_dict': <HTTP response details>
1492 }
1493 Any failure uploading an object will return a failure dictionary as
1495 described below:
1496 {
1498 'action': 'download_object',
1499 'container': <container>,
1500 'object': <object name>,
1501 'success': False,
1502 'path': <local path of the failed download>,
1503 'pseudodir': <if true, the failed download was an empty directory>,
1504 'attempts': <attempt count>,
1505 'error': <error>,
1506 'traceback': <trace>,
1507 'error_timestamp': <timestamp>,
1508 'response_dict': <HTTP response details>
1509 }
1510 Example
1512 The code below demonstrates the use of download to download all PNG
1513 images from a dated archive folder in a given container:
1514 import logging
1516 from swiftclient.service import SwiftService, SwiftError
1518 from sys import argv
1519 logging.basicConfig(level=logging.ERROR)
1521 logging.getLogger("requests").setLevel(logging.CRITICAL)
1522 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1523 logger = logging.getLogger(__name__)
1524 def is_png(obj):
1526 return (
1527 obj["name"].lower().endswith('.png') or
1528 obj["content_type"] == 'image/png'
1529 )
1530 container = argv[1]
1532 with SwiftService() as swift:
1533 try:
1534 list_options = {"prefix": "archive_2016-01-01/"}
1535 list_parts_gen = swift.list(container=container)
1536 for page in list_parts_gen:
1537 if page["success"]:
1538 objects = [
1539 obj["name"] for obj in page["listing"] if is_png(obj)
1540 ]
1541 for down_res in swift.download(
1542 container=container,
1543 objects=objects):
1544 if down_res['success']:
1545 print("'%s' downloaded" % down_res['object'])
1546 else:
1547 print("'%s' download failed" % down_res['object'])
1548 else:
1549 raise page["error"]
1550 except SwiftError as e:
1551 logger.error(e.value)
1552 Upload
1555 Upload is always called against an account and container and with a
1556 list of objects to upload. Each element of the object list may be a
1557 plain string detailing the path of the object to upload, or a SwiftU‐
1558 ploadObject that allows finer control over some aspects of the individ‐
1559 ual operations.
1560 When a simple string is supplied to specify a file to upload, the name
1562 of the object uploaded is the full path of the specified file and the
1563 options used for the upload are those supplied to the call to upload.
1564 Constructing a SwiftUploadObject allows the user to supply an object
1566 name for the uploaded file, and modify the options used by upload at
1567 the granularity of individual files.
1568 If the given container or account does not exist, the upload method
1570 will raise a SwiftError, otherwise an iterator over the results gener‐
1571 ated for each object upload is returned.
1572 See swiftclient.service.SwiftService.upload for docs generated from the
1574 method docstring.
1575 For each successfully uploaded object (or object segment), the results
1577 returned by the iterator will be a dictionary as described below:
1578 {
1580 'action': 'upload_object',
1581 'container': <container>,
1582 'object': <object name>,
1583 'success': True,
1584 'status': <'uploaded'|'skipped-identical'|'skipped-changed'>,
1585 'attempts': <attempt count>,
1586 'response_dict': <HTTP response details>
1587 }
1588 {
1590 'action': 'upload_segment',
1591 'for_container': <container>,
1592 'for_object': <object name>,
1593 'segment_index': <segment_index>,
1594 'segment_size': <segment_size>,
1595 'segment_location': <segment_path>
1596 'segment_etag': <etag>,
1597 'log_line': <object segment n>
1598 'success': True,
1599 'response_dict': <HTTP response details>,
1600 'attempts': <attempt count>
1601 }
1602 Any failure uploading an object will return a failure dictionary as
1604 described below:
1605 {
1607 'action': 'upload_object',
1608 'container': <container>,
1609 'object': <object name>,
1610 'success': False,
1611 'attempts': <attempt count>,
1612 'error': <error>,
1613 'traceback': <trace>,
1614 'error_timestamp': <timestamp>,
1615 'response_dict': <HTTP response details>
1616 }
1617 {
1619 'action': 'upload_segment',
1620 'for_container': <container>,
1621 'for_object': <object name>,
1622 'segment_index': <segment_index>,
1623 'segment_size': <segment_size>,
1624 'segment_location': <segment_path>,
1625 'log_line': <object segment n>,
1626 'success': False,
1627 'error': <error>,
1628 'traceback': <trace>,
1629 'error_timestamp': <timestamp>,
1630 'response_dict': <HTTP response details>,
1631 'attempts': <attempt count>
1632 }
1633 Example
1635 The code below demonstrates the use of upload to upload all files and
1636 folders in a given directory, and rename each object by replacing the
1637 root directory name with ‘my-<d>-objects’, where <d> is the name of the
1638 uploaded directory:
1639 import logging
1641 from os import walk
1643 from os.path import join
1644 from swiftclient.multithreading import OutputManager
1645 from swiftclient.service import SwiftError, SwiftService, SwiftUploadObject
1646 from sys import argv
1647 logging.basicConfig(level=logging.ERROR)
1649 logging.getLogger("requests").setLevel(logging.CRITICAL)
1650 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1651 logger = logging.getLogger(__name__)
1652 _opts = {'object_uu_threads': 20}
1654 dir = argv[1]
1655 container = argv[2]
1656 with SwiftService(options=_opts) as swift, OutputManager() as out_manager:
1657 try:
1658 # Collect all the files and folders in the given directory
1659 objs = []
1660 dir_markers = []
1661 for (_dir, _ds, _fs) in walk(dir):
1662 if not (_ds + _fs):
1663 dir_markers.append(_dir)
1664 else:
1665 objs.extend([join(_dir, _f) for _f in _fs])
1666 # Now that we've collected all the required files and dir markers
1668 # build the ``SwiftUploadObject``s for the call to upload
1669 objs = [
1670 SwiftUploadObject(
1671 o, object_name=o.replace(
1672 dir, 'my-%s-objects' % dir, 1
1673 )
1674 ) for o in objs
1675 ]
1676 dir_markers = [
1677 SwiftUploadObject(
1678 None, object_name=d.replace(
1679 dir, 'my-%s-objects' % dir, 1
1680 ), options={'dir_marker': True}
1681 ) for d in dir_markers
1682 ]
1683 # Schedule uploads on the SwiftService thread pool and iterate
1685 # over the results
1686 for r in swift.upload(container, objs + dir_markers):
1687 if r['success']:
1688 if 'object' in r:
1689 print(r['object'])
1690 elif 'for_object' in r:
1691 print(
1692 '%s segment %s' % (r['for_object'],
1693 r['segment_index'])
1694 )
1695 else:
1696 error = r['error']
1697 if r['action'] == "create_container":
1698 logger.warning(
1699 'Warning: failed to create container '
1700 "'%s'%s", container, error
1701 )
1702 elif r['action'] == "upload_object":
1703 logger.error(
1704 "Failed to upload object %s to container %s: %s" %
1705 (container, r['object'], error)
1706 )
1707 else:
1708 logger.error("%s" % error)
1709 except SwiftError as e:
1711 logger.error(e.value)
1712 Delete
1715 Delete can be called against an account or a container to remove the
1716 containers or objects contained within them. Each call to delete
1717 returns an iterator over results of each resulting sub-request.
1718 If the number of requested delete operations is large and the target
1720 swift cluster is running the bulk middleware, the call to SwiftSer‐
1721 vice.delete will make use of bulk operations and the returned result
1722 iterator will return bulk_delete results rather than individual
1723 delete_object, delete_container or delete_segment results.
1724 See swiftclient.service.SwiftService.delete for docs generated from the
1726 method docstring.
1727 For each successfully deleted container, object or segment, the results
1729 returned by the iterator will be a dictionary as described below:
1730 {
1732 'action': <'delete_object'|'delete_segment'>,
1733 'container': <container>,
1734 'object': <object name>,
1735 'success': True,
1736 'attempts': <attempt count>,
1737 'response_dict': <HTTP response details>
1738 }
1739 {
1741 'action': 'delete_container',
1742 'container': <container>,
1743 'success': True,
1744 'response_dict': <HTTP response details>,
1745 'attempts': <attempt count>
1746 }
1747 {
1749 'action': 'bulk_delete',
1750 'container': <container>,
1751 'objects': <[objects]>,
1752 'success': True,
1753 'attempts': <attempt count>,
1754 'response_dict': <HTTP response details>
1755 }
1756 Any failure in a delete operation will return a failure dictionary as
1758 described below:
1759 {
1761 'action': ('delete_object'|'delete_segment'),
1762 'container': <container>,
1763 'object': <object name>,
1764 'success': False,
1765 'attempts': <attempt count>,
1766 'error': <error>,
1767 'traceback': <trace>,
1768 'error_timestamp': <timestamp>,
1769 'response_dict': <HTTP response details>
1770 }
1771 {
1773 'action': 'delete_container',
1774 'container': <container>,
1775 'success': False,
1776 'error': <error>,
1777 'traceback': <trace>,
1778 'error_timestamp': <timestamp>,
1779 'response_dict': <HTTP response details>,
1780 'attempts': <attempt count>
1781 }
1782 {
1784 'action': 'bulk_delete',
1785 'container': <container>,
1786 'objects': <[objects]>,
1787 'success': False,
1788 'attempts': <attempt count>,
1789 'error': <error>,
1790 'traceback': <trace>,
1791 'error_timestamp': <timestamp>,
1792 'response_dict': <HTTP response details>
1793 }
1794 Example
1796 The code below demonstrates the use of delete to remove a given list of
1797 objects from a specified container. As the objects are deleted the
1798 transaction ID of the relevant request is printed along with the object
1799 name and number of attempts required. By printing the transaction ID,
1800 the printed operations can be easily linked to events in the swift
1801 server logs:
1802 import logging
1804 from swiftclient.service import SwiftService
1806 from sys import argv
1807 logging.basicConfig(level=logging.ERROR)
1810 logging.getLogger("requests").setLevel(logging.CRITICAL)
1811 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1812 logger = logging.getLogger(__name__)
1813 _opts = {'object_dd_threads': 20}
1815 container = argv[1]
1816 objects = argv[2:]
1817 with SwiftService(options=_opts) as swift:
1818 del_iter = swift.delete(container=container, objects=objects)
1819 for del_res in del_iter:
1820 c = del_res.get('container', '')
1821 o = del_res.get('object', '')
1822 a = del_res.get('attempts')
1823 if del_res['success'] and not del_res['action'] == 'bulk_delete':
1824 rd = del_res.get('response_dict')
1825 if rd is not None:
1826 t = dict(rd.get('headers', {}))
1827 if t:
1828 print(
1829 'Successfully deleted {0}/{1} in {2} attempts '
1830 '(transaction id: {3})'.format(c, o, a, t)
1831 )
1832 else:
1833 print(
1834 'Successfully deleted {0}/{1} in {2} '
1835 'attempts'.format(c, o, a)
1836 )
1837 Copy
1840 Copy can be called to copy an object or update the metadata on the
1841 given items.
1842 Each element of the object list may be a plain string of the object
1844 name, or a SwiftCopyObject that allows finer control over the options
1845 applied to each of the individual copy operations (destination,
1846 fresh_metadata, options).
1847 Destination should be in format /container/object; if not set, the
1849 object will be copied onto itself. Fresh_metadata sets mode of opera‐
1850 tion on metadata. If not set, current object user metadata will be
1851 copied/preserved; if set, all current user metadata will be removed.
1852 Returns an iterator over the results generated for each object copy
1854 (and may also include the results of creating destination containers).
1855 When a string is given for the object name, destination and fresh meta‐
1857 data will default to None and None, which result in adding metadata to
1858 existing objects.
1859 Successful copy results are dictionaries as described below:
1861 {
1863 'action': 'copy_object',
1864 'success': True,
1865 'container': <container>,
1866 'object': <object>,
1867 'destination': <destination>,
1868 'headers': {},
1869 'fresh_metadata': <boolean>,
1870 'response_dict': <HTTP response details>
1871 }
1872 Any failure in a copy operation will return a failure dictionary as
1874 described below:
1875 {
1877 'action': 'copy_object',
1878 'success': False,
1879 'container': <container>,
1880 'object': <object>,
1881 'destination': <destination>,
1882 'headers': {},
1883 'fresh_metadata': <boolean>,
1884 'response_dict': <HTTP response details>,
1885 'error': <error>,
1886 'traceback': <traceback>,
1887 'error_timestamp': <timestamp>
1888 }
1889 Example
1891 The code below demonstrates the use of copy to add new user metadata
1892 for objects a and b, and to copy object c to d (with added metadata).
1893 import logging
1895 from swiftclient.service import SwiftService, SwiftCopyObject, SwiftError
1897 logging.basicConfig(level=logging.ERROR)
1899 logging.getLogger("requests").setLevel(logging.CRITICAL)
1900 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
1901 logger = logging.getLogger(__name__)
1902 with SwiftService() as swift:
1904 try:
1905 obj = SwiftCopyObject("c", {"Destination": "/cont/d"})
1906 for i in swift.copy(
1907 "cont", ["a", "b", obj],
1908 {"meta": ["foo:bar"], "Destination": "/cc"}):
1909 if i["success"]:
1910 if i["action"] == "copy_object":
1911 print(
1912 "object %s copied from /%s/%s" %
1913 (i["destination"], i["container"], i["object"])
1914 )
1915 if i["action"] == "create_container":
1916 print(
1917 "container %s created" % i["container"]
1918 )
1919 else:
1920 if "error" in i and isinstance(i["error"], Exception):
1921 raise i["error"]
1922 except SwiftError as e:
1923 logger.error(e.value)
1924 Capabilities
1927 Capabilities can be called against an account or a particular proxy URL
1928 in order to determine the capabilities of the swift cluster. These
1929 capabilities include details about configuration options and the mid‐
1930 dlewares that are installed in the proxy pipeline.
1931 See swiftclient.service.SwiftService.capabilities for docs generated
1933 from the method docstring.
1934 For each successful call to list capabilities, a result dictionary will
1936 be returned with the contents described below:
1937 {
1939 'action': 'capabilities',
1940 'timestamp': <time of the call>,
1941 'success': True,
1942 'capabilities': <dictionary containing capability details>
1943 }
1944 The contents of the capabilities dictionary contain the core swift
1946 capabilities under the key swift; all other keys show the configuration
1947 options for additional middlewares deployed in the proxy pipeline. An
1948 example capabilities dictionary is given below:
1949 {
1951 'account_quotas': {},
1952 'bulk_delete': {
1953 'max_deletes_per_request': 10000,
1954 'max_failed_deletes': 1000
1955 },
1956 'bulk_upload': {
1957 'max_containers_per_extraction': 10000,
1958 'max_failed_extractions': 1000
1959 },
1960 'container_quotas': {},
1961 'container_sync': {'realms': {}},
1962 'formpost': {},
1963 'keystoneauth': {},
1964 'slo': {
1965 'max_manifest_segments': 1000,
1966 'max_manifest_size': 2097152,
1967 'min_segment_size': 1048576
1968 },
1969 'swift': {
1970 'account_autocreate': True,
1971 'account_listing_limit': 10000,
1972 'allow_account_management': True,
1973 'container_listing_limit': 10000,
1974 'extra_header_count': 0,
1975 'max_account_name_length': 256,
1976 'max_container_name_length': 256,
1977 'max_file_size': 5368709122,
1978 'max_header_size': 8192,
1979 'max_meta_count': 90,
1980 'max_meta_name_length': 128,
1981 'max_meta_overall_size': 4096,
1982 'max_meta_value_length': 256,
1983 'max_object_name_length': 1024,
1984 'policies': [
1985 {'default': True, 'name': 'Policy-0'}
1986 ],
1987 'strict_cors_mode': False,
1988 'version': '2.2.2'
1989 },
1990 'tempurl': {
1991 'methods': ['GET', 'HEAD', 'PUT']
1992 }
1993 }
1994 Example
1996 The code below demonstrates the use of capabilities to determine if the
1997 Swift cluster supports static large objects, and if so, the maximum
1998 number of segments that can be described in a single manifest file,
1999 along with the size restrictions on those objects:
2000 import logging
2002 from swiftclient.exceptions import ClientException
2004 from swiftclient.service import SwiftService
2005 logging.basicConfig(level=logging.ERROR)
2007 logging.getLogger("requests").setLevel(logging.CRITICAL)
2008 logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
2009 logger = logging.getLogger(__name__)
2010 with SwiftService() as swift:
2012 try:
2013 capabilities_result = swift.capabilities()
2014 capabilities = capabilities_result['capabilities']
2015 if 'slo' in capabilities:
2016 print('SLO is supported')
2017 else:
2018 print('SLO is not supported')
2019 except ClientException as e:
2020 logger.error(e.value)
2021 The swiftclient.Connection API
2024 A low level API that provides methods for authentication and methods
2025 that correspond to the individual REST API calls described in the swift
2026 documentation.
2027 For usage details see the client docs: swiftclient.client.
2029 Authentication
2031 This section covers the various combinations of kwargs required when
2032 creating an instance of the Connection object for communicating with a
2033 swift object store. The combinations of options required for each
2034 authentication version are detailed below, but are just a subset of
2035 those that can be used to successfully authenticate. These are the most
2036 common and recommended combinations.
2037 Keystone Session
2039 from keystoneauth1 import session
2040 from keystoneauth1.identity import v3
2041 # Create a password auth plugin
2043 auth = v3.Password(auth_url='http://127.0.0.1:5000/v3/',
2044 username='tester',
2045 password='testing',
2046 user_domain_name='Default',
2047 project_name='Default',
2048 project_domain_name='Default')
2049 # Create session
2051 keystone_session = session.Session(auth=auth)
2052 # Create swiftclient Connection
2054 swift_conn = Connection(session=keystone_session)
2055 Keystone v3
2057 _authurl = 'http://127.0.0.1:5000/v3/'
2058 _auth_version = '3'
2059 _user = 'tester'
2060 _key = 'testing'
2061 _os_options = {
2062 'user_domain_name': 'Default',
2063 'project_domain_name': 'Default',
2064 'project_name': 'Default'
2065 }
2066 conn = Connection(
2068 authurl=_authurl,
2069 user=_user,
2070 key=_key,
2071 os_options=_os_options,
2072 auth_version=_auth_version
2073 )
2074 Keystone v2
2076 _authurl = 'http://127.0.0.1:5000/v2.0/'
2077 _auth_version = '2'
2078 _user = 'tester'
2079 _key = 'testing'
2080 _tenant_name = 'test'
2081 conn = Connection(
2083 authurl=_authurl,
2084 user=_user,
2085 key=_key,
2086 tenant_name=_tenant_name,
2087 auth_version=_auth_version
2088 )
2089 Legacy Auth
2091 _authurl = 'http://127.0.0.1:8080/'
2092 _auth_version = '1'
2093 _user = 'tester'
2094 _key = 'testing'
2095 _tenant_name = 'test'
2096 conn = Connection(
2098 authurl=_authurl,
2099 user=_user,
2100 key=_key,
2101 tenant_name=_tenant_name,
2102 auth_version=_auth_version
2103 )
2104 Examples
2106 In this section we present some simple code examples that demonstrate
2107 the usage of the Connection API. You can find full details of the
2108 options and methods available to the Connection API in the docstring
2109 generated documentation: swiftclient.client.
2110 List the available containers:
2112 resp_headers, containers = conn.get_account()
2114 print("Response headers: %s" % resp_headers)
2115 for container in containers:
2116 print(container)
2117 Create a new container:
2119 container = 'new-container'
2121 conn.put_container(container)
2122 resp_headers, containers = conn.get_account()
2123 if container in containers:
2124 print("The container was created")
2125 Create a new object with the contents of a local text file:
2127 container = 'new-container'
2129 with open('local.txt', 'r') as local:
2130 conn.put_object(
2131 container,
2132 'local_object.txt',
2133 contents=local,
2134 content_type='text/plain'
2135 )
2136 Confirm presence of the object:
2138 obj = 'local_object.txt'
2140 container = 'new-container'
2141 try:
2142 resp_headers = conn.head_object(container, obj)
2143 print('The object was successfully created')
2144 except ClientException as e:
2145 if e.http_status = '404':
2146 print('The object was not found')
2147 else:
2148 print('An error occurred checking for the existence of the object')
2149 Download the created object:
2151 obj = 'local_object.txt'
2153 container = 'new-container'
2154 resp_headers, obj_contents = conn.get_object(container, obj)
2155 with open('local_copy.txt', 'w') as local:
2156 local.write(obj_contents)
2157 Delete the created object:
2159 obj = 'local_object.txt'
2161 container = 'new-container'
2162 try:
2163 conn.delete_object(container, obj)
2164 print("Successfully deleted the object")
2165 except ClientException as e:
2166 print("Failed to delete the object with error: %s" % e)
2167
2169 swiftclient
2170 OpenStack Swift Python client binding.
2171 swiftclient.authv1
2173 swiftclient.client
2174 OpenStack Swift client library used internally
2175 swiftclient.service
2177 swiftclient.exceptions
2178 swiftclient.multithreading
2179 swiftclient.utils
2180 Miscellaneous utility functions for use with Swift.
2181
2183 · genindex
2184 · modindex
2186 · search
2188
2190 Copyright 2013 OpenStack, LLC.
2191 Licensed under the Apache License, Version 2.0 (the “License”); you may
2193 not use this file except in compliance with the License. You may
2194 obtain a copy of the License at
2195 · http://www.apache.org/licenses/LICENSE-2.0
2197 Unless required by applicable law or agreed to in writing, software
2199 distributed under the License is distributed on an “AS IS” BASIS, WITH‐
2200 OUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
2201 See the License for the specific language governing permissions and
2202 limitations under the License.
2203
2205 unknown
2206
2208 2013-2020 OpenStack, LLC.
22093.10.1 Sep 17, 2020 PYTHON-SWIFTCLIENT(1)