1S3(n) Amazon S3 Web Service Utilities S3(n)
2______________________________________________________________________________
6
8 S3 - Amazon S3 Web Service Interface
9
11 package require Tcl 8.5
12 package require S3 ?1.0.3?
14 package require sha1 1.0
16 package require md5 2.0
18 package require base64 2.3
20 package require xsxp 1.0
22 S3::Configure ?-reset boolean? ?-retries integer? ?-accesskeyid id‐
24 string? ?-secretaccesskey idstring? ?-service-access-point FQDN? ?-use-
25 tls boolean? ?-default-compare always|never|exists|miss‐
26 ing|newer|date|checksum|different? ?-default-separator string? ?-de‐
27 fault-acl private|public-read|public-read-write|authenticated-
28 read|keep|calc? ?-default-bucket bucketname?
29 S3::SuggestBucket ?name?
31 S3::REST dict
33 S3::ListAllMyBuckets ?-blocking boolean? ?-parse-xml xmlstring? ?-re‐
35 sult-type REST|xml|pxml|dict|names|owner?
36 S3::PutBucket ?-bucket bucketname? ?-blocking boolean? ?-acl {}|pri‐
38 vate|public-read|public-read-write|authenticated-read?
39 S3::DeleteBucket ?-bucket bucketname? ?-blocking boolean?
41 S3::GetBucket ?-bucket bucketname? ?-blocking boolean? ?-parse-xml xml‐
43 string? ?-max-count integer? ?-prefix prefixstring? ?-delimiter delim‐
44 iterstring? ?-result-type REST|xml|pxml|names|dict?
45 S3::Put ?-bucket bucketname? -resource resourcename ?-blocking boolean?
47 ?-file filename? ?-content contentstring? ?-acl private|public-
48 read|public-read-write|authenticated-read|calc|keep? ?-content-type
49 contenttypestring? ?-x-amz-meta-* metadatatext? ?-compare comparemode?
50 S3::Get ?-bucket bucketname? -resource resourcename ?-blocking boolean?
52 ?-compare comparemode? ?-file filename? ?-content contentvarname?
53 ?-timestamp aws|now? ?-headers headervarname?
54 S3::Head ?-bucket bucketname? -resource resourcename ?-blocking bool‐
56 ean? ?-dict dictvarname? ?-headers headersvarname? ?-status statusvar‐
57 name?
58 S3::GetAcl ?-blocking boolean? ?-bucket bucketname? -resource resource‐
60 name ?-result-type REST|xml|pxml?
61 S3::PutAcl ?-blocking boolean? ?-bucket bucketname? -resource resource‐
63 name ?-acl new-acl?
64 S3::Delete ?-bucket bucketname? -resource resourcename ?-blocking bool‐
66 ean? ?-status statusvar?
67 S3::Push ?-bucket bucketname? -directory directoryname ?-prefix pre‐
69 fixstring? ?-compare comparemode? ?-x-amz-meta-* metastring? ?-acl
70 aclcode? ?-delete boolean? ?-error throw|break|continue? ?-progress
71 scriptprefix?
72 S3::Pull ?-bucket bucketname? -directory directoryname ?-prefix pre‐
74 fixstring? ?-blocking boolean? ?-compare comparemode? ?-delete boolean?
75 ?-timestamp aws|now? ?-error throw|break|continue? ?-progress script‐
76 prefix?
77 S3::Toss ?-bucket bucketname? -prefix prefixstring ?-blocking boolean?
79 ?-error throw|break|continue? ?-progress scriptprefix?
80______________________________________________________________________________
82
84 This package provides access to Amazon's Simple Storage Solution web
85 service.
86 As a quick summary, Amazon Simple Storage Solution provides a for-fee
88 web service allowing the storage of arbitrary data as "resources"
89 within "buckets" online. See http://www.amazonaws.com/ for details on
90 that system. Access to the service is via HTTP (SOAP or REST). Much
91 of this documentation will not make sense if you're not familiar with
92 the terms and functionality of the Amazon S3 service.
93 This package provides services for reading and writing the data items
95 via the REST interface. It also provides some higher-level operations.
96 Other packages in the same distribution provide for even more function‐
97 ality.
98 Copyright 2006 Darren New. All Rights Reserved. NO WARRANTIES OF ANY
100 TYPE ARE PROVIDED. COPYING OR USE INDEMNIFIES THE AUTHOR IN ALL WAYS.
101 This software is licensed under essentially the same terms as Tcl. See
102 LICENSE.txt for the terms.
103
105 The error reporting from this package makes use of $errorCode to pro‐
106 vide more details on what happened than simply throwing an error. Any
107 error caught by the S3 package (and we try to catch them all) will re‐
108 turn with an $errorCode being a list having at least three elements. In
109 all cases, the first element will be "S3". The second element will take
110 on one of six values, with that element defining the value of the third
111 and subsequent elements. S3::REST does not throw an error, but rather
112 returns a dictionary with the keys "error", "errorInfo", and "error‐
113 Code" set. This allows for reliable background use. The possible second
114 elements are these:
115 usage The usage of the package is incorrect. For example, a command
117 has been invoked which requires the library to be configured be‐
118 fore the library has been configured, or an invalid combination
119 of options has been specified. The third element of $errorCode
120 supplies the name of the parameter that was wrong. The fourth
121 usually provides the arguments that were actually supplied to
122 the throwing proc, unless the usage error isn't confined to a
123 single proc.
124 local Something happened on the local system which threw an error. For
126 example, a request to upload or download a file was made and the
127 file permissions denied that sort of access. The third element
128 of $errorCode is the original $errorCode.
129 socket Something happened with the socket. It closed prematurely, or
131 some other condition of failure-to-communicate-with-Amazon was
132 detected. The third element of $errorCode is the original $er‐
133 rorCode, or sometimes the message from fcopy, or ...?
134 remote The Amazon web service returned an error code outside the 2xx
136 range in the HTTP header. In other words, everything went as
137 documented, except this particular case was documented not to
138 work. The third element is the dictionary returned from
139 ::S3::REST. Note that S3::REST itself never throws this error,
140 but just returns the dictionary. Most of the higher-level com‐
141 mands throw for convenience, unless an argument indicates they
142 should not. If something is documented as "not throwing an S3
143 remote error", it means a status return is set rather than
144 throwing an error if Amazon returns a non-2XX HTTP result code.
145 notyet The user obeyed the documentation, but the author has not yet
147 gotten around to implementing this feature. (Right now, only TLS
148 support and sophisticated permissions fall into this category,
149 as well as the S3::Acl command.)
150 xml The service has returned invalid XML, or XML whose schema is un‐
152 expected. For the high-level commands that accept service XML as
153 input for parsing, this may also be thrown.
154
156 This package provides several separate levels of complexity.
157 • The lowest level simply takes arguments to be sent to the ser‐
159 vice, sends them, retrieves the result, and provides it to the
160 caller. Note: This layer allows both synchronous and event-
161 driven processing. It depends on the MD5 and SHA1 and base64
162 packages from Tcllib (available at http://core.tcl.tk/tcllib/).
163 Note that S3::Configure is required for S3::REST to work due to
164 the authentication portion, so we put that in the "lowest
165 level."
166 • The next layer parses the results of calls, allowing for func‐
168 tionality such as uploading only changed files, synchronizing
169 directories, and so on. This layer depends on the TclXML pack‐
170 age as well as the included xsxp package. These packages are
171 package required when these more-sophisticated routines are
172 called, so nothing breaks if they are not correctly installed.
173 • Also included is a separate program that uses the library. It
175 provides code to parse $argv0 and $argv from the command line,
176 allowing invocation as a tclkit, etc. (Not yet implmented.)
177 • Another separate program provides a GUI interface allowing drag-
179 and-drop and other such functionality. (Not yet implemented.)
180 • Also built on this package is the OddJob program. It is a sepa‐
182 rate program designed to allow distribution of computational
183 work units over Amazon's Elastic Compute Cloud web service.
184 The goal is to have at least the bottom-most layers implemented in pure
186 Tcl using only that which comes from widely-available sources, such as
187 Tcllib.
188
190 These commands do not require any packages not listed above. They talk
191 directly to the service, or they are utility or configuration routines.
192 Note that the "xsxp" package was written to support this package, so it
193 should be available wherever you got this package.
194 S3::Configure ?-reset boolean? ?-retries integer? ?-accesskeyid id‐
196 string? ?-secretaccesskey idstring? ?-service-access-point FQDN? ?-use-
197 tls boolean? ?-default-compare always|never|exists|miss‐
198 ing|newer|date|checksum|different? ?-default-separator string? ?-de‐
199 fault-acl private|public-read|public-read-write|authenticated-
200 read|keep|calc? ?-default-bucket bucketname?
201 There is one command for configuration, and that is S3::Config‐
202 ure. If called with no arguments, it returns a dictionary of
203 key/value pairs listing all current settings. If called with
204 one argument, it returns the value of that single argument. If
205 called with two or more arguments, it must be called with pairs
206 of arguments, and it applies the changes in order. There is
207 only one set of configuration information per interpreter.
208 The following options are accepted:
210 -reset boolean
212 By default, false. If true, any previous changes and any
213 changes on the same call before the reset option will be
214 returned to default values.
215 -retries integer
217 Default value is 3. If Amazon returns a 500 error, a
218 retry after an exponential backoff delay will be tried
219 this many times before finally throwing the 500 error.
220 This applies to each call to S3::REST from the higher-
221 level commands, but not to S3::REST itself. That is,
222 S3::REST will always return httpstatus 500 if that's what
223 it receives. Functions like S3::Put will retry the PUT
224 call, and will also retry the GET and HEAD calls used to
225 do content comparison. Changing this to 0 will prevent
226 retries and their associated delays. In addition, socket
227 errors (i.e., errors whose errorCode starts with "S3
228 socket") will be similarly retried after backoffs.
229 -accesskeyid idstring
231 -secretaccesskey idstring
233 Each defaults to an empty string. These must be set be‐
234 fore any calls are made. This is your S3 ID. Once you
235 sign up for an account, go to http://www.amazonaws.com/,
236 sign in, go to the "Your Web Services Account" button,
237 pick "AWS Access Identifiers", and your access key ID and
238 secret access keys will be available. All S3::REST calls
239 are authenticated. Blame Amazon for the poor choice of
240 names.
241 -service-access-point FQDN
243 Defaults to "s3.amazonaws.com". This is the fully-quali‐
244 fied domain name of the server to contact for S3::REST
245 calls. You should probably never need to touch this, un‐
246 less someone else implements a compatible service, or you
247 wish to test something by pointing the library at your
248 own service.
249 -slop-seconds integer
251 When comparing dates between Amazon and the local ma‐
252 chine, two dates within this many seconds of each other
253 are considered the same. Useful for clock drift correc‐
254 tion, processing overhead time, and so on.
255 -use-tls boolean
257 Defaults to false. This is not yet implemented. If true,
258 S3::REST will negotiate a TLS connection to Amazon. If
259 false, unencrypted connections are used.
260 -bucket-prefix string
262 Defaults to "TclS3". This string is used by S3::Suggest‐
263 BucketName if that command is passed an empty string as
264 an argument. It is used to distinguish different applica‐
265 tions using the Amazon service. Your application should
266 always set this to keep from interfering with the buckets
267 of other users of Amazon S3 or with other buckets of the
268 same user.
269 -default-compare always|never|exists|missing|newer|date|check‐
271 sum|different
272 Defaults to "always." If no -compare is specified on
273 S3::Put, S3::Get, or S3::Delete, this comparison is used.
274 See those commands for a description of the meaning.
275 -default-separator string
277 Defaults to "/". This is currently unused. It might make
278 sense to use this for S3::Push and S3::Pull, but allowing
279 resources to have slashes in their names that aren't
280 marking directories would be problematic. Hence, this
281 currently does nothing.
282 -default-acl private|public-read|public-read-write|authenti‐
284 cated-read|keep|calc
285 Defaults to an empty string. If no -acl argument is pro‐
286 vided to S3::Put or S3::Push, this string is used (given
287 as the x-amz-acl header if not keep or calc). If this is
288 also empty, no x-amz-acl header is generated. This is
289 not used by S3::REST.
290 -default-bucket bucketname
292 If no bucket is given to S3::GetBucket, S3::PutBucket,
293 S3::Get, S3::Put, S3::Head, S3::Acl, S3::Delete,
294 S3::Push, S3::Pull, or S3::Toss, and if this configura‐
295 tion variable is not an empty string (and not simply
296 "/"), then this value will be used for the bucket. This
297 is useful if one program does a large amount of resource
298 manipulation within a single bucket.
299 S3::SuggestBucket ?name?
302 The S3::SuggestBucket command accepts an optional string as a
303 prefix and returns a valid bucket containing the name argument
304 and the Access Key ID. This makes the name unique to the owner
305 and to the application (assuming the application picks a good
306 name argument). If no name is provided, the name from S3::Con‐
307 figure -bucket-prefix is used. If that too is empty (which is
308 not the default), an error is thrown.
309 S3::REST dict
311 The S3::REST command takes as an argument a dictionary and re‐
312 turns a dictionary. The return dictionary has the same keys as
313 the input dictionary, and includes additional keys as the re‐
314 sult. The presence or absence of keys in the input dictionary
315 can control the behavior of the routine. It never throws an er‐
316 ror directly, but includes keys "error", "errorInfo", and "er‐
317 rorCode" if necessary. Some keys are required, some optional.
318 The routine can run either in blocking or non-blocking mode,
319 based on the presense of resultvar in the input dictionary. This
320 requires the -accesskeyid and -secretaccesskey to be configured
321 via S3::Configure before being called.
322 The possible input keys are these:
324 verb GET|PUT|DELETE|HEAD
326 This required item indicates the verb to be used.
327 resource string
329 This required item indicates the resource to be accessed.
330 A leading / is added if not there already. It will be
331 URL-encoded for you if necessary. Do not supply a re‐
332 source name that is already URL-encoded.
333 ?rtype torrent|acl?
335 This indicates a torrent or acl resource is being manipu‐
336 lated. Do not include this in the resource key, or the
337 "?" separator will get URL-encoded.
338 ?parameters dict?
340 This optional dictionary provides parameters added to the
341 URL for the transaction. The keys must be in the correct
342 case (which is confusing in the Amazon documentation) and
343 the values must be valid. This can be an empty dictionary
344 or omitted entirely if no parameters are desired. No
345 other error checking on parameters is performed.
346 ?headers dict?
348 This optional dictionary provides headers to be added to
349 the HTTP request. The keys must be in lower case for the
350 authentication to work. The values must not contain em‐
351 bedded newlines or carriage returns. This is primarily
352 useful for adding x-amz-* headers. Since authentication
353 is calculated by S3::REST, do not add that header here.
354 Since content-type gets its own key, also do not add that
355 header here.
356 ?inbody contentstring?
358 This optional item, if provided, gives the content that
359 will be sent. It is sent with a tranfer encoding of bi‐
360 nary, and only the low bytes are used, so use [encoding
361 convertto utf-8] if the string is a utf-8 string. This is
362 written all in one blast, so if you are using non-block‐
363 ing mode and the inbody is especially large, you may wind
364 up blocking on the write socket.
365 ?infile filename?
367 This optional item, if provided, and if inbody is not
368 provided, names the file from which the body of the HTTP
369 message will be constructed. The file is opened for read‐
370 ing and sent progressively by [fcopy], so it should not
371 block in non-blocking mode even if the file is very
372 large. The file is transfered in binary mode, so the
373 bytes on your disk will match the bytes in your resource.
374 Due to HTTP restrictions, it must be possible to use
375 [file size] on this file to determine the size at the
376 start of the transaction.
377 ?S3chan channel?
379 This optional item, if provided, indicates the already-
380 open socket over which the transaction should be con‐
381 ducted. If not provided, a connection is made to the ser‐
382 vice access point specified via S3::Configure, which is
383 normally s3.amazonaws.com. If this is provided, the chan‐
384 nel is not closed at the end of the transaction.
385 ?outchan channel?
387 This optional item, if provided, indicates the already-
388 open channel to which the body returned from S3 should be
389 written. That is, to retrieve a large resource, open a
390 file, set the translation mode, and pass the channel as
391 the value of the key outchan. Output will be written to
392 the channel in pieces so memory does not fill up unneces‐
393 sarily. The channel is not closed at the end of the
394 transaction.
395 ?resultvar varname?
397 This optional item, if provided, indicates that S3::REST
398 should run in non-blocking mode. The varname should be
399 fully qualified with respect to namespaces and cannot be
400 local to a proc. If provided, the result of the S3::REST
401 call is assigned to this variable once everything has
402 completed; use trace or vwait to know when this has hap‐
403 pened. If this key is not provided, the result is simply
404 returned from the call to S3::REST and no calls to the
405 eventloop are invoked from within this call.
406 ?throwsocket throw|return?
408 This optional item, if provided, indicates that S3::REST
409 should throw an error if throwmode is throw and a socket
410 error is encountered. It indicates that S3::REST should
411 return the error code in the returned dictionary if a
412 socket error is encountered and this is set to return. If
413 throwsocket is set to return or if the call is not block‐
414 ing, then a socket error (i.e., an error whose error code
415 starts with "S3 socket" will be returned in the dictio‐
416 nary as error, errorInfo, and errorCode. If a foreground
417 call is made (i.e., resultvar is not provided), and this
418 option is not provided or is set to throw, then error
419 will be invoked instead.
420 Once the call to S3::REST completes, a new dict is returned, either in
422 the resultvar or as the result of execution. This dict is a copy of the
423 original dict with the results added as new keys. The possible new keys
424 are these:
425 error errorstring
427 errorInfo errorstring
429 errorCode errorstring
431 If an error is caught, these three keys will be set in
432 the result. Note that S3::REST does not consider a
433 non-2XX HTTP return code as an error. The errorCode value
434 will be formatted according to the ERROR REPORTING de‐
435 scription. If these are present, other keys described
436 here might not be.
437 httpstatus threedigits
439 The three-digit code from the HTTP transaction. 2XX for
440 good, 5XX for server error, etc.
441 httpmessage text
443 The textual result after the status code. "OK" or "For‐
444 bidden" or etc.
445 outbody contentstring
447 If outchan was not specified, this key will hold a refer‐
448 ence to the (unencoded) contents of the body returned.
449 If Amazon returned an error (a la the httpstatus not a
450 2XX value), the error message will be in outbody or writ‐
451 ten to outchan as appropriate.
452 outheaders dict
454 This contains a dictionary of headers returned by Amazon.
455 The keys are always lower case. It's mainly useful for
456 finding the x-amz-meta-* headers, if any, although things
457 like last-modified and content-type are also useful. The
458 keys of this dictionary are always lower case. Both keys
459 and values are trimmed of extraneous whitespace.
460
462 The routines in this section all make use of one or more calls to
463 S3::REST to do their work, then parse and manage the data in a conve‐
464 nient way. All these commands throw errors as described in ERROR RE‐
465 PORTING unless otherwise noted.
466 In all these commands, all arguments are presented as name/value pairs,
468 in any order. All the argument names start with a hyphen.
469 There are a few options that are common to many of the commands, and
471 those common options are documented here.
472 -blocking boolean
474 If provided and specified as false, then any calls to S3:REST
475 will be non-blocking, and internally these routines will call
476 [vwait] to get the results. In other words, these routines will
477 return the same value, but they'll have event loops running
478 while waiting for Amazon.
479 -parse-xml xmlstring
481 If provided, the routine skips actually communicating with Ama‐
482 zon, and instead behaves as if the XML string provided was re‐
483 turned as the body of the call. Since several of these routines
484 allow the return of data in various formats, this argument can
485 be used to parse existing XML to extract the bits of information
486 that are needed. It's also helpful for testing.
487 -bucket bucketname
489 Almost every high-level command needs to know what bucket the
490 resources are in. This option specifies that. (Only the command
491 to list available buckets does not require this parameter.)
492 This does not need to be URL-encoded, even if it contains spe‐
493 cial or non-ASCII characters. May or may not contain leading or
494 trailing spaces - commands normalize the bucket. If this is not
495 supplied, the value is taken from S3::Configure -default-bucket
496 if that string isn't empty. Note that spaces and slashes are al‐
497 ways trimmed from both ends and the rest must leave a valid
498 bucket.
499 -resource resourcename
501 This specifies the resource of interest within the bucket. It
502 may or may not start with a slash - both cases are handled.
503 This does not need to be URL-encoded, even if it contains spe‐
504 cial or non-ASCII characters.
505 -compare always|never|exists|missing|newer|date|checksum|different
507 When commands copy resources to files or files to resources, the
508 caller may specify that the copy should be skipped if the con‐
509 tents are the same. This argument specifies the conditions under
510 which the files should be copied. If it is not passed, the re‐
511 sult of S3::Configure -default-compare is used, which in turn
512 defaults to "always." The meanings of the various values are
513 these:
514 always Always copy the data. This is the default.
516 never Never copy the data. This is essentially a no-op, except
518 in S3::Push and S3::Pull where the -delete flag might
519 make a difference.
520 exists Copy the data only if the destination already exists.
522 missing
524 Copy the data only if the destination does not already
525 exist.
526 newer Copy the data if the destination is missing, or if the
528 date on the source is newer than the date on the destina‐
529 tion by at least S3::Configure -slop-seconds seconds. If
530 the source is Amazon, the date is taken from the Last-
531 Modified header. If the source is local, it is taken as
532 the mtime of the file. If the source data is specified in
533 a string rather than a file, it is taken as right now,
534 via [clock seconds].
535 date Like newer, except copy if the date is newer or older.
537 checksum
539 Calculate the MD5 checksum on the local file or string,
540 ask Amazon for the eTag of the resource, and copy the
541 data if they're different. Copy the data also if the des‐
542 tination is missing. Note that this can be slow with
543 large local files unless the C version of the MD5 support
544 is available.
545 different
547 Copy the data if the destination does not exist. If the
548 destination exists and an actual file name was specified
549 (rather than a content string), and the date on the file
550 differs from the date on the resource, copy the data. If
551 the data is provided as a content string, the "date" is
552 treated as "right now", so it will likely always differ
553 unless slop-seconds is large. If the dates are the same,
554 the MD5 checksums are compared, and the data is copied if
555 the checksums differ.
556 Note that "newer" and "date" don't care about the contents, and "check‐
558 sum" doesn't care about the dates, but "different" checks both.
559 S3::ListAllMyBuckets ?-blocking boolean? ?-parse-xml xmlstring? ?-re‐
561 sult-type REST|xml|pxml|dict|names|owner?
562 This routine performs a GET on the Amazon S3 service, which is
563 defined to return a list of buckets owned by the account identi‐
564 fied by the authorization header. (Blame Amazon for the dumb
565 names.)
566 -blocking boolean
568 See above for standard definition.
569 -parse-xml xmlstring
571 See above for standard definition.
572 -result-type REST
574 The dictionary returned by S3::REST is the return value
575 of S3::ListAllMyBuckets. In this case, a non-2XX httpsta‐
576 tus will not throw an error. You may not combine this
577 with -parse-xml.
578 -result-type xml
580 The raw XML of the body is returned as the result (with
581 no encoding applied).
582 -result-type pxml
584 The XML of the body as parsed by xsxp::parse is returned.
585 -result-type dict
587 A dictionary of interesting portions of the XML is re‐
588 turned. The dictionary contains the following keys:
589 Owner/ID
591 The Amazon AWS ID (in hex) of the owner of the
592 bucket.
593 Owner/DisplayName
595 The Amazon AWS ID's Display Name.
596 Bucket/Name
598 A list of names, one for each bucket.
599 Bucket/CreationDate
601 A list of dates, one for each bucket, in the same
602 order as Bucket/Name, in ISO format (as returned
603 by Amazon).
604 -result-type names
607 A list of bucket names is returned with all other infor‐
608 mation stripped out. This is the default result type for
609 this command.
610 -result-type owner
612 A list containing two elements is returned. The first el‐
613 ement is the owner's ID, and the second is the owner's
614 display name.
615 S3::PutBucket ?-bucket bucketname? ?-blocking boolean? ?-acl {}|pri‐
618 vate|public-read|public-read-write|authenticated-read?
619 This command creates a bucket if it does not already exist.
620 Bucket names are globally unique, so you may get a "Forbidden"
621 error from Amazon even if you cannot see the bucket in
622 S3::ListAllMyBuckets. See S3::SuggestBucket for ways to minimize
623 this risk. The x-amz-acl header comes from the -acl option, or
624 from S3::Configure -default-acl if not specified.
625 S3::DeleteBucket ?-bucket bucketname? ?-blocking boolean?
627 This command deletes a bucket if it is empty and you have such
628 permission. Note that Amazon's list of buckets is a global re‐
629 source, requiring far-flung synchronization. If you delete a
630 bucket, it may be quite a few minutes (or hours) before you can
631 recreate it, yielding "Conflict" errors until then.
632 S3::GetBucket ?-bucket bucketname? ?-blocking boolean? ?-parse-xml xml‐
634 string? ?-max-count integer? ?-prefix prefixstring? ?-delimiter delim‐
635 iterstring? ?-result-type REST|xml|pxml|names|dict?
636 This lists the contents of a bucket. That is, it returns a di‐
637 rectory listing of resources within a bucket, rather than trans‐
638 fering any user data.
639 -bucket bucketname
641 The standard bucket argument.
642 -blocking boolean
644 The standard blocking argument.
645 -parse-xml xmlstring
647 The standard parse-xml argument.
648 -max-count integer
650 If supplied, this is the most number of records to be re‐
651 turned. If not supplied, the code will iterate until all
652 records have been found. Not compatible with -parse-xml.
653 Note that if this is supplied, only one call to S3::REST
654 will be made. Otherwise, enough calls will be made to ex‐
655 haust the listing, buffering results in memory, so take
656 care if you may have huge buckets.
657 -prefix prefixstring
659 If present, restricts listing to resources with a partic‐
660 ular prefix. One leading / is stripped if present.
661 -delimiter delimiterstring
663 If present, specifies a delimiter for the listing. The
664 presence of this will summarize multiple resources into
665 one entry, as if S3 supported directories. See the Amazon
666 documentation for details.
667 -result-type REST|xml|pxml|names|dict
669 This indicates the format of the return result of the
670 command.
671 REST If -max-count is specified, the dictionary re‐
673 turned from S3::REST is returned. If -max-count is
674 not specified, a list of all the dictionaries re‐
675 turned from the one or more calls to S3::REST is
676 returned.
677 xml If -max-count is specified, the body returned from
679 S3::REST is returned. If -max-count is not speci‐
680 fied, a list of all the bodies returned from the
681 one or more calls to S3::REST is returned.
682 pxml If -max-count is specified, the body returned from
684 S3::REST is passed throught xsxp::parse and then
685 returned. If -max-count is not specified, a list
686 of all the bodies returned from the one or more
687 calls to S3::REST are each passed through
688 xsxp::parse and then returned.
689 names Returns a list of all names found in either the
691 Contents/Key fields or the CommonPrefixes/Prefix
692 fields. If no -delimiter is specified and no -max-
693 count is specified, this returns a list of all re‐
694 sources with the specified -prefix.
695 dict Returns a dictionary. (Returns only one dictionary
697 even if -max-count wasn't specified.) The keys of
698 the dictionary are as follows:
699 Name The name of the bucket (from the final call
701 to S3::REST).
702 Prefix From the final call to S3::REST.
704 Marker From the final call to S3::REST.
706 MaxKeys
708 From the final call to S3::REST.
709 IsTruncated
711 From the final call to S3::REST, so always
712 false if -max-count is not specified.
713 NextMarker
715 Always provided if IsTruncated is true, and
716 calculated of Amazon does not provide it.
717 May be empty if IsTruncated is false.
718 Key A list of names of resources in the bucket
720 matching the -prefix and -delimiter re‐
721 strictions.
722 LastModified
724 A list of times of resources in the bucket,
725 in the same order as Key, in the format re‐
726 turned by Amazon. (I.e., it is not parsed
727 into a seconds-from-epoch.)
728 ETag A list of entity tags (a.k.a. MD5 check‐
730 sums) in the same order as Key.
731 Size A list of sizes in bytes of the resources,
733 in the same order as Key.
734 Owner/ID
736 A list of owners of the resources in the
737 bucket, in the same order as Key.
738 Owner/DisplayName
740 A list of owners of the resources in the
741 bucket, in the same order as Key. These are
742 the display names.
743 CommonPrefixes/Prefix
745 A list of prefixes common to multiple enti‐
746 ties. This is present only if -delimiter
747 was supplied.
748 S3::Put ?-bucket bucketname? -resource resourcename ?-blocking boolean?
750 ?-file filename? ?-content contentstring? ?-acl private|public-
751 read|public-read-write|authenticated-read|calc|keep? ?-content-type
752 contenttypestring? ?-x-amz-meta-* metadatatext? ?-compare comparemode?
753 This command sends data to a resource on Amazon's servers for
754 storage, using the HTTP PUT command. It returns 0 if the -com‐
755 pare mode prevented the transfer, 1 if the transfer worked, or
756 throws an error if the transfer was attempted but failed.
757 Server 5XX errors and S3 socket errors are retried according to
758 S3:Configure -retries settings before throwing an error; other
759 errors throw immediately.
760 -bucket
762 This specifies the bucket into which the resource will be
763 written. Leading and/or trailing slashes are removed for
764 you, as are spaces.
765 -resource
767 This is the full name of the resource within the bucket.
768 A single leading slash is removed, but not a trailing
769 slash. Spaces are not trimmed.
770 -blocking
772 The standard blocking flag.
773 -file If this is specified, the filename must exist, must be
775 readable, and must not be a special or directory file.
776 [file size] must apply to it and must not change for the
777 lifetime of the call. The default content-type is calcu‐
778 lated based on the name and/or contents of the file.
779 Specifying this is an error if -content is also speci‐
780 fied, but at least one of -file or -content must be spec‐
781 ified. (The file is allowed to not exist or not be read‐
782 able if -compare never is specified.)
783 -content
785 If this is specified, the contentstring is sent as the
786 body of the resource. The content-type defaults to "ap‐
787 plication/octet-string". Only the low bytes are sent, so
788 non-ASCII should use the appropriate encoding (such as
789 [encoding convertto utf-8]) before passing it to this
790 routine, if necessary. Specifying this is an error if
791 -file is also specified, but at least one of -file or
792 -content must be specified.
793 -acl This defaults to S3::Configure -default-acl if not speci‐
795 fied. It sets the x-amz-acl header on the PUT operation.
796 If the value provided is calc, the x-amz-acl header is
797 calculated based on the I/O permissions of the file to be
798 uploaded; it is an error to specify calc and -content.
799 If the value provided is keep, the acl of the resource is
800 read before the PUT (or the default is used if the re‐
801 source does not exist), then set back to what it was af‐
802 ter the PUT (if it existed). An error will occur if the
803 resource is successfully written but the kept ACL cannot
804 be then applied. This should never happen. Note: calc
805 is not currently fully implemented.
806 -x-amz-meta-*
808 If any header starts with "-x-amz-meta-", its contents
809 are added to the PUT command to be stored as metadata
810 with the resource. Again, no encoding is performed, and
811 the metadata should not contain characters like newlines,
812 carriage returns, and so on. It is best to stick with
813 simple ASCII strings, or to fix the library in several
814 places.
815 -content-type
817 This overrides the content-type calculated by -file or
818 sets the content-type for -content.
819 -compare
821 This is the standard compare mode argument. S3::Put re‐
822 turns 1 if the data was copied or 0 if the data was
823 skipped due to the comparison mode so indicating it
824 should be skipped.
825 S3::Get ?-bucket bucketname? -resource resourcename ?-blocking boolean?
828 ?-compare comparemode? ?-file filename? ?-content contentvarname?
829 ?-timestamp aws|now? ?-headers headervarname?
830 This command retrieves data from a resource on Amazon's S3
831 servers, using the HTTP GET command. It returns 0 if the -com‐
832 pare mode prevented the transfer, 1 if the transfer worked, or
833 throws an error if the transfer was attempted but failed. Server
834 5XX errors and S3 socket errors are are retried according to
835 S3:Configure settings before throwing an error; other errors
836 throw immediately. Note that this is always authenticated as the
837 user configured in via S3::Configure -accesskeyid. Use the
838 Tcllib http for unauthenticated GETs.
839 -bucket
841 This specifies the bucket from which the resource will be
842 read. Leading and/or trailing slashes are removed for
843 you, as are spaces.
844 -resource
846 This is the full name of the resource within the bucket.
847 A single leading slash is removed, but not a trailing
848 slash. Spaces are not trimmed.
849 -blocking
851 The standard blocking flag.
852 -file If this is specified, the body of the resource will be
854 read into this file, incrementally without pulling it en‐
855 tirely into memory first. The parent directory must al‐
856 ready exist. If the file already exists, it must be
857 writable. If an error is thrown part-way through the
858 process and the file already existed, it may be clob‐
859 bered. If an error is thrown part-way through the process
860 and the file did not already exist, any partial bits will
861 be deleted. Specifying this is an error if -content is
862 also specified, but at least one of -file or -content
863 must be specified.
864 -timestamp
866 This is only valid in conjunction with -file. It may be
867 specified as now or aws. The default is now. If now, the
868 file's modification date is left up to the system. If
869 aws, the file's mtime is set to match the Last-Modified
870 header on the resource, synchronizing the two appropri‐
871 ately for -compare date or -compare newer.
872 -content
874 If this is specified, the contentvarname is a variable in
875 the caller's scope (not necessarily global) that receives
876 the value of the body of the resource. No encoding is
877 done, so if the resource (for example) represents a UTF-8
878 byte sequence, use [encoding convertfrom utf-8] to get a
879 valid UTF-8 string. If this is specified, the -compare is
880 ignored unless it is never, in which case no assignment
881 to contentvarname is performed. Specifying this is an er‐
882 ror if -file is also specified, but at least one of -file
883 or -content must be specified.
884 -compare
886 This is the standard compare mode argument. S3::Get re‐
887 turns 1 if the data was copied or 0 if the data was
888 skipped due to the comparison mode so indicating it
889 should be skipped.
890 -headers
892 If this is specified, the headers resulting from the
893 fetch are stored in the provided variable, as a dictio‐
894 nary. This will include content-type and x-amz-meta-*
895 headers, as well as the usual HTTP headers, the x-amz-id
896 debugging headers, and so on. If no file is fetched (due
897 to -compare or other errors), no assignment to this vari‐
898 able is performed.
899 S3::Head ?-bucket bucketname? -resource resourcename ?-blocking bool‐
902 ean? ?-dict dictvarname? ?-headers headersvarname? ?-status statusvar‐
903 name?
904 This command requests HEAD from the resource. It returns
905 whether a 2XX code was returned as a result of the request,
906 never throwing an S3 remote error. That is, if this returns 1,
907 the resource exists and is accessible. If this returns 0, some‐
908 thing went wrong, and the -status result can be consulted for
909 details.
910 -bucket
912 This specifies the bucket from which the resource will be
913 read. Leading and/or trailing slashes are removed for
914 you, as are spaces.
915 -resource
917 This is the full name of the resource within the bucket.
918 A single leading slash is removed, but not a trailing
919 slash. Spaces are not trimmed.
920 -blocking
922 The standard blocking flag.
923 -dict If specified, the resulting dictionary from the S3::REST
925 call is assigned to the indicated (not necessarily
926 global) variable in the caller's scope.
927 -headers
929 If specified, the dictionary of headers from the result
930 are assigned to the indicated (not necessarily global)
931 variable in the caller's scope.
932 -status
934 If specified, the indicated (not necessarily global)
935 variable in the caller's scope is assigned a 2-element
936 list. The first element is the 3-digit HTTP status code,
937 while the second element is the HTTP message (such as
938 "OK" or "Forbidden").
939 S3::GetAcl ?-blocking boolean? ?-bucket bucketname? -resource resource‐
941 name ?-result-type REST|xml|pxml?
942 This command gets the ACL of the indicated resource or throws an
943 error if it is unavailable.
944 -blocking boolean
946 See above for standard definition.
947 -bucket
949 This specifies the bucket from which the resource will be
950 read. Leading and/or trailing slashes are removed for
951 you, as are spaces.
952 -resource
954 This is the full name of the resource within the bucket.
955 A single leading slash is removed, but not a trailing
956 slash. Spaces are not trimmed.
957 -parse-xml xml
959 The XML from a previous GetACL can be passed in to be
960 parsed into dictionary form. In this case, -result-type
961 must be pxml or dict.
962 -result-type REST
964 The dictionary returned by S3::REST is the return value
965 of S3::GetAcl. In this case, a non-2XX httpstatus will
966 not throw an error.
967 -result-type xml
969 The raw XML of the body is returned as the result (with
970 no encoding applied).
971 -result-type pxml
973 The XML of the body as parsed by xsxp::parse is returned.
974 -result-type dict
976 This fetches the ACL, parses it, and returns a dictionary
977 of two elements.
978 The first element has the key "owner" whose value is the
980 canonical ID of the owner of the resource.
981 The second element has the key "acl" whose value is a
983 dictionary. Each key in the dictionary is one of Ama‐
984 zon's permissions, namely "READ", "WRITE", "READ_ACP",
985 "WRITE_ACP", or "FULL_CONTROL". Each value of each key
986 is a list of canonical IDs or group URLs that have that
987 permission. Elements are not in the list in any particu‐
988 lar order, and not all keys are necessarily present.
989 Display names are not returned, as they are not espe‐
990 cially useful; use pxml to obtain them if necessary.
991 S3::PutAcl ?-blocking boolean? ?-bucket bucketname? -resource resource‐
993 name ?-acl new-acl?
994 This sets the ACL on the indicated resource. It returns the XML
995 written to the ACL, or throws an error if anything went wrong.
996 -blocking boolean
998 See above for standard definition.
999 -bucket
1001 This specifies the bucket from which the resource will be
1002 read. Leading and/or trailing slashes are removed for
1003 you, as are spaces.
1004 -resource
1006 This is the full name of the resource within the bucket.
1007 A single leading slash is removed, but not a trailing
1008 slash. Spaces are not trimmed.
1009 -owner If this is provided, it is assumed to match the owner of
1011 the resource. Otherwise, a GET may need to be issued
1012 against the resource to find the owner. If you already
1013 have the owner (such as from a call to S3::GetAcl, you
1014 can pass the value of the "owner" key as the value of
1015 this option, and it will be used in the construction of
1016 the XML.
1017 -acl If this option is specified, it provides the ACL the
1019 caller wishes to write to the resource. If this is not
1020 supplied or is empty, the value is taken from S3::Config‐
1021 ure -default-acl. The ACL is written with a PUT to the
1022 ?acl resource.
1023 If the value passed to this option starts with "<", it is
1025 taken to be a body to be PUT to the ACL resource.
1026 If the value matches one of the standard Amazon x-amz-acl
1028 headers (i.e., a canned access policy), that header is
1029 translated to XML and then applied. The canned access
1030 policies are private, public-read, public-read-write, and
1031 authenticated-read (in lower case).
1032 Otherwise, the value is assumed to be a dictionary for‐
1034 matted as the "acl" sub-entry within the dict returns by
1035 S3::GetAcl -result-type dict. The proper XML is gener‐
1036 ated and applied to the resource. Note that a value con‐
1037 taining "//" is assumed to be a group, a value containing
1038 "@" is assumed to be an AmazonCustomerByEmail, and other‐
1039 wise the value is assumed to be a canonical Amazon ID.
1040 Note that you cannot change the owner, so calling GetAcl
1042 on a resource owned by one user and applying it via
1043 PutAcl on a resource owned by another user may not do ex‐
1044 actly what you expect.
1045 S3::Delete ?-bucket bucketname? -resource resourcename ?-blocking bool‐
1047 ean? ?-status statusvar?
1048 This command deletes the specified resource from the specified
1049 bucket. It returns 1 if the resource was deleted successfully,
1050 0 otherwise. It returns 0 rather than throwing an S3 remote er‐
1051 ror.
1052 -bucket
1054 This specifies the bucket from which the resource will be
1055 deleted. Leading and/or trailing slashes are removed for
1056 you, as are spaces.
1057 -resource
1059 This is the full name of the resource within the bucket.
1060 A single leading slash is removed, but not a trailing
1061 slash. Spaces are not trimmed.
1062 -blocking
1064 The standard blocking flag.
1065 -status
1067 If specified, the indicated (not necessarily global)
1068 variable in the caller's scope is set to a two-element
1069 list. The first element is the 3-digit HTTP status code.
1070 The second element is the HTTP message (such as "OK" or
1071 "Forbidden"). Note that Amazon's DELETE result is 204 on
1072 success, that being the code indicating no content in the
1073 returned body.
1074 S3::Push ?-bucket bucketname? -directory directoryname ?-prefix pre‐
1077 fixstring? ?-compare comparemode? ?-x-amz-meta-* metastring? ?-acl
1078 aclcode? ?-delete boolean? ?-error throw|break|continue? ?-progress
1079 scriptprefix?
1080 This synchronises a local directory with a remote bucket by
1081 pushing the differences using S3::Put. Note that if something
1082 has changed in the bucket but not locally, those changes could
1083 be lost. Thus, this is not a general two-way synchronization
1084 primitive. (See S3::Sync for that.) Note too that resource names
1085 are case sensitive, so changing the case of a file on a Windows
1086 machine may lead to otherwise-unnecessary transfers. Note that
1087 only regular files are considered, so devices, pipes, symlinks,
1088 and directories are not copied.
1089 -bucket
1091 This names the bucket into which data will be pushed.
1092 -directory
1094 This names the local directory from which files will be
1095 taken. It must exist, be readable via [glob] and so on.
1096 If only some of the files therein are readable, S3::Push
1097 will PUT those files that are readable and return in its
1098 results the list of files that could not be opened.
1099 -prefix
1101 This names the prefix that will be added to all re‐
1102 sources. That is, it is the remote equivalent of -direc‐
1103 tory. If it is not specified, the root of the bucket
1104 will be treated as the remote directory. An example may
1105 clarify.
1106 S3::Push -bucket test -directory /tmp/xyz -prefix hello/world
1109 In this example, /tmp/xyz/pdq.html will be stored as
1112 http://s3.amazonaws.com/test/hello/world/pdq.html in Ama‐
1113 zon's servers. Also, /tmp/xyz/abc/def/Hello will be
1114 stored as http://s3.amazon‐
1115 aws.com/test/hello/world/abc/def/Hello in Amazon's
1116 servers. Without the -prefix option, /tmp/xyz/pdq.html
1117 would be stored as http://s3.amazonaws.com/test/pdq.html.
1118 -blocking
1120 This is the standard blocking option.
1121 -compare
1123 If present, this is passed to each invocation of S3::Put.
1124 Naturally, S3::Configure -default-compare is used if this
1125 is not specified.
1126 -x-amz-meta-*
1128 If present, this is passed to each invocation of S3::Put.
1129 All copied files will have the same metadata.
1130 -acl If present, this is passed to each invocation of S3::Put.
1132 -delete
1134 This defaults to false. If true, resources in the desti‐
1135 nation that are not in the source directory are deleted
1136 with S3::Delete. Since only regular files are consid‐
1137 ered, the existance of a symlink, pipe, device, or direc‐
1138 tory in the local source will not prevent the deletion of
1139 a remote resource with a corresponding name.
1140 -error This controls the behavior of S3::Push in the event that
1142 S3::Put throws an error. Note that errors encountered on
1143 the local file system or in reading the list of resources
1144 in the remote bucket always throw errors. This option
1145 allows control over "partial" errors, when some files
1146 were copied and some were not. S3::Delete is always fin‐
1147 ished up, with errors simply recorded in the return re‐
1148 sult.
1149 throw The error is rethrown with the same errorCode.
1151 break Processing stops without throwing an error, the
1153 error is recorded in the return value, and the
1154 command returns with a normal return. The calls
1155 to S3::Delete are not started.
1156 continue
1158 This is the default. Processing continues without
1159 throwing, recording the error in the return re‐
1160 sult, and resuming with the next file in the local
1161 directory to be copied.
1162 -progress
1164 If this is specified and the indicated script prefix is
1165 not empty, the indicated script prefix will be invoked
1166 several times in the caller's context with additional ar‐
1167 guments at various points in the processing. This allows
1168 progress reporting without backgrounding. The provided
1169 prefix will be invoked with additional arguments, with
1170 the first additional argument indicating what part of the
1171 process is being reported on. The prefix is initially
1172 invoked with args as the first additional argument and a
1173 dictionary representing the normalized arguments to the
1174 S3::Push call as the second additional argument. Then
1175 the prefix is invoked with local as the first additional
1176 argument and a list of suffixes of the files to be con‐
1177 sidered as the second argument. Then the prefix is in‐
1178 voked with remote as the first additional argument and a
1179 list of suffixes existing in the remote bucket as the
1180 second additional argument. Then, for each file in the
1181 local list, the prefix will be invoked with start as the
1182 first additional argument and the common suffix as the
1183 second additional argument. When S3::Put returns for
1184 that file, the prefix will be invoked with copy as the
1185 first additional argument, the common suffix as the sec‐
1186 ond additional argument, and a third argument that will
1187 be "copied" (if S3::Put sent the resource), "skipped" (if
1188 S3::Put decided not to based on -compare), or the error‐
1189 Code that S3::Put threw due to unexpected errors (in
1190 which case the third argument is a list that starts with
1191 "S3"). When all files have been transfered, the prefix
1192 may be invoked zero or more times with delete as the
1193 first additional argument and the suffix of the resource
1194 being deleted as the second additional argument, with a
1195 third argument being either an empty string (if the
1196 delete worked) or the errorCode from S3::Delete if it
1197 failed. Finally, the prefix will be invoked with finished
1198 as the first additional argument and the return value as
1199 the second additional argument.
1200 The return result from this command is a dictionary. They keys
1202 are the suffixes (i.e., the common portion of the path after the
1203 -directory and -prefix), while the values are either "copied",
1204 "skipped" (if -compare indicated not to copy the file), or the
1205 errorCode thrown by S3::Put, as appropriate. If -delete was
1206 true, there may also be entries for suffixes with the value
1207 "deleted" or "notdeleted", indicating whether the attempted
1208 S3::Delete worked or not, respectively. There is one additional
1209 pair in the return result, whose key is the empty string and
1210 whose value is a nested dictionary. The keys of this nested
1211 dictionary include "filescopied" (the number of files success‐
1212 fully copied), "bytescopied" (the number of data bytes in the
1213 files copied, excluding headers, metadata, etc), "com‐
1214 pareskipped" (the number of files not copied due to -compare
1215 mode), "errorskipped" (the number of files not copied due to
1216 thrown errors), "filesdeleted" (the number of resources deleted
1217 due to not having corresponding files locally, or 0 if -delete
1218 is false), and "filesnotdeleted" (the number of resources whose
1219 deletion was attempted but failed).
1220 Note that this is currently implemented somewhat inefficiently.
1222 It fetches the bucket listing (including timestamps and eTags),
1223 then calls S3::Put, which uses HEAD to find the timestamps and
1224 eTags again. Correcting this with no API change is planned for a
1225 future upgrade.
1226 S3::Pull ?-bucket bucketname? -directory directoryname ?-prefix pre‐
1229 fixstring? ?-blocking boolean? ?-compare comparemode? ?-delete boolean?
1230 ?-timestamp aws|now? ?-error throw|break|continue? ?-progress script‐
1231 prefix?
1232 This synchronises a remote bucket with a local directory by
1233 pulling the differences using S3::Get If something has been
1234 changed locally but not in the bucket, those difference may be
1235 lost. This is not a general two-way synchronization mechanism.
1236 (See S3::Sync for that.) This creates directories if needed;
1237 new directories are created with default permissions. Note that
1238 resource names are case sensitive, so changing the case of a
1239 file on a Windows machine may lead to otherwise-unnecessary
1240 transfers. Also, try not to store data in resources that end
1241 with a slash, or which are prefixes of resources that otherwise
1242 would start with a slash; i.e., don't use this if you store data
1243 in resources whose names have to be directories locally.
1244 Note that this is currently implemented somewhat inefficiently.
1246 It fetches the bucket listing (including timestamps and eTags),
1247 then calls S3::Get, which uses HEAD to find the timestamps and
1248 eTags again. Correcting this with no API change is planned for a
1249 future upgrade.
1250 -bucket
1252 This names the bucket from which data will be pulled.
1253 -directory
1255 This names the local directory into which files will be
1256 written It must exist, be readable via [glob], writable
1257 for file creation, and so on. If only some of the files
1258 therein are writable, S3::Pull will GET those files that
1259 are writable and return in its results the list of files
1260 that could not be opened.
1261 -prefix
1263 The prefix of resources that will be considered for re‐
1264 trieval. See S3::Push for more details, examples, etc.
1265 (Of course, S3::Pull reads rather than writes, but the
1266 prefix is treated similarly.)
1267 -blocking
1269 This is the standard blocking option.
1270 -compare
1272 This is passed to each invocation of S3::Get if provided.
1273 Naturally, S3::Configure -default-compare is used if this
1274 is not provided.
1275 -timestamp
1277 This is passed to each invocation of S3::Get if provided.
1278 -delete
1280 If this is specified and true, files that exist in the
1281 -directory that are not in the -prefix will be deleted
1282 after all resources have been copied. In addition, empty
1283 directories (other than the top-level -directory) will be
1284 deleted, as Amazon S3 has no concept of an empty direc‐
1285 tory.
1286 -error See S3::Push for a description of this option.
1288 -progress
1290 See S3::Push for a description of this option. It dif‐
1291 fers slightly in that local directories may be included
1292 with a trailing slash to indicate they are directories.
1293 The return value from this command is a dictionary. It is iden‐
1295 tical in form and meaning to the description of the return re‐
1296 sult of S3::Push. It differs only in that directories may be in‐
1297 cluded, with a trailing slash in their name, if they are empty
1298 and get deleted.
1299 S3::Toss ?-bucket bucketname? -prefix prefixstring ?-blocking boolean?
1301 ?-error throw|break|continue? ?-progress scriptprefix?
1302 This deletes some or all resources within a bucket. It would be
1303 considered a "recursive delete" had Amazon implemented actual
1304 directories.
1305 -bucket
1307 The bucket from which resources will be deleted.
1308 -blocking
1310 The standard blocking option.
1311 -prefix
1313 The prefix for resources to be deleted. Any resource that
1314 starts with this string will be deleted. This is re‐
1315 quired. To delete everything in the bucket, pass an
1316 empty string for the prefix.
1317 -error If this is "throw", S3::Toss rethrows any errors it en‐
1319 counters. If this is "break", S3::Toss returns with a
1320 normal return after the first error, recording that error
1321 in the return result. If this is "continue", which is the
1322 default, S3::Toss continues on and lists all errors in
1323 the return result.
1324 -progress
1326 If this is specified and not an empty string, the script
1327 prefix will be invoked several times in the context of
1328 the caller with additional arguments appended. Ini‐
1329 tially, it will be invoked with the first additional ar‐
1330 gument being args and the second being the processed list
1331 of arguments to S3::Toss. Then it is invoked with remote
1332 as the first additional argument and the list of suffixes
1333 in the bucket to be deleted as the second additional ar‐
1334 gument. Then it is invoked with the first additional ar‐
1335 gument being delete and the second additional argument
1336 being the suffix deleted and the third additional argu‐
1337 ment being "deleted" or "notdeleted" depending on whether
1338 S3::Delete threw an error. Finally, the script prefix is
1339 invoked with a first additional argument of "finished"
1340 and a second additional argument of the return value.
1341 The return value is a dictionary. The keys are the suffixes of
1343 files that S3::Toss attempted to delete, and whose values are
1344 either the string "deleted" or "notdeleted". There is also one
1345 additional pair, whose key is the empty string and whose value
1346 is an embedded dictionary. The keys of this embedded dictionary
1347 include "filesdeleted" and "filesnotdeleted", each of which has
1348 integer values.
1349
1351 • The pure-Tcl MD5 checking is slow. If you are processing files
1352 in the megabyte range, consider ensuring binary support is
1353 available.
1354 • The commands S3::Pull and S3::Push fetch a directory listing
1356 which includes timestamps and MD5 hashes, then invoke S3::Get
1357 and S3::Put. If a complex -compare mode is specified, S3::Get
1358 and S3::Put will invoke a HEAD operation for each file to fetch
1359 timestamps and MD5 hashes of each resource again. It is expected
1360 that a future release of this package will solve this without
1361 any API changes.
1362 • The commands S3::Pull and S3::Push fetch a directory listing
1364 without using -max-count. The entire directory is pulled into
1365 memory at once. For very large buckets, this could be a perfor‐
1366 mance problem. The author, at this time, does not plan to change
1367 this behavior. Welcome to Open Source.
1368 • S3::Sync is neither designed nor implemented yet. The intention
1370 would be to keep changes synchronised, so changes could be made
1371 to both the bucket and the local directory and be merged by
1372 S3::Sync.
1373 • Nor is -compare calc fully implemented. This is primarily due to
1375 Windows not providing a convenient method for distinguishing be‐
1376 tween local files that are "public-read" or "public-read-write".
1377 Assistance figuring out TWAPI for this would be appreciated. The
1378 U**X semantics are difficult to map directly as well. See the
1379 source for details. Note that there are not tests for calc,
1380 since it isn't done yet.
1381 • The HTTP processing is implemented within the library, rather
1383 than using a "real" HTTP package. Hence, multi-line headers are
1384 not (yet) handled correctly. Do not include carriage returns or
1385 linefeeds in x-amz-meta-* headers, content-type values, and so
1386 on. The author does not at this time expect to improve this.
1387 • Internally, S3::Push and S3::Pull and S3::Toss are all very sim‐
1389 ilar and should be refactored.
1390 • The idea of using -compare never -delete true to delete files
1392 that have been deleted from one place but not the other yet not
1393 copying changed files is untested.
1394
1396 To fetch a "directory" out of a bucket, make changes, and store it
1397 back:
1398 file mkdir ./tempfiles
1401 S3::Pull -bucket sample -prefix of/interest -directory ./tempfiles \
1402 -timestamp aws
1403 do_my_process ./tempfiles other arguments
1404 S3::Push -bucket sample -prefix of/interest -directory ./tempfiles \
1405 -compare newer -delete true
1406 To delete files locally that were deleted off of S3 but not otherwise
1409 update files:
1410 S3::Pull -bucket sample -prefix of/interest -directory ./myfiles \
1413 -compare never -delete true
1414
1417 The author intends to work on several additional projects related to
1418 this package, in addition to finishing the unfinished features.
1419 First, a command-line program allowing browsing of buckets and transfer
1421 of files from shell scripts and command prompts is useful.
1422 Second, a GUI-based program allowing visual manipulation of bucket and
1424 resource trees not unlike Windows Explorer would be useful.
1425 Third, a command-line (and perhaps a GUI-based) program called "OddJob"
1427 that will use S3 to synchronize computation amongst multiple servers
1428 running OddJob. An S3 bucket will be set up with a number of scripts to
1429 run, and the OddJob program can be invoked on multiple machines to run
1430 scripts on all the machines, each moving on to the next unstarted task
1431 as it finishes each. This is still being designed, and it is intended
1432 primarily to be run on Amazon's Elastic Compute Cloud.
1433
1435 This package uses the TLS package to handle the security for https urls
1436 and other socket connections.
1437 Policy decisions like the set of protocols to support and what ciphers
1439 to use are not the responsibility of TLS, nor of this package itself
1440 however. Such decisions are the responsibility of whichever applica‐
1441 tion is using the package, and are likely influenced by the set of
1442 servers the application will talk to as well.
1443 For example, in light of the recent POODLE attack [http://googleonli‐
1445 nesecurity.blogspot.co.uk/2014/10/this-poodle-bites-exploiting-
1446 ssl-30.html] discovered by Google many servers will disable support for
1447 the SSLv3 protocol. To handle this change the applications using TLS
1448 must be patched, and not this package, nor TLS itself. Such a patch
1449 may be as simple as generally activating tls1 support, as shown in the
1450 example below.
1451 package require tls
1454 tls::init -tls1 1 ;# forcibly activate support for the TLS1 protocol
1455 ... your own application code ...
1457
1460 This document, and the package it describes, will undoubtedly contain
1461 bugs and other problems. Please report such in the category amazon-s3
1462 of the Tcllib Trackers [http://core.tcl.tk/tcllib/reportlist]. Please
1463 also report any ideas for enhancements you may have for either package
1464 and/or documentation.
1465 When proposing code changes, please provide unified diffs, i.e the out‐
1467 put of diff -u.
1468 Note further that attachments are strongly preferred over inlined
1470 patches. Attachments can be made by going to the Edit form of the
1471 ticket immediately after its creation, and then using the left-most
1472 button in the secondary navigation bar.
1473
1475 amazon, cloud, s3
1476
1478 Networking
1479
1481 2006,2008 Darren New. All Rights Reserved. See LICENSE.TXT for terms.
1482tcllib 1.0.3 S3(n)