1S3(n) Amazon S3 Web Service Interface S3(n)
2______________________________________________________________________________
6
8 S3 - Amazon S3 Web Service Interface
9
11 package require Tcl 8.5
12 package require sha1 1.0
14 package require md5 2.0
16 package require base64 2.3
18 package require xsxp 1.0
20 S3::Configure ?-reset boolean? ?-retries integer? ?-accesskeyid
22 idstring? ?-secretaccesskey idstring? ?-service-access-point FQDN?
23 ?-use-tls boolean? ?-default-compare always|never|exists|miss‐
24 ing|newer|date|checksum|different? ?-default-separator string?
25 ?-default-acl private|public-read|public-read-write|authenticated-
26 read|keep|calc? ?-default-bucket bucketname?
27 S3::SuggestBucket ?name?
29 S3::REST dict
31 S3::ListAllMyBuckets ?-blocking boolean? ?-parse-xml xmlstring?
33 ?-result-type REST|xml|pxml|dict|names|owner?
34 S3::PutBucket ?-bucket bucketname? ?-blocking boolean? ?-acl {}|pri‐
36 vate|public-read|public-read-write|authenticated-read?
37 S3::DeleteBucket ?-bucket bucketname? ?-blocking boolean?
39 S3::GetBucket ?-bucket bucketname? ?-blocking boolean? ?-parse-xml xml‐
41 string? ?-max-count integer? ?-prefix prefixstring? ?-delimiter delim‐
42 iterstring? ?-result-type REST|xml|pxml|names|dict?
43 S3::Put ?-bucket bucketname? -resource resourcename ?-blocking boolean?
45 ?-file filename? ?-content contentstring? ?-acl private|public-
46 read|public-read-write|authenticated-read|calc|keep? ?-content-type
47 contenttypestring? ?-x-amz-meta-* metadatatext? ?-compare comparemode?
48 S3::Get ?-bucket bucketname? -resource resourcename ?-blocking boolean?
50 ?-compare comparemode? ?-file filename? ?-content contentvarname?
51 ?-timestamp aws|now? ?-headers headervarname?
52 S3::Head ?-bucket bucketname? -resource resourcename ?-blocking bool‐
54 ean? ?-dict dictvarname? ?-headers headersvarname? ?-status statusvar‐
55 name?
56 S3::GetAcl ?-blocking boolean? ?-bucket bucketname? -resource resource‐
58 name ?-result-type REST|xml|pxml?
59 S3::PutAcl ?-blocking boolean? ?-bucket bucketname? -resource resource‐
61 name ?-acl new-acl?
62 S3::Delete ?-bucket bucketname? -resource resourcename ?-blocking bool‐
64 ean? ?-status statusvar?
65 S3::Push ?-bucket bucketname? -directory directoryname ?-prefix pre‐
67 fixstring? ?-compare comparemode? ?-x-amz-meta-* metastring? ?-acl
68 aclcode? ?-delete boolean? ?-error throw|break|continue? ?-progress
69 scriptprefix?
70 S3::Pull ?-bucket bucketname? -directory directoryname ?-prefix pre‐
72 fixstring? ?-blocking boolean? ?-compare comparemode? ?-delete boolean?
73 ?-timestamp aws|now? ?-error throw|break|continue? ?-progress script‐
74 prefix?
75 S3::Toss ?-bucket bucketname? -prefix prefixstring ?-blocking boolean?
77 ?-error throw|break|continue? ?-progress scriptprefix?
78_________________________________________________________________
80
82 This package provides access to Amazon's Simple Storage Solution web
83 service.
84 As a quick summary, Amazon Simple Storage Solution provides a for-fee
86 web service allowing the storage of arbitrary data as "resources"
87 within "buckets" online. See http://www.amazonaws.com/ for details on
88 that system. Access to the service is via HTTP (SOAP or REST). Much
89 of this documentation will not make sense if you're not familiar with
90 the terms and functionality of the Amazon S3 service.
91 This package provides services for reading and writing the data items
93 via the REST interface. It also provides some higher-level operations.
94 Other packages in the same distribution provide for even more function‐
95 ality.
96 Copyright 2006 Darren New. All Rights Reserved. NO WARRANTIES OF ANY
98 TYPE ARE PROVIDED. COPYING OR USE INDEMNIFIES THE AUTHOR IN ALL WAYS.
99 This software is licensed under essentially the same terms as Tcl. See
100 LICENSE.txt for the terms.
101
103 The error reporting from this package makes use of $errorCode to pro‐
104 vide more details on what happened than simply throwing an error. Any
105 error caught by the S3 package (and we try to catch them all) will
106 return with an $errorCode being a list having at least three elements.
107 In all cases, the first element will be "S3". The second element will
108 take on one of six values, with that element defining the value of the
109 third and subsequent elements. S3::REST does not throw an error, but
110 rather returns a dictionary with the keys "error", "errorInfo", and
111 "errorCode" set. This allows for reliable background use. The possible
112 second elements are these:
113 usage The usage of the package is incorrect. For example, a command
115 has been invoked which requires the library to be configured
116 before the library has been configured, or an invalid combina‐
117 tion of options has been specified. The third element of $error‐
118 Code supplies the name of the parameter that was wrong. The
119 fourth usually provides the arguments that were actually sup‐
120 plied to the throwing proc, unless the usage error isn't con‐
121 fined to a single proc.
122 local Something happened on the local system which threw an error. For
124 example, a request to upload or download a file was made and the
125 file permissions denied that sort of access. The third element
126 of $errorCode is the original $errorCode.
127 socket Something happened with the socket. It closed prematurely, or
129 some other condition of failure-to-communicate-with-Amazon was
130 detected. The third element of $errorCode is the original
131 $errorCode, or sometimes the message from fcopy, or ...?
132 remote The Amazon web service returned an error code outside the 2xx
134 range in the HTTP header. In other words, everything went as
135 documented, except this particular case was documented not to
136 work. The third element is the dictionary returned from
137 ::S3::REST. Note that S3::REST itself never throws this error,
138 but just returns the dictionary. Most of the higher-level com‐
139 mands throw for convenience, unless an argument indicates they
140 should not. If something is documented as "not throwing an S3
141 remote error", it means a status return is set rather than
142 throwing an error if Amazon returns a non-2XX HTTP result code.
143 notyet The user obeyed the documentation, but the author has not yet
145 gotten around to implementing this feature. (Right now, only TLS
146 support and sophisticated permissions fall into this category,
147 as well as the S3::Acl command.)
148 xml The service has returned invalid XML, or XML whose schema is
150 unexpected. For the high-level commands that accept service XML
151 as input for parsing, this may also be thrown.
152
154 This package provides several separate levels of complexity.
155 · The lowest level simply takes arguments to be sent to the ser‐
157 vice, sends them, retrieves the result, and provides it to the
158 caller. Note: This layer allows both synchronous and event-
159 driven processing. It depends on the MD5 and SHA1 and base64
160 packages from Tcllib (available at http://tcllib.source‐
161 forge.net/). Note that S3::Configure is required for S3::REST
162 to work due to the authentication portion, so we put that in the
163 "lowest level."
164 · The next layer parses the results of calls, allowing for func‐
166 tionality such as uploading only changed files, synchronizing
167 directories, and so on. This layer depends on the TclXML pack‐
168 age as well as the included xsxp package. These packages are
169 package required when these more-sophisticated routines are
170 called, so nothing breaks if they are not correctly installed.
171 · Also included is a separate program that uses the library. It
173 provides code to parse $argv0 and $argv from the command line,
174 allowing invocation as a tclkit, etc. (Not yet implmented.)
175 · Another separate program provides a GUI interface allowing drag-
177 and-drop and other such functionality. (Not yet implemented.)
178 · Also built on this package is the OddJob program. It is a sepa‐
180 rate program designed to allow distribution of computational
181 work units over Amazon's Elastic Compute Cloud web service.
182 The goal is to have at least the bottom-most layers implemented in pure
184 Tcl using only that which comes from widely-available sources, such as
185 Tcllib.
186
188 These commands do not require any packages not listed above. They talk
189 directly to the service, or they are utility or configuration routines.
190 Note that the "xsxp" package was written to support this package, so it
191 should be available wherever you got this package.
192 S3::Configure ?-reset boolean? ?-retries integer? ?-accesskeyid
194 idstring? ?-secretaccesskey idstring? ?-service-access-point FQDN?
195 ?-use-tls boolean? ?-default-compare always|never|exists|miss‐
196 ing|newer|date|checksum|different? ?-default-separator string?
197 ?-default-acl private|public-read|public-read-write|authenticated-
198 read|keep|calc? ?-default-bucket bucketname?
199 There is one command for configuration, and that is S3::Config‐
200 ure. If called with no arguments, it returns a dictionary of
201 key/value pairs listing all current settings. If called with
202 one argument, it returns the value of that single argument. If
203 called with two or more arguments, it must be called with pairs
204 of arguments, and it applies the changes in order. There is
205 only one set of configuration information per interpreter.
206 The following options are accepted:
208 -reset boolean
210 By default, false. If true, any previous changes and any
211 changes on the same call before the reset option will be
212 returned to default values.
213 -retries integer
215 Default value is 3. If Amazon returns a 500 error, a
216 retry after an exponential backoff delay will be tried
217 this many times before finally throwing the 500 error.
218 This applies to each call to S3::REST from the higher-
219 level commands, but not to S3::REST itself. That is,
220 S3::REST will always return httpstatus 500 if that's what
221 it receives. Functions like S3::Put will retry the PUT
222 call, and will also retry the GET and HEAD calls used to
223 do content comparison. Changing this to 0 will prevent
224 retries and their associated delays. In addition, socket
225 errors (i.e., errors whose errorCode starts with "S3
226 socket") will be similarly retried after backoffs.
227 -accesskeyid idstring
229 -secretaccesskey idstring
231 Each defaults to an empty string. These must be set
232 before any calls are made. This is your S3 ID. Once you
233 sign up for an account, go to http://www.amazonaws.com/,
234 sign in, go to the "Your Web Services Account" button,
235 pick "AWS Access Identifiers", and your access key ID and
236 secret access keys will be available. All S3::REST calls
237 are authenticated. Blame Amazon for the poor choice of
238 names.
239 -service-access-point FQDN
241 Defaults to "s3.amazonaws.com". This is the fully-quali‐
242 fied domain name of the server to contact for S3::REST
243 calls. You should probably never need to touch this,
244 unless someone else implements a compatible service, or
245 you wish to test something by pointing the library at
246 your own service.
247 -slop-seconds integer
249 When comparing dates between Amazon and the local
250 machine, two dates within this many seconds of each other
251 are considered the same. Useful for clock drift correc‐
252 tion, processing overhead time, and so on.
253 -use-tls boolean
255 Defaults to false. This is not yet implemented. If true,
256 S3::REST will negotiate a TLS connection to Amazon. If
257 false, unencrypted connections are used.
258 -bucket-prefix string
260 Defaults to "TclS3". This string is used by S3::Suggest‐
261 BucketName if that command is passed an empty string as
262 an argument. It is used to distinguish different applica‐
263 tions using the Amazon service. Your application should
264 always set this to keep from interfering with the buckets
265 of other users of Amazon S3 or with other buckets of the
266 same user.
267 -default-compare always|never|exists|missing|newer|date|check‐
269 sum|different
270 Defaults to "always." If no -compare is specified on
271 S3::Put, S3::Get, or S3::Delete, this comparison is used.
272 See those commands for a description of the meaning.
273 -default-separator string
275 Defaults to "/". This is currently unused. It might make
276 sense to use this for S3::Push and S3::Pull, but allowing
277 resources to have slashes in their names that aren't
278 marking directories would be problematic. Hence, this
279 currently does nothing.
280 -default-acl private|public-read|public-read-write|authenti‐
282 cated-read|keep|calc
283 Defaults to an empty string. If no -acl argument is pro‐
284 vided to S3::Put or S3::Push, this string is used (given
285 as the x-amz-acl header if not keep or calc). If this is
286 also empty, no x-amz-acl header is generated. This is
287 not used by S3::REST.
288 -default-bucket bucketname
290 If no bucket is given to S3::GetBucket, S3::PutBucket,
291 S3::Get, S3::Put, S3::Head, S3::Acl, S3::Delete,
292 S3::Push, S3::Pull, or S3::Toss, and if this configura‐
293 tion variable is not an empty string (and not simply
294 "/"), then this value will be used for the bucket. This
295 is useful if one program does a large amount of resource
296 manipulation within a single bucket.
297 S3::SuggestBucket ?name?
300 The S3::SuggestBucket command accepts an optional string as a
301 prefix and returns a valid bucket containing the name argument
302 and the Access Key ID. This makes the name unique to the owner
303 and to the application (assuming the application picks a good
304 name argument). If no name is provided, the name from S3::Con‐
305 figure -bucket-prefix is used. If that too is empty (which is
306 not the default), an error is thrown.
307 S3::REST dict
309 The S3::REST command takes as an argument a dictionary and
310 returns a dictionary. The return dictionary has the same keys
311 as the input dictionary, and includes additional keys as the
312 result. The presence or absence of keys in the input dictionary
313 can control the behavior of the routine. It never throws an
314 error directly, but includes keys "error", "errorInfo", and
315 "errorCode" if necessary. Some keys are required, some
316 optional. The routine can run either in blocking or non-blocking
317 mode, based on the presense of resultvar in the input dictio‐
318 nary. This requires the -accesskeyid and -secretaccesskey to be
319 configured via S3::Configure before being called.
320 The possible input keys are these:
322 verb GET|PUT|DELETE|HEAD
324 This required item indicates the verb to be used.
325 resource string
327 This required item indicates the resource to be accessed.
328 A leading / is added if not there already. It will be
329 URL-encoded for you if necessary. Do not supply a
330 resource name that is already URL-encoded.
331 ?rtype torrent|acl?
333 This indicates a torrent or acl resource is being manipu‐
334 lated. Do not include this in the resource key, or the
335 "?" separator will get URL-encoded.
336 ?parameters dict?
338 This optional dictionary provides parameters added to the
339 URL for the transaction. The keys must be in the correct
340 case (which is confusing in the Amazon documentation) and
341 the values must be valid. This can be an empty dictionary
342 or omitted entirely if no parameters are desired. No
343 other error checking on parameters is performed.
344 ?headers dict?
346 This optional dictionary provides headers to be added to
347 the HTTP request. The keys must be in lower case for the
348 authentication to work. The values must not contain
349 embedded newlines or carriage returns. This is primarily
350 useful for adding x-amz-* headers. Since authentication
351 is calculated by S3::REST, do not add that header here.
352 Since content-type gets its own key, also do not add that
353 header here.
354 ?inbody contentstring?
356 This optional item, if provided, gives the content that
357 will be sent. It is sent with a tranfer encoding of
358 binary, and only the low bytes are used, so use [encoding
359 convertto utf-8] if the string is a utf-8 string. This is
360 written all in one blast, so if you are using non-block‐
361 ing mode and the inbody is especially large, you may wind
362 up blocking on the write socket.
363 ?infile filename?
365 This optional item, if provided, and if inbody is not
366 provided, names the file from which the body of the HTTP
367 message will be constructed. The file is opened for read‐
368 ing and sent progressively by [fcopy], so it should not
369 block in non-blocking mode even if the file is very
370 large. The file is transfered in binary mode, so the
371 bytes on your disk will match the bytes in your resource.
372 Due to HTTP restrictions, it must be possible to use
373 [file size] on this file to determine the size at the
374 start of the transaction.
375 ?S3chan channel?
377 This optional item, if provided, indicates the already-
378 open socket over which the transaction should be con‐
379 ducted. If not provided, a connection is made to the ser‐
380 vice access point specified via S3::Configure, which is
381 normally s3.amazonaws.com. If this is provided, the chan‐
382 nel is not closed at the end of the transaction.
383 ?outchan channel?
385 This optional item, if provided, indicates the already-
386 open channel to which the body returned from S3 should be
387 written. That is, to retrieve a large resource, open a
388 file, set the translation mode, and pass the channel as
389 the value of the key outchan. Output will be written to
390 the channel in pieces so memory does not fill up unneces‐
391 sarily. The channel is not closed at the end of the
392 transaction.
393 ?resultvar varname?
395 This optional item, if provided, indicates that S3::REST
396 should run in non-blocking mode. The varname should be
397 fully qualified with respect to namespaces and cannot be
398 local to a proc. If provided, the result of the S3::REST
399 call is assigned to this variable once everything has
400 completed; use trace or vwait to know when this has hap‐
401 pened. If this key is not provided, the result is simply
402 returned from the call to S3::REST and no calls to the
403 eventloop are invoked from within this call.
404 ?throwsocket throw|return?
406 This optional item, if provided, indicates that S3::REST
407 should throw an error if throwmode is throw and a socket
408 error is encountered. It indicates that S3::REST should
409 return the error code in the returned dictionary if a
410 socket error is encountered and this is set to return. If
411 throwsocket is set to return or if the call is not block‐
412 ing, then a socket error (i.e., an error whose error code
413 starts with "S3 socket" will be returned in the dictio‐
414 nary as error, errorInfo, and errorCode. If a foreground
415 call is made (i.e., resultvar is not provided), and this
416 option is not provided or is set to throw, then error
417 will be invoked instead.
418 Once the call to S3::REST completes, a new dict is returned, either in
420 the resultvar or as the result of execution. This dict is a copy of the
421 original dict with the results added as new keys. The possible new keys
422 are these:
423 error errorstring
425 errorInfo errorstring
427 errorCode errorstring
429 If an error is caught, these three keys will be set in
430 the result. Note that S3::REST does not consider a
431 non-2XX HTTP return code as an error. The errorCode value
432 will be formatted according to the ERROR REPORTING
433 description. If these are present, other keys described
434 here might not be.
435 httpstatus threedigits
437 The three-digit code from the HTTP transaction. 2XX for
438 good, 5XX for server error, etc.
439 httpmessage text
441 The textual result after the status code. "OK" or "For‐
442 bidden" or etc.
443 outbody contentstring
445 If outchan was not specified, this key will hold a refer‐
446 ence to the (unencoded) contents of the body returned.
447 If Amazon returned an error (a la the httpstatus not a
448 2XX value), the error message will be in outbody or writ‐
449 ten to outchan as appropriate.
450 outheaders dict
452 This contains a dictionary of headers returned by Amazon.
453 The keys are always lower case. It's mainly useful for
454 finding the x-amz-meta-* headers, if any, although things
455 like last-modified and content-type are also useful. The
456 keys of this dictionary are always lower case. Both keys
457 and values are trimmed of extraneous whitespace.
458
460 The routines in this section all make use of one or more calls to
461 S3::REST to do their work, then parse and manage the data in a conve‐
462 nient way. All these commands throw errors as described in ERROR
463 REPORTING unless otherwise noted.
464 In all these commands, all arguments are presented as name/value pairs,
466 in any order. All the argument names start with a hyphen.
467 There are a few options that are common to many of the commands, and
469 those common options are documented here.
470 -blocking boolean
472 If provided and specified as false, then any calls to S3:REST
473 will be non-blocking, and internally these routines will call
474 [vwait] to get the results. In other words, these routines will
475 return the same value, but they'll have event loops running
476 while waiting for Amazon.
477 -parse-xml xmlstring
479 If provided, the routine skips actually communicating with Ama‐
480 zon, and instead behaves as if the XML string provided was
481 returned as the body of the call. Since several of these rou‐
482 tines allow the return of data in various formats, this argument
483 can be used to parse existing XML to extract the bits of infor‐
484 mation that are needed. It's also helpful for testing.
485 -bucket bucketname
487 Almost every high-level command needs to know what bucket the
488 resources are in. This option specifies that. (Only the command
489 to list available buckets does not require this parameter.)
490 This does not need to be URL-encoded, even if it contains spe‐
491 cial or non-ASCII characters. May or may not contain leading or
492 trailing spaces - commands normalize the bucket. If this is not
493 supplied, the value is taken from S3::Configure -default-bucket
494 if that string isn't empty. Note that spaces and slashes are
495 always trimmed from both ends and the rest must leave a valid
496 bucket.
497 -resource resourcename
499 This specifies the resource of interest within the bucket. It
500 may or may not start with a slash - both cases are handled.
501 This does not need to be URL-encoded, even if it contains spe‐
502 cial or non-ASCII characters.
503 -compare always|never|exists|missing|newer|date|checksum|different
505 When commands copy resources to files or files to resources, the
506 caller may specify that the copy should be skipped if the con‐
507 tents are the same. This argument specifies the conditions under
508 which the files should be copied. If it is not passed, the
509 result of S3::Configure -default-compare is used, which in turn
510 defaults to "always." The meanings of the various values are
511 these:
512 always Always copy the data. This is the default.
514 never Never copy the data. This is essentially a no-op, except
516 in S3::Push and S3::Pull where the -delete flag might
517 make a difference.
518 exists Copy the data only if the destination already exists.
520 missing
522 Copy the data only if the destination does not already
523 exist.
524 newer Copy the data if the destination is missing, or if the
526 date on the source is newer than the date on the destina‐
527 tion by at least S3::Configure -slop-seconds seconds. If
528 the source is Amazon, the date is taken from the Last-
529 Modified header. If the source is local, it is taken as
530 the mtime of the file. If the source data is specified in
531 a string rather than a file, it is taken as right now,
532 via [clock seconds].
533 date Like newer, except copy if the date is newer or older.
535 checksum
537 Calculate the MD5 checksum on the local file or string,
538 ask Amazon for the eTag of the resource, and copy the
539 data if they're different. Copy the data also if the des‐
540 tination is missing. Note that this can be slow with
541 large local files unless the C version of the MD5 support
542 is available.
543 different
545 Copy the data if the destination does not exist. If the
546 destination exists and an actual file name was specified
547 (rather than a content string), and the date on the file
548 differs from the date on the resource, copy the data. If
549 the data is provided as a content string, the "date" is
550 treated as "right now", so it will likely always differ
551 unless slop-seconds is large. If the dates are the same,
552 the MD5 checksums are compared, and the data is copied if
553 the checksums differ.
554 Note that "newer" and "date" don't care about the contents, and "check‐
556 sum" doesn't care about the dates, but "different" checks both.
557 S3::ListAllMyBuckets ?-blocking boolean? ?-parse-xml xmlstring?
559 ?-result-type REST|xml|pxml|dict|names|owner?
560 This routine performs a GET on the Amazon S3 service, which is
561 defined to return a list of buckets owned by the account identi‐
562 fied by the authorization header. (Blame Amazon for the dumb
563 names.)
564 -blocking boolean
566 See above for standard definition.
567 -parse-xml xmlstring
569 See above for standard definition.
570 -result-type REST
572 The dictionary returned by S3::REST is the return value
573 of S3::ListAllMyBuckets. In this case, a non-2XX httpsta‐
574 tus will not throw an error. You may not combine this
575 with -parse-xml.
576 -result-type xml
578 The raw XML of the body is returned as the result (with
579 no encoding applied).
580 -result-type pxml
582 The XML of the body as parsed by xsxp::parse is returned.
583 -result-type dict
585 A dictionary of interesting portions of the XML is
586 returned. The dictionary contains the following keys:
587 Owner/ID
589 The Amazon AWS ID (in hex) of the owner of the
590 bucket.
591 Owner/DisplayName
593 The Amazon AWS ID's Display Name.
594 Bucket/Name
596 A list of names, one for each bucket.
597 Bucket/CreationDate
599 A list of dates, one for each bucket, in the same
600 order as Bucket/Name, in ISO format (as returned
601 by Amazon).
602 -result-type names
605 A list of bucket names is returned with all other infor‐
606 mation stripped out. This is the default result type for
607 this command.
608 -result-type owner
610 A list containing two elements is returned. The first
611 element is the owner's ID, and the second is the owner's
612 display name.
613 S3::PutBucket ?-bucket bucketname? ?-blocking boolean? ?-acl {}|pri‐
616 vate|public-read|public-read-write|authenticated-read?
617 This command creates a bucket if it does not already exist.
618 Bucket names are globally unique, so you may get a "Forbidden"
619 error from Amazon even if you cannot see the bucket in
620 S3::ListAllMyBuckets. See S3::SuggestBucket for ways to minimize
621 this risk. The x-amz-acl header comes from the -acl option, or
622 from S3::Configure -default-acl if not specified.
623 S3::DeleteBucket ?-bucket bucketname? ?-blocking boolean?
625 This command deletes a bucket if it is empty and you have such
626 permission. Note that Amazon's list of buckets is a global
627 resource, requiring far-flung synchronization. If you delete a
628 bucket, it may be quite a few minutes (or hours) before you can
629 recreate it, yielding "Conflict" errors until then.
630 S3::GetBucket ?-bucket bucketname? ?-blocking boolean? ?-parse-xml xml‐
632 string? ?-max-count integer? ?-prefix prefixstring? ?-delimiter delim‐
633 iterstring? ?-result-type REST|xml|pxml|names|dict?
634 This lists the contents of a bucket. That is, it returns a
635 directory listing of resources within a bucket, rather than
636 transfering any user data.
637 -bucket bucketname
639 The standard bucket argument.
640 -blocking boolean
642 The standard blocking argument.
643 -parse-xml xmlstring
645 The standard parse-xml argument.
646 -max-count integer
648 If supplied, this is the most number of records to be
649 returned. If not supplied, the code will iterate until
650 all records have been found. Not compatible with -parse-
651 xml. Note that if this is supplied, only one call to
652 S3::REST will be made. Otherwise, enough calls will be
653 made to exhaust the listing, buffering results in memory,
654 so take care if you may have huge buckets.
655 -prefix prefixstring
657 If present, restricts listing to resources with a partic‐
658 ular prefix. One leading / is stripped if present.
659 -delimiter delimiterstring
661 If present, specifies a delimiter for the listing. The
662 presence of this will summarize multiple resources into
663 one entry, as if S3 supported directories. See the Amazon
664 documentation for details.
665 -result-type REST|xml|pxml|names|dict
667 This indicates the format of the return result of the
668 command.
669 REST If -max-count is specified, the dictionary
671 returned from S3::REST is returned. If -max-count
672 is not specified, a list of all the dictionaries
673 returned from the one or more calls to S3::REST is
674 returned.
675 xml If -max-count is specified, the body returned from
677 S3::REST is returned. If -max-count is not speci‐
678 fied, a list of all the bodies returned from the
679 one or more calls to S3::REST is returned.
680 pxml If -max-count is specified, the body returned from
682 S3::REST is passed throught xsxp::parse and then
683 returned. If -max-count is not specified, a list
684 of all the bodies returned from the one or more
685 calls to S3::REST are each passed through
686 xsxp::parse and then returned.
687 names Returns a list of all names found in either the
689 Contents/Key fields or the CommonPrefixes/Prefix
690 fields. If no -delimiter is specified and no -max-
691 count is specified, this returns a list of all
692 resources with the specified -prefix.
693 dict Returns a dictionary. (Returns only one dictionary
695 even if -max-count wasn't specified.) The keys of
696 the dictionary are as follows:
697 Name The name of the bucket (from the final call
699 to S3::REST).
700 Prefix From the final call to S3::REST.
702 Marker From the final call to S3::REST.
704 MaxKeys
706 From the final call to S3::REST.
707 IsTruncated
709 From the final call to S3::REST, so always
710 false if -max-count is not specified.
711 NextMarker
713 Always provided if IsTruncated is true, and
714 calculated of Amazon does not provide it.
715 May be empty if IsTruncated is false.
716 Key A list of names of resources in the bucket
718 matching the -prefix and -delimiter
719 restrictions.
720 LastModified
722 A list of times of resources in the bucket,
723 in the same order as Key, in the format
724 returned by Amazon. (I.e., it is not parsed
725 into a seconds-from-epoch.)
726 ETag A list of entity tags (a.k.a. MD5 check‐
728 sums) in the same order as Key.
729 Size A list of sizes in bytes of the resources,
731 in the same order as Key.
732 Owner/ID
734 A list of owners of the resources in the
735 bucket, in the same order as Key.
736 Owner/DisplayName
738 A list of owners of the resources in the
739 bucket, in the same order as Key. These are
740 the display names.
741 CommonPrefixes/Prefix
743 A list of prefixes common to multiple enti‐
744 ties. This is present only if -delimiter
745 was supplied.
746 S3::Put ?-bucket bucketname? -resource resourcename ?-blocking boolean?
748 ?-file filename? ?-content contentstring? ?-acl private|public-
749 read|public-read-write|authenticated-read|calc|keep? ?-content-type
750 contenttypestring? ?-x-amz-meta-* metadatatext? ?-compare comparemode?
751 This command sends data to a resource on Amazon's servers for
752 storage, using the HTTP PUT command. It returns 0 if the -com‐
753 pare mode prevented the transfer, 1 if the transfer worked, or
754 throws an error if the transfer was attempted but failed.
755 Server 5XX errors and S3 socket errors are retried according to
756 S3:Configure -retries settings before throwing an error; other
757 errors throw immediately.
758 -bucket
760 This specifies the bucket into which the resource will be
761 written. Leading and/or trailing slashes are removed for
762 you, as are spaces.
763 -resource
765 This is the full name of the resource within the bucket.
766 A single leading slash is removed, but not a trailing
767 slash. Spaces are not trimmed.
768 -blocking
770 The standard blocking flag.
771 -file If this is specified, the filename must exist, must be
773 readable, and must not be a special or directory file.
774 [file size] must apply to it and must not change for the
775 lifetime of the call. The default content-type is calcu‐
776 lated based on the name and/or contents of the file.
777 Specifying this is an error if -content is also speci‐
778 fied, but at least one of -file or -content must be spec‐
779 ified. (The file is allowed to not exist or not be read‐
780 able if -compare never is specified.)
781 -content
783 If this is specified, the contentstring is sent as the
784 body of the resource. The content-type defaults to
785 "application/octet-string". Only the low bytes are sent,
786 so non-ASCII should use the appropriate encoding (such as
787 [encoding convertto utf-8]) before passing it to this
788 routine, if necessary. Specifying this is an error if
789 -file is also specified, but at least one of -file or
790 -content must be specified.
791 -acl This defaults to S3::Configure -default-acl if not speci‐
793 fied. It sets the x-amz-acl header on the PUT operation.
794 If the value provided is calc, the x-amz-acl header is
795 calculated based on the I/O permissions of the file to be
796 uploaded; it is an error to specify calc and -content.
797 If the value provided is keep, the acl of the resource is
798 read before the PUT (or the default is used if the
799 resource does not exist), then set back to what it was
800 after the PUT (if it existed). An error will occur if the
801 resource is successfully written but the kept ACL cannot
802 be then applied. This should never happen. Note: calc
803 is not currently fully implemented.
804 -x-amz-meta-*
806 If any header starts with "-x-amz-meta-", its contents
807 are added to the PUT command to be stored as metadata
808 with the resource. Again, no encoding is performed, and
809 the metadata should not contain characters like newlines,
810 carriage returns, and so on. It is best to stick with
811 simple ASCII strings, or to fix the library in several
812 places.
813 -content-type
815 This overrides the content-type calculated by -file or
816 sets the content-type for -content.
817 -compare
819 This is the standard compare mode argument. S3::Put
820 returns 1 if the data was copied or 0 if the data was
821 skipped due to the comparison mode so indicating it
822 should be skipped.
823 S3::Get ?-bucket bucketname? -resource resourcename ?-blocking boolean?
826 ?-compare comparemode? ?-file filename? ?-content contentvarname?
827 ?-timestamp aws|now? ?-headers headervarname?
828 This command retrieves data from a resource on Amazon's S3
829 servers, using the HTTP GET command. It returns 0 if the -com‐
830 pare mode prevented the transfer, 1 if the transfer worked, or
831 throws an error if the transfer was attempted but failed. Server
832 5XX errors and S3 socket errors are are retried according to
833 S3:Configure settings before throwing an error; other errors
834 throw immediately. Note that this is always authenticated as the
835 user configured in via S3::Configure -accesskeyid. Use the
836 Tcllib http for unauthenticated GETs.
837 -bucket
839 This specifies the bucket from which the resource will be
840 read. Leading and/or trailing slashes are removed for
841 you, as are spaces.
842 -resource
844 This is the full name of the resource within the bucket.
845 A single leading slash is removed, but not a trailing
846 slash. Spaces are not trimmed.
847 -blocking
849 The standard blocking flag.
850 -file If this is specified, the body of the resource will be
852 read into this file, incrementally without pulling it
853 entirely into memory first. The parent directory must
854 already exist. If the file already exists, it must be
855 writable. If an error is thrown part-way through the
856 process and the file already existed, it may be clob‐
857 bered. If an error is thrown part-way through the process
858 and the file did not already exist, any partial bits will
859 be deleted. Specifying this is an error if -content is
860 also specified, but at least one of -file or -content
861 must be specified.
862 -timestamp
864 This is only valid in conjunction with -file. It may be
865 specified as now or aws. The default is now. If now, the
866 file's modification date is left up to the system. If
867 aws, the file's mtime is set to match the Last-Modified
868 header on the resource, synchronizing the two appropri‐
869 ately for -compare date or -compare newer.
870 -content
872 If this is specified, the contentvarname is a variable in
873 the caller's scope (not necessarily global) that receives
874 the value of the body of the resource. No encoding is
875 done, so if the resource (for example) represents a UTF-8
876 byte sequence, use [encoding convertfrom utf-8] to get a
877 valid UTF-8 string. If this is specified, the -compare is
878 ignored unless it is never, in which case no assignment
879 to contentvarname is performed. Specifying this is an
880 error if -file is also specified, but at least one of
881 -file or -content must be specified.
882 -compare
884 This is the standard compare mode argument. S3::Get
885 returns 1 if the data was copied or 0 if the data was
886 skipped due to the comparison mode so indicating it
887 should be skipped.
888 -headers
890 If this is specified, the headers resulting from the
891 fetch are stored in the provided variable, as a dictio‐
892 nary. This will include content-type and x-amz-meta-*
893 headers, as well as the usual HTTP headers, the x-amz-id
894 debugging headers, and so on. If no file is fetched (due
895 to -compare or other errors), no assignment to this vari‐
896 able is performed.
897 S3::Head ?-bucket bucketname? -resource resourcename ?-blocking bool‐
900 ean? ?-dict dictvarname? ?-headers headersvarname? ?-status statusvar‐
901 name?
902 This command requests HEAD from the resource. It returns
903 whether a 2XX code was returned as a result of the request,
904 never throwing an S3 remote error. That is, if this returns 1,
905 the resource exists and is accessible. If this returns 0, some‐
906 thing went wrong, and the -status result can be consulted for
907 details.
908 -bucket
910 This specifies the bucket from which the resource will be
911 read. Leading and/or trailing slashes are removed for
912 you, as are spaces.
913 -resource
915 This is the full name of the resource within the bucket.
916 A single leading slash is removed, but not a trailing
917 slash. Spaces are not trimmed.
918 -blocking
920 The standard blocking flag.
921 -dict If specified, the resulting dictionary from the S3::REST
923 call is assigned to the indicated (not necessarily
924 global) variable in the caller's scope.
925 -headers
927 If specified, the dictionary of headers from the result
928 are assigned to the indicated (not necessarily global)
929 variable in the caller's scope.
930 -status
932 If specified, the indicated (not necessarily global)
933 variable in the caller's scope is assigned a 2-element
934 list. The first element is the 3-digit HTTP status code,
935 while the second element is the HTTP message (such as
936 "OK" or "Forbidden").
937 S3::GetAcl ?-blocking boolean? ?-bucket bucketname? -resource resource‐
939 name ?-result-type REST|xml|pxml?
940 This command gets the ACL of the indicated resource or throws an
941 error if it is unavailable.
942 -blocking boolean
944 See above for standard definition.
945 -bucket
947 This specifies the bucket from which the resource will be
948 read. Leading and/or trailing slashes are removed for
949 you, as are spaces.
950 -resource
952 This is the full name of the resource within the bucket.
953 A single leading slash is removed, but not a trailing
954 slash. Spaces are not trimmed.
955 -parse-xml xml
957 The XML from a previous GetACL can be passed in to be
958 parsed into dictionary form. In this case, -result-type
959 must be pxml or dict.
960 -result-type REST
962 The dictionary returned by S3::REST is the return value
963 of S3::GetAcl. In this case, a non-2XX httpstatus will
964 not throw an error.
965 -result-type xml
967 The raw XML of the body is returned as the result (with
968 no encoding applied).
969 -result-type pxml
971 The XML of the body as parsed by xsxp::parse is returned.
972 -result-type dict
974 This fetches the ACL, parses it, and returns a dictionary
975 of two elements.
976 The first element has the key "owner" whose value is the
978 canonical ID of the owner of the resource.
979 The second element has the key "acl" whose value is a
981 dictionary. Each key in the dictionary is one of Ama‐
982 zon's permissions, namely "READ", "WRITE", "READ_ACP",
983 "WRITE_ACP", or "FULL_CONTROL". Each value of each key
984 is a list of canonical IDs or group URLs that have that
985 permission. Elements are not in the list in any particu‐
986 lar order, and not all keys are necessarily present.
987 Display names are not returned, as they are not espe‐
988 cially useful; use pxml to obtain them if necessary.
989 S3::PutAcl ?-blocking boolean? ?-bucket bucketname? -resource resource‐
991 name ?-acl new-acl?
992 This sets the ACL on the indicated resource. It returns the XML
993 written to the ACL, or throws an error if anything went wrong.
994 -blocking boolean
996 See above for standard definition.
997 -bucket
999 This specifies the bucket from which the resource will be
1000 read. Leading and/or trailing slashes are removed for
1001 you, as are spaces.
1002 -resource
1004 This is the full name of the resource within the bucket.
1005 A single leading slash is removed, but not a trailing
1006 slash. Spaces are not trimmed.
1007 -owner If this is provided, it is assumed to match the owner of
1009 the resource. Otherwise, a GET may need to be issued
1010 against the resource to find the owner. If you already
1011 have the owner (such as from a call to S3::GetAcl, you
1012 can pass the value of the "owner" key as the value of
1013 this option, and it will be used in the construction of
1014 the XML.
1015 -acl If this option is specified, it provides the ACL the
1017 caller wishes to write to the resource. If this is not
1018 supplied or is empty, the value is taken from S3::Config‐
1019 ure -default-acl. The ACL is written with a PUT to the
1020 ?acl resource.
1021 If the value passed to this option starts with "<", it is
1023 taken to be a body to be PUT to the ACL resource.
1024 If the value matches one of the standard Amazon x-amz-acl
1026 headers (i.e., a canned access policy), that header is
1027 translated to XML and then applied. The canned access
1028 policies are private, public-read, public-read-write, and
1029 authenticated-read (in lower case).
1030 Otherwise, the value is assumed to be a dictionary for‐
1032 matted as the "acl" sub-entry within the dict returns by
1033 S3::GetAcl -result-type dict. The proper XML is gener‐
1034 ated and applied to the resource. Note that a value con‐
1035 taining "//" is assumed to be a group, a value containing
1036 "@" is assumed to be an AmazonCustomerByEmail, and other‐
1037 wise the value is assumed to be a canonical Amazon ID.
1038 Note that you cannot change the owner, so calling GetAcl
1040 on a resource owned by one user and applying it via
1041 PutAcl on a resource owned by another user may not do
1042 exactly what you expect.
1043 S3::Delete ?-bucket bucketname? -resource resourcename ?-blocking bool‐
1045 ean? ?-status statusvar?
1046 This command deletes the specified resource from the specified
1047 bucket. It returns 1 if the resource was deleted successfully,
1048 0 otherwise. It returns 0 rather than throwing an S3 remote
1049 error.
1050 -bucket
1052 This specifies the bucket from which the resource will be
1053 deleted. Leading and/or trailing slashes are removed for
1054 you, as are spaces.
1055 -resource
1057 This is the full name of the resource within the bucket.
1058 A single leading slash is removed, but not a trailing
1059 slash. Spaces are not trimmed.
1060 -blocking
1062 The standard blocking flag.
1063 -status
1065 If specified, the indicated (not necessarily global)
1066 variable in the caller's scope is set to a two-element
1067 list. The first element is the 3-digit HTTP status code.
1068 The second element is the HTTP message (such as "OK" or
1069 "Forbidden"). Note that Amazon's DELETE result is 204 on
1070 success, that being the code indicating no content in the
1071 returned body.
1072 S3::Push ?-bucket bucketname? -directory directoryname ?-prefix pre‐
1075 fixstring? ?-compare comparemode? ?-x-amz-meta-* metastring? ?-acl
1076 aclcode? ?-delete boolean? ?-error throw|break|continue? ?-progress
1077 scriptprefix?
1078 This synchronises a local directory with a remote bucket by
1079 pushing the differences using S3::Put. Note that if something
1080 has changed in the bucket but not locally, those changes could
1081 be lost. Thus, this is not a general two-way synchronization
1082 primitive. (See S3::Sync for that.) Note too that resource names
1083 are case sensitive, so changing the case of a file on a Windows
1084 machine may lead to otherwise-unnecessary transfers. Note that
1085 only regular files are considered, so devices, pipes, symlinks,
1086 and directories are not copied.
1087 -bucket
1089 This names the bucket into which data will be pushed.
1090 -directory
1092 This names the local directory from which files will be
1093 taken. It must exist, be readable via [glob] and so on.
1094 If only some of the files therein are readable, S3::Push
1095 will PUT those files that are readable and return in its
1096 results the list of files that could not be opened.
1097 -prefix
1099 This names the prefix that will be added to all
1100 resources. That is, it is the remote equivalent of
1101 -directory. If it is not specified, the root of the
1102 bucket will be treated as the remote directory. An exam‐
1103 ple may clarify.
1104 S3::Push -bucket test -directory /tmp/xyz -prefix hello/world
1106 In this example, /tmp/xyz/pdq.html will be stored as
1108 http://s3.amazonaws.com/test/hello/world/pdq.html in Ama‐
1109 zon's servers. Also, /tmp/xyz/abc/def/Hello will be
1110 stored as http://s3.amazon‐
1111 aws.com/test/hello/world/abc/def/Hello in Amazon's
1112 servers. Without the -prefix option, /tmp/xyz/pdq.html
1113 would be stored as http://s3.amazonaws.com/test/pdq.html.
1114 -blocking
1116 This is the standard blocking option.
1117 -compare
1119 If present, this is passed to each invocation of S3::Put.
1120 Naturally, S3::Configure -default-compare is used if this
1121 is not specified.
1122 -x-amz-meta-*
1124 If present, this is passed to each invocation of S3::Put.
1125 All copied files will have the same metadata.
1126 -acl If present, this is passed to each invocation of S3::Put.
1128 -delete
1130 This defaults to false. If true, resources in the desti‐
1131 nation that are not in the source directory are deleted
1132 with S3::Delete. Since only regular files are consid‐
1133 ered, the existance of a symlink, pipe, device, or direc‐
1134 tory in the local source will not prevent the deletion of
1135 a remote resource with a corresponding name.
1136 -error This controls the behavior of S3::Push in the event that
1138 S3::Put throws an error. Note that errors encountered on
1139 the local file system or in reading the list of resources
1140 in the remote bucket always throw errors. This option
1141 allows control over "partial" errors, when some files
1142 were copied and some were not. S3::Delete is always fin‐
1143 ished up, with errors simply recorded in the return
1144 result.
1145 throw The error is rethrown with the same errorCode.
1147 break Processing stops without throwing an error, the
1149 error is recorded in the return value, and the
1150 command returns with a normal return. The calls
1151 to S3::Delete are not started.
1152 continue
1154 This is the default. Processing continues without
1155 throwing, recording the error in the return
1156 result, and resuming with the next file in the
1157 local directory to be copied.
1158 -progress
1160 If this is specified and the indicated script prefix is
1161 not empty, the indicated script prefix will be invoked
1162 several times in the caller's context with additional
1163 arguments at various points in the processing. This
1164 allows progress reporting without backgrounding. The
1165 provided prefix will be invoked with additional argu‐
1166 ments, with the first additional argument indicating what
1167 part of the process is being reported on. The prefix is
1168 initially invoked with args as the first additional argu‐
1169 ment and a dictionary representing the normalized argu‐
1170 ments to the S3::Push call as the second additional argu‐
1171 ment. Then the prefix is invoked with local as the first
1172 additional argument and a list of suffixes of the files
1173 to be considered as the second argument. Then the prefix
1174 is invoked with remote as the first additional argument
1175 and a list of suffixes existing in the remote bucket as
1176 the second additional argument. Then, for each file in
1177 the local list, the prefix will be invoked with start as
1178 the first additional argument and the common suffix as
1179 the second additional argument. When S3::Put returns for
1180 that file, the prefix will be invoked with copy as the
1181 first additional argument, the common suffix as the sec‐
1182 ond additional argument, and a third argument that will
1183 be "copied" (if S3::Put sent the resource), "skipped" (if
1184 S3::Put decided not to based on -compare), or the error‐
1185 Code that S3::Put threw due to unexpected errors (in
1186 which case the third argument is a list that starts with
1187 "S3"). When all files have been transfered, the prefix
1188 may be invoked zero or more times with delete as the
1189 first additional argument and the suffix of the resource
1190 being deleted as the second additional argument, with a
1191 third argument being either an empty string (if the
1192 delete worked) or the errorCode from S3::Delete if it
1193 failed. Finally, the prefix will be invoked with finished
1194 as the first additional argument and the return value as
1195 the second additional argument.
1196 The return result from this command is a dictionary. They keys are the
1197 suffixes (i.e., the common portion of the path after the -directory and
1198 -prefix), while the values are either "copied", "skipped" (if -compare
1199 indicated not to copy the file), or the errorCode thrown by S3::Put, as
1200 appropriate. If -delete was true, there may also be entries for suf‐
1201 fixes with the value "deleted" or "notdeleted", indicating whether the
1202 attempted S3::Delete worked or not, respectively. There is one addi‐
1203 tional pair in the return result, whose key is the empty string and
1204 whose value is a nested dictionary. The keys of this nested dictionary
1205 include "filescopied" (the number of files successfully copied),
1206 "bytescopied" (the number of data bytes in the files copied, excluding
1207 headers, metadata, etc), "compareskipped" (the number of files not
1208 copied due to -compare mode), "errorskipped" (the number of files not
1209 copied due to thrown errors), "filesdeleted" (the number of resources
1210 deleted due to not having corresponding files locally, or 0 if -delete
1211 is false), and "filesnotdeleted" (the number of resources whose dele‐
1212 tion was attempted but failed).
1213 Note that this is currently implemented somewhat inefficiently. It
1215 fetches the bucket listing (including timestamps and eTags), then calls
1216 S3::Put, which uses HEAD to find the timestamps and eTags again. Cor‐
1217 recting this with no API change is planned for a future upgrade.
1218 S3::Pull ?-bucket bucketname? -directory directoryname ?-prefix pre‐
1221 fixstring? ?-blocking boolean? ?-compare comparemode? ?-delete boolean?
1222 ?-timestamp aws|now? ?-error throw|break|continue? ?-progress script‐
1223 prefix?
1224 This synchronises a remote bucket with a local directory by
1225 pulling the differences using S3::Get If something has been
1226 changed locally but not in the bucket, those difference may be
1227 lost. This is not a general two-way synchronization mechanism.
1228 (See S3::Sync for that.) This creates directories if needed;
1229 new directories are created with default permissions. Note that
1230 resource names are case sensitive, so changing the case of a
1231 file on a Windows machine may lead to otherwise-unnecessary
1232 transfers. Also, try not to store data in resources that end
1233 with a slash, or which are prefixes of resources that otherwise
1234 would start with a slash; i.e., don't use this if you store data
1235 in resources whose names have to be directories locally.
1236 Note that this is currently implemented somewhat inefficiently.
1238 It fetches the bucket listing (including timestamps and eTags),
1239 then calls S3::Get, which uses HEAD to find the timestamps and
1240 eTags again. Correcting this with no API change is planned for a
1241 future upgrade.
1242 -bucket
1244 This names the bucket from which data will be pulled.
1245 -directory
1247 This names the local directory into which files will be
1248 written It must exist, be readable via [glob], writable
1249 for file creation, and so on. If only some of the files
1250 therein are writable, S3::Pull will GET those files that
1251 are writable and return in its results the list of files
1252 that could not be opened.
1253 -prefix
1255 The prefix of resources that will be considered for
1256 retrieval. See S3::Push for more details, examples, etc.
1257 (Of course, S3::Pull reads rather than writes, but the
1258 prefix is treated similarly.)
1259 -blocking
1261 This is the standard blocking option.
1262 -compare
1264 This is passed to each invocation of S3::Get if provided.
1265 Naturally, S3::Configure -default-compare is used if this
1266 is not provided.
1267 -timestamp
1269 This is passed to each invocation of S3::Get if provided.
1270 -delete
1272 If this is specified and true, files that exist in the
1273 -directory that are not in the -prefix will be deleted
1274 after all resources have been copied. In addition, empty
1275 directories (other than the top-level -directory) will be
1276 deleted, as Amazon S3 has no concept of an empty direc‐
1277 tory.
1278 -error See S3::Push for a description of this option.
1280 -progress
1282 See S3::Push for a description of this option. It dif‐
1283 fers slightly in that local directories may be included
1284 with a trailing slash to indicate they are directories.
1285 The return value from this command is a dictionary. It is identical in
1286 form and meaning to the description of the return result of S3::Push.
1287 It differs only in that directories may be included, with a trailing
1288 slash in their name, if they are empty and get deleted.
1289 S3::Toss ?-bucket bucketname? -prefix prefixstring ?-blocking boolean?
1291 ?-error throw|break|continue? ?-progress scriptprefix?
1292 This deletes some or all resources within a bucket. It would be
1293 considered a "recursive delete" had Amazon implemented actual
1294 directories.
1295 -bucket
1297 The bucket from which resources will be deleted.
1298 -blocking
1300 The standard blocking option.
1301 -prefix
1303 The prefix for resources to be deleted. Any resource that
1304 starts with this string will be deleted. This is
1305 required. To delete everything in the bucket, pass an
1306 empty string for the prefix.
1307 -error If this is "throw", S3::Toss rethrows any errors it
1309 encounters. If this is "break", S3::Toss returns with a
1310 normal return after the first error, recording that error
1311 in the return result. If this is "continue", which is the
1312 default, S3::Toss continues on and lists all errors in
1313 the return result.
1314 -progress
1316 If this is specified and not an empty string, the script
1317 prefix will be invoked several times in the context of
1318 the caller with additional arguments appended. Ini‐
1319 tially, it will be invoked with the first additional
1320 argument being args and the second being the processed
1321 list of arguments to S3::Toss. Then it is invoked with
1322 remote as the first additional argument and the list of
1323 suffixes in the bucket to be deleted as the second addi‐
1324 tional argument. Then it is invoked with the first addi‐
1325 tional argument being delete and the second additional
1326 argument being the suffix deleted and the third addi‐
1327 tional argument being "deleted" or "notdeleted" depending
1328 on whether S3::Delete threw an error. Finally, the
1329 script prefix is invoked with a first additional argument
1330 of "finished" and a second additional argument of the
1331 return value.
1332 The return value is a dictionary. The keys are the suffixes of files
1333 that S3::Toss attempted to delete, and whose values are either the
1334 string "deleted" or "notdeleted". There is also one additional pair,
1335 whose key is the empty string and whose value is an embedded dictio‐
1336 nary. The keys of this embedded dictionary include "filesdeleted" and
1337 "filesnotdeleted", each of which has integer values.
1338
1340 · The pure-Tcl MD5 checking is slow. If you are processing files
1341 in the megabyte range, consider ensuring binary support is
1342 available.
1343 · The commands S3::Pull and S3::Push fetch a directory listing
1345 which includes timestamps and MD5 hashes, then invoke S3::Get
1346 and S3::Put. If a complex -compare mode is specified, S3::Get
1347 and S3::Put will invoke a HEAD operation for each file to fetch
1348 timestamps and MD5 hashes of each resource again. It is expected
1349 that a future release of this package will solve this without
1350 any API changes.
1351 · The commands S3::Pull and S3::Push fetch a directory listing
1353 without using -max-count. The entire directory is pulled into
1354 memory at once. For very large buckets, this could be a perfor‐
1355 mance problem. The author, at this time, does not plan to change
1356 this behavior. Welcome to Open Source.
1357 · S3::Sync is neither designed nor implemented yet. The intention
1359 would be to keep changes synchronised, so changes could be made
1360 to both the bucket and the local directory and be merged by
1361 S3::Sync.
1362 · Nor is -compare calc fully implemented. This is primarily due to
1364 Windows not providing a convenient method for distinguishing
1365 between local files that are "public-read" or "public-read-
1366 write". Assistance figuring out TWAPI for this would be appreci‐
1367 ated. The U**X semantics are difficult to map directly as well.
1368 See the source for details. Note that there are not tests for
1369 calc, since it isn't done yet.
1370 · The HTTP processing is implemented within the library, rather
1372 than using a "real" HTTP package. Hence, multi-line headers are
1373 not (yet) handled correctly. Do not include carriage returns or
1374 linefeeds in x-amz-meta-* headers, content-type values, and so
1375 on. The author does not at this time expect to improve this.
1376 · Internally, S3::Push and S3::Pull and S3::Toss are all very sim‐
1378 ilar and should be refactored.
1379 · The idea of using -compare never -delete true to delete files
1381 that have been deleted from one place but not the other yet not
1382 copying changed files is untested.
1383
1385 To fetch a "directory" out of a bucket, make changes, and store it
1386 back:
1387 file mkdir ./tempfiles
1389 S3::Pull -bucket sample -prefix of/interest -directory ./tempfiles \
1390 -timestamp aws
1391 do_my_process ./tempfiles other arguments
1392 S3::Push -bucket sample -prefix of/interest -directory ./tempfiles \
1393 -compare newer -delete true
1394 To delete files locally that were deleted off of S3 but not otherwise
1397 update files:
1398 S3::Pull -bucket sample -prefix of/interest -directory ./myfiles \
1400 -compare never -delete true
1401
1404 The author intends to work on several additional projects related to
1405 this package, in addition to finishing the unfinished features.
1406 First, a command-line program allowing browsing of buckets and transfer
1408 of files from shell scripts and command prompts is useful.
1409 Second, a GUI-based program allowing visual manipulation of bucket and
1411 resource trees not unlike Windows Explorer would be useful.
1412 Third, a command-line (and perhaps a GUI-based) program called "OddJob"
1414 that will use S3 to synchronize computation amongst multiple servers
1415 running OddJob. An S3 bucket will be set up with a number of scripts to
1416 run, and the OddJob program can be invoked on multiple machines to run
1417 scripts on all the machines, each moving on to the next unstarted task
1418 as it finishes each. This is still being designed, and it is intended
1419 primarily to be run on Amazon's Elastic Compute Cloud.
1420
1422 This document, and the package it describes, will undoubtedly contain
1423 bugs and other problems. Please report such in the category amazon-s3
1424 of the Tcllib SF Trackers [http://source‐
1425 forge.net/tracker/?group_id=12883]. Please also report any ideas for
1426 enhancements you may have for either package and/or documentation.
1427
1429 Copyright (c) Copyright 2006,2008 Darren New. All Rights Reserved. See LICENSE.TXT for terms.
1430amazon-s3 1.0.0 S3(n)