1s3fs(1) General Commands Manual s3fs(1)
2
6 s3fs - The S3 FUSE filesystem disk management utility
7
9 s3fs [<-C> [-h] | [-cdrf <bucket>] [-p <access_key>] [-s
10 secret_access_key] ] | [ -o <options> <mountpoint>]
11
13 s3fs is a FUSE filesystem application backed by amazon web services
14 simple storage service (s3, http://aws.amazon.com). s3fs can operate in
15 a command mode or a mount mode.
16 In mount mode, s3fs will mount an amazon s3 bucket (that has been prop‐
18 erly formatted) as a local file system.
19 In command mode, s3fs is capable of manipulating amazon s3 buckets in
21 various usefull ways
22
24 Options are used in command mode. To enter command mode, you must
25 specify -C as the first command line option. Note these options are
26 only available in command mode
27 -C Enter command mode. This must be the first option on the com‐
30 mand line when using s3fs in command mode
31 -h Display usage information on command mode
34 -c <bucket>
36 Create the named s3 bucket
37 -d <bucket>
39 Delete the named s3 bucket (and any data contained there, use
40 with caution!)
41 -r <bucket>
43 Interactively repair a broken s3 filesystem (not yet imple‐
44 mented)
45 -f <bucket>
47 Format an s3 bucket to make it suitable for mounting
48 -k <bucket>
50 Toggle the lock bit on a bucket. buckets must be unlocked to
51 format or delete them
52 -p <access_key>
54 Provide the AWS access key if not set in your environment
55 -s <secret_access_key>
57 Provide the AWS secret access key if not set in your environment
58 -o <options>
60 Specify mount options when s3fs is operating in mount mode
61
63 Note these options are only available when operating s3fs in mount mode
64 bucket=<bucket>
66 Specify the name of the Amazon S3 bucket you wish to mount
67 (default: none)
68 preserve_cache=[no|yes]
70 If set to yes, the cache for the given mount will not be deleted
71 on unmount, saving the need for downloads on a subsequent
72 remount (default: no)
73 cachedir=<directory>
75 Sets the base directory where cached s3fs files will be stored.
76 (default=$HOME/.fuse-s3fs-cache/)
77 host=<host name>
79 Allows overriding of default amazon s3 hostname. defaults to
80 s3.aws.amazon.com
81 lazy_fsdata=[no|yes]
83 If set to yes, filesystem metadata will only be written back to
84 S3 on unmount (default: yes)
85 writeback_time=<seconds>
87 Number of seconds of hysteresis to pause when considering
88 uploading a file to S3. If, after the pause, s3fs detects that
89 other processes have opened a file, uploading is postponed
90 (default=10)
91 AWS_ACCESS_KEY_ID=<key>
93 Specify the AWS_ACCESS_KEY_ID for your aws account if you don't
94 wish to place it in your environment (default: none)
95 AWS_SECRET_ACCESS_KEY=<key>
97 Specify your AWS_SECRET_ACCESS_KEY for your aws account if you
98 don't wish to place it in your environment. NOTE: If you add a
99 mount to /etc/fstab, it is discouraged for you to use this
100 option. Doing so will publish your secret key and allow others
101 to use your account. (default: none)
102
104 Note that to unmount FUSE filesystems the fusermount utility should be
105 used.
106
108 AWS_ACCESS_KEY_ID
109 This is your Amazon web service public key. It must be set so
110 that s3fs can identify you to amazon
111 AWS_SECRET_ACCESS_KEY
113 This is your Amazon web service private key. It must be set so
114 that s3fs can identify you to amazon.
115 Both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be set and
117 exported by the
118 user preforming a mount with s3fs in order to successfully use
119 s3fs
120
122 Create an S3 Bucket
123 s3fs -C -c <bucket_name>
124 Format an S3 Bucket
126 s3fs -C -f <bucket_name>
127 Mount an S3 Bucket
129 s3fs -o [other mount options],bucket=<bucket_name> <mount point>
130
132 /etc/fstab
133 The format for an s3fs entry in /etc/fstab should look like this
135 fstab <mnt‐
137 point> fuse [options],suid,dev,exec,noauto,users,bucket=<s3_bucket>
138 0 0
139 Note that the <mntpoint> must be readable by the mounting user
141
144 Multi User capability
145 While it is possible to share s3 buckets among multiple users,
146 the current data consistency model for Amazons S3 service pre‐
147 vent the safe usage of multiple mounts from multiple users.
148 While s3fs will currently allow multiple mounts, data corruption
149 may result from such activity. A future release will contain a
150 locking mechanism to safely guard against multiple read-write
151 mounts. multiple read-only mounts following a single read-write
152 mount is safe, but will not reflect changes made by the write‐
153 able mount to any of the file or filesystem metadata, limiting
154 its usefulness there
155 File System Layout & Limitations
158 The s3fs filesystem is designed to be very, very simple. S3
159 maintains a flat storage system, meaning no directory heirarchy
160 is allowed on the backing store. Directory information is
161 stored in the filesystem metadata that is retrieved during the
162 mount operation from a file called fsdata. This file is a
163 python class that has been pickled, and maintains all the direc‐
164 tory heirarchy. Files are stored against their fully qualified
165 path names within the file system, which makes for easy file
166 retrieval via any web based interface to S3, should the metadata
167 become corrupted. Note that the metadata (stored on S3 in a
168 file called fsdata), has a layout that is defined by the pyhon
169 class definition in the s3fs executable. To understand or view
170 the layout of this data, one must read the python code. Specif‐
171 ically the S3DriveMetaData class should be investigated, as that
172 is the base class that is pickled. Given that files are stored
173 as individual objects on S3, coupled with the fact that there is
174 a 5GB limit to each object in S3, s3fs currently has a natural
175 limit of 5GB per file
176 Compatibility with other S3 Access mechanisms
179 Amazon S3 is simply a storage back end. s3fs is simply a stor‐
180 age api that exports that storage in the form of a local file
181 system. Note there are many other mechanisms for using S3, some
182 filesystem oriented, like s3fs, as well as other methods. It is
183 important to note, that this application creates a unique
184 filesystem structure on the targeted S3 storage bucket. As such
185 it is incompatible with other S3 fuse implementations. One
186 should not use alternate s3fs clients (of which several are
187 available) to mount buckets created with this utility. This is
188 true for all S3 filesystem implementations, as each has a unique
189 filesystem structure. That being said, the design of this par‐
190 ticular S3 filesystem layout does make it compatible with web
191 based S3 access mechanism. Using a web browser, it is possible
192 to retrieve individual files from a bucket that has been format‐
193 ted for use as an s3fs file system.
194Neil Horman Feb 2008 s3fs(1)