nbdkit-S3-plugin(1)

1nbdkit-S3-plugin(1)                 NBDKIT                 nbdkit-S3-plugin(1)
2
3
4

NAME

6       nbdkit-S3-plugin - expose data in Amazon S3 or Ceph buckets as block
7       device
8

SYNOPSIS

10        nbdkit S3 [access-key=...] [secret-key=...] [session-token=...]
11                  [endpoint-url=...]
12                  [size=NN object-size=NN]
13                  bucket=BUCKET key=STRING
14

DESCRIPTION

16       "nbdkit-S3-plugin" is a plugin for nbdkit(1) which lets you open
17       objects stored in Amazon S3 or Ceph as disk images.
18
19       This plugin uses the Python Amazon Web Services SDK called Boto3.
20

EXAMPLES

22        nbdkit S3 endpoint-url=https://ceph.example.com \
23                  bucket=MY-BUCKET key=disk.img
24
25       Provides read only block device holding the data contained in the
26       "disk.img" object.
27
28        nbdkit S3 endpoint-url=https://ceph.example.com \
29                  size=50G object-size=128k \
30                  bucket=MY-BUCKET key=disk
31
32       Provides a read-write block device with size 50G, whose contents are
33       stored multiple in objects of size 128k, prefixed with disk/
34

PARAMETERS

36       access-key=ACCESS_KEY
37       secret-key=SECRET_KEY
38       session-token=SESSION_TOKEN
39           The AWS credentials may be specified on the command line using
40           these parameters, but it is more secure to pass them using files
41           (see "CREDENTIALS" below).
42
43       endpoint-url=ENDPOINT
44           If accessing Ceph or another compatible S3 service, provide the
45           endpoint URL through this parameter.
46
47       bucket=BUCKET
48           The bucket containing the object(s).  This parameter is required.
49
50       key=STRING
51           The object name (if "size" is not specified) or object prefix (if
52           "size" is specified) to use within the bucket.  This parameter is
53           required.
54
55       size=SIZE
56       object-size=SIZE
57           These two parameters must always be specified together. If set,
58           data will be split into blocks of "object-size" and stored as
59           separate objects. The block device will report a total size of
60           "size" and be writeable and trim-able.
61
62           Object names will have the form key/%16x, where %16x is the
63           16-digit hexadecimal block number. If there are existing objects
64           under such name that do not have the expected size, the plugin will
65           crash.
66

PERFORMANCE CONSIDERATIONS

68       It is highly recommended that clients do their utmost to issue requests
69       that exactly match the object size: Smaller write requests will incur a
70       performance penalty due to the need for read-modify-write cycles (thus
71       also incurring latency from two network round-trips). Larger read and
72       write requests will incur a performance penalty because of sequential
73       execution.
74
75       The nbdkit-blocksize-filter(1) can be used to alleviate the impact of
76       requests larger than the object size, but does not help if the client
77       issues requests smaller than the block size.
78
79       The nbdkit-stats-filter(1) can be used to investigate what block sizes
80       and alignments are used by the client.
81
82       When connecting through the Linux kernel's NBD module, consider setting
83       "/sys/block/nbd<X>/queue/max_sectors_kb" to match the object size.
84

CREDENTIALS

86       Although AWS credentials can be passed to nbdkit on the command line,
87       this is not secure since a user on the same machine could read them
88       using ps(1).
89
90       A better way is to pass the credentials through the ~/.aws/credentials
91       file.  This takes the form:
92
93        [default]
94        aws_access_key_id = XXX
95        aws_secret_access_key = YYY
96
97        [profile]
98        aws_access_key_id = XXX
99        aws_secret_access_key = YYY
100
101       Different profiles from the file can be selected by setting the
102       "AWS_PROFILE" environment variable.
103
104       There is much more information about credentials in the Boto3
105       documentation.
106

COMPARISON TO S3FS-FUSE AND S3BACKER

108       s3fs-fuse (https://github.com/s3fs-fuse/s3fs-fuse) and s3backer
109       (https://github.com/archiecobbs/s3backer) provide similar functionality
110       but are based on FUSE (rather than NBD). They provide a regular file
111       (backed by S3) which can then be loopback-mounted to provide a block
112       device. s3backer also supports to optionally encrypt and compress
113       objects.
114
115       In theory, NBD should provide better performance than FUSE, because:
116
117       •   The kernel no longer serializes write and read requests but issues
118           them concurrently.
119
120       •   Read and write request size can exceed 128 kB
121
122       •   The system can still be reliably hibernated (a running FUSE daemon
123           may prevent this)
124
125       •   Requests pass through the VFS only once, not twice
126
127       •   Data is present in the page cache only once, not twice
128
129       However, for high-bandwith network connections s3backer and s3fs-fuse
130       may be faster because they are written in C rather than Python.
131

FILES

133       $plugindir/nbdkit-S3-plugin
134           The plugin.
135
136           Use "nbdkit --dump-config" to find the location of $plugindir.
137
138       $HOME/.aws/credentials
139           AWS credentials can be passed to boto3 using this file.
140

ENVIRONMENT VARIABLES

142       "AWS_*"
143           Boto3 reads some credential information from "AWS_*" environment
144           variables.
145

VERSION

147       "nbdkit-S3-plugin" first appeared in nbdkit 1.24.
148

AUTHORS

156       Richard W.M. Jones
157
158       Nikolaus Rath
159

COPYRIGHT

161       Copyright Red Hat
162

LICENSE

164       Redistribution and use in source and binary forms, with or without
165       modification, are permitted provided that the following conditions are
166       met:
167
168       •   Redistributions of source code must retain the above copyright
169           notice, this list of conditions and the following disclaimer.
170
171       •   Redistributions in binary form must reproduce the above copyright
172           notice, this list of conditions and the following disclaimer in the
173           documentation and/or other materials provided with the
174           distribution.
175
176       •   Neither the name of Red Hat nor the names of its contributors may
177           be used to endorse or promote products derived from this software
178           without specific prior written permission.
179
180       THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND ANY
181       EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
182       IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
183       PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR CONTRIBUTORS BE
184       LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
185       CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
186       SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
187       BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
188       WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
189       OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
190       ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
191
192
193
194nbdkit-1.34.4                     2023-09-26               nbdkit-S3-plugin(1)