htslib-s3-plugin(7)

1htslib-s3-plugin(7)          Bioinformatics tools          htslib-s3-plugin(7)
2
3
4

NAME

6       s3 plugin - htslib AWS S3 plugin
7

DESCRIPTION

9       The  S3 plugin allows htslib file functions to communicate with servers
10       that use the AWS S3 protocol.  Files are identified by their bucket and
11       object key in a URL format e.g.
12
13       s3://mybucket/path/to/file
14
15       With path/to/file being the object key.
16
17       Necessary  security  information can be provided in as part of the URL,
18       in environment variables or from configuration files.
19
20       The full URL format is:
21
22       s3[+SCHEME]://[ID[:SECRET[:TOKEN]]@]BUCKET/PATH
23
24       The elements are:
25
26       SCHEME The protocol used.  Defaults to https.
27
28       ID     The user AWS access key.
29
30       SECRET The secret key for use with the access key.
31
32       TOKEN  Token used for temporary security credentials.
33
34       BUCKET AWS S3 bucket.
35
36       PATH   Path to the object under the bucket.
37
38       The environment variables below will be used if the user ID is not set.
39
40       AWS_ACCESS_KEY_ID
41              The user AWS access key.
42
43       AWS_SECRET_ACCESS_KEY
44              The secret key for use with the access key.
45
46       AWS_DEFAULT_REGION
47              The region to use. Defaults to us-east-1.
48
49       AWS_SESSION_TOKEN
50              Token used for temporary security credentials.
51
52       AWS_DEFAULT_PROFILE
53              The profile to use in credentials, config or s3cfg  files.   De‐
54              faults to default.
55
56       AWS_PROFILE
57              Same as above.
58
59       AWS_SHARED_CREDENTIALS_FILE
60              Location  of  the  credentials file.  Defaults to ~/.aws/creden‐
61              tials.
62
63       HTS_S3_S3CFG
64              Location of the s3cfg file.  Defaults to ~/.s3cfg.
65
66       HTS_S3_HOST
67              Sets the host.  Defaults to s3.amazonaws.com.
68
69       HTS_S3_V2
70              If set use signature v2 rather the default v4.  This will  limit
71              the plugin to reading only.
72
73       HTS_S3_PART_SIZE
74              Sets  the upload part size in Mb, the minimum being 5Mb.  By de‐
75              fault the part size starts at 5Mb and expands at regular  inter‐
76              vals to accommodate bigger files (up to 2.5 Tbytes with the cur‐
77              rent rate).  Using this setting disables the automatic part size
78              expansion.
79
80       HTS_S3_ADDRESS_STYLE
81              Sets  the  URL  style.   Options  are auto (default), virtual or
82              path.
83
84       In the absence of an ID from  the  previous  two  methods  the  creden‐
85       tial/config  files will be used.  The default file locations are either
86       ~/.aws/credentials or ~/.s3cfg (in that order).
87
88       Entries used in  aws  style  credentials  file  are  aws_access_key_id,
89       aws_secret_access_key,  aws_session_token, region and addressing_style.
90       Only the first two are usually needed.
91
92       Entries used in s3cmd style config files  are  access_key,  secret_key,
93       access_token,  host_base,  bucket_location  and host_bucket. Again only
94       the first two are usually needed. The host_bucket option is  only  used
95       to set a path-style URL, see below.
96
97

NOTES

99       In most cases this plugin transforms the given URL into a virtual host-
100       style format e.g. https://bucket.host/path/to/file.  A path-style  for‐
101       mat  is used where the URL is not DNS compliant or the bucket name con‐
102       tains a dot e.g.  https://host/bu.cket/path/to/file.
103
104       Path-style can be forced by setting  one  either  HTS_S3_ADDRESS_STYLE,
105       addressing_style  or  host_bucket.   The  first  two can be set to path
106       while host_bucket must not include the %(bucket).s string.
107
108

NAME

DESCRIPTION

NOTES

SEE ALSO