1uri_string(3) Erlang Module Definition uri_string(3)
2
6 uri_string - URI processing functions.
7
9 This module contains functions for parsing and handling URIs (RFC 3986)
10 and form-urlencoded query strings (HTML 5.2).
11 Parsing and serializing non-UTF-8 form-urlencoded query strings are
13 also supported (HTML 5.0).
14 A URI is an identifier consisting of a sequence of characters matching
16 the syntax rule named URI in RFC 3986.
17 The generic URI syntax consists of a hierarchical sequence of compo‐
19 nents referred to as the scheme, authority, path, query, and fragment:
20 URI = scheme ":" hier-part [ "?" query ] [ "#" fragment ]
22 hier-part = "//" authority path-abempty
23 / path-absolute
24 / path-rootless
25 / path-empty
26 scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )
27 authority = [ userinfo "@" ] host [ ":" port ]
28 userinfo = *( unreserved / pct-encoded / sub-delims / ":" )
29 reserved = gen-delims / sub-delims
31 gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"
32 sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
33 / "*" / "+" / "," / ";" / "="
34 unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"
36 The interpretation of a URI depends only on the characters used and not
40 on how those characters are represented in a network protocol.
41 The functions implemented by this module cover the following use cases:
43 * Parsing URIs into its components and returing a map
45 parse/1
46 * Recomposing a map of URI components into a URI string
48 recompose/1
49 * Changing inbound binary and percent-encoding of URIs
51 transcode/2
52 * Transforming URIs into a normalized form
54 normalize/1
55 normalize/2
56 * Composing form-urlencoded query strings from a list of key-value
58 pairs
59 compose_query/1
60 compose_query/2
61 * Dissecting form-urlencoded query strings into a list of key-value
63 pairs
64 dissect_query/1
65 * Decoding percent-encoded triplets
67 percent_decode/1
68 There are four different encodings present during the handling of URIs:
70 * Inbound binary encoding in binaries
72 * Inbound percent-encoding in lists and binaries
74 * Outbound binary encoding in binaries
76 * Outbound percent-encoding in lists and binaries
78 Functions with uri_string() argument accept lists, binaries and mixed
80 lists (lists with binary elements) as input type. All of the functions
81 but transcode/2 expects input as lists of unicode codepoints, UTF-8
82 encoded binaries and UTF-8 percent-encoded URI parts ("%C3%B6" corre‐
83 sponds to the unicode character "ö").
84 Unless otherwise specified the return value type and encoding are the
86 same as the input type and encoding. That is, binary input returns
87 binary output, list input returns a list output but mixed input returns
88 list output.
89 In case of lists there is only percent-encoding. In binaries, however,
91 both binary encoding and percent-encoding shall be considered.
92 transcode/2 provides the means to convert between the supported encod‐
93 ings, it takes a uri_string() and a list of options specifying inbound
94 and outbound encodings.
95 RFC 3986 does not mandate any specific character encoding and it is
97 usually defined by the protocol or surrounding text. This library takes
98 the same assumption, binary and percent-encoding are handled as one
99 configuration unit, they cannot be set to different values.
100
102 error() = {error, atom(), term()}
103 Error tuple indicating the type of error. Possible values of the
105 second component:
106 * invalid_character
108 * invalid_encoding
110 * invalid_input
112 * invalid_map
114 * invalid_percent_encoding
116 * invalid_scheme
118 * invalid_uri
120 * invalid_utf8
122 * missing_value
124 The third component is a term providing additional information
126 about the cause of the error.
127 uri_map() =
129 #{fragment => unicode:chardata(),
130 host => unicode:chardata(),
131 path => unicode:chardata(),
132 port => integer() >= 0 | undefined,
133 query => unicode:chardata(),
134 scheme => unicode:chardata(),
135 userinfo => unicode:chardata()}
136 Map holding the main components of a URI.
138 uri_string() = iodata()
140 List of unicode codepoints, a UTF-8 encoded binary, or a mix of
142 the two, representing an RFC 3986 compliant URI (percent-encoded
143 form). A URI is a sequence of characters from a very limited
144 set: the letters of the basic Latin alphabet, digits, and a few
145 special characters.
146
148 allowed_characters() -> [{atom(), list()}]
149 This is a utility function meant to be used in the shell for
151 printing the allowed characters in each major URI component, and
152 also in the most important characters sets. Please note that
153 this function does not replace the ABNF rules defined by the
154 standards, these character sets are derived directly from those
155 aformentioned rules. For more information see the Uniform
156 Resource Identifiers chapter in stdlib's Users Guide.
157 compose_query(QueryList) -> QueryString
159 Types:
161 QueryList = [{unicode:chardata(), unicode:chardata() | true}]
163 QueryString = uri_string() | error()
164 Composes a form-urlencoded QueryString based on a QueryList, a
166 list of non-percent-encoded key-value pairs. Form-urlencoding is
167 defined in section 4.10.21.6 of the HTML 5.2 specification and
168 in section 4.10.22.6 of the HTML 5.0 specification for non-UTF-8
169 encodings.
170 See also the opposite operation dissect_query/1.
172 Example:
174 1> uri_string:compose_query([{"foo bar","1"},{"city","örebro"}]).
176 "foo+bar=1&city=%C3%B6rebro"
177 2> uri_string:compose_query([{<<"foo bar">>,<<"1">>},
178 2> {<<"city">>,<<"örebro"/utf8>>}]).
179 <<"foo+bar=1&city=%C3%B6rebro">>
180 compose_query(QueryList, Options) -> QueryString
183 Types:
185 QueryList = [{unicode:chardata(), unicode:chardata() | true}]
187 Options = [{encoding, atom()}]
188 QueryString = uri_string() | error()
189 Same as compose_query/1 but with an additional Options parame‐
191 ter, that controls the encoding ("charset") used by the encoding
192 algorithm. There are two supported encodings: utf8 (or unicode)
193 and latin1.
194 Each character in the entry's name and value that cannot be
196 expressed using the selected character encoding, is replaced by
197 a string consisting of a U+0026 AMPERSAND character (&), a "#"
198 (U+0023) character, one or more ASCII digits representing the
199 Unicode code point of the character in base ten, and finally a
200 ";" (U+003B) character.
201 Bytes that are out of the range 0x2A, 0x2D, 0x2E, 0x30 to 0x39,
203 0x41 to 0x5A, 0x5F, 0x61 to 0x7A, are percent-encoded (U+0025
204 PERCENT SIGN character (%) followed by uppercase ASCII hex dig‐
205 its representing the hexadecimal value of the byte).
206 See also the opposite operation dissect_query/1.
208 Example:
210 1> uri_string:compose_query([{"foo bar","1"},{"city","örebro"}],
212 1> [{encoding, latin1}]).
213 "foo+bar=1&city=%F6rebro"
214 2> uri_string:compose_query([{<<"foo bar">>,<<"1">>},
215 2> {<<"city">>,<<"東京"/utf8>>}], [{encoding, latin1}]).
216 <<"foo+bar=1&city=%26%2326481%3B%26%2320140%3B">>
217 dissect_query(QueryString) -> QueryList
220 Types:
222 QueryString = uri_string()
224 QueryList =
225 [{unicode:chardata(), unicode:chardata() | true}] |
226 error()
227 Dissects an urlencoded QueryString and returns a QueryList, a
229 list of non-percent-encoded key-value pairs. Form-urlencoding is
230 defined in section 4.10.21.6 of the HTML 5.2 specification and
231 in section 4.10.22.6 of the HTML 5.0 specification for non-UTF-8
232 encodings.
233 See also the opposite operation compose_query/1.
235 Example:
237 1> uri_string:dissect_query("foo+bar=1&city=%C3%B6rebro").
239 [{"foo bar","1"},{"city","örebro"}]
240 2> uri_string:dissect_query(<<"foo+bar=1&city=%26%2326481%3B%26%2320140%3B">>).
241 [{<<"foo bar">>,<<"1">>},
242 {<<"city">>,<<230,157,177,228,186,172>>}]
243 normalize(URI) -> NormalizedURI
246 Types:
248 URI = uri_string() | uri_map()
250 NormalizedURI = uri_string() | error()
251 Transforms an URI into a normalized form using Syntax-Based Nor‐
253 malization as defined by RFC 3986.
254 This function implements case normalization, percent-encoding
256 normalization, path segment normalization and scheme based nor‐
257 malization for HTTP(S) with basic support for FTP, SSH, SFTP and
258 TFTP.
259 Example:
261 1> uri_string:normalize("/a/b/c/./../../g").
263 "/a/g"
264 2> uri_string:normalize(<<"mid/content=5/../6">>).
265 <<"mid/6">>
266 3> uri_string:normalize("http://localhost:80").
267 "https://localhost/"
268 4> uri_string:normalize(#{scheme => "http",port => 80,path => "/a/b/c/./../../g",
269 4> host => "localhost-örebro"}).
270 "http://localhost-%C3%B6rebro/a/g"
271 normalize(URI, Options) -> NormalizedURI
274 Types:
276 URI = uri_string() | uri_map()
278 Options = [return_map]
279 NormalizedURI = uri_string() | uri_map() | error()
280 Same as normalize/1 but with an additional Options parameter,
282 that controls whether the normalized URI shall be returned as an
283 uri_map(). There is one supported option: return_map.
284 Example:
286 1> uri_string:normalize("/a/b/c/./../../g", [return_map]).
288 #{path => "/a/g"}
289 2> uri_string:normalize(<<"mid/content=5/../6">>, [return_map]).
290 #{path => <<"mid/6">>}
291 3> uri_string:normalize("http://localhost:80", [return_map]).
292 #{scheme => "http",path => "/",host => "localhost"}
293 4> uri_string:normalize(#{scheme => "http",port => 80,path => "/a/b/c/./../../g",
294 4> host => "localhost-örebro"}, [return_map]).
295 #{scheme => "http",path => "/a/g",host => "localhost-örebro"}
296 parse(URIString) -> URIMap
299 Types:
301 URIString = uri_string()
303 URIMap = uri_map() | error()
304 Parses an RFC 3986 compliant uri_string() into a uri_map(), that
306 holds the parsed components of the URI. If parsing fails, an
307 error tuple is returned.
308 See also the opposite operation recompose/1.
310 Example:
312 1> uri_string:parse("foo://user@example.com:8042/over/there?name=ferret#nose").
314 #{fragment => "nose",host => "example.com",
315 path => "/over/there",port => 8042,query => "name=ferret",
316 scheme => foo,userinfo => "user"}
317 2> uri_string:parse(<<"foo://user@example.com:8042/over/there?name=ferret">>).
318 #{host => <<"example.com">>,path => <<"/over/there">>,
319 port => 8042,query => <<"name=ferret">>,scheme => <<"foo">>,
320 userinfo => <<"user">>}
321 percent_decode(URI) -> Result
324 Types:
326 URI = uri_string() | uri_map()
328 Result =
329 uri_string() |
330 uri_map() |
331 {error, {invalid, {atom(), {term(), term()}}}}
332 Decodes all percent-encoded triplets in the input that can be
334 both a uri_string() and a uri_map(). Note, that this function
335 performs raw decoding and it shall be used on already parsed URI
336 components. Applying this function directly on a standard URI
337 can effectively change it.
338 If the input encoding is not UTF-8, an error tuple is returned.
340 Example:
342 1> uri_string:percent_decode(#{host => "localhost-%C3%B6rebro",path => [],
344 1> scheme => "http"}).
345 #{host => "localhost-örebro",path => [],scheme => "http"}
346 2> uri_string:percent_decode(<<"%C3%B6rebro">>).
347 <<"örebro"/utf8>>
348 Warning:
351 Using uri_string:percent_decode/1 directly on a URI is not safe.
352 This example shows, that after each consecutive application of
353 the function the resulting URI will be changed. None of these
354 URIs refer to the same resource.
355 3> uri_string:percent_decode(<<"http://local%252Fhost/path">>).
357 <<"http://local%2Fhost/path">>
358 4> uri_string:percent_decode(<<"http://local%2Fhost/path">>).
359 <<"http://local/host/path">>
360 recompose(URIMap) -> URIString
364 Types:
366 URIMap = uri_map()
368 URIString = uri_string() | error()
369 Creates an RFC 3986 compliant URIString (percent-encoded), based
371 on the components of URIMap. If the URIMap is invalid, an error
372 tuple is returned.
373 See also the opposite operation parse/1.
375 Example:
377 1> URIMap = #{fragment => "nose", host => "example.com", path => "/over/there",
379 1> port => 8042, query => "name=ferret", scheme => "foo", userinfo => "user"}.
380 #{fragment => "nose",host => "example.com",
381 path => "/over/there",port => 8042,query => "name=ferret",
382 scheme => "foo",userinfo => "user"}
383 2> uri_string:recompose(URIMap).
385 "foo://example.com:8042/over/there?name=ferret#nose"
386 resolve(RefURI, BaseURI) -> TargetURI
388 Types:
390 RefURI = BaseURI = uri_string() | uri_map()
392 TargetURI = uri_string() | error()
393 Convert a RefURI reference that might be relative to a given
395 base URI into the parsed components of the reference's target,
396 which can then be recomposed to form the target URI.
397 Example:
399 1> uri_string:resolve("/abs/ol/ute", "http://localhost/a/b/c?q").
401 "http://localhost/abs/ol/ute"
402 2> uri_string:resolve("../relative", "http://localhost/a/b/c?q").
403 "http://localhost/a/relative"
404 3> uri_string:resolve("http://localhost/full", "http://localhost/a/b/c?q").
405 "http://localhost/full"
406 4> uri_string:resolve(#{path => "path", query => "xyz"}, "http://localhost/a/b/c?q").
407 "http://localhost/a/b/path?xyz"
408 resolve(RefURI, BaseURI, Options) -> TargetURI
411 Types:
413 RefURI = BaseURI = uri_string() | uri_map()
415 Options = [return_map]
416 TargetURI = uri_string() | uri_map() | error()
417 Same as resolve/2 but with an additional Options parameter, that
419 controls whether the target URI shall be returned as an
420 uri_map(). There is one supported option: return_map.
421 Example:
423 1> uri_string:resolve("/abs/ol/ute", "http://localhost/a/b/c?q", [return_map]).
425 #{host => "localhost",path => "/abs/ol/ute",scheme => "http"}
426 2> uri_string:resolve(#{path => "/abs/ol/ute"}, #{scheme => "http",
427 2> host => "localhost", path => "/a/b/c?q"}, [return_map]).
428 #{host => "localhost",path => "/abs/ol/ute",scheme => "http"}
429 transcode(URIString, Options) -> Result
432 Types:
434 URIString = uri_string()
436 Options =
437 [{in_encoding, unicode:encoding()} |
438 {out_encoding, unicode:encoding()}]
439 Result = uri_string() | error()
440 Transcodes an RFC 3986 compliant URIString, where Options is a
442 list of tagged tuples, specifying the inbound (in_encoding) and
443 outbound (out_encoding) encodings. in_encoding and out_encoding
444 specifies both binary encoding and percent-encoding for the
445 input and output data. Mixed encoding, where binary encoding is
446 not the same as percent-encoding, is not supported. If an argu‐
447 ment is invalid, an error tuple is returned.
448 Example:
450 1> uri_string:transcode(<<"foo%00%00%00%F6bar"/utf32>>,
452 1> [{in_encoding, utf32},{out_encoding, utf8}]).
453 <<"foo%C3%B6bar"/utf8>>
454 2> uri_string:transcode("foo%F6bar", [{in_encoding, latin1},
455 2> {out_encoding, utf8}]).
456 "foo%C3%B6bar"
457Ericsson AB stdlib 3.14.1 uri_string(3)