1http(n) Tcl Bundled Packages http(n)
2______________________________________________________________________________
6
8 http - Client-side implementation of the HTTP/1.1 protocol
9
11 package require http ?2.9?
12 ::http::config ?-option value ...?
14 ::http::geturl url ?-option value ...?
16 ::http::formatQuery key value ?key value ...?
18 ::http::quoteString value
20 ::http::reset token ?why?
22 ::http::wait token
24 ::http::status token
26 ::http::size token
28 ::http::code token
30 ::http::ncode token
32 ::http::meta token
34 ::http::data token
36 ::http::error token
38 ::http::cleanup token
40 ::http::register proto port command
42 ::http::registerError port ?message?
44 ::http::unregister proto
46______________________________________________________________________________
47
49 The http package provides the client side of the HTTP/1.1 protocol, as
50 defined in RFC 7230 to RFC 7235, which supersede RFC 2616. The package
51 implements the GET, POST, and HEAD operations of HTTP/1.1. It allows
52 configuration of a proxy host to get through firewalls. The package is
53 compatible with the Safesock security policy, so it can be used by un‐
54 trusted applets to do URL fetching from a restricted set of hosts. This
55 package can be extended to support additional HTTP transport protocols,
56 such as HTTPS, by providing a custom socket command, via ::http::regis‐
57 ter.
58 The ::http::geturl procedure does a HTTP transaction. Its options de‐
60 termine whether a GET, POST, or HEAD transaction is performed. The re‐
61 turn value of ::http::geturl is a token for the transaction. The value
62 is also the name of an array in the ::http namespace that contains
63 state information about the transaction. The elements of this array
64 are described in the STATE ARRAY section.
65 If the -command option is specified, then the HTTP operation is done in
67 the background. ::http::geturl returns immediately after generating
68 the HTTP request and the callback is invoked when the transaction com‐
69 pletes. For this to work, the Tcl event loop must be active. In Tk
70 applications this is always true. For pure-Tcl applications, the
71 caller can use ::http::wait after calling ::http::geturl to start the
72 event loop.
73 Note: The event queue is even used without the -command option. As a
75 side effect, arbitrary commands may be processed while http::geturl is
76 running.
77
79 ::http::config ?options?
80 The ::http::config command is used to set and query the name of
81 the proxy server and port, and the User-Agent name used in the
82 HTTP requests. If no options are specified, then the current
83 configuration is returned. If a single argument is specified,
84 then it should be one of the flags described below. In this
85 case the current value of that setting is returned. Otherwise,
86 the options should be a set of flags and values that define the
87 configuration:
88 -accept mimetypes
90 The Accept header of the request. The default is */*,
91 which means that all types of documents are accepted.
92 Otherwise you can supply a comma-separated list of mime
93 type patterns that you are willing to receive. For exam‐
94 ple, “image/gif, image/jpeg, text/*”.
95 -pipeline boolean
97 Specifies whether HTTP/1.1 transactions on a persistent
98 socket will be pipelined. See the PERSISTENT SOCKETS
99 section for details. The default is 1.
100 -postfresh boolean
102 Specifies whether requests that use the POST method will
103 always use a fresh socket, overriding the -keepalive op‐
104 tion of command http::geturl. See the PERSISTENT SOCKETS
105 section for details. The default is 0.
106 -proxyhost hostname
108 The name of the proxy host, if any. If this value is the
109 empty string, the URL host is contacted directly.
110 -proxyport number
112 The proxy port number.
113 -proxyfilter command
115 The command is a callback that is made during
116 ::http::geturl to determine if a proxy is required for a
117 given host. One argument, a host name, is added to com‐
118 mand when it is invoked. If a proxy is required, the
119 callback should return a two-element list containing the
120 proxy server and proxy port. Otherwise the filter should
121 return an empty list. The default filter returns the
122 values of the -proxyhost and -proxyport settings if they
123 are non-empty.
124 -repost boolean
126 Specifies what to do if a POST request over a persistent
127 connection fails because the server has half-closed the
128 connection. If boolean true, the request will be auto‐
129 matically retried; if boolean false it will not, and the
130 application that uses http::geturl is expected to seek
131 user confirmation before retrying the POST. The value
132 true should be used only under certain conditions. See
133 the PERSISTENT SOCKETS section for details. The default
134 is 0.
135 -urlencoding encoding
137 The encoding used for creating the x-url-encoded URLs
138 with ::http::formatQuery and ::http::quoteString. The
139 default is utf-8, as specified by RFC 2718. Prior to
140 http 2.5 this was unspecified, and that behavior can be
141 returned by specifying the empty string ({}), although
142 iso8859-1 is recommended to restore similar behavior but
143 without the ::http::formatQuery or ::http::quoteString
144 throwing an error processing non-latin-1 characters.
145 -useragent string
147 The value of the User-Agent header in the HTTP request.
148 In an unsafe interpreter, the default value depends upon
149 the operating system, and the version numbers of http and
150 Tcl, and is (for example) “Mozilla/5.0 (Windows; U; Win‐
151 dows NT 10.0) http/2.9.0 Tcl/8.6.9”. A safe interpreter
152 cannot determine its operating system, and so the default
153 in a safe interpreter is to use a Windows 10 value with
154 the current version numbers of http and Tcl.
155 -zip boolean
157 If the value is boolean true, then by default requests
158 will send a header “Accept-Encoding: gzip,deflate,com‐
159 press”. If the value is boolean false, then by default
160 this header will not be sent. In either case the default
161 can be overridden for an individual request by supplying
162 a custom Accept-Encoding header in the -headers option of
163 http::geturl. The default is 1.
164 ::http::geturl url ?options?
166 The ::http::geturl command is the main procedure in the package.
167 The -query option causes a POST operation and the -validate op‐
168 tion causes a HEAD operation; otherwise, a GET operation is per‐
169 formed. The ::http::geturl command returns a token value that
170 can be used to get information about the transaction. See the
171 STATE ARRAY and ERRORS section for details. The ::http::geturl
172 command blocks until the operation completes, unless the -com‐
173 mand option specifies a callback that is invoked when the HTTP
174 transaction completes. ::http::geturl takes several options:
175 -binary boolean
177 Specifies whether to force interpreting the URL data as
178 binary. Normally this is auto-detected (anything not be‐
179 ginning with a text content type or whose content encod‐
180 ing is gzip or compress is considered binary data).
181 -blocksize size
183 The block size used when reading the URL. At most size
184 bytes are read at once. After each block, a call to the
185 -progress callback is made (if that option is specified).
186 -channel name
188 Copy the URL contents to channel name instead of saving
189 it in state(body).
190 -command callback
192 Invoke callback after the HTTP transaction completes.
193 This option causes ::http::geturl to return immediately.
194 The callback gets an additional argument that is the to‐
195 ken returned from ::http::geturl. This token is the name
196 of an array that is described in the STATE ARRAY section.
197 Here is a template for the callback:
198 proc httpCallback {token} {
200 upvar #0 $token state
201 # Access state as a Tcl array
202 }
203 -handler callback
205 Invoke callback whenever HTTP data is available; if
206 present, nothing else will be done with the HTTP data.
207 This procedure gets two additional arguments: the socket
208 for the HTTP data and the token returned from
209 ::http::geturl. The token is the name of a global array
210 that is described in the STATE ARRAY section. The proce‐
211 dure is expected to return the number of bytes read from
212 the socket. Here is a template for the callback:
213 proc httpHandlerCallback {socket token} {
215 upvar #0 $token state
216 # Access socket, and state as a Tcl array
217 # For example...
218 ...
219 set data [read $socket 1000]
220 set nbytes [string length $data]
221 ...
222 return $nbytes
223 }
224 The http::geturl code for the -handler option is not com‐
226 patible with either compression or chunked transfer-en‐
227 coding. If -handler is specified, then to work around
228 these issues http::geturl will reduce the HTTP protocol
229 to 1.0, and override the -zip option (i.e. it will not
230 send the header "Accept-Encoding: gzip,deflate,com‐
231 press").
232 If options -handler and -channel are used together, the
234 handler is responsible for copying the data from the HTTP
235 socket to the specified channel. The name of the channel
236 is available to the handler as element -channel of the
237 token array.
238 -headers keyvaluelist
240 This option is used to add headers not already specified
241 by ::http::config to the HTTP request. The keyvaluelist
242 argument must be a list with an even number of elements
243 that alternate between keys and values. The keys become
244 header field names. Newlines are stripped from the val‐
245 ues so the header cannot be corrupted. For example, if
246 keyvaluelist is Pragma no-cache then the following header
247 is included in the HTTP request:
248 Pragma: no-cache
250 -keepalive boolean
252 If boolean true, attempt to keep the connection open for
253 servicing multiple requests. Default is 0.
254 -method type
256 Force the HTTP request method to type. ::http::geturl
257 will auto-select GET, POST or HEAD based on other op‐
258 tions, but this option enables choices like PUT and
259 DELETE for webdav support.
260 -myaddr address
262 Pass an specific local address to the underlying socket
263 call in case multiple interfaces are available.
264 -progress callback
266 The callback is made after each transfer of data from the
267 URL. The callback gets three additional arguments: the
268 token from ::http::geturl, the expected total size of the
269 contents from the Content-Length meta-data, and the cur‐
270 rent number of bytes transferred so far. The expected
271 total size may be unknown, in which case zero is passed
272 to the callback. Here is a template for the progress
273 callback:
274 proc httpProgress {token total current} {
276 upvar #0 $token state
277 }
278 -protocol version
280 Select the HTTP protocol version to use. This should be
281 1.0 or 1.1 (the default). Should only be necessary for
282 servers that do not understand or otherwise complain
283 about HTTP/1.1.
284 -query query
286 This flag causes ::http::geturl to do a POST request that
287 passes the query as payload verbatim to the server. The
288 content format (and encoding) of query is announced by
289 the header field content-type set by the option -type.
290 query is an x-url-encoding formatted query, if used for
291 html forms. The ::http::formatQuery procedure can be
292 used to do the formatting.
293 -queryblocksize size
295 The block size used when posting query data to the URL.
296 At most size bytes are written at once. After each
297 block, a call to the -queryprogress callback is made (if
298 that option is specified).
299 -querychannel channelID
301 This flag causes ::http::geturl to do a POST request that
302 passes the data contained in channelID to the server. The
303 data contained in channelID must be an x-url-encoding
304 formatted query unless the -type option below is used.
305 If a Content-Length header is not specified via the
306 -headers options, ::http::geturl attempts to determine
307 the size of the post data in order to create that header.
308 If it is unable to determine the size, it returns an er‐
309 ror.
310 -queryprogress callback
312 The callback is made after each transfer of data to the
313 URL (i.e. POST) and acts exactly like the -progress op‐
314 tion (the callback format is the same).
315 -strict boolean
317 Whether to enforce RFC 3986 URL validation on the re‐
318 quest. Default is 1.
319 -timeout milliseconds
321 If milliseconds is non-zero, then ::http::geturl sets up
322 a timeout to occur after the specified number of mil‐
323 liseconds. A timeout results in a call to ::http::reset
324 and to the -command callback, if specified. The return
325 value of ::http::status is timeout after a timeout has
326 occurred.
327 -type mime-type
329 Use mime-type as the Content-Type value, instead of the
330 default value (application/x-www-form-urlencoded) during
331 a POST operation.
332 -validate boolean
334 If boolean is non-zero, then ::http::geturl does an HTTP
335 HEAD request. This request returns meta information
336 about the URL, but the contents are not returned. The
337 meta information is available in the state(meta) vari‐
338 able after the transaction. See the STATE ARRAY section
339 for details.
340 ::http::formatQuery key value ?key value ...?
342 This procedure does x-url-encoding of query data. It takes an
343 even number of arguments that are the keys and values of the
344 query. It encodes the keys and values, and generates one string
345 that has the proper & and = separators. The result is suitable
346 for the -query value passed to ::http::geturl.
347 ::http::quoteString value
349 This procedure does x-url-encoding of string. It takes a single
350 argument and encodes it.
351 ::http::reset token ?why?
353 This command resets the HTTP transaction identified by token, if
354 any. This sets the state(status) value to why, which defaults
355 to reset, and then calls the registered -command callback.
356 ::http::wait token
358 This is a convenience procedure that blocks and waits for the
359 transaction to complete. This only works in trusted code be‐
360 cause it uses vwait. Also, it is not useful for the case where
361 ::http::geturl is called without the -command option because in
362 this case the ::http::geturl call does not return until the HTTP
363 transaction is complete, and thus there is nothing to wait for.
364 ::http::data token
366 This is a convenience procedure that returns the body element
367 (i.e., the URL data) of the state array.
368 ::http::error token
370 This is a convenience procedure that returns the error element
371 of the state array.
372 ::http::status token
374 This is a convenience procedure that returns the status element
375 of the state array.
376 ::http::code token
378 This is a convenience procedure that returns the http element of
379 the state array.
380 ::http::ncode token
382 This is a convenience procedure that returns just the numeric
383 return code (200, 404, etc.) from the http element of the state
384 array.
385 ::http::size token
387 This is a convenience procedure that returns the currentsize el‐
388 ement of the state array, which represents the number of bytes
389 received from the URL in the ::http::geturl call.
390 ::http::meta token
392 This is a convenience procedure that returns the meta element of
393 the state array which contains the HTTP response headers. See
394 below for an explanation of this element.
395 ::http::cleanup token
397 This procedure cleans up the state associated with the connec‐
398 tion identified by token. After this call, the procedures like
399 ::http::data cannot be used to get information about the opera‐
400 tion. It is strongly recommended that you call this function
401 after you are done with a given HTTP request. Not doing so will
402 result in memory not being freed, and if your app calls
403 ::http::geturl enough times, the memory leak could cause a per‐
404 formance hit...or worse.
405 ::http::register proto port command
407 This procedure allows one to provide custom HTTP transport types
408 such as HTTPS, by registering a prefix, the default port, and
409 the command to execute to create the Tcl channel. E.g.:
410 package require http
412 package require tls
413 ::http::register https 443 ::tls::socket
415 set token [::http::geturl https://my.secure.site/]
417 ::http::registerError port ?message?
419 This procedure allows a registered protocol handler to deliver
420 an error message for use by http. Calling this command does not
421 raise an error. The command is useful when a registered protocol
422 detects an problem (for example, an invalid TLS certificate)
423 that will cause an error to propagate to http. The command al‐
424 lows http to provide a precise error message rather than a gen‐
425 eral one. The command returns the value provided by the last
426 call with argument message, or the empty string if no such call
427 has been made.
428 ::http::unregister proto
430 This procedure unregisters a protocol handler that was previ‐
431 ously registered via ::http::register, returning a two-item list
432 of the default port and handler command that was previously in‐
433 stalled (via ::http::register) if there was such a handler, and
434 an error if there was no such handler.
435
437 The ::http::geturl procedure will raise errors in the following cases:
438 invalid command line options, an invalid URL, a URL on a non-existent
439 host, or a URL at a bad port on an existing host. These errors mean
440 that it cannot even start the network transaction. It will also raise
441 an error if it gets an I/O error while writing out the HTTP request
442 header. For synchronous ::http::geturl calls (where -command is not
443 specified), it will raise an error if it gets an I/O error while read‐
444 ing the HTTP reply headers or data. Because ::http::geturl does not
445 return a token in these cases, it does all the required cleanup and
446 there is no issue of your app having to call ::http::cleanup.
447 For asynchronous ::http::geturl calls, all of the above error situa‐
449 tions apply, except that if there is any error while reading the HTTP
450 reply headers or data, no exception is thrown. This is because after
451 writing the HTTP headers, ::http::geturl returns, and the rest of the
452 HTTP transaction occurs in the background. The command callback can
453 check if any error occurred during the read by calling ::http::status
454 to check the status and if its error, calling ::http::error to get the
455 error message.
456 Alternatively, if the main program flow reaches a point where it needs
458 to know the result of the asynchronous HTTP request, it can call
459 ::http::wait and then check status and error, just as the callback
460 does.
461 In any case, you must still call ::http::cleanup to delete the state
463 array when you are done.
464 There are other possible results of the HTTP transaction determined by
466 examining the status from ::http::status. These are described below.
467 ok If the HTTP transaction completes entirely, then status will be
469 ok. However, you should still check the ::http::code value to
470 get the HTTP status. The ::http::ncode procedure provides just
471 the numeric error (e.g., 200, 404 or 500) while the ::http::code
472 procedure returns a value like “HTTP 404 File not found”.
473 eof If the server closes the socket without replying, then no error
475 is raised, but the status of the transaction will be eof.
476 error The error message will also be stored in the error status array
478 element, accessible via ::http::error.
479 timeout
481 A timeout occurred before the transaction could complete
482 reset user-reset
484 Another error possibility is that ::http::geturl is unable to write all
486 the post query data to the server before the server responds and closes
487 the socket. The error message is saved in the posterror status array
488 element and then ::http::geturl attempts to complete the transaction.
489 If it can read the server's response it will end up with an ok status,
490 otherwise it will have an eof status.
491
493 The ::http::geturl procedure returns a token that can be used to get to
494 the state of the HTTP transaction in the form of a Tcl array. Use this
495 construct to create an easy-to-use array variable:
496 upvar #0 $token state
498 Once the data associated with the URL is no longer needed, the state
500 array should be unset to free up storage. The ::http::cleanup proce‐
501 dure is provided for that purpose. The following elements of the array
502 are supported:
503 binary This is boolean true if (after decoding any compression
505 specified by the “Content-Encoding” response header) the
506 HTTP response is binary. It is boolean false if the HTTP
507 response is text.
508 body The contents of the URL. This will be empty if the
510 -channel option has been specified. This value is re‐
511 turned by the ::http::data command.
512 charset
514 The value of the charset attribute from the Content-Type
515 meta-data value. If none was specified, this defaults to
516 the RFC standard iso8859-1, or the value of $::http::de‐
517 faultCharset. Incoming text data will be automatically
518 converted from this charset to utf-8.
519 coding A copy of the Content-Encoding meta-data value.
521 currentsize
523 The current number of bytes fetched from the URL. This
524 value is returned by the ::http::size command.
525 error If defined, this is the error string seen when the HTTP
527 transaction was aborted.
528 http The HTTP status reply from the server. This value is re‐
530 turned by the ::http::code command. The format of this
531 value is:
532 HTTP/1.1 code string
534 The code is a three-digit number defined in the HTTP
536 standard. A code of 200 is OK. Codes beginning with 4
537 or 5 indicate errors. Codes beginning with 3 are redi‐
538 rection errors. In this case the Location meta-data
539 specifies a new URL that contains the requested informa‐
540 tion.
541 meta The HTTP protocol returns meta-data that describes the
543 URL contents. The meta element of the state array is a
544 list of the keys and values of the meta-data. This is in
545 a format useful for initializing an array that just con‐
546 tains the meta-data:
547 array set meta $state(meta)
549 Some of the meta-data keys are listed below, but the HTTP
551 standard defines more, and servers are free to add their
552 own.
553 Content-Type
555 The type of the URL contents. Examples include
556 text/html, image/gif, application/postscript and
557 application/x-tcl.
558 Content-Length
560 The advertised size of the contents. The actual
561 size obtained by ::http::geturl is available as
562 state(currentsize).
563 Location
565 An alternate URL that contains the requested data.
566 posterror
568 The error, if any, that occurred while writing the post
569 query data to the server.
570 status See description in the chapter ERRORS above for a list
572 and description of status. During the transaction this
573 value is the empty string.
574 totalsize
576 A copy of the Content-Length meta-data value.
577 type A copy of the Content-Type meta-data value.
579 url The requested URL.
581
583 BASICS
584 See RFC 7230 Sec 6, which supersedes RFC 2616 Sec 8.1.
585 A persistent connection allows multiple HTTP/1.1 transactions to be
587 carried over the same TCP connection. Pipelining allows a client to
588 make multiple requests over a persistent connection without waiting for
589 each response. The server sends responses in the same order that the
590 requests were received.
591 If a POST request fails to complete, typically user confirmation is
593 needed before sending the request again. The user may wish to verify
594 whether the server was modified by the failed POST request, before
595 sending the same request again.
596 A HTTP request will use a persistent socket if the call to http::geturl
598 has the option -keepalive true. It will use pipelining where permitted
599 if the http::config option -pipeline is boolean true (its default
600 value).
601 The http package maintains no more than one persistent connection to
603 each server (i.e. each value of “domain:port”). If http::geturl is
604 called to make a request over a persistent connection while the connec‐
605 tion is busy with another request, the new request will be held in a
606 queue until the connection is free.
607 The http package does not support HTTP/1.0 persistent connections con‐
609 trolled by the Keep-Alive header.
610 SPECIAL CASES
612 This subsection discusses issues related to closure of the persistent
613 connection by the server, automatic retry of failed requests, the spe‐
614 cial treatment necessary for POST requests, and the options for dealing
615 with these cases.
616 In accordance with RFC 7230, http::geturl does not pipeline requests
618 that use the POST method. If a POST uses a persistent connection and
619 is not the first request on that connection, http::geturl waits until
620 it has received the response for the previous request; or (if
621 http::config option -postfresh is boolean true) it uses a new connec‐
622 tion for each POST.
623 If the server is processing a number of pipelined requests, and sends a
625 response header “Connection: close” with one of the responses (other
626 than the last), then subsequent responses are unfulfilled. http::geturl
627 will send the unfulfilled requests again over a new connection.
628 A difficulty arises when a HTTP client sends a request over a persis‐
630 tent connection that has been idle for a while. The HTTP server may
631 half-close an apparently idle connection while the client is sending a
632 request, but before the request arrives at the server: in this case (an
633 “asynchronous close event”) the request will fail. The difficulty
634 arises because the client cannot be certain whether the POST modified
635 the state of the server. For HEAD or GET requests, http::geturl opens
636 another connection and retransmits the failed request. However, if the
637 request was a POST, RFC 7230 forbids automatic retry by default, sug‐
638 gesting either user confirmation, or confirmation by user-agent soft‐
639 ware that has semantic understanding of the application. The
640 http::config option -repost allows for either possibility.
641 Asynchronous close events can occur only in a short interval of time.
643 The http package monitors each persistent connection for closure by the
644 server. Upon detection, the connection is also closed at the client
645 end, and subsequent requests will use a fresh connection.
646 If the http::geturl command is called with option -keepalive true, then
648 it will both try to use an existing persistent connection (if one is
649 available), and it will send the server a “Connection: keep-alive” re‐
650 quest header asking to keep the connection open for future requests.
651 The http::config options -pipeline, -postfresh, and -repost relate to
653 persistent connections.
654 Option -pipeline, if boolean true, will pipeline GET and HEAD requests
656 made over a persistent connection. POST requests will not be pipelined
657 - if the POST is not the first transaction on the connection, its re‐
658 quest will not be sent until the previous response has finished. GET
659 and HEAD requests made after a POST will not be sent until the POST re‐
660 sponse has been delivered, and will not be sent if the POST fails.
661 Option -postfresh, if boolean true, will override the http::geturl op‐
663 tion -keepalive, and always open a fresh connection for a POST request.
664 Option -repost, if true, permits automatic retry of a POST request that
666 fails because it uses a persistent connection that the server has half-
667 closed (an “asynchronous close event”). Subsequent GET and HEAD re‐
668 quests in a failed pipeline will also be retried. The -repost option
669 should be used only if the application understands that the retry is
670 appropriate - specifically, the application must know that if the
671 failed POST successfully modified the state of the server, a repeat
672 POST would have no adverse effect.
673
675 This example creates a procedure to copy a URL to a file while printing
676 a progress meter, and prints the meta-data associated with the URL.
677 proc httpcopy { url file {chunk 4096} } {
679 set out [open $file w]
680 set token [::http::geturl $url -channel $out \
681 -progress httpCopyProgress -blocksize $chunk]
682 close $out
683 # This ends the line started by httpCopyProgress
685 puts stderr ""
686 upvar #0 $token state
688 set max 0
689 foreach {name value} $state(meta) {
690 if {[string length $name] > $max} {
691 set max [string length $name]
692 }
693 if {[regexp -nocase ^location$ $name]} {
694 # Handle URL redirects
695 puts stderr "Location:$value"
696 return [httpcopy [string trim $value] $file $chunk]
697 }
698 }
699 incr max
700 foreach {name value} $state(meta) {
701 puts [format "%-*s %s" $max $name: $value]
702 }
703 return $token
705 }
706 proc httpCopyProgress {args} {
707 puts -nonewline stderr .
708 flush stderr
709 }
710
712 safe(n), socket(n), safesock(n)
713
715 internet, security policy, socket, www
716http 2.9 http(n)