1WGET(1) GNU Wget WGET(1)
2
6 Wget - The non-interactive network downloader.
7
9 wget [option]... [URL]...
10
12 GNU Wget is a free utility for non-interactive download of files from
13 the Web. It supports HTTP, HTTPS, and FTP protocols, as well as
14 retrieval through HTTP proxies.
15 Wget is non-interactive, meaning that it can work in the background,
17 while the user is not logged on. This allows you to start a retrieval
18 and disconnect from the system, letting Wget finish the work. By
19 contrast, most of the Web browsers require constant user's presence,
20 which can be a great hindrance when transferring a lot of data.
21 Wget can follow links in HTML, XHTML, and CSS pages, to create local
23 versions of remote web sites, fully recreating the directory structure
24 of the original site. This is sometimes referred to as "recursive
25 downloading." While doing that, Wget respects the Robot Exclusion
26 Standard (/robots.txt). Wget can be instructed to convert the links in
27 downloaded files to point at the local files, for offline viewing.
28 Wget has been designed for robustness over slow or unstable network
30 connections; if a download fails due to a network problem, it will keep
31 retrying until the whole file has been retrieved. If the server
32 supports regetting, it will instruct the server to continue the
33 download from where it left off.
34
36 Option Syntax
37 Since Wget uses GNU getopt to process command-line arguments, every
38 option has a long form along with the short one. Long options are more
39 convenient to remember, but take time to type. You may freely mix
40 different option styles, or specify options after the command-line
41 arguments. Thus you may write:
42 wget -r --tries=10 http://fly.srk.fer.hr/ -o log
44 The space between the option accepting an argument and the argument may
46 be omitted. Instead of -o log you can write -olog.
47 You may put several options that do not require arguments together,
49 like:
50 wget -drc <URL>
52 This is completely equivalent to:
54 wget -d -r -c <URL>
56 Since the options can be specified after the arguments, you may
58 terminate them with --. So the following will try to download URL -x,
59 reporting failure to log:
60 wget -o log -- -x
62 The options that accept comma-separated lists all respect the
64 convention that specifying an empty list clears its value. This can be
65 useful to clear the .wgetrc settings. For instance, if your .wgetrc
66 sets "exclude_directories" to /cgi-bin, the following example will
67 first reset it, and then set it to exclude /~nobody and /~somebody.
68 You can also clear the lists in .wgetrc.
69 wget -X " -X /~nobody,/~somebody
71 Most options that do not accept arguments are boolean options, so named
73 because their state can be captured with a yes-or-no ("boolean")
74 variable. For example, --follow-ftp tells Wget to follow FTP links
75 from HTML files and, on the other hand, --no-glob tells it not to
76 perform file globbing on FTP URLs. A boolean option is either
77 affirmative or negative (beginning with --no). All such options share
78 several properties.
79 Unless stated otherwise, it is assumed that the default behavior is the
81 opposite of what the option accomplishes. For example, the documented
82 existence of --follow-ftp assumes that the default is to not follow FTP
83 links from HTML pages.
84 Affirmative options can be negated by prepending the --no- to the
86 option name; negative options can be negated by omitting the --no-
87 prefix. This might seem superfluous---if the default for an
88 affirmative option is to not do something, then why provide a way to
89 explicitly turn it off? But the startup file may in fact change the
90 default. For instance, using "follow_ftp = on" in .wgetrc makes Wget
91 follow FTP links by default, and using --no-follow-ftp is the only way
92 to restore the factory default from the command line.
93 Basic Startup Options
95 -V
96 --version
97 Display the version of Wget.
98 -h
100 --help
101 Print a help message describing all of Wget's command-line options.
102 -b
104 --background
105 Go to background immediately after startup. If no output file is
106 specified via the -o, output is redirected to wget-log.
107 -e command
109 --execute command
110 Execute command as if it were a part of .wgetrc. A command thus
111 invoked will be executed after the commands in .wgetrc, thus taking
112 precedence over them. If you need to specify more than one wgetrc
113 command, use multiple instances of -e.
114 Logging and Input File Options
116 -o logfile
117 --output-file=logfile
118 Log all messages to logfile. The messages are normally reported to
119 standard error.
120 -a logfile
122 --append-output=logfile
123 Append to logfile. This is the same as -o, only it appends to
124 logfile instead of overwriting the old log file. If logfile does
125 not exist, a new file is created.
126 -d
128 --debug
129 Turn on debug output, meaning various information important to the
130 developers of Wget if it does not work properly. Your system
131 administrator may have chosen to compile Wget without debug
132 support, in which case -d will not work. Please note that
133 compiling with debug support is always safe---Wget compiled with
134 the debug support will not print any debug info unless requested
135 with -d.
136 -q
138 --quiet
139 Turn off Wget's output.
140 -v
142 --verbose
143 Turn on verbose output, with all the available data. The default
144 output is verbose.
145 -nv
147 --no-verbose
148 Turn off verbose without being completely quiet (use -q for that),
149 which means that error messages and basic information still get
150 printed.
151 --report-speed=type
153 Output bandwidth as type. The only accepted value is bits.
154 -i file
156 --input-file=file
157 Read URLs from a local or external file. If - is specified as
158 file, URLs are read from the standard input. (Use ./- to read from
159 a file literally named -.)
160 If this function is used, no URLs need be present on the command
162 line. If there are URLs both on the command line and in an input
163 file, those on the command lines will be the first ones to be
164 retrieved. If --force-html is not specified, then file should
165 consist of a series of URLs, one per line.
166 However, if you specify --force-html, the document will be regarded
168 as html. In that case you may have problems with relative links,
169 which you can solve either by adding "<base href="url">" to the
170 documents or by specifying --base=url on the command line.
171 If the file is an external one, the document will be automatically
173 treated as html if the Content-Type matches text/html.
174 Furthermore, the file's location will be implicitly used as base
175 href if none was specified.
176 -F
178 --force-html
179 When input is read from a file, force it to be treated as an HTML
180 file. This enables you to retrieve relative links from existing
181 HTML files on your local disk, by adding "<base href="url">" to
182 HTML, or using the --base command-line option.
183 -B URL
185 --base=URL
186 Resolves relative links using URL as the point of reference, when
187 reading links from an HTML file specified via the -i/--input-file
188 option (together with --force-html, or when the input file was
189 fetched remotely from a server describing it as HTML). This is
190 equivalent to the presence of a "BASE" tag in the HTML input file,
191 with URL as the value for the "href" attribute.
192 For instance, if you specify http://foo/bar/a.html for URL, and
194 Wget reads ../baz/b.html from the input file, it would be resolved
195 to http://foo/baz/b.html.
196 --config=FILE
198 Specify the location of a startup file you wish to use.
199 Download Options
201 --bind-address=ADDRESS
202 When making client TCP/IP connections, bind to ADDRESS on the local
203 machine. ADDRESS may be specified as a hostname or IP address.
204 This option can be useful if your machine is bound to multiple IPs.
205 -t number
207 --tries=number
208 Set number of retries to number. Specify 0 or inf for infinite
209 retrying. The default is to retry 20 times, with the exception of
210 fatal errors like "connection refused" or "not found" (404), which
211 are not retried.
212 -O file
214 --output-document=file
215 The documents will not be written to the appropriate files, but all
216 will be concatenated together and written to file. If - is used as
217 file, documents will be printed to standard output, disabling link
218 conversion. (Use ./- to print to a file literally named -.)
219 Use of -O is not intended to mean simply "use the name file instead
221 of the one in the URL;" rather, it is analogous to shell
222 redirection: wget -O file http://foo is intended to work like wget
223 -O - http://foo > file; file will be truncated immediately, and all
224 downloaded content will be written there.
225 For this reason, -N (for timestamp-checking) is not supported in
227 combination with -O: since file is always newly created, it will
228 always have a very new timestamp. A warning will be issued if this
229 combination is used.
230 Similarly, using -r or -p with -O may not work as you expect: Wget
232 won't just download the first file to file and then download the
233 rest to their normal names: all downloaded content will be placed
234 in file. This was disabled in version 1.11, but has been reinstated
235 (with a warning) in 1.11.2, as there are some cases where this
236 behavior can actually have some use.
237 Note that a combination with -k is only permitted when downloading
239 a single document, as in that case it will just convert all
240 relative URIs to external ones; -k makes no sense for multiple URIs
241 when they're all being downloaded to a single file; -k can be used
242 only when the output is a regular file.
243 -nc
245 --no-clobber
246 If a file is downloaded more than once in the same directory,
247 Wget's behavior depends on a few options, including -nc. In
248 certain cases, the local file will be clobbered, or overwritten,
249 upon repeated download. In other cases it will be preserved.
250 When running Wget without -N, -nc, -r, or -p, downloading the same
252 file in the same directory will result in the original copy of file
253 being preserved and the second copy being named file.1. If that
254 file is downloaded yet again, the third copy will be named file.2,
255 and so on. (This is also the behavior with -nd, even if -r or -p
256 are in effect.) When -nc is specified, this behavior is
257 suppressed, and Wget will refuse to download newer copies of file.
258 Therefore, ""no-clobber"" is actually a misnomer in this
259 mode---it's not clobbering that's prevented (as the numeric
260 suffixes were already preventing clobbering), but rather the
261 multiple version saving that's prevented.
262 When running Wget with -r or -p, but without -N, -nd, or -nc, re-
264 downloading a file will result in the new copy simply overwriting
265 the old. Adding -nc will prevent this behavior, instead causing
266 the original version to be preserved and any newer copies on the
267 server to be ignored.
268 When running Wget with -N, with or without -r or -p, the decision
270 as to whether or not to download a newer copy of a file depends on
271 the local and remote timestamp and size of the file. -nc may not
272 be specified at the same time as -N.
273 Note that when -nc is specified, files with the suffixes .html or
275 .htm will be loaded from the local disk and parsed as if they had
276 been retrieved from the Web.
277 --backups=backups
279 Before (over)writing a file, back up an existing file by adding a
280 .1 suffix (_1 on VMS) to the file name. Such backup files are
281 rotated to .2, .3, and so on, up to backups (and lost beyond that).
282 -c
284 --continue
285 Continue getting a partially-downloaded file. This is useful when
286 you want to finish up a download started by a previous instance of
287 Wget, or by another program. For instance:
288 wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z
290 If there is a file named ls-lR.Z in the current directory, Wget
292 will assume that it is the first portion of the remote file, and
293 will ask the server to continue the retrieval from an offset equal
294 to the length of the local file.
295 Note that you don't need to specify this option if you just want
297 the current invocation of Wget to retry downloading a file should
298 the connection be lost midway through. This is the default
299 behavior. -c only affects resumption of downloads started prior to
300 this invocation of Wget, and whose local files are still sitting
301 around.
302 Without -c, the previous example would just download the remote
304 file to ls-lR.Z.1, leaving the truncated ls-lR.Z file alone.
305 Beginning with Wget 1.7, if you use -c on a non-empty file, and it
307 turns out that the server does not support continued downloading,
308 Wget will refuse to start the download from scratch, which would
309 effectively ruin existing contents. If you really want the
310 download to start from scratch, remove the file.
311 Also beginning with Wget 1.7, if you use -c on a file which is of
313 equal size as the one on the server, Wget will refuse to download
314 the file and print an explanatory message. The same happens when
315 the file is smaller on the server than locally (presumably because
316 it was changed on the server since your last download
317 attempt)---because "continuing" is not meaningful, no download
318 occurs.
319 On the other side of the coin, while using -c, any file that's
321 bigger on the server than locally will be considered an incomplete
322 download and only "(length(remote) - length(local))" bytes will be
323 downloaded and tacked onto the end of the local file. This
324 behavior can be desirable in certain cases---for instance, you can
325 use wget -c to download just the new portion that's been appended
326 to a data collection or log file.
327 However, if the file is bigger on the server because it's been
329 changed, as opposed to just appended to, you'll end up with a
330 garbled file. Wget has no way of verifying that the local file is
331 really a valid prefix of the remote file. You need to be
332 especially careful of this when using -c in conjunction with -r,
333 since every file will be considered as an "incomplete download"
334 candidate.
335 Another instance where you'll get a garbled file if you try to use
337 -c is if you have a lame HTTP proxy that inserts a "transfer
338 interrupted" string into the local file. In the future a
339 "rollback" option may be added to deal with this case.
340 Note that -c only works with FTP servers and with HTTP servers that
342 support the "Range" header.
343 --progress=type
345 Select the type of the progress indicator you wish to use. Legal
346 indicators are "dot" and "bar".
347 The "bar" indicator is used by default. It draws an ASCII progress
349 bar graphics (a.k.a "thermometer" display) indicating the status of
350 retrieval. If the output is not a TTY, the "dot" bar will be used
351 by default.
352 Use --progress=dot to switch to the "dot" display. It traces the
354 retrieval by printing dots on the screen, each dot representing a
355 fixed amount of downloaded data.
356 When using the dotted retrieval, you may also set the style by
358 specifying the type as dot:style. Different styles assign
359 different meaning to one dot. With the "default" style each dot
360 represents 1K, there are ten dots in a cluster and 50 dots in a
361 line. The "binary" style has a more "computer"-like
362 orientation---8K dots, 16-dots clusters and 48 dots per line (which
363 makes for 384K lines). The "mega" style is suitable for
364 downloading very large files---each dot represents 64K retrieved,
365 there are eight dots in a cluster, and 48 dots on each line (so
366 each line contains 3M).
367 Note that you can set the default style using the "progress"
369 command in .wgetrc. That setting may be overridden from the
370 command line. The exception is that, when the output is not a TTY,
371 the "dot" progress will be favored over "bar". To force the bar
372 output, use --progress=bar:force.
373 -N
375 --timestamping
376 Turn on time-stamping.
377 --no-use-server-timestamps
379 Don't set the local file's timestamp by the one on the server.
380 By default, when a file is downloaded, it's timestamps are set to
382 match those from the remote file. This allows the use of
383 --timestamping on subsequent invocations of wget. However, it is
384 sometimes useful to base the local file's timestamp on when it was
385 actually downloaded; for that purpose, the
386 --no-use-server-timestamps option has been provided.
387 -S
389 --server-response
390 Print the headers sent by HTTP servers and responses sent by FTP
391 servers.
392 --spider
394 When invoked with this option, Wget will behave as a Web spider,
395 which means that it will not download the pages, just check that
396 they are there. For example, you can use Wget to check your
397 bookmarks:
398 wget --spider --force-html -i bookmarks.html
400 This feature needs much more work for Wget to get close to the
402 functionality of real web spiders.
403 -T seconds
405 --timeout=seconds
406 Set the network timeout to seconds seconds. This is equivalent to
407 specifying --dns-timeout, --connect-timeout, and --read-timeout,
408 all at the same time.
409 When interacting with the network, Wget can check for timeout and
411 abort the operation if it takes too long. This prevents anomalies
412 like hanging reads and infinite connects. The only timeout enabled
413 by default is a 900-second read timeout. Setting a timeout to 0
414 disables it altogether. Unless you know what you are doing, it is
415 best not to change the default timeout settings.
416 All timeout-related options accept decimal values, as well as
418 subsecond values. For example, 0.1 seconds is a legal (though
419 unwise) choice of timeout. Subsecond timeouts are useful for
420 checking server response times or for testing network latency.
421 --dns-timeout=seconds
423 Set the DNS lookup timeout to seconds seconds. DNS lookups that
424 don't complete within the specified time will fail. By default,
425 there is no timeout on DNS lookups, other than that implemented by
426 system libraries.
427 --connect-timeout=seconds
429 Set the connect timeout to seconds seconds. TCP connections that
430 take longer to establish will be aborted. By default, there is no
431 connect timeout, other than that implemented by system libraries.
432 --read-timeout=seconds
434 Set the read (and write) timeout to seconds seconds. The "time" of
435 this timeout refers to idle time: if, at any point in the download,
436 no data is received for more than the specified number of seconds,
437 reading fails and the download is restarted. This option does not
438 directly affect the duration of the entire download.
439 Of course, the remote server may choose to terminate the connection
441 sooner than this option requires. The default read timeout is 900
442 seconds.
443 --limit-rate=amount
445 Limit the download speed to amount bytes per second. Amount may be
446 expressed in bytes, kilobytes with the k suffix, or megabytes with
447 the m suffix. For example, --limit-rate=20k will limit the
448 retrieval rate to 20KB/s. This is useful when, for whatever
449 reason, you don't want Wget to consume the entire available
450 bandwidth.
451 This option allows the use of decimal numbers, usually in
453 conjunction with power suffixes; for example, --limit-rate=2.5k is
454 a legal value.
455 Note that Wget implements the limiting by sleeping the appropriate
457 amount of time after a network read that took less time than
458 specified by the rate. Eventually this strategy causes the TCP
459 transfer to slow down to approximately the specified rate.
460 However, it may take some time for this balance to be achieved, so
461 don't be surprised if limiting the rate doesn't work well with very
462 small files.
463 -w seconds
465 --wait=seconds
466 Wait the specified number of seconds between the retrievals. Use
467 of this option is recommended, as it lightens the server load by
468 making the requests less frequent. Instead of in seconds, the time
469 can be specified in minutes using the "m" suffix, in hours using
470 "h" suffix, or in days using "d" suffix.
471 Specifying a large value for this option is useful if the network
473 or the destination host is down, so that Wget can wait long enough
474 to reasonably expect the network error to be fixed before the
475 retry. The waiting interval specified by this function is
476 influenced by "--random-wait", which see.
477 --waitretry=seconds
479 If you don't want Wget to wait between every retrieval, but only
480 between retries of failed downloads, you can use this option. Wget
481 will use linear backoff, waiting 1 second after the first failure
482 on a given file, then waiting 2 seconds after the second failure on
483 that file, up to the maximum number of seconds you specify.
484 By default, Wget will assume a value of 10 seconds.
486 --random-wait
488 Some web sites may perform log analysis to identify retrieval
489 programs such as Wget by looking for statistically significant
490 similarities in the time between requests. This option causes the
491 time between requests to vary between 0.5 and 1.5 * wait seconds,
492 where wait was specified using the --wait option, in order to mask
493 Wget's presence from such analysis.
494 A 2001 article in a publication devoted to development on a popular
496 consumer platform provided code to perform this analysis on the
497 fly. Its author suggested blocking at the class C address level to
498 ensure automated retrieval programs were blocked despite changing
499 DHCP-supplied addresses.
500 The --random-wait option was inspired by this ill-advised
502 recommendation to block many unrelated users from a web site due to
503 the actions of one.
504 --no-proxy
506 Don't use proxies, even if the appropriate *_proxy environment
507 variable is defined.
508 -Q quota
510 --quota=quota
511 Specify download quota for automatic retrievals. The value can be
512 specified in bytes (default), kilobytes (with k suffix), or
513 megabytes (with m suffix).
514 Note that quota will never affect downloading a single file. So if
516 you specify wget -Q10k ftp://wuarchive.wustl.edu/ls-lR.gz, all of
517 the ls-lR.gz will be downloaded. The same goes even when several
518 URLs are specified on the command-line. However, quota is
519 respected when retrieving either recursively, or from an input
520 file. Thus you may safely type wget -Q2m -i sites---download will
521 be aborted when the quota is exceeded.
522 Setting quota to 0 or to inf unlimits the download quota.
524 --no-dns-cache
526 Turn off caching of DNS lookups. Normally, Wget remembers the IP
527 addresses it looked up from DNS so it doesn't have to repeatedly
528 contact the DNS server for the same (typically small) set of hosts
529 it retrieves from. This cache exists in memory only; a new Wget
530 run will contact DNS again.
531 However, it has been reported that in some situations it is not
533 desirable to cache host names, even for the duration of a short-
534 running application like Wget. With this option Wget issues a new
535 DNS lookup (more precisely, a new call to "gethostbyname" or
536 "getaddrinfo") each time it makes a new connection. Please note
537 that this option will not affect caching that might be performed by
538 the resolving library or by an external caching layer, such as
539 NSCD.
540 If you don't understand exactly what this option does, you probably
542 won't need it.
543 --restrict-file-names=modes
545 Change which characters found in remote URLs must be escaped during
546 generation of local filenames. Characters that are restricted by
547 this option are escaped, i.e. replaced with %HH, where HH is the
548 hexadecimal number that corresponds to the restricted character.
549 This option may also be used to force all alphabetical cases to be
550 either lower- or uppercase.
551 By default, Wget escapes the characters that are not valid or safe
553 as part of file names on your operating system, as well as control
554 characters that are typically unprintable. This option is useful
555 for changing these defaults, perhaps because you are downloading to
556 a non-native partition, or because you want to disable escaping of
557 the control characters, or you want to further restrict characters
558 to only those in the ASCII range of values.
559 The modes are a comma-separated set of text values. The acceptable
561 values are unix, windows, nocontrol, ascii, lowercase, and
562 uppercase. The values unix and windows are mutually exclusive (one
563 will override the other), as are lowercase and uppercase. Those
564 last are special cases, as they do not change the set of characters
565 that would be escaped, but rather force local file paths to be
566 converted either to lower- or uppercase.
567 When "unix" is specified, Wget escapes the character / and the
569 control characters in the ranges 0--31 and 128--159. This is the
570 default on Unix-like operating systems.
571 When "windows" is given, Wget escapes the characters \, |, /, :, ?,
573 ", *, <, >, and the control characters in the ranges 0--31 and
574 128--159. In addition to this, Wget in Windows mode uses + instead
575 of : to separate host and port in local file names, and uses @
576 instead of ? to separate the query portion of the file name from
577 the rest. Therefore, a URL that would be saved as
578 www.xemacs.org:4300/search.pl?input=blah in Unix mode would be
579 saved as www.xemacs.org+4300/search.pl@input=blah in Windows mode.
580 This mode is the default on Windows.
581 If you specify nocontrol, then the escaping of the control
583 characters is also switched off. This option may make sense when
584 you are downloading URLs whose names contain UTF-8 characters, on a
585 system which can save and display filenames in UTF-8 (some possible
586 byte values used in UTF-8 byte sequences fall in the range of
587 values designated by Wget as "controls").
588 The ascii mode is used to specify that any bytes whose values are
590 outside the range of ASCII characters (that is, greater than 127)
591 shall be escaped. This can be useful when saving filenames whose
592 encoding does not match the one used locally.
593 -4
595 --inet4-only
596 -6
597 --inet6-only
598 Force connecting to IPv4 or IPv6 addresses. With --inet4-only or
599 -4, Wget will only connect to IPv4 hosts, ignoring AAAA records in
600 DNS, and refusing to connect to IPv6 addresses specified in URLs.
601 Conversely, with --inet6-only or -6, Wget will only connect to IPv6
602 hosts and ignore A records and IPv4 addresses.
603 Neither options should be needed normally. By default, an
605 IPv6-aware Wget will use the address family specified by the host's
606 DNS record. If the DNS responds with both IPv4 and IPv6 addresses,
607 Wget will try them in sequence until it finds one it can connect
608 to. (Also see "--prefer-family" option described below.)
609 These options can be used to deliberately force the use of IPv4 or
611 IPv6 address families on dual family systems, usually to aid
612 debugging or to deal with broken network configuration. Only one
613 of --inet6-only and --inet4-only may be specified at the same time.
614 Neither option is available in Wget compiled without IPv6 support.
615 --prefer-family=none/IPv4/IPv6
617 When given a choice of several addresses, connect to the addresses
618 with specified address family first. The address order returned by
619 DNS is used without change by default.
620 This avoids spurious errors and connect attempts when accessing
622 hosts that resolve to both IPv6 and IPv4 addresses from IPv4
623 networks. For example, www.kame.net resolves to
624 2001:200:0:8002:203:47ff:fea5:3085 and to 203.178.141.194. When
625 the preferred family is "IPv4", the IPv4 address is used first;
626 when the preferred family is "IPv6", the IPv6 address is used
627 first; if the specified value is "none", the address order returned
628 by DNS is used without change.
629 Unlike -4 and -6, this option doesn't inhibit access to any address
631 family, it only changes the order in which the addresses are
632 accessed. Also note that the reordering performed by this option
633 is stable---it doesn't affect order of addresses of the same
634 family. That is, the relative order of all IPv4 addresses and of
635 all IPv6 addresses remains intact in all cases.
636 --retry-connrefused
638 Consider "connection refused" a transient error and try again.
639 Normally Wget gives up on a URL when it is unable to connect to the
640 site because failure to connect is taken as a sign that the server
641 is not running at all and that retries would not help. This option
642 is for mirroring unreliable sites whose servers tend to disappear
643 for short periods of time.
644 --user=user
646 --password=password
647 Specify the username user and password password for both FTP and
648 HTTP file retrieval. These parameters can be overridden using the
649 --ftp-user and --ftp-password options for FTP connections and the
650 --http-user and --http-password options for HTTP connections.
651 --ask-password
653 Prompt for a password for each connection established. Cannot be
654 specified when --password is being used, because they are mutually
655 exclusive.
656 --no-iri
658 Turn off internationalized URI (IRI) support. Use --iri to turn it
659 on. IRI support is activated by default.
660 You can set the default state of IRI support using the "iri"
662 command in .wgetrc. That setting may be overridden from the command
663 line.
664 --local-encoding=encoding
666 Force Wget to use encoding as the default system encoding. That
667 affects how Wget converts URLs specified as arguments from locale
668 to UTF-8 for IRI support.
669 Wget use the function "nl_langinfo()" and then the "CHARSET"
671 environment variable to get the locale. If it fails, ASCII is used.
672 You can set the default local encoding using the "local_encoding"
674 command in .wgetrc. That setting may be overridden from the command
675 line.
676 --remote-encoding=encoding
678 Force Wget to use encoding as the default remote server encoding.
679 That affects how Wget converts URIs found in files from remote
680 encoding to UTF-8 during a recursive fetch. This options is only
681 useful for IRI support, for the interpretation of non-ASCII
682 characters.
683 For HTTP, remote encoding can be found in HTTP "Content-Type"
685 header and in HTML "Content-Type http-equiv" meta tag.
686 You can set the default encoding using the "remoteencoding" command
688 in .wgetrc. That setting may be overridden from the command line.
689 --unlink
691 Force Wget to unlink file instead of clobbering existing file. This
692 option is useful for downloading to the directory with hardlinks.
693 Directory Options
695 -nd
696 --no-directories
697 Do not create a hierarchy of directories when retrieving
698 recursively. With this option turned on, all files will get saved
699 to the current directory, without clobbering (if a name shows up
700 more than once, the filenames will get extensions .n).
701 -x
703 --force-directories
704 The opposite of -nd---create a hierarchy of directories, even if
705 one would not have been created otherwise. E.g. wget -x
706 http://fly.srk.fer.hr/robots.txt will save the downloaded file to
707 fly.srk.fer.hr/robots.txt.
708 -nH
710 --no-host-directories
711 Disable generation of host-prefixed directories. By default,
712 invoking Wget with -r http://fly.srk.fer.hr/ will create a
713 structure of directories beginning with fly.srk.fer.hr/. This
714 option disables such behavior.
715 --protocol-directories
717 Use the protocol name as a directory component of local file names.
718 For example, with this option, wget -r http://host will save to
719 http/host/... rather than just to host/....
720 --cut-dirs=number
722 Ignore number directory components. This is useful for getting a
723 fine-grained control over the directory where recursive retrieval
724 will be saved.
725 Take, for example, the directory at
727 ftp://ftp.xemacs.org/pub/xemacs/. If you retrieve it with -r, it
728 will be saved locally under ftp.xemacs.org/pub/xemacs/. While the
729 -nH option can remove the ftp.xemacs.org/ part, you are still stuck
730 with pub/xemacs. This is where --cut-dirs comes in handy; it makes
731 Wget not "see" number remote directory components. Here are
732 several examples of how --cut-dirs option works.
733 No options -> ftp.xemacs.org/pub/xemacs/
735 -nH -> pub/xemacs/
736 -nH --cut-dirs=1 -> xemacs/
737 -nH --cut-dirs=2 -> .
738 --cut-dirs=1 -> ftp.xemacs.org/xemacs/
740 ...
741 If you just want to get rid of the directory structure, this option
743 is similar to a combination of -nd and -P. However, unlike -nd,
744 --cut-dirs does not lose with subdirectories---for instance, with
745 -nH --cut-dirs=1, a beta/ subdirectory will be placed to
746 xemacs/beta, as one would expect.
747 -P prefix
749 --directory-prefix=prefix
750 Set directory prefix to prefix. The directory prefix is the
751 directory where all other files and subdirectories will be saved
752 to, i.e. the top of the retrieval tree. The default is . (the
753 current directory).
754 HTTP Options
756 --default-page=name
757 Use name as the default file name when it isn't known (i.e., for
758 URLs that end in a slash), instead of index.html.
759 -E
761 --adjust-extension
762 If a file of type application/xhtml+xml or text/html is downloaded
763 and the URL does not end with the regexp \.[Hh][Tt][Mm][Ll]?, this
764 option will cause the suffix .html to be appended to the local
765 filename. This is useful, for instance, when you're mirroring a
766 remote site that uses .asp pages, but you want the mirrored pages
767 to be viewable on your stock Apache server. Another good use for
768 this is when you're downloading CGI-generated materials. A URL
769 like http://site.com/article.cgi?25 will be saved as
770 article.cgi?25.html.
771 Note that filenames changed in this way will be re-downloaded every
773 time you re-mirror a site, because Wget can't tell that the local
774 X.html file corresponds to remote URL X (since it doesn't yet know
775 that the URL produces output of type text/html or
776 application/xhtml+xml.
777 As of version 1.12, Wget will also ensure that any downloaded files
779 of type text/css end in the suffix .css, and the option was renamed
780 from --html-extension, to better reflect its new behavior. The old
781 option name is still acceptable, but should now be considered
782 deprecated.
783 At some point in the future, this option may well be expanded to
785 include suffixes for other types of content, including content
786 types that are not parsed by Wget.
787 --http-user=user
789 --http-password=password
790 Specify the username user and password password on an HTTP server.
791 According to the type of the challenge, Wget will encode them using
792 either the "basic" (insecure), the "digest", or the Windows "NTLM"
793 authentication scheme.
794 Another way to specify username and password is in the URL itself.
796 Either method reveals your password to anyone who bothers to run
797 "ps". To prevent the passwords from being seen, store them in
798 .wgetrc or .netrc, and make sure to protect those files from other
799 users with "chmod". If the passwords are really important, do not
800 leave them lying in those files either---edit the files and delete
801 them after Wget has started the download.
802 --no-http-keep-alive
804 Turn off the "keep-alive" feature for HTTP downloads. Normally,
805 Wget asks the server to keep the connection open so that, when you
806 download more than one document from the same server, they get
807 transferred over the same TCP connection. This saves time and at
808 the same time reduces the load on the server.
809 This option is useful when, for some reason, persistent (keep-
811 alive) connections don't work for you, for example due to a server
812 bug or due to the inability of server-side scripts to cope with the
813 connections.
814 --no-cache
816 Disable server-side cache. In this case, Wget will send the remote
817 server an appropriate directive (Pragma: no-cache) to get the file
818 from the remote service, rather than returning the cached version.
819 This is especially useful for retrieving and flushing out-of-date
820 documents on proxy servers.
821 Caching is allowed by default.
823 --no-cookies
825 Disable the use of cookies. Cookies are a mechanism for
826 maintaining server-side state. The server sends the client a
827 cookie using the "Set-Cookie" header, and the client responds with
828 the same cookie upon further requests. Since cookies allow the
829 server owners to keep track of visitors and for sites to exchange
830 this information, some consider them a breach of privacy. The
831 default is to use cookies; however, storing cookies is not on by
832 default.
833 --load-cookies file
835 Load cookies from file before the first HTTP retrieval. file is a
836 textual file in the format originally used by Netscape's
837 cookies.txt file.
838 You will typically use this option when mirroring sites that
840 require that you be logged in to access some or all of their
841 content. The login process typically works by the web server
842 issuing an HTTP cookie upon receiving and verifying your
843 credentials. The cookie is then resent by the browser when
844 accessing that part of the site, and so proves your identity.
845 Mirroring such a site requires Wget to send the same cookies your
847 browser sends when communicating with the site. This is achieved
848 by --load-cookies---simply point Wget to the location of the
849 cookies.txt file, and it will send the same cookies your browser
850 would send in the same situation. Different browsers keep textual
851 cookie files in different locations:
852 "Netscape 4.x."
854 The cookies are in ~/.netscape/cookies.txt.
855 "Mozilla and Netscape 6.x."
857 Mozilla's cookie file is also named cookies.txt, located
858 somewhere under ~/.mozilla, in the directory of your profile.
859 The full path usually ends up looking somewhat like
860 ~/.mozilla/default/some-weird-string/cookies.txt.
861 "Internet Explorer."
863 You can produce a cookie file Wget can use by using the File
864 menu, Import and Export, Export Cookies. This has been tested
865 with Internet Explorer 5; it is not guaranteed to work with
866 earlier versions.
867 "Other browsers."
869 If you are using a different browser to create your cookies,
870 --load-cookies will only work if you can locate or produce a
871 cookie file in the Netscape format that Wget expects.
872 If you cannot use --load-cookies, there might still be an
874 alternative. If your browser supports a "cookie manager", you can
875 use it to view the cookies used when accessing the site you're
876 mirroring. Write down the name and value of the cookie, and
877 manually instruct Wget to send those cookies, bypassing the
878 "official" cookie support:
879 wget --no-cookies --header "Cookie: <name>=<value>"
881 --save-cookies file
883 Save cookies to file before exiting. This will not save cookies
884 that have expired or that have no expiry time (so-called "session
885 cookies"), but also see --keep-session-cookies.
886 --keep-session-cookies
888 When specified, causes --save-cookies to also save session cookies.
889 Session cookies are normally not saved because they are meant to be
890 kept in memory and forgotten when you exit the browser. Saving
891 them is useful on sites that require you to log in or to visit the
892 home page before you can access some pages. With this option,
893 multiple Wget runs are considered a single browser session as far
894 as the site is concerned.
895 Since the cookie file format does not normally carry session
897 cookies, Wget marks them with an expiry timestamp of 0. Wget's
898 --load-cookies recognizes those as session cookies, but it might
899 confuse other browsers. Also note that cookies so loaded will be
900 treated as other session cookies, which means that if you want
901 --save-cookies to preserve them again, you must use
902 --keep-session-cookies again.
903 --ignore-length
905 Unfortunately, some HTTP servers (CGI programs, to be more precise)
906 send out bogus "Content-Length" headers, which makes Wget go wild,
907 as it thinks not all the document was retrieved. You can spot this
908 syndrome if Wget retries getting the same document again and again,
909 each time claiming that the (otherwise normal) connection has
910 closed on the very same byte.
911 With this option, Wget will ignore the "Content-Length" header---as
913 if it never existed.
914 --header=header-line
916 Send header-line along with the rest of the headers in each HTTP
917 request. The supplied header is sent as-is, which means it must
918 contain name and value separated by colon, and must not contain
919 newlines.
920 You may define more than one additional header by specifying
922 --header more than once.
923 wget --header='Accept-Charset: iso-8859-2' \
925 --header='Accept-Language: hr' \
926 http://fly.srk.fer.hr/
927 Specification of an empty string as the header value will clear all
929 previous user-defined headers.
930 As of Wget 1.10, this option can be used to override headers
932 otherwise generated automatically. This example instructs Wget to
933 connect to localhost, but to specify foo.bar in the "Host" header:
934 wget --header="Host: foo.bar" http://localhost/
936 In versions of Wget prior to 1.10 such use of --header caused
938 sending of duplicate headers.
939 --max-redirect=number
941 Specifies the maximum number of redirections to follow for a
942 resource. The default is 20, which is usually far more than
943 necessary. However, on those occasions where you want to allow more
944 (or fewer), this is the option to use.
945 --proxy-user=user
947 --proxy-password=password
948 Specify the username user and password password for authentication
949 on a proxy server. Wget will encode them using the "basic"
950 authentication scheme.
951 Security considerations similar to those with --http-password
953 pertain here as well.
954 --referer=url
956 Include `Referer: url' header in HTTP request. Useful for
957 retrieving documents with server-side processing that assume they
958 are always being retrieved by interactive web browsers and only
959 come out properly when Referer is set to one of the pages that
960 point to them.
961 --save-headers
963 Save the headers sent by the HTTP server to the file, preceding the
964 actual contents, with an empty line as the separator.
965 -U agent-string
967 --user-agent=agent-string
968 Identify as agent-string to the HTTP server.
969 The HTTP protocol allows the clients to identify themselves using a
971 "User-Agent" header field. This enables distinguishing the WWW
972 software, usually for statistical purposes or for tracing of
973 protocol violations. Wget normally identifies as Wget/version,
974 version being the current version number of Wget.
975 However, some sites have been known to impose the policy of
977 tailoring the output according to the "User-Agent"-supplied
978 information. While this is not such a bad idea in theory, it has
979 been abused by servers denying information to clients other than
980 (historically) Netscape or, more frequently, Microsoft Internet
981 Explorer. This option allows you to change the "User-Agent" line
982 issued by Wget. Use of this option is discouraged, unless you
983 really know what you are doing.
984 Specifying empty user agent with --user-agent="" instructs Wget not
986 to send the "User-Agent" header in HTTP requests.
987 --post-data=string
989 --post-file=file
990 Use POST as the method for all HTTP requests and send the specified
991 data in the request body. --post-data sends string as data,
992 whereas --post-file sends the contents of file. Other than that,
993 they work in exactly the same way. In particular, they both expect
994 content of the form "key1=value1&key2=value2", with percent-
995 encoding for special characters; the only difference is that one
996 expects its content as a command-line parameter and the other
997 accepts its content from a file. In particular, --post-file is not
998 for transmitting files as form attachments: those must appear as
999 "key=value" data (with appropriate percent-coding) just like
1000 everything else. Wget does not currently support
1001 "multipart/form-data" for transmitting POST data; only
1002 "application/x-www-form-urlencoded". Only one of --post-data and
1003 --post-file should be specified.
1004 Please be aware that Wget needs to know the size of the POST data
1006 in advance. Therefore the argument to "--post-file" must be a
1007 regular file; specifying a FIFO or something like /dev/stdin won't
1008 work. It's not quite clear how to work around this limitation
1009 inherent in HTTP/1.0. Although HTTP/1.1 introduces chunked
1010 transfer that doesn't require knowing the request length in
1011 advance, a client can't use chunked unless it knows it's talking to
1012 an HTTP/1.1 server. And it can't know that until it receives a
1013 response, which in turn requires the request to have been completed
1014 -- a chicken-and-egg problem.
1015 Note: if Wget is redirected after the POST request is completed, it
1017 will not send the POST data to the redirected URL. This is because
1018 URLs that process POST often respond with a redirection to a
1019 regular page, which does not desire or accept POST. It is not
1020 completely clear that this behavior is optimal; if it doesn't work
1021 out, it might be changed in the future.
1022 This example shows how to log to a server using POST and then
1024 proceed to download the desired pages, presumably only accessible
1025 to authorized users:
1026 # Log in to the server. This can be done only once.
1028 wget --save-cookies cookies.txt \
1029 --post-data 'user=foo&password=bar' \
1030 http://server.com/auth.php
1031 # Now grab the page or pages we care about.
1033 wget --load-cookies cookies.txt \
1034 -p http://server.com/interesting/article.php
1035 If the server is using session cookies to track user
1037 authentication, the above will not work because --save-cookies will
1038 not save them (and neither will browsers) and the cookies.txt file
1039 will be empty. In that case use --keep-session-cookies along with
1040 --save-cookies to force saving of session cookies.
1041 --content-disposition
1043 If this is set to on, experimental (not fully-functional) support
1044 for "Content-Disposition" headers is enabled. This can currently
1045 result in extra round-trips to the server for a "HEAD" request, and
1046 is known to suffer from a few bugs, which is why it is not
1047 currently enabled by default.
1048 This option is useful for some file-downloading CGI programs that
1050 use "Content-Disposition" headers to describe what the name of a
1051 downloaded file should be.
1052 --content-on-error
1054 If this is set to on, wget will not skip the content when the
1055 server responds with a http status code that indicates error.
1056 --trust-server-names
1058 If this is set to on, on a redirect the last component of the
1059 redirection URL will be used as the local file name. By default it
1060 is used the last component in the original URL.
1061 --auth-no-challenge
1063 If this option is given, Wget will send Basic HTTP authentication
1064 information (plaintext username and password) for all requests,
1065 just like Wget 1.10.2 and prior did by default.
1066 Use of this option is not recommended, and is intended only to
1068 support some few obscure servers, which never send HTTP
1069 authentication challenges, but accept unsolicited auth info, say,
1070 in addition to form-based authentication.
1071 HTTPS (SSL/TLS) Options
1073 To support encrypted HTTP (HTTPS) downloads, Wget must be compiled with
1074 an external SSL library, currently OpenSSL. If Wget is compiled
1075 without SSL support, none of these options are available.
1076 --secure-protocol=protocol
1078 Choose the secure protocol to be used. Legal values are auto,
1079 SSLv2, SSLv3, TLSv1, TLSv1_1 and TLSv1_2. If auto is used, the SSL
1080 library is given the liberty of choosing the appropriate protocol
1081 automatically, which is achieved by sending a SSLv2 greeting and
1082 announcing support for SSLv3 and TLSv1. This is the default.
1083 Specifying SSLv2, SSLv3, TLSv1, TLSv1_1 or TLSv1_2 forces the use
1085 of the corresponding protocol. This is useful when talking to old
1086 and buggy SSL server implementations that make it hard for the
1087 underlying SSL library to choose the correct protocol version.
1088 Fortunately, such servers are quite rare.
1089 --no-check-certificate
1091 Don't check the server certificate against the available
1092 certificate authorities. Also don't require the URL host name to
1093 match the common name presented by the certificate.
1094 As of Wget 1.10, the default is to verify the server's certificate
1096 against the recognized certificate authorities, breaking the SSL
1097 handshake and aborting the download if the verification fails.
1098 Although this provides more secure downloads, it does break
1099 interoperability with some sites that worked with previous Wget
1100 versions, particularly those using self-signed, expired, or
1101 otherwise invalid certificates. This option forces an "insecure"
1102 mode of operation that turns the certificate verification errors
1103 into warnings and allows you to proceed.
1104 If you encounter "certificate verification" errors or ones saying
1106 that "common name doesn't match requested host name", you can use
1107 this option to bypass the verification and proceed with the
1108 download. Only use this option if you are otherwise convinced of
1109 the site's authenticity, or if you really don't care about the
1110 validity of its certificate. It is almost always a bad idea not to
1111 check the certificates when transmitting confidential or important
1112 data.
1113 --certificate=file
1115 Use the client certificate stored in file. This is needed for
1116 servers that are configured to require certificates from the
1117 clients that connect to them. Normally a certificate is not
1118 required and this switch is optional.
1119 --certificate-type=type
1121 Specify the type of the client certificate. Legal values are PEM
1122 (assumed by default) and DER, also known as ASN1.
1123 --private-key=file
1125 Read the private key from file. This allows you to provide the
1126 private key in a file separate from the certificate.
1127 --private-key-type=type
1129 Specify the type of the private key. Accepted values are PEM (the
1130 default) and DER.
1131 --ca-certificate=file
1133 Use file as the file with the bundle of certificate authorities
1134 ("CA") to verify the peers. The certificates must be in PEM
1135 format.
1136 Without this option Wget looks for CA certificates at the system-
1138 specified locations, chosen at OpenSSL installation time.
1139 --ca-directory=directory
1141 Specifies directory containing CA certificates in PEM format. Each
1142 file contains one CA certificate, and the file name is based on a
1143 hash value derived from the certificate. This is achieved by
1144 processing a certificate directory with the "c_rehash" utility
1145 supplied with OpenSSL. Using --ca-directory is more efficient than
1146 --ca-certificate when many certificates are installed because it
1147 allows Wget to fetch certificates on demand.
1148 Without this option Wget looks for CA certificates at the system-
1150 specified locations, chosen at OpenSSL installation time.
1151 --random-file=file
1153 Use file as the source of random data for seeding the pseudo-random
1154 number generator on systems without /dev/random.
1155 On such systems the SSL library needs an external source of
1157 randomness to initialize. Randomness may be provided by EGD (see
1158 --egd-file below) or read from an external source specified by the
1159 user. If this option is not specified, Wget looks for random data
1160 in $RANDFILE or, if that is unset, in $HOME/.rnd. If none of those
1161 are available, it is likely that SSL encryption will not be usable.
1162 If you're getting the "Could not seed OpenSSL PRNG; disabling SSL."
1164 error, you should provide random data using some of the methods
1165 described above.
1166 --egd-file=file
1168 Use file as the EGD socket. EGD stands for Entropy Gathering
1169 Daemon, a user-space program that collects data from various
1170 unpredictable system sources and makes it available to other
1171 programs that might need it. Encryption software, such as the SSL
1172 library, needs sources of non-repeating randomness to seed the
1173 random number generator used to produce cryptographically strong
1174 keys.
1175 OpenSSL allows the user to specify his own source of entropy using
1177 the "RAND_FILE" environment variable. If this variable is unset,
1178 or if the specified file does not produce enough randomness,
1179 OpenSSL will read random data from EGD socket specified using this
1180 option.
1181 If this option is not specified (and the equivalent startup command
1183 is not used), EGD is never contacted. EGD is not needed on modern
1184 Unix systems that support /dev/random.
1185 --warc-file=file
1187 Use file as the destination WARC file.
1188 --warc-header=string
1190 Use string into as the warcinfo record.
1191 --warc-max-size=size
1193 Set the maximum size of the WARC files to size.
1194 --warc-cdx
1196 Write CDX index files.
1197 --warc-dedup=file
1199 Do not store records listed in this CDX file.
1200 --no-warc-compression
1202 Do not compress WARC files with GZIP.
1203 --no-warc-digests
1205 Do not calculate SHA1 digests.
1206 --no-warc-keep-log
1208 Do not store the log file in a WARC record.
1209 --warc-tempdir=dir
1211 Specify the location for temporary files created by the WARC
1212 writer.
1213 FTP Options
1215 --ftp-user=user
1216 --ftp-password=password
1217 Specify the username user and password password on an FTP server.
1218 Without this, or the corresponding startup option, the password
1219 defaults to -wget@, normally used for anonymous FTP.
1220 Another way to specify username and password is in the URL itself.
1222 Either method reveals your password to anyone who bothers to run
1223 "ps". To prevent the passwords from being seen, store them in
1224 .wgetrc or .netrc, and make sure to protect those files from other
1225 users with "chmod". If the passwords are really important, do not
1226 leave them lying in those files either---edit the files and delete
1227 them after Wget has started the download.
1228 --no-remove-listing
1230 Don't remove the temporary .listing files generated by FTP
1231 retrievals. Normally, these files contain the raw directory
1232 listings received from FTP servers. Not removing them can be
1233 useful for debugging purposes, or when you want to be able to
1234 easily check on the contents of remote server directories (e.g. to
1235 verify that a mirror you're running is complete).
1236 Note that even though Wget writes to a known filename for this
1238 file, this is not a security hole in the scenario of a user making
1239 .listing a symbolic link to /etc/passwd or something and asking
1240 "root" to run Wget in his or her directory. Depending on the
1241 options used, either Wget will refuse to write to .listing, making
1242 the globbing/recursion/time-stamping operation fail, or the
1243 symbolic link will be deleted and replaced with the actual .listing
1244 file, or the listing will be written to a .listing.number file.
1245 Even though this situation isn't a problem, though, "root" should
1247 never run Wget in a non-trusted user's directory. A user could do
1248 something as simple as linking index.html to /etc/passwd and asking
1249 "root" to run Wget with -N or -r so the file will be overwritten.
1250 --no-glob
1252 Turn off FTP globbing. Globbing refers to the use of shell-like
1253 special characters (wildcards), like *, ?, [ and ] to retrieve more
1254 than one file from the same directory at once, like:
1255 wget ftp://gnjilux.srk.fer.hr/*.msg
1257 By default, globbing will be turned on if the URL contains a
1259 globbing character. This option may be used to turn globbing on or
1260 off permanently.
1261 You may have to quote the URL to protect it from being expanded by
1263 your shell. Globbing makes Wget look for a directory listing,
1264 which is system-specific. This is why it currently works only with
1265 Unix FTP servers (and the ones emulating Unix "ls" output).
1266 --no-passive-ftp
1268 Disable the use of the passive FTP transfer mode. Passive FTP
1269 mandates that the client connect to the server to establish the
1270 data connection rather than the other way around.
1271 If the machine is connected to the Internet directly, both passive
1273 and active FTP should work equally well. Behind most firewall and
1274 NAT configurations passive FTP has a better chance of working.
1275 However, in some rare firewall configurations, active FTP actually
1276 works when passive FTP doesn't. If you suspect this to be the
1277 case, use this option, or set "passive_ftp=off" in your init file.
1278 --preserve-permissions
1280 Preserve remote file permissions instead of permissions set by
1281 umask.
1282 --retr-symlinks
1284 By default, when retrieving FTP directories recursively and a
1285 symbolic link is encountered, the symbolic link is traversed and
1286 the pointed-to files are retrieved. Currently, Wget does not
1287 traverse symbolic links to directories to download them
1288 recursively, though this feature may be added in the future.
1289 When --retr-symlinks=no is specified, the linked-to file is not
1291 downloaded. Instead, a matching symbolic link is created on the
1292 local filesystem. The pointed-to file will not be retrieved unless
1293 this recursive retrieval would have encountered it separately and
1294 downloaded it anyway. This option poses a security risk where a
1295 malicious FTP Server may cause Wget to write to files outside of
1296 the intended directories through a specially crafted .LISTING file.
1297 Note that when retrieving a file (not a directory) because it was
1299 specified on the command-line, rather than because it was recursed
1300 to, this option has no effect. Symbolic links are always traversed
1301 in this case.
1302 Recursive Retrieval Options
1304 -r
1305 --recursive
1306 Turn on recursive retrieving. The default maximum depth is 5.
1307 -l depth
1309 --level=depth
1310 Specify recursion maximum depth level depth.
1311 --delete-after
1313 This option tells Wget to delete every single file it downloads,
1314 after having done so. It is useful for pre-fetching popular pages
1315 through a proxy, e.g.:
1316 wget -r -nd --delete-after http://whatever.com/~popular/page/
1318 The -r option is to retrieve recursively, and -nd to not create
1320 directories.
1321 Note that --delete-after deletes files on the local machine. It
1323 does not issue the DELE command to remote FTP sites, for instance.
1324 Also note that when --delete-after is specified, --convert-links is
1325 ignored, so .orig files are simply not created in the first place.
1326 -k
1328 --convert-links
1329 After the download is complete, convert the links in the document
1330 to make them suitable for local viewing. This affects not only the
1331 visible hyperlinks, but any part of the document that links to
1332 external content, such as embedded images, links to style sheets,
1333 hyperlinks to non-HTML content, etc.
1334 Each link will be changed in one of the two ways:
1336 · The links to files that have been downloaded by Wget will be
1338 changed to refer to the file they point to as a relative link.
1339 Example: if the downloaded file /foo/doc.html links to
1341 /bar/img.gif, also downloaded, then the link in doc.html will
1342 be modified to point to ../bar/img.gif. This kind of
1343 transformation works reliably for arbitrary combinations of
1344 directories.
1345 · The links to files that have not been downloaded by Wget will
1347 be changed to include host name and absolute path of the
1348 location they point to.
1349 Example: if the downloaded file /foo/doc.html links to
1351 /bar/img.gif (or to ../bar/img.gif), then the link in doc.html
1352 will be modified to point to http://hostname/bar/img.gif.
1353 Because of this, local browsing works reliably: if a linked file
1355 was downloaded, the link will refer to its local name; if it was
1356 not downloaded, the link will refer to its full Internet address
1357 rather than presenting a broken link. The fact that the former
1358 links are converted to relative links ensures that you can move the
1359 downloaded hierarchy to another directory.
1360 Note that only at the end of the download can Wget know which links
1362 have been downloaded. Because of that, the work done by -k will be
1363 performed at the end of all the downloads.
1364 -K
1366 --backup-converted
1367 When converting a file, back up the original version with a .orig
1368 suffix. Affects the behavior of -N.
1369 -m
1371 --mirror
1372 Turn on options suitable for mirroring. This option turns on
1373 recursion and time-stamping, sets infinite recursion depth and
1374 keeps FTP directory listings. It is currently equivalent to -r -N
1375 -l inf --no-remove-listing.
1376 -p
1378 --page-requisites
1379 This option causes Wget to download all the files that are
1380 necessary to properly display a given HTML page. This includes
1381 such things as inlined images, sounds, and referenced stylesheets.
1382 Ordinarily, when downloading a single HTML page, any requisite
1384 documents that may be needed to display it properly are not
1385 downloaded. Using -r together with -l can help, but since Wget
1386 does not ordinarily distinguish between external and inlined
1387 documents, one is generally left with "leaf documents" that are
1388 missing their requisites.
1389 For instance, say document 1.html contains an "<IMG>" tag
1391 referencing 1.gif and an "<A>" tag pointing to external document
1392 2.html. Say that 2.html is similar but that its image is 2.gif and
1393 it links to 3.html. Say this continues up to some arbitrarily high
1394 number.
1395 If one executes the command:
1397 wget -r -l 2 http://<site>/1.html
1399 then 1.html, 1.gif, 2.html, 2.gif, and 3.html will be downloaded.
1401 As you can see, 3.html is without its requisite 3.gif because Wget
1402 is simply counting the number of hops (up to 2) away from 1.html in
1403 order to determine where to stop the recursion. However, with this
1404 command:
1405 wget -r -l 2 -p http://<site>/1.html
1407 all the above files and 3.html's requisite 3.gif will be
1409 downloaded. Similarly,
1410 wget -r -l 1 -p http://<site>/1.html
1412 will cause 1.html, 1.gif, 2.html, and 2.gif to be downloaded. One
1414 might think that:
1415 wget -r -l 0 -p http://<site>/1.html
1417 would download just 1.html and 1.gif, but unfortunately this is not
1419 the case, because -l 0 is equivalent to -l inf---that is, infinite
1420 recursion. To download a single HTML page (or a handful of them,
1421 all specified on the command-line or in a -i URL input file) and
1422 its (or their) requisites, simply leave off -r and -l:
1423 wget -p http://<site>/1.html
1425 Note that Wget will behave as if -r had been specified, but only
1427 that single page and its requisites will be downloaded. Links from
1428 that page to external documents will not be followed. Actually, to
1429 download a single page and all its requisites (even if they exist
1430 on separate websites), and make sure the lot displays properly
1431 locally, this author likes to use a few options in addition to -p:
1432 wget -E -H -k -K -p http://<site>/<document>
1434 To finish off this topic, it's worth knowing that Wget's idea of an
1436 external document link is any URL specified in an "<A>" tag, an
1437 "<AREA>" tag, or a "<LINK>" tag other than "<LINK
1438 REL="stylesheet">".
1439 --strict-comments
1441 Turn on strict parsing of HTML comments. The default is to
1442 terminate comments at the first occurrence of -->.
1443 According to specifications, HTML comments are expressed as SGML
1445 declarations. Declaration is special markup that begins with <!
1446 and ends with >, such as <!DOCTYPE ...>, that may contain comments
1447 between a pair of -- delimiters. HTML comments are "empty
1448 declarations", SGML declarations without any non-comment text.
1449 Therefore, <!--foo--> is a valid comment, and so is <!--one--
1450 --two-->, but <!--1--2--> is not.
1451 On the other hand, most HTML writers don't perceive comments as
1453 anything other than text delimited with <!-- and -->, which is not
1454 quite the same. For example, something like <!------------> works
1455 as a valid comment as long as the number of dashes is a multiple of
1456 four (!). If not, the comment technically lasts until the next --,
1457 which may be at the other end of the document. Because of this,
1458 many popular browsers completely ignore the specification and
1459 implement what users have come to expect: comments delimited with
1460 <!-- and -->.
1461 Until version 1.9, Wget interpreted comments strictly, which
1463 resulted in missing links in many web pages that displayed fine in
1464 browsers, but had the misfortune of containing non-compliant
1465 comments. Beginning with version 1.9, Wget has joined the ranks of
1466 clients that implements "naive" comments, terminating each comment
1467 at the first occurrence of -->.
1468 If, for whatever reason, you want strict comment parsing, use this
1470 option to turn it on.
1471 Recursive Accept/Reject Options
1473 -A acclist --accept acclist
1474 -R rejlist --reject rejlist
1475 Specify comma-separated lists of file name suffixes or patterns to
1476 accept or reject. Note that if any of the wildcard characters, *,
1477 ?, [ or ], appear in an element of acclist or rejlist, it will be
1478 treated as a pattern, rather than a suffix.
1479 --accept-regex urlregex
1481 --reject-regex urlregex
1482 Specify a regular expression to accept or reject the complete URL.
1483 --regex-type regextype
1485 Specify the regular expression type. Possible types are posix or
1486 pcre. Note that to be able to use pcre type, wget has to be
1487 compiled with libpcre support.
1488 -D domain-list
1490 --domains=domain-list
1491 Set domains to be followed. domain-list is a comma-separated list
1492 of domains. Note that it does not turn on -H.
1493 --exclude-domains domain-list
1495 Specify the domains that are not to be followed.
1496 --follow-ftp
1498 Follow FTP links from HTML documents. Without this option, Wget
1499 will ignore all the FTP links.
1500 --follow-tags=list
1502 Wget has an internal table of HTML tag / attribute pairs that it
1503 considers when looking for linked documents during a recursive
1504 retrieval. If a user wants only a subset of those tags to be
1505 considered, however, he or she should be specify such tags in a
1506 comma-separated list with this option.
1507 --ignore-tags=list
1509 This is the opposite of the --follow-tags option. To skip certain
1510 HTML tags when recursively looking for documents to download,
1511 specify them in a comma-separated list.
1512 In the past, this option was the best bet for downloading a single
1514 page and its requisites, using a command-line like:
1515 wget --ignore-tags=a,area -H -k -K -r http://<site>/<document>
1517 However, the author of this option came across a page with tags
1519 like "<LINK REL="home" HREF="/">" and came to the realization that
1520 specifying tags to ignore was not enough. One can't just tell Wget
1521 to ignore "<LINK>", because then stylesheets will not be
1522 downloaded. Now the best bet for downloading a single page and its
1523 requisites is the dedicated --page-requisites option.
1524 --ignore-case
1526 Ignore case when matching files and directories. This influences
1527 the behavior of -R, -A, -I, and -X options, as well as globbing
1528 implemented when downloading from FTP sites. For example, with
1529 this option, -A *.txt will match file1.txt, but also file2.TXT,
1530 file3.TxT, and so on.
1531 -H
1533 --span-hosts
1534 Enable spanning across hosts when doing recursive retrieving.
1535 -L
1537 --relative
1538 Follow relative links only. Useful for retrieving a specific home
1539 page without any distractions, not even those from the same hosts.
1540 -I list
1542 --include-directories=list
1543 Specify a comma-separated list of directories you wish to follow
1544 when downloading. Elements of list may contain wildcards.
1545 -X list
1547 --exclude-directories=list
1548 Specify a comma-separated list of directories you wish to exclude
1549 from download. Elements of list may contain wildcards.
1550 -np
1552 --no-parent
1553 Do not ever ascend to the parent directory when retrieving
1554 recursively. This is a useful option, since it guarantees that
1555 only the files below a certain hierarchy will be downloaded.
1556
1558 Wget supports proxies for both HTTP and FTP retrievals. The standard
1559 way to specify proxy location, which Wget recognizes, is using the
1560 following environment variables:
1561 http_proxy
1563 https_proxy
1564 If set, the http_proxy and https_proxy variables should contain the
1565 URLs of the proxies for HTTP and HTTPS connections respectively.
1566 ftp_proxy
1568 This variable should contain the URL of the proxy for FTP
1569 connections. It is quite common that http_proxy and ftp_proxy are
1570 set to the same URL.
1571 no_proxy
1573 This variable should contain a comma-separated list of domain
1574 extensions proxy should not be used for. For instance, if the
1575 value of no_proxy is .mit.edu, proxy will not be used to retrieve
1576 documents from MIT.
1577
1579 Wget may return one of several error codes if it encounters problems.
1580 0 No problems occurred.
1582 1 Generic error code.
1584 2 Parse error---for instance, when parsing command-line options, the
1586 .wgetrc or .netrc...
1587 3 File I/O error.
1589 4 Network failure.
1591 5 SSL verification failure.
1593 6 Username/password authentication failure.
1595 7 Protocol errors.
1597 8 Server issued an error response.
1599 With the exceptions of 0 and 1, the lower-numbered exit codes take
1601 precedence over higher-numbered ones, when multiple types of errors are
1602 encountered.
1603 In versions of Wget prior to 1.12, Wget's exit status tended to be
1605 unhelpful and inconsistent. Recursive downloads would virtually always
1606 return 0 (success), regardless of any issues encountered, and non-
1607 recursive fetches only returned the status corresponding to the most
1608 recently-attempted download.
1609
1611 /etc/wgetrc
1612 Default location of the global startup file.
1613 .wgetrc
1615 User startup file.
1616
1618 You are welcome to submit bug reports via the GNU Wget bug tracker (see
1619 <http://wget.addictivecode.org/BugTracker>).
1620 Before actually submitting a bug report, please try to follow a few
1622 simple guidelines.
1623 1. Please try to ascertain that the behavior you see really is a bug.
1625 If Wget crashes, it's a bug. If Wget does not behave as
1626 documented, it's a bug. If things work strange, but you are not
1627 sure about the way they are supposed to work, it might well be a
1628 bug, but you might want to double-check the documentation and the
1629 mailing lists.
1630 2. Try to repeat the bug in as simple circumstances as possible. E.g.
1632 if Wget crashes while downloading wget -rl0 -kKE -t5 --no-proxy
1633 http://yoyodyne.com -o /tmp/log, you should try to see if the crash
1634 is repeatable, and if will occur with a simpler set of options.
1635 You might even try to start the download at the page where the
1636 crash occurred to see if that page somehow triggered the crash.
1637 Also, while I will probably be interested to know the contents of
1639 your .wgetrc file, just dumping it into the debug message is
1640 probably a bad idea. Instead, you should first try to see if the
1641 bug repeats with .wgetrc moved out of the way. Only if it turns
1642 out that .wgetrc settings affect the bug, mail me the relevant
1643 parts of the file.
1644 3. Please start Wget with -d option and send us the resulting output
1646 (or relevant parts thereof). If Wget was compiled without debug
1647 support, recompile it---it is much easier to trace bugs with debug
1648 support on.
1649 Note: please make sure to remove any potentially sensitive
1651 information from the debug log before sending it to the bug
1652 address. The "-d" won't go out of its way to collect sensitive
1653 information, but the log will contain a fairly complete transcript
1654 of Wget's communication with the server, which may include
1655 passwords and pieces of downloaded data. Since the bug address is
1656 publically archived, you may assume that all bug reports are
1657 visible to the public.
1658 4. If Wget has crashed, try to run it in a debugger, e.g. "gdb `which
1660 wget` core" and type "where" to get the backtrace. This may not
1661 work if the system administrator has disabled core files, but it is
1662 safe to try.
1663
1665 This is not the complete manual for GNU Wget. For more complete
1666 information, including more detailed explanations of some of the
1667 options, and a number of commands available for use with .wgetrc files
1668 and the -e option, see the GNU Info entry for wget.
1669
1671 Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
1672
1674 Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004,
1675 2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation, Inc.
1676 Permission is granted to copy, distribute and/or modify this document
1678 under the terms of the GNU Free Documentation License, Version 1.2 or
1679 any later version published by the Free Software Foundation; with no
1680 Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A
1681 copy of the license is included in the section entitled "GNU Free
1682 Documentation License".
1683GNU Wget 1.14 2019-05-15 WGET(1)
