1WGET(1) GNU Wget WGET(1)
2
6 Wget - The non-interactive network downloader.
7
9 wget [option]... [URL]...
10
12 GNU Wget is a free utility for non-interactive download of files from
13 the Web. It supports HTTP, HTTPS, and FTP protocols, as well as
14 retrieval through HTTP proxies.
15 Wget is non-interactive, meaning that it can work in the background,
17 while the user is not logged on. This allows you to start a retrieval
18 and disconnect from the system, letting Wget finish the work. By
19 contrast, most of the Web browsers require constant user's presence,
20 which can be a great hindrance when transferring a lot of data.
21 Wget can follow links in HTML, XHTML, and CSS pages, to create local
23 versions of remote web sites, fully recreating the directory structure
24 of the original site. This is sometimes referred to as "recursive
25 downloading." While doing that, Wget respects the Robot Exclusion
26 Standard (/robots.txt). Wget can be instructed to convert the links in
27 downloaded files to point at the local files, for offline viewing.
28 Wget has been designed for robustness over slow or unstable network
30 connections; if a download fails due to a network problem, it will keep
31 retrying until the whole file has been retrieved. If the server
32 supports regetting, it will instruct the server to continue the
33 download from where it left off.
34
36 Option Syntax
37 Since Wget uses GNU getopt to process command-line arguments, every
38 option has a long form along with the short one. Long options are more
39 convenient to remember, but take time to type. You may freely mix
40 different option styles, or specify options after the command-line
41 arguments. Thus you may write:
42 wget -r --tries=10 http://fly.srk.fer.hr/ -o log
44 The space between the option accepting an argument and the argument may
46 be omitted. Instead of -o log you can write -olog.
47 You may put several options that do not require arguments together,
49 like:
50 wget -drc <URL>
52 This is completely equivalent to:
54 wget -d -r -c <URL>
56 Since the options can be specified after the arguments, you may
58 terminate them with --. So the following will try to download URL -x,
59 reporting failure to log:
60 wget -o log -- -x
62 The options that accept comma-separated lists all respect the
64 convention that specifying an empty list clears its value. This can be
65 useful to clear the .wgetrc settings. For instance, if your .wgetrc
66 sets "exclude_directories" to /cgi-bin, the following example will
67 first reset it, and then set it to exclude /~nobody and /~somebody.
68 You can also clear the lists in .wgetrc.
69 wget -X " -X /~nobody,/~somebody
71 Most options that do not accept arguments are boolean options, so named
73 because their state can be captured with a yes-or-no ("boolean")
74 variable. For example, --follow-ftp tells Wget to follow FTP links
75 from HTML files and, on the other hand, --no-glob tells it not to
76 perform file globbing on FTP URLs. A boolean option is either
77 affirmative or negative (beginning with --no). All such options share
78 several properties.
79 Unless stated otherwise, it is assumed that the default behavior is the
81 opposite of what the option accomplishes. For example, the documented
82 existence of --follow-ftp assumes that the default is to not follow FTP
83 links from HTML pages.
84 Affirmative options can be negated by prepending the --no- to the
86 option name; negative options can be negated by omitting the --no-
87 prefix. This might seem superfluous---if the default for an
88 affirmative option is to not do something, then why provide a way to
89 explicitly turn it off? But the startup file may in fact change the
90 default. For instance, using "follow_ftp = on" in .wgetrc makes Wget
91 follow FTP links by default, and using --no-follow-ftp is the only way
92 to restore the factory default from the command line.
93 Basic Startup Options
95 -V
96 --version
97 Display the version of Wget.
98 -h
100 --help
101 Print a help message describing all of Wget's command-line options.
102 -b
104 --background
105 Go to background immediately after startup. If no output file is
106 specified via the -o, output is redirected to wget-log.
107 -e command
109 --execute command
110 Execute command as if it were a part of .wgetrc. A command thus
111 invoked will be executed after the commands in .wgetrc, thus taking
112 precedence over them. If you need to specify more than one wgetrc
113 command, use multiple instances of -e.
114 Logging and Input File Options
116 -o logfile
117 --output-file=logfile
118 Log all messages to logfile. The messages are normally reported to
119 standard error.
120 -a logfile
122 --append-output=logfile
123 Append to logfile. This is the same as -o, only it appends to
124 logfile instead of overwriting the old log file. If logfile does
125 not exist, a new file is created.
126 -d
128 --debug
129 Turn on debug output, meaning various information important to the
130 developers of Wget if it does not work properly. Your system
131 administrator may have chosen to compile Wget without debug
132 support, in which case -d will not work. Please note that
133 compiling with debug support is always safe---Wget compiled with
134 the debug support will not print any debug info unless requested
135 with -d.
136 -q
138 --quiet
139 Turn off Wget's output.
140 -v
142 --verbose
143 Turn on verbose output, with all the available data. The default
144 output is verbose.
145 -nv
147 --no-verbose
148 Turn off verbose without being completely quiet (use -q for that),
149 which means that error messages and basic information still get
150 printed.
151 --report-speed=type
153 Output bandwidth as type. The only accepted value is bits.
154 -i file
156 --input-file=file
157 Read URLs from a local or external file. If - is specified as
158 file, URLs are read from the standard input. (Use ./- to read from
159 a file literally named -.)
160 If this function is used, no URLs need be present on the command
162 line. If there are URLs both on the command line and in an input
163 file, those on the command lines will be the first ones to be
164 retrieved. If --force-html is not specified, then file should
165 consist of a series of URLs, one per line.
166 However, if you specify --force-html, the document will be regarded
168 as html. In that case you may have problems with relative links,
169 which you can solve either by adding "<base href="url">" to the
170 documents or by specifying --base=url on the command line.
171 If the file is an external one, the document will be automatically
173 treated as html if the Content-Type matches text/html.
174 Furthermore, the file's location will be implicitly used as base
175 href if none was specified.
176 --input-metalink=file
178 Downloads files covered in local Metalink file. Metalink version 3
179 and 4 are supported.
180 --keep-badhash
182 Keeps downloaded Metalink's files with a bad hash. It appends
183 .badhash to the name of Metalink's files which have a checksum
184 mismatch, except without overwriting existing files.
185 --metalink-over-http
187 Issues HTTP HEAD request instead of GET and extracts Metalink
188 metadata from response headers. Then it switches to Metalink
189 download. If no valid Metalink metadata is found, it falls back to
190 ordinary HTTP download. Enables Content-Type:
191 application/metalink4+xml files download/processing.
192 --metalink-index=number
194 Set the Metalink application/metalink4+xml metaurl ordinal NUMBER.
195 From 1 to the total number of "application/metalink4+xml"
196 available. Specify 0 or inf to choose the first good one.
197 Metaurls, such as those from a --metalink-over-http, may have been
198 sorted by priority key's value; keep this in mind to choose the
199 right NUMBER.
200 --preferred-location
202 Set preferred location for Metalink resources. This has effect if
203 multiple resources with same priority are available.
204 --xattr
206 Enable use of file system's extended attributes to save the
207 original URL and the Referer HTTP header value if used.
208 Be aware that the URL might contain private information like access
210 tokens or credentials.
211 -F
213 --force-html
214 When input is read from a file, force it to be treated as an HTML
215 file. This enables you to retrieve relative links from existing
216 HTML files on your local disk, by adding "<base href="url">" to
217 HTML, or using the --base command-line option.
218 -B URL
220 --base=URL
221 Resolves relative links using URL as the point of reference, when
222 reading links from an HTML file specified via the -i/--input-file
223 option (together with --force-html, or when the input file was
224 fetched remotely from a server describing it as HTML). This is
225 equivalent to the presence of a "BASE" tag in the HTML input file,
226 with URL as the value for the "href" attribute.
227 For instance, if you specify http://foo/bar/a.html for URL, and
229 Wget reads ../baz/b.html from the input file, it would be resolved
230 to http://foo/baz/b.html.
231 --config=FILE
233 Specify the location of a startup file you wish to use instead of
234 the default one(s). Use --no-config to disable reading of config
235 files. If both --config and --no-config are given, --no-config is
236 ignored.
237 --rejected-log=logfile
239 Logs all URL rejections to logfile as comma separated values. The
240 values include the reason of rejection, the URL and the parent URL
241 it was found in.
242 Download Options
244 --bind-address=ADDRESS
245 When making client TCP/IP connections, bind to ADDRESS on the local
246 machine. ADDRESS may be specified as a hostname or IP address.
247 This option can be useful if your machine is bound to multiple IPs.
248 --bind-dns-address=ADDRESS
250 [libcares only] This address overrides the route for DNS requests.
251 If you ever need to circumvent the standard settings from
252 /etc/resolv.conf, this option together with --dns-servers is your
253 friend. ADDRESS must be specified either as IPv4 or IPv6 address.
254 Wget needs to be built with libcares for this option to be
255 available.
256 --dns-servers=ADDRESSES
258 [libcares only] The given address(es) override the standard
259 nameserver addresses, e.g. as configured in /etc/resolv.conf.
260 ADDRESSES may be specified either as IPv4 or IPv6 addresses, comma-
261 separated. Wget needs to be built with libcares for this option to
262 be available.
263 -t number
265 --tries=number
266 Set number of tries to number. Specify 0 or inf for infinite
267 retrying. The default is to retry 20 times, with the exception of
268 fatal errors like "connection refused" or "not found" (404), which
269 are not retried.
270 -O file
272 --output-document=file
273 The documents will not be written to the appropriate files, but all
274 will be concatenated together and written to file. If - is used as
275 file, documents will be printed to standard output, disabling link
276 conversion. (Use ./- to print to a file literally named -.)
277 Use of -O is not intended to mean simply "use the name file instead
279 of the one in the URL;" rather, it is analogous to shell
280 redirection: wget -O file http://foo is intended to work like wget
281 -O - http://foo > file; file will be truncated immediately, and all
282 downloaded content will be written there.
283 For this reason, -N (for timestamp-checking) is not supported in
285 combination with -O: since file is always newly created, it will
286 always have a very new timestamp. A warning will be issued if this
287 combination is used.
288 Similarly, using -r or -p with -O may not work as you expect: Wget
290 won't just download the first file to file and then download the
291 rest to their normal names: all downloaded content will be placed
292 in file. This was disabled in version 1.11, but has been reinstated
293 (with a warning) in 1.11.2, as there are some cases where this
294 behavior can actually have some use.
295 A combination with -nc is only accepted if the given output file
297 does not exist.
298 Note that a combination with -k is only permitted when downloading
300 a single document, as in that case it will just convert all
301 relative URIs to external ones; -k makes no sense for multiple URIs
302 when they're all being downloaded to a single file; -k can be used
303 only when the output is a regular file.
304 -nc
306 --no-clobber
307 If a file is downloaded more than once in the same directory,
308 Wget's behavior depends on a few options, including -nc. In
309 certain cases, the local file will be clobbered, or overwritten,
310 upon repeated download. In other cases it will be preserved.
311 When running Wget without -N, -nc, -r, or -p, downloading the same
313 file in the same directory will result in the original copy of file
314 being preserved and the second copy being named file.1. If that
315 file is downloaded yet again, the third copy will be named file.2,
316 and so on. (This is also the behavior with -nd, even if -r or -p
317 are in effect.) When -nc is specified, this behavior is
318 suppressed, and Wget will refuse to download newer copies of file.
319 Therefore, ""no-clobber"" is actually a misnomer in this
320 mode---it's not clobbering that's prevented (as the numeric
321 suffixes were already preventing clobbering), but rather the
322 multiple version saving that's prevented.
323 When running Wget with -r or -p, but without -N, -nd, or -nc, re-
325 downloading a file will result in the new copy simply overwriting
326 the old. Adding -nc will prevent this behavior, instead causing
327 the original version to be preserved and any newer copies on the
328 server to be ignored.
329 When running Wget with -N, with or without -r or -p, the decision
331 as to whether or not to download a newer copy of a file depends on
332 the local and remote timestamp and size of the file. -nc may not
333 be specified at the same time as -N.
334 A combination with -O/--output-document is only accepted if the
336 given output file does not exist.
337 Note that when -nc is specified, files with the suffixes .html or
339 .htm will be loaded from the local disk and parsed as if they had
340 been retrieved from the Web.
341 --backups=backups
343 Before (over)writing a file, back up an existing file by adding a
344 .1 suffix (_1 on VMS) to the file name. Such backup files are
345 rotated to .2, .3, and so on, up to backups (and lost beyond that).
346 --no-netrc
348 Do not try to obtain credentials from .netrc file. By default
349 .netrc file is searched for credentials in case none have been
350 passed on command line and authentication is required.
351 -c
353 --continue
354 Continue getting a partially-downloaded file. This is useful when
355 you want to finish up a download started by a previous instance of
356 Wget, or by another program. For instance:
357 wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z
359 If there is a file named ls-lR.Z in the current directory, Wget
361 will assume that it is the first portion of the remote file, and
362 will ask the server to continue the retrieval from an offset equal
363 to the length of the local file.
364 Note that you don't need to specify this option if you just want
366 the current invocation of Wget to retry downloading a file should
367 the connection be lost midway through. This is the default
368 behavior. -c only affects resumption of downloads started prior to
369 this invocation of Wget, and whose local files are still sitting
370 around.
371 Without -c, the previous example would just download the remote
373 file to ls-lR.Z.1, leaving the truncated ls-lR.Z file alone.
374 If you use -c on a non-empty file, and the server does not support
376 continued downloading, Wget will restart the download from scratch
377 and overwrite the existing file entirely.
378 Beginning with Wget 1.7, if you use -c on a file which is of equal
380 size as the one on the server, Wget will refuse to download the
381 file and print an explanatory message. The same happens when the
382 file is smaller on the server than locally (presumably because it
383 was changed on the server since your last download
384 attempt)---because "continuing" is not meaningful, no download
385 occurs.
386 On the other side of the coin, while using -c, any file that's
388 bigger on the server than locally will be considered an incomplete
389 download and only "(length(remote) - length(local))" bytes will be
390 downloaded and tacked onto the end of the local file. This
391 behavior can be desirable in certain cases---for instance, you can
392 use wget -c to download just the new portion that's been appended
393 to a data collection or log file.
394 However, if the file is bigger on the server because it's been
396 changed, as opposed to just appended to, you'll end up with a
397 garbled file. Wget has no way of verifying that the local file is
398 really a valid prefix of the remote file. You need to be
399 especially careful of this when using -c in conjunction with -r,
400 since every file will be considered as an "incomplete download"
401 candidate.
402 Another instance where you'll get a garbled file if you try to use
404 -c is if you have a lame HTTP proxy that inserts a "transfer
405 interrupted" string into the local file. In the future a
406 "rollback" option may be added to deal with this case.
407 Note that -c only works with FTP servers and with HTTP servers that
409 support the "Range" header.
410 --start-pos=OFFSET
412 Start downloading at zero-based position OFFSET. Offset may be
413 expressed in bytes, kilobytes with the `k' suffix, or megabytes
414 with the `m' suffix, etc.
415 --start-pos has higher precedence over --continue. When
417 --start-pos and --continue are both specified, wget will emit a
418 warning then proceed as if --continue was absent.
419 Server support for continued download is required, otherwise
421 --start-pos cannot help. See -c for details.
422 --progress=type
424 Select the type of the progress indicator you wish to use. Legal
425 indicators are "dot" and "bar".
426 The "bar" indicator is used by default. It draws an ASCII progress
428 bar graphics (a.k.a "thermometer" display) indicating the status of
429 retrieval. If the output is not a TTY, the "dot" bar will be used
430 by default.
431 Use --progress=dot to switch to the "dot" display. It traces the
433 retrieval by printing dots on the screen, each dot representing a
434 fixed amount of downloaded data.
435 The progress type can also take one or more parameters. The
437 parameters vary based on the type selected. Parameters to type are
438 passed by appending them to the type sperated by a colon (:) like
439 this: --progress=type:parameter1:parameter2.
440 When using the dotted retrieval, you may set the style by
442 specifying the type as dot:style. Different styles assign
443 different meaning to one dot. With the "default" style each dot
444 represents 1K, there are ten dots in a cluster and 50 dots in a
445 line. The "binary" style has a more "computer"-like
446 orientation---8K dots, 16-dots clusters and 48 dots per line (which
447 makes for 384K lines). The "mega" style is suitable for
448 downloading large files---each dot represents 64K retrieved, there
449 are eight dots in a cluster, and 48 dots on each line (so each line
450 contains 3M). If "mega" is not enough then you can use the "giga"
451 style---each dot represents 1M retrieved, there are eight dots in a
452 cluster, and 32 dots on each line (so each line contains 32M).
453 With --progress=bar, there are currently two possible parameters,
455 force and noscroll.
456 When the output is not a TTY, the progress bar always falls back to
458 "dot", even if --progress=bar was passed to Wget during invocation.
459 This behaviour can be overridden and the "bar" output forced by
460 using the "force" parameter as --progress=bar:force.
461 By default, the bar style progress bar scroll the name of the file
463 from left to right for the file being downloaded if the filename
464 exceeds the maximum length allotted for its display. In certain
465 cases, such as with --progress=bar:force, one may not want the
466 scrolling filename in the progress bar. By passing the "noscroll"
467 parameter, Wget can be forced to display as much of the filename as
468 possible without scrolling through it.
469 Note that you can set the default style using the "progress"
471 command in .wgetrc. That setting may be overridden from the
472 command line. For example, to force the bar output without
473 scrolling, use --progress=bar:force:noscroll.
474 --show-progress
476 Force wget to display the progress bar in any verbosity.
477 By default, wget only displays the progress bar in verbose mode.
479 One may however, want wget to display the progress bar on screen in
480 conjunction with any other verbosity modes like --no-verbose or
481 --quiet. This is often a desired a property when invoking wget to
482 download several small/large files. In such a case, wget could
483 simply be invoked with this parameter to get a much cleaner output
484 on the screen.
485 This option will also force the progress bar to be printed to
487 stderr when used alongside the --logfile option.
488 -N
490 --timestamping
491 Turn on time-stamping.
492 --no-if-modified-since
494 Do not send If-Modified-Since header in -N mode. Send preliminary
495 HEAD request instead. This has only effect in -N mode.
496 --no-use-server-timestamps
498 Don't set the local file's timestamp by the one on the server.
499 By default, when a file is downloaded, its timestamps are set to
501 match those from the remote file. This allows the use of
502 --timestamping on subsequent invocations of wget. However, it is
503 sometimes useful to base the local file's timestamp on when it was
504 actually downloaded; for that purpose, the
505 --no-use-server-timestamps option has been provided.
506 -S
508 --server-response
509 Print the headers sent by HTTP servers and responses sent by FTP
510 servers.
511 --spider
513 When invoked with this option, Wget will behave as a Web spider,
514 which means that it will not download the pages, just check that
515 they are there. For example, you can use Wget to check your
516 bookmarks:
517 wget --spider --force-html -i bookmarks.html
519 This feature needs much more work for Wget to get close to the
521 functionality of real web spiders.
522 -T seconds
524 --timeout=seconds
525 Set the network timeout to seconds seconds. This is equivalent to
526 specifying --dns-timeout, --connect-timeout, and --read-timeout,
527 all at the same time.
528 When interacting with the network, Wget can check for timeout and
530 abort the operation if it takes too long. This prevents anomalies
531 like hanging reads and infinite connects. The only timeout enabled
532 by default is a 900-second read timeout. Setting a timeout to 0
533 disables it altogether. Unless you know what you are doing, it is
534 best not to change the default timeout settings.
535 All timeout-related options accept decimal values, as well as
537 subsecond values. For example, 0.1 seconds is a legal (though
538 unwise) choice of timeout. Subsecond timeouts are useful for
539 checking server response times or for testing network latency.
540 --dns-timeout=seconds
542 Set the DNS lookup timeout to seconds seconds. DNS lookups that
543 don't complete within the specified time will fail. By default,
544 there is no timeout on DNS lookups, other than that implemented by
545 system libraries.
546 --connect-timeout=seconds
548 Set the connect timeout to seconds seconds. TCP connections that
549 take longer to establish will be aborted. By default, there is no
550 connect timeout, other than that implemented by system libraries.
551 --read-timeout=seconds
553 Set the read (and write) timeout to seconds seconds. The "time" of
554 this timeout refers to idle time: if, at any point in the download,
555 no data is received for more than the specified number of seconds,
556 reading fails and the download is restarted. This option does not
557 directly affect the duration of the entire download.
558 Of course, the remote server may choose to terminate the connection
560 sooner than this option requires. The default read timeout is 900
561 seconds.
562 --limit-rate=amount
564 Limit the download speed to amount bytes per second. Amount may be
565 expressed in bytes, kilobytes with the k suffix, or megabytes with
566 the m suffix. For example, --limit-rate=20k will limit the
567 retrieval rate to 20KB/s. This is useful when, for whatever
568 reason, you don't want Wget to consume the entire available
569 bandwidth.
570 This option allows the use of decimal numbers, usually in
572 conjunction with power suffixes; for example, --limit-rate=2.5k is
573 a legal value.
574 Note that Wget implements the limiting by sleeping the appropriate
576 amount of time after a network read that took less time than
577 specified by the rate. Eventually this strategy causes the TCP
578 transfer to slow down to approximately the specified rate.
579 However, it may take some time for this balance to be achieved, so
580 don't be surprised if limiting the rate doesn't work well with very
581 small files.
582 -w seconds
584 --wait=seconds
585 Wait the specified number of seconds between the retrievals. Use
586 of this option is recommended, as it lightens the server load by
587 making the requests less frequent. Instead of in seconds, the time
588 can be specified in minutes using the "m" suffix, in hours using
589 "h" suffix, or in days using "d" suffix.
590 Specifying a large value for this option is useful if the network
592 or the destination host is down, so that Wget can wait long enough
593 to reasonably expect the network error to be fixed before the
594 retry. The waiting interval specified by this function is
595 influenced by "--random-wait", which see.
596 --waitretry=seconds
598 If you don't want Wget to wait between every retrieval, but only
599 between retries of failed downloads, you can use this option. Wget
600 will use linear backoff, waiting 1 second after the first failure
601 on a given file, then waiting 2 seconds after the second failure on
602 that file, up to the maximum number of seconds you specify.
603 By default, Wget will assume a value of 10 seconds.
605 --random-wait
607 Some web sites may perform log analysis to identify retrieval
608 programs such as Wget by looking for statistically significant
609 similarities in the time between requests. This option causes the
610 time between requests to vary between 0.5 and 1.5 * wait seconds,
611 where wait was specified using the --wait option, in order to mask
612 Wget's presence from such analysis.
613 A 2001 article in a publication devoted to development on a popular
615 consumer platform provided code to perform this analysis on the
616 fly. Its author suggested blocking at the class C address level to
617 ensure automated retrieval programs were blocked despite changing
618 DHCP-supplied addresses.
619 The --random-wait option was inspired by this ill-advised
621 recommendation to block many unrelated users from a web site due to
622 the actions of one.
623 --no-proxy
625 Don't use proxies, even if the appropriate *_proxy environment
626 variable is defined.
627 -Q quota
629 --quota=quota
630 Specify download quota for automatic retrievals. The value can be
631 specified in bytes (default), kilobytes (with k suffix), or
632 megabytes (with m suffix).
633 Note that quota will never affect downloading a single file. So if
635 you specify wget -Q10k https://example.com/ls-lR.gz, all of the
636 ls-lR.gz will be downloaded. The same goes even when several URLs
637 are specified on the command-line. However, quota is respected
638 when retrieving either recursively, or from an input file. Thus
639 you may safely type wget -Q2m -i sites---download will be aborted
640 when the quota is exceeded.
641 Setting quota to 0 or to inf unlimits the download quota.
643 --no-dns-cache
645 Turn off caching of DNS lookups. Normally, Wget remembers the IP
646 addresses it looked up from DNS so it doesn't have to repeatedly
647 contact the DNS server for the same (typically small) set of hosts
648 it retrieves from. This cache exists in memory only; a new Wget
649 run will contact DNS again.
650 However, it has been reported that in some situations it is not
652 desirable to cache host names, even for the duration of a short-
653 running application like Wget. With this option Wget issues a new
654 DNS lookup (more precisely, a new call to "gethostbyname" or
655 "getaddrinfo") each time it makes a new connection. Please note
656 that this option will not affect caching that might be performed by
657 the resolving library or by an external caching layer, such as
658 NSCD.
659 If you don't understand exactly what this option does, you probably
661 won't need it.
662 --restrict-file-names=modes
664 Change which characters found in remote URLs must be escaped during
665 generation of local filenames. Characters that are restricted by
666 this option are escaped, i.e. replaced with %HH, where HH is the
667 hexadecimal number that corresponds to the restricted character.
668 This option may also be used to force all alphabetical cases to be
669 either lower- or uppercase.
670 By default, Wget escapes the characters that are not valid or safe
672 as part of file names on your operating system, as well as control
673 characters that are typically unprintable. This option is useful
674 for changing these defaults, perhaps because you are downloading to
675 a non-native partition, or because you want to disable escaping of
676 the control characters, or you want to further restrict characters
677 to only those in the ASCII range of values.
678 The modes are a comma-separated set of text values. The acceptable
680 values are unix, windows, nocontrol, ascii, lowercase, and
681 uppercase. The values unix and windows are mutually exclusive (one
682 will override the other), as are lowercase and uppercase. Those
683 last are special cases, as they do not change the set of characters
684 that would be escaped, but rather force local file paths to be
685 converted either to lower- or uppercase.
686 When "unix" is specified, Wget escapes the character / and the
688 control characters in the ranges 0--31 and 128--159. This is the
689 default on Unix-like operating systems.
690 When "windows" is given, Wget escapes the characters \, |, /, :, ?,
692 ", *, <, >, and the control characters in the ranges 0--31 and
693 128--159. In addition to this, Wget in Windows mode uses + instead
694 of : to separate host and port in local file names, and uses @
695 instead of ? to separate the query portion of the file name from
696 the rest. Therefore, a URL that would be saved as
697 www.xemacs.org:4300/search.pl?input=blah in Unix mode would be
698 saved as www.xemacs.org+4300/search.pl@input=blah in Windows mode.
699 This mode is the default on Windows.
700 If you specify nocontrol, then the escaping of the control
702 characters is also switched off. This option may make sense when
703 you are downloading URLs whose names contain UTF-8 characters, on a
704 system which can save and display filenames in UTF-8 (some possible
705 byte values used in UTF-8 byte sequences fall in the range of
706 values designated by Wget as "controls").
707 The ascii mode is used to specify that any bytes whose values are
709 outside the range of ASCII characters (that is, greater than 127)
710 shall be escaped. This can be useful when saving filenames whose
711 encoding does not match the one used locally.
712 -4
714 --inet4-only
715 -6
716 --inet6-only
717 Force connecting to IPv4 or IPv6 addresses. With --inet4-only or
718 -4, Wget will only connect to IPv4 hosts, ignoring AAAA records in
719 DNS, and refusing to connect to IPv6 addresses specified in URLs.
720 Conversely, with --inet6-only or -6, Wget will only connect to IPv6
721 hosts and ignore A records and IPv4 addresses.
722 Neither options should be needed normally. By default, an
724 IPv6-aware Wget will use the address family specified by the host's
725 DNS record. If the DNS responds with both IPv4 and IPv6 addresses,
726 Wget will try them in sequence until it finds one it can connect
727 to. (Also see "--prefer-family" option described below.)
728 These options can be used to deliberately force the use of IPv4 or
730 IPv6 address families on dual family systems, usually to aid
731 debugging or to deal with broken network configuration. Only one
732 of --inet6-only and --inet4-only may be specified at the same time.
733 Neither option is available in Wget compiled without IPv6 support.
734 --prefer-family=none/IPv4/IPv6
736 When given a choice of several addresses, connect to the addresses
737 with specified address family first. The address order returned by
738 DNS is used without change by default.
739 This avoids spurious errors and connect attempts when accessing
741 hosts that resolve to both IPv6 and IPv4 addresses from IPv4
742 networks. For example, www.kame.net resolves to
743 2001:200:0:8002:203:47ff:fea5:3085 and to 203.178.141.194. When
744 the preferred family is "IPv4", the IPv4 address is used first;
745 when the preferred family is "IPv6", the IPv6 address is used
746 first; if the specified value is "none", the address order returned
747 by DNS is used without change.
748 Unlike -4 and -6, this option doesn't inhibit access to any address
750 family, it only changes the order in which the addresses are
751 accessed. Also note that the reordering performed by this option
752 is stable---it doesn't affect order of addresses of the same
753 family. That is, the relative order of all IPv4 addresses and of
754 all IPv6 addresses remains intact in all cases.
755 --retry-connrefused
757 Consider "connection refused" a transient error and try again.
758 Normally Wget gives up on a URL when it is unable to connect to the
759 site because failure to connect is taken as a sign that the server
760 is not running at all and that retries would not help. This option
761 is for mirroring unreliable sites whose servers tend to disappear
762 for short periods of time.
763 --user=user
765 --password=password
766 Specify the username user and password password for both FTP and
767 HTTP file retrieval. These parameters can be overridden using the
768 --ftp-user and --ftp-password options for FTP connections and the
769 --http-user and --http-password options for HTTP connections.
770 --ask-password
772 Prompt for a password for each connection established. Cannot be
773 specified when --password is being used, because they are mutually
774 exclusive.
775 --use-askpass=command
777 Prompt for a user and password using the specified command. If no
778 command is specified then the command in the environment variable
779 WGET_ASKPASS is used. If WGET_ASKPASS is not set then the command
780 in the environment variable SSH_ASKPASS is used.
781 You can set the default command for use-askpass in the .wgetrc.
783 That setting may be overridden from the command line.
784 --no-iri
786 Turn off internationalized URI (IRI) support. Use --iri to turn it
787 on. IRI support is activated by default.
788 You can set the default state of IRI support using the "iri"
790 command in .wgetrc. That setting may be overridden from the command
791 line.
792 --local-encoding=encoding
794 Force Wget to use encoding as the default system encoding. That
795 affects how Wget converts URLs specified as arguments from locale
796 to UTF-8 for IRI support.
797 Wget use the function "nl_langinfo()" and then the "CHARSET"
799 environment variable to get the locale. If it fails, ASCII is used.
800 You can set the default local encoding using the "local_encoding"
802 command in .wgetrc. That setting may be overridden from the command
803 line.
804 --remote-encoding=encoding
806 Force Wget to use encoding as the default remote server encoding.
807 That affects how Wget converts URIs found in files from remote
808 encoding to UTF-8 during a recursive fetch. This options is only
809 useful for IRI support, for the interpretation of non-ASCII
810 characters.
811 For HTTP, remote encoding can be found in HTTP "Content-Type"
813 header and in HTML "Content-Type http-equiv" meta tag.
814 You can set the default encoding using the "remoteencoding" command
816 in .wgetrc. That setting may be overridden from the command line.
817 --unlink
819 Force Wget to unlink file instead of clobbering existing file. This
820 option is useful for downloading to the directory with hardlinks.
821 Directory Options
823 -nd
824 --no-directories
825 Do not create a hierarchy of directories when retrieving
826 recursively. With this option turned on, all files will get saved
827 to the current directory, without clobbering (if a name shows up
828 more than once, the filenames will get extensions .n).
829 -x
831 --force-directories
832 The opposite of -nd---create a hierarchy of directories, even if
833 one would not have been created otherwise. E.g. wget -x
834 http://fly.srk.fer.hr/robots.txt will save the downloaded file to
835 fly.srk.fer.hr/robots.txt.
836 -nH
838 --no-host-directories
839 Disable generation of host-prefixed directories. By default,
840 invoking Wget with -r http://fly.srk.fer.hr/ will create a
841 structure of directories beginning with fly.srk.fer.hr/. This
842 option disables such behavior.
843 --protocol-directories
845 Use the protocol name as a directory component of local file names.
846 For example, with this option, wget -r http://host will save to
847 http/host/... rather than just to host/....
848 --cut-dirs=number
850 Ignore number directory components. This is useful for getting a
851 fine-grained control over the directory where recursive retrieval
852 will be saved.
853 Take, for example, the directory at
855 ftp://ftp.xemacs.org/pub/xemacs/. If you retrieve it with -r, it
856 will be saved locally under ftp.xemacs.org/pub/xemacs/. While the
857 -nH option can remove the ftp.xemacs.org/ part, you are still stuck
858 with pub/xemacs. This is where --cut-dirs comes in handy; it makes
859 Wget not "see" number remote directory components. Here are
860 several examples of how --cut-dirs option works.
861 No options -> ftp.xemacs.org/pub/xemacs/
863 -nH -> pub/xemacs/
864 -nH --cut-dirs=1 -> xemacs/
865 -nH --cut-dirs=2 -> .
866 --cut-dirs=1 -> ftp.xemacs.org/xemacs/
868 ...
869 If you just want to get rid of the directory structure, this option
871 is similar to a combination of -nd and -P. However, unlike -nd,
872 --cut-dirs does not lose with subdirectories---for instance, with
873 -nH --cut-dirs=1, a beta/ subdirectory will be placed to
874 xemacs/beta, as one would expect.
875 -P prefix
877 --directory-prefix=prefix
878 Set directory prefix to prefix. The directory prefix is the
879 directory where all other files and subdirectories will be saved
880 to, i.e. the top of the retrieval tree. The default is . (the
881 current directory).
882 HTTP Options
884 --default-page=name
885 Use name as the default file name when it isn't known (i.e., for
886 URLs that end in a slash), instead of index.html.
887 -E
889 --adjust-extension
890 If a file of type application/xhtml+xml or text/html is downloaded
891 and the URL does not end with the regexp \.[Hh][Tt][Mm][Ll]?, this
892 option will cause the suffix .html to be appended to the local
893 filename. This is useful, for instance, when you're mirroring a
894 remote site that uses .asp pages, but you want the mirrored pages
895 to be viewable on your stock Apache server. Another good use for
896 this is when you're downloading CGI-generated materials. A URL
897 like http://site.com/article.cgi?25 will be saved as
898 article.cgi?25.html.
899 Note that filenames changed in this way will be re-downloaded every
901 time you re-mirror a site, because Wget can't tell that the local
902 X.html file corresponds to remote URL X (since it doesn't yet know
903 that the URL produces output of type text/html or
904 application/xhtml+xml.
905 As of version 1.12, Wget will also ensure that any downloaded files
907 of type text/css end in the suffix .css, and the option was renamed
908 from --html-extension, to better reflect its new behavior. The old
909 option name is still acceptable, but should now be considered
910 deprecated.
911 As of version 1.19.2, Wget will also ensure that any downloaded
913 files with a "Content-Encoding" of br, compress, deflate or gzip
914 end in the suffix .br, .Z, .zlib and .gz respectively.
915 At some point in the future, this option may well be expanded to
917 include suffixes for other types of content, including content
918 types that are not parsed by Wget.
919 --http-user=user
921 --http-password=password
922 Specify the username user and password password on an HTTP server.
923 According to the type of the challenge, Wget will encode them using
924 either the "basic" (insecure), the "digest", or the Windows "NTLM"
925 authentication scheme.
926 Another way to specify username and password is in the URL itself.
928 Either method reveals your password to anyone who bothers to run
929 "ps". To prevent the passwords from being seen, use the
930 --use-askpass or store them in .wgetrc or .netrc, and make sure to
931 protect those files from other users with "chmod". If the
932 passwords are really important, do not leave them lying in those
933 files either---edit the files and delete them after Wget has
934 started the download.
935 --no-http-keep-alive
937 Turn off the "keep-alive" feature for HTTP downloads. Normally,
938 Wget asks the server to keep the connection open so that, when you
939 download more than one document from the same server, they get
940 transferred over the same TCP connection. This saves time and at
941 the same time reduces the load on the server.
942 This option is useful when, for some reason, persistent (keep-
944 alive) connections don't work for you, for example due to a server
945 bug or due to the inability of server-side scripts to cope with the
946 connections.
947 --no-cache
949 Disable server-side cache. In this case, Wget will send the remote
950 server an appropriate directive (Pragma: no-cache) to get the file
951 from the remote service, rather than returning the cached version.
952 This is especially useful for retrieving and flushing out-of-date
953 documents on proxy servers.
954 Caching is allowed by default.
956 --no-cookies
958 Disable the use of cookies. Cookies are a mechanism for
959 maintaining server-side state. The server sends the client a
960 cookie using the "Set-Cookie" header, and the client responds with
961 the same cookie upon further requests. Since cookies allow the
962 server owners to keep track of visitors and for sites to exchange
963 this information, some consider them a breach of privacy. The
964 default is to use cookies; however, storing cookies is not on by
965 default.
966 --load-cookies file
968 Load cookies from file before the first HTTP retrieval. file is a
969 textual file in the format originally used by Netscape's
970 cookies.txt file.
971 You will typically use this option when mirroring sites that
973 require that you be logged in to access some or all of their
974 content. The login process typically works by the web server
975 issuing an HTTP cookie upon receiving and verifying your
976 credentials. The cookie is then resent by the browser when
977 accessing that part of the site, and so proves your identity.
978 Mirroring such a site requires Wget to send the same cookies your
980 browser sends when communicating with the site. This is achieved
981 by --load-cookies---simply point Wget to the location of the
982 cookies.txt file, and it will send the same cookies your browser
983 would send in the same situation. Different browsers keep textual
984 cookie files in different locations:
985 "Netscape 4.x."
987 The cookies are in ~/.netscape/cookies.txt.
988 "Mozilla and Netscape 6.x."
990 Mozilla's cookie file is also named cookies.txt, located
991 somewhere under ~/.mozilla, in the directory of your profile.
992 The full path usually ends up looking somewhat like
993 ~/.mozilla/default/some-weird-string/cookies.txt.
994 "Internet Explorer."
996 You can produce a cookie file Wget can use by using the File
997 menu, Import and Export, Export Cookies. This has been tested
998 with Internet Explorer 5; it is not guaranteed to work with
999 earlier versions.
1000 "Other browsers."
1002 If you are using a different browser to create your cookies,
1003 --load-cookies will only work if you can locate or produce a
1004 cookie file in the Netscape format that Wget expects.
1005 If you cannot use --load-cookies, there might still be an
1007 alternative. If your browser supports a "cookie manager", you can
1008 use it to view the cookies used when accessing the site you're
1009 mirroring. Write down the name and value of the cookie, and
1010 manually instruct Wget to send those cookies, bypassing the
1011 "official" cookie support:
1012 wget --no-cookies --header "Cookie: <name>=<value>"
1014 --save-cookies file
1016 Save cookies to file before exiting. This will not save cookies
1017 that have expired or that have no expiry time (so-called "session
1018 cookies"), but also see --keep-session-cookies.
1019 --keep-session-cookies
1021 When specified, causes --save-cookies to also save session cookies.
1022 Session cookies are normally not saved because they are meant to be
1023 kept in memory and forgotten when you exit the browser. Saving
1024 them is useful on sites that require you to log in or to visit the
1025 home page before you can access some pages. With this option,
1026 multiple Wget runs are considered a single browser session as far
1027 as the site is concerned.
1028 Since the cookie file format does not normally carry session
1030 cookies, Wget marks them with an expiry timestamp of 0. Wget's
1031 --load-cookies recognizes those as session cookies, but it might
1032 confuse other browsers. Also note that cookies so loaded will be
1033 treated as other session cookies, which means that if you want
1034 --save-cookies to preserve them again, you must use
1035 --keep-session-cookies again.
1036 --ignore-length
1038 Unfortunately, some HTTP servers (CGI programs, to be more precise)
1039 send out bogus "Content-Length" headers, which makes Wget go wild,
1040 as it thinks not all the document was retrieved. You can spot this
1041 syndrome if Wget retries getting the same document again and again,
1042 each time claiming that the (otherwise normal) connection has
1043 closed on the very same byte.
1044 With this option, Wget will ignore the "Content-Length" header---as
1046 if it never existed.
1047 --header=header-line
1049 Send header-line along with the rest of the headers in each HTTP
1050 request. The supplied header is sent as-is, which means it must
1051 contain name and value separated by colon, and must not contain
1052 newlines.
1053 You may define more than one additional header by specifying
1055 --header more than once.
1056 wget --header='Accept-Charset: iso-8859-2' \
1058 --header='Accept-Language: hr' \
1059 http://fly.srk.fer.hr/
1060 Specification of an empty string as the header value will clear all
1062 previous user-defined headers.
1063 As of Wget 1.10, this option can be used to override headers
1065 otherwise generated automatically. This example instructs Wget to
1066 connect to localhost, but to specify foo.bar in the "Host" header:
1067 wget --header="Host: foo.bar" http://localhost/
1069 In versions of Wget prior to 1.10 such use of --header caused
1071 sending of duplicate headers.
1072 --compression=type
1074 Choose the type of compression to be used. Legal values are auto,
1075 gzip and none.
1076 If auto or gzip are specified, Wget asks the server to compress the
1078 file using the gzip compression format. If the server compresses
1079 the file and responds with the "Content-Encoding" header field set
1080 appropriately, the file will be decompressed automatically.
1081 If none is specified, wget will not ask the server to compress the
1083 file and will not decompress any server responses. This is the
1084 default.
1085 Compression support is currently experimental. In case it is turned
1087 on, please report any bugs to "bug-wget@gnu.org".
1088 --max-redirect=number
1090 Specifies the maximum number of redirections to follow for a
1091 resource. The default is 20, which is usually far more than
1092 necessary. However, on those occasions where you want to allow more
1093 (or fewer), this is the option to use.
1094 --proxy-user=user
1096 --proxy-password=password
1097 Specify the username user and password password for authentication
1098 on a proxy server. Wget will encode them using the "basic"
1099 authentication scheme.
1100 Security considerations similar to those with --http-password
1102 pertain here as well.
1103 --referer=url
1105 Include `Referer: url' header in HTTP request. Useful for
1106 retrieving documents with server-side processing that assume they
1107 are always being retrieved by interactive web browsers and only
1108 come out properly when Referer is set to one of the pages that
1109 point to them.
1110 --save-headers
1112 Save the headers sent by the HTTP server to the file, preceding the
1113 actual contents, with an empty line as the separator.
1114 -U agent-string
1116 --user-agent=agent-string
1117 Identify as agent-string to the HTTP server.
1118 The HTTP protocol allows the clients to identify themselves using a
1120 "User-Agent" header field. This enables distinguishing the WWW
1121 software, usually for statistical purposes or for tracing of
1122 protocol violations. Wget normally identifies as Wget/version,
1123 version being the current version number of Wget.
1124 However, some sites have been known to impose the policy of
1126 tailoring the output according to the "User-Agent"-supplied
1127 information. While this is not such a bad idea in theory, it has
1128 been abused by servers denying information to clients other than
1129 (historically) Netscape or, more frequently, Microsoft Internet
1130 Explorer. This option allows you to change the "User-Agent" line
1131 issued by Wget. Use of this option is discouraged, unless you
1132 really know what you are doing.
1133 Specifying empty user agent with --user-agent="" instructs Wget not
1135 to send the "User-Agent" header in HTTP requests.
1136 --post-data=string
1138 --post-file=file
1139 Use POST as the method for all HTTP requests and send the specified
1140 data in the request body. --post-data sends string as data,
1141 whereas --post-file sends the contents of file. Other than that,
1142 they work in exactly the same way. In particular, they both expect
1143 content of the form "key1=value1&key2=value2", with percent-
1144 encoding for special characters; the only difference is that one
1145 expects its content as a command-line parameter and the other
1146 accepts its content from a file. In particular, --post-file is not
1147 for transmitting files as form attachments: those must appear as
1148 "key=value" data (with appropriate percent-coding) just like
1149 everything else. Wget does not currently support
1150 "multipart/form-data" for transmitting POST data; only
1151 "application/x-www-form-urlencoded". Only one of --post-data and
1152 --post-file should be specified.
1153 Please note that wget does not require the content to be of the
1155 form "key1=value1&key2=value2", and neither does it test for it.
1156 Wget will simply transmit whatever data is provided to it. Most
1157 servers however expect the POST data to be in the above format when
1158 processing HTML Forms.
1159 When sending a POST request using the --post-file option, Wget
1161 treats the file as a binary file and will send every character in
1162 the POST request without stripping trailing newline or formfeed
1163 characters. Any other control characters in the text will also be
1164 sent as-is in the POST request.
1165 Please be aware that Wget needs to know the size of the POST data
1167 in advance. Therefore the argument to "--post-file" must be a
1168 regular file; specifying a FIFO or something like /dev/stdin won't
1169 work. It's not quite clear how to work around this limitation
1170 inherent in HTTP/1.0. Although HTTP/1.1 introduces chunked
1171 transfer that doesn't require knowing the request length in
1172 advance, a client can't use chunked unless it knows it's talking to
1173 an HTTP/1.1 server. And it can't know that until it receives a
1174 response, which in turn requires the request to have been completed
1175 -- a chicken-and-egg problem.
1176 Note: As of version 1.15 if Wget is redirected after the POST
1178 request is completed, its behaviour will depend on the response
1179 code returned by the server. In case of a 301 Moved Permanently,
1180 302 Moved Temporarily or 307 Temporary Redirect, Wget will, in
1181 accordance with RFC2616, continue to send a POST request. In case
1182 a server wants the client to change the Request method upon
1183 redirection, it should send a 303 See Other response code.
1184 This example shows how to log in to a server using POST and then
1186 proceed to download the desired pages, presumably only accessible
1187 to authorized users:
1188 # Log in to the server. This can be done only once.
1190 wget --save-cookies cookies.txt \
1191 --post-data 'user=foo&password=bar' \
1192 http://example.com/auth.php
1193 # Now grab the page or pages we care about.
1195 wget --load-cookies cookies.txt \
1196 -p http://example.com/interesting/article.php
1197 If the server is using session cookies to track user
1199 authentication, the above will not work because --save-cookies will
1200 not save them (and neither will browsers) and the cookies.txt file
1201 will be empty. In that case use --keep-session-cookies along with
1202 --save-cookies to force saving of session cookies.
1203 --method=HTTP-Method
1205 For the purpose of RESTful scripting, Wget allows sending of other
1206 HTTP Methods without the need to explicitly set them using
1207 --header=Header-Line. Wget will use whatever string is passed to
1208 it after --method as the HTTP Method to the server.
1209 --body-data=Data-String
1211 --body-file=Data-File
1212 Must be set when additional data needs to be sent to the server
1213 along with the Method specified using --method. --body-data sends
1214 string as data, whereas --body-file sends the contents of file.
1215 Other than that, they work in exactly the same way.
1216 Currently, --body-file is not for transmitting files as a whole.
1218 Wget does not currently support "multipart/form-data" for
1219 transmitting data; only "application/x-www-form-urlencoded". In the
1220 future, this may be changed so that wget sends the --body-file as a
1221 complete file instead of sending its contents to the server. Please
1222 be aware that Wget needs to know the contents of BODY Data in
1223 advance, and hence the argument to --body-file should be a regular
1224 file. See --post-file for a more detailed explanation. Only one of
1225 --body-data and --body-file should be specified.
1226 If Wget is redirected after the request is completed, Wget will
1228 suspend the current method and send a GET request till the
1229 redirection is completed. This is true for all redirection
1230 response codes except 307 Temporary Redirect which is used to
1231 explicitly specify that the request method should not change.
1232 Another exception is when the method is set to "POST", in which
1233 case the redirection rules specified under --post-data are
1234 followed.
1235 --content-disposition
1237 If this is set to on, experimental (not fully-functional) support
1238 for "Content-Disposition" headers is enabled. This can currently
1239 result in extra round-trips to the server for a "HEAD" request, and
1240 is known to suffer from a few bugs, which is why it is not
1241 currently enabled by default.
1242 This option is useful for some file-downloading CGI programs that
1244 use "Content-Disposition" headers to describe what the name of a
1245 downloaded file should be.
1246 When combined with --metalink-over-http and --trust-server-names, a
1248 Content-Type: application/metalink4+xml file is named using the
1249 "Content-Disposition" filename field, if available.
1250 --content-on-error
1252 If this is set to on, wget will not skip the content when the
1253 server responds with a http status code that indicates error.
1254 --trust-server-names
1256 If this is set, on a redirect, the local file name will be based on
1257 the redirection URL. By default the local file name is based on
1258 the original URL. When doing recursive retrieving this can be
1259 helpful because in many web sites redirected URLs correspond to an
1260 underlying file structure, while link URLs do not.
1261 --auth-no-challenge
1263 If this option is given, Wget will send Basic HTTP authentication
1264 information (plaintext username and password) for all requests,
1265 just like Wget 1.10.2 and prior did by default.
1266 Use of this option is not recommended, and is intended only to
1268 support some few obscure servers, which never send HTTP
1269 authentication challenges, but accept unsolicited auth info, say,
1270 in addition to form-based authentication.
1271 --retry-on-http-error=code[,code,...]
1273 Consider given HTTP response codes as non-fatal, transient errors.
1274 Supply a comma-separated list of 3-digit HTTP response codes as
1275 argument. Useful to work around special circumstances where retries
1276 are required, but the server responds with an error code normally
1277 not retried by Wget. Such errors might be 503 (Service Unavailable)
1278 and 429 (Too Many Requests). Retries enabled by this option are
1279 performed subject to the normal retry timing and retry count
1280 limitations of Wget.
1281 Using this option is intended to support special use cases only and
1283 is generally not recommended, as it can force retries even in cases
1284 where the server is actually trying to decrease its load. Please
1285 use wisely and only if you know what you are doing.
1286 HTTPS (SSL/TLS) Options
1288 To support encrypted HTTP (HTTPS) downloads, Wget must be compiled with
1289 an external SSL library. The current default is GnuTLS. In addition,
1290 Wget also supports HSTS (HTTP Strict Transport Security). If Wget is
1291 compiled without SSL support, none of these options are available.
1292 --secure-protocol=protocol
1294 Choose the secure protocol to be used. Legal values are auto,
1295 SSLv2, SSLv3, TLSv1, TLSv1_1, TLSv1_2, TLSv1_3 and PFS. If auto is
1296 used, the SSL library is given the liberty of choosing the
1297 appropriate protocol automatically, which is achieved by sending a
1298 TLSv1 greeting. This is the default.
1299 Specifying SSLv2, SSLv3, TLSv1, TLSv1_1, TLSv1_2 or TLSv1_3 forces
1301 the use of the corresponding protocol. This is useful when talking
1302 to old and buggy SSL server implementations that make it hard for
1303 the underlying SSL library to choose the correct protocol version.
1304 Fortunately, such servers are quite rare.
1305 Specifying PFS enforces the use of the so-called Perfect Forward
1307 Security cipher suites. In short, PFS adds security by creating a
1308 one-time key for each SSL connection. It has a bit more CPU impact
1309 on client and server. We use known to be secure ciphers (e.g. no
1310 MD4) and the TLS protocol. This mode also explicitly excludes non-
1311 PFS key exchange methods, such as RSA.
1312 --https-only
1314 When in recursive mode, only HTTPS links are followed.
1315 --ciphers
1317 Set the cipher list string. Typically this string sets the cipher
1318 suites and other SSL/TLS options that the user wish should be used,
1319 in a set order of preference (GnuTLS calls it 'priority string').
1320 This string will be fed verbatim to the SSL/TLS engine (OpenSSL or
1321 GnuTLS) and hence its format and syntax is dependant on that. Wget
1322 will not process or manipulate it in any way. Refer to the OpenSSL
1323 or GnuTLS documentation for more information.
1324 --no-check-certificate
1326 Don't check the server certificate against the available
1327 certificate authorities. Also don't require the URL host name to
1328 match the common name presented by the certificate.
1329 As of Wget 1.10, the default is to verify the server's certificate
1331 against the recognized certificate authorities, breaking the SSL
1332 handshake and aborting the download if the verification fails.
1333 Although this provides more secure downloads, it does break
1334 interoperability with some sites that worked with previous Wget
1335 versions, particularly those using self-signed, expired, or
1336 otherwise invalid certificates. This option forces an "insecure"
1337 mode of operation that turns the certificate verification errors
1338 into warnings and allows you to proceed.
1339 If you encounter "certificate verification" errors or ones saying
1341 that "common name doesn't match requested host name", you can use
1342 this option to bypass the verification and proceed with the
1343 download. Only use this option if you are otherwise convinced of
1344 the site's authenticity, or if you really don't care about the
1345 validity of its certificate. It is almost always a bad idea not to
1346 check the certificates when transmitting confidential or important
1347 data. For self-signed/internal certificates, you should download
1348 the certificate and verify against that instead of forcing this
1349 insecure mode. If you are really sure of not desiring any
1350 certificate verification, you can specify --check-certificate=quiet
1351 to tell wget to not print any warning about invalid certificates,
1352 albeit in most cases this is the wrong thing to do.
1353 --certificate=file
1355 Use the client certificate stored in file. This is needed for
1356 servers that are configured to require certificates from the
1357 clients that connect to them. Normally a certificate is not
1358 required and this switch is optional.
1359 --certificate-type=type
1361 Specify the type of the client certificate. Legal values are PEM
1362 (assumed by default) and DER, also known as ASN1.
1363 --private-key=file
1365 Read the private key from file. This allows you to provide the
1366 private key in a file separate from the certificate.
1367 --private-key-type=type
1369 Specify the type of the private key. Accepted values are PEM (the
1370 default) and DER.
1371 --ca-certificate=file
1373 Use file as the file with the bundle of certificate authorities
1374 ("CA") to verify the peers. The certificates must be in PEM
1375 format.
1376 Without this option Wget looks for CA certificates at the system-
1378 specified locations, chosen at OpenSSL installation time.
1379 --ca-directory=directory
1381 Specifies directory containing CA certificates in PEM format. Each
1382 file contains one CA certificate, and the file name is based on a
1383 hash value derived from the certificate. This is achieved by
1384 processing a certificate directory with the "c_rehash" utility
1385 supplied with OpenSSL. Using --ca-directory is more efficient than
1386 --ca-certificate when many certificates are installed because it
1387 allows Wget to fetch certificates on demand.
1388 Without this option Wget looks for CA certificates at the system-
1390 specified locations, chosen at OpenSSL installation time.
1391 --crl-file=file
1393 Specifies a CRL file in file. This is needed for certificates that
1394 have been revocated by the CAs.
1395 --pinnedpubkey=file/hashes
1397 Tells wget to use the specified public key file (or hashes) to
1398 verify the peer. This can be a path to a file which contains a
1399 single public key in PEM or DER format, or any number of base64
1400 encoded sha256 hashes preceded by "sha256//" and separated by ";"
1401 When negotiating a TLS or SSL connection, the server sends a
1403 certificate indicating its identity. A public key is extracted from
1404 this certificate and if it does not exactly match the public key(s)
1405 provided to this option, wget will abort the connection before
1406 sending or receiving any data.
1407 --random-file=file
1409 [OpenSSL and LibreSSL only] Use file as the source of random data
1410 for seeding the pseudo-random number generator on systems without
1411 /dev/urandom.
1412 On such systems the SSL library needs an external source of
1414 randomness to initialize. Randomness may be provided by EGD (see
1415 --egd-file below) or read from an external source specified by the
1416 user. If this option is not specified, Wget looks for random data
1417 in $RANDFILE or, if that is unset, in $HOME/.rnd.
1418 If you're getting the "Could not seed OpenSSL PRNG; disabling SSL."
1420 error, you should provide random data using some of the methods
1421 described above.
1422 --egd-file=file
1424 [OpenSSL only] Use file as the EGD socket. EGD stands for Entropy
1425 Gathering Daemon, a user-space program that collects data from
1426 various unpredictable system sources and makes it available to
1427 other programs that might need it. Encryption software, such as
1428 the SSL library, needs sources of non-repeating randomness to seed
1429 the random number generator used to produce cryptographically
1430 strong keys.
1431 OpenSSL allows the user to specify his own source of entropy using
1433 the "RAND_FILE" environment variable. If this variable is unset,
1434 or if the specified file does not produce enough randomness,
1435 OpenSSL will read random data from EGD socket specified using this
1436 option.
1437 If this option is not specified (and the equivalent startup command
1439 is not used), EGD is never contacted. EGD is not needed on modern
1440 Unix systems that support /dev/urandom.
1441 --no-hsts
1443 Wget supports HSTS (HTTP Strict Transport Security, RFC 6797) by
1444 default. Use --no-hsts to make Wget act as a non-HSTS-compliant
1445 UA. As a consequence, Wget would ignore all the
1446 "Strict-Transport-Security" headers, and would not enforce any
1447 existing HSTS policy.
1448 --hsts-file=file
1450 By default, Wget stores its HSTS database in ~/.wget-hsts. You can
1451 use --hsts-file to override this. Wget will use the supplied file
1452 as the HSTS database. Such file must conform to the correct HSTS
1453 database format used by Wget. If Wget cannot parse the provided
1454 file, the behaviour is unspecified.
1455 The Wget's HSTS database is a plain text file. Each line contains
1457 an HSTS entry (ie. a site that has issued a
1458 "Strict-Transport-Security" header and that therefore has specified
1459 a concrete HSTS policy to be applied). Lines starting with a dash
1460 ("#") are ignored by Wget. Please note that in spite of this
1461 convenient human-readability hand-hacking the HSTS database is
1462 generally not a good idea.
1463 An HSTS entry line consists of several fields separated by one or
1465 more whitespace:
1466 "<hostname> SP [<port>] SP <include subdomains> SP <created> SP
1468 <max-age>"
1469 The hostname and port fields indicate the hostname and port to
1471 which the given HSTS policy applies. The port field may be zero,
1472 and it will, in most of the cases. That means that the port number
1473 will not be taken into account when deciding whether such HSTS
1474 policy should be applied on a given request (only the hostname will
1475 be evaluated). When port is different to zero, both the target
1476 hostname and the port will be evaluated and the HSTS policy will
1477 only be applied if both of them match. This feature has been
1478 included for testing/development purposes only. The Wget testsuite
1479 (in testenv/) creates HSTS databases with explicit ports with the
1480 purpose of ensuring Wget's correct behaviour. Applying HSTS
1481 policies to ports other than the default ones is discouraged by RFC
1482 6797 (see Appendix B "Differences between HSTS Policy and Same-
1483 Origin Policy"). Thus, this functionality should not be used in
1484 production environments and port will typically be zero. The last
1485 three fields do what they are expected to. The field
1486 include_subdomains can either be 1 or 0 and it signals whether the
1487 subdomains of the target domain should be part of the given HSTS
1488 policy as well. The created and max-age fields hold the timestamp
1489 values of when such entry was created (first seen by Wget) and the
1490 HSTS-defined value 'max-age', which states how long should that
1491 HSTS policy remain active, measured in seconds elapsed since the
1492 timestamp stored in created. Once that time has passed, that HSTS
1493 policy will no longer be valid and will eventually be removed from
1494 the database.
1495 If you supply your own HSTS database via --hsts-file, be aware that
1497 Wget may modify the provided file if any change occurs between the
1498 HSTS policies requested by the remote servers and those in the
1499 file. When Wget exists, it effectively updates the HSTS database by
1500 rewriting the database file with the new entries.
1501 If the supplied file does not exist, Wget will create one. This
1503 file will contain the new HSTS entries. If no HSTS entries were
1504 generated (no "Strict-Transport-Security" headers were sent by any
1505 of the servers) then no file will be created, not even an empty
1506 one. This behaviour applies to the default database file
1507 (~/.wget-hsts) as well: it will not be created until some server
1508 enforces an HSTS policy.
1509 Care is taken not to override possible changes made by other Wget
1511 processes at the same time over the HSTS database. Before dumping
1512 the updated HSTS entries on the file, Wget will re-read it and
1513 merge the changes.
1514 Using a custom HSTS database and/or modifying an existing one is
1516 discouraged. For more information about the potential security
1517 threats arised from such practice, see section 14 "Security
1518 Considerations" of RFC 6797, specially section 14.9 "Creative
1519 Manipulation of HSTS Policy Store".
1520 --warc-file=file
1522 Use file as the destination WARC file.
1523 --warc-header=string
1525 Use string into as the warcinfo record.
1526 --warc-max-size=size
1528 Set the maximum size of the WARC files to size.
1529 --warc-cdx
1531 Write CDX index files.
1532 --warc-dedup=file
1534 Do not store records listed in this CDX file.
1535 --no-warc-compression
1537 Do not compress WARC files with GZIP.
1538 --no-warc-digests
1540 Do not calculate SHA1 digests.
1541 --no-warc-keep-log
1543 Do not store the log file in a WARC record.
1544 --warc-tempdir=dir
1546 Specify the location for temporary files created by the WARC
1547 writer.
1548 FTP Options
1550 --ftp-user=user
1551 --ftp-password=password
1552 Specify the username user and password password on an FTP server.
1553 Without this, or the corresponding startup option, the password
1554 defaults to -wget@, normally used for anonymous FTP.
1555 Another way to specify username and password is in the URL itself.
1557 Either method reveals your password to anyone who bothers to run
1558 "ps". To prevent the passwords from being seen, store them in
1559 .wgetrc or .netrc, and make sure to protect those files from other
1560 users with "chmod". If the passwords are really important, do not
1561 leave them lying in those files either---edit the files and delete
1562 them after Wget has started the download.
1563 --no-remove-listing
1565 Don't remove the temporary .listing files generated by FTP
1566 retrievals. Normally, these files contain the raw directory
1567 listings received from FTP servers. Not removing them can be
1568 useful for debugging purposes, or when you want to be able to
1569 easily check on the contents of remote server directories (e.g. to
1570 verify that a mirror you're running is complete).
1571 Note that even though Wget writes to a known filename for this
1573 file, this is not a security hole in the scenario of a user making
1574 .listing a symbolic link to /etc/passwd or something and asking
1575 "root" to run Wget in his or her directory. Depending on the
1576 options used, either Wget will refuse to write to .listing, making
1577 the globbing/recursion/time-stamping operation fail, or the
1578 symbolic link will be deleted and replaced with the actual .listing
1579 file, or the listing will be written to a .listing.number file.
1580 Even though this situation isn't a problem, though, "root" should
1582 never run Wget in a non-trusted user's directory. A user could do
1583 something as simple as linking index.html to /etc/passwd and asking
1584 "root" to run Wget with -N or -r so the file will be overwritten.
1585 --no-glob
1587 Turn off FTP globbing. Globbing refers to the use of shell-like
1588 special characters (wildcards), like *, ?, [ and ] to retrieve more
1589 than one file from the same directory at once, like:
1590 wget ftp://gnjilux.srk.fer.hr/*.msg
1592 By default, globbing will be turned on if the URL contains a
1594 globbing character. This option may be used to turn globbing on or
1595 off permanently.
1596 You may have to quote the URL to protect it from being expanded by
1598 your shell. Globbing makes Wget look for a directory listing,
1599 which is system-specific. This is why it currently works only with
1600 Unix FTP servers (and the ones emulating Unix "ls" output).
1601 --no-passive-ftp
1603 Disable the use of the passive FTP transfer mode. Passive FTP
1604 mandates that the client connect to the server to establish the
1605 data connection rather than the other way around.
1606 If the machine is connected to the Internet directly, both passive
1608 and active FTP should work equally well. Behind most firewall and
1609 NAT configurations passive FTP has a better chance of working.
1610 However, in some rare firewall configurations, active FTP actually
1611 works when passive FTP doesn't. If you suspect this to be the
1612 case, use this option, or set "passive_ftp=off" in your init file.
1613 --preserve-permissions
1615 Preserve remote file permissions instead of permissions set by
1616 umask.
1617 --retr-symlinks
1619 By default, when retrieving FTP directories recursively and a
1620 symbolic link is encountered, the symbolic link is traversed and
1621 the pointed-to files are retrieved. Currently, Wget does not
1622 traverse symbolic links to directories to download them
1623 recursively, though this feature may be added in the future.
1624 When --retr-symlinks=no is specified, the linked-to file is not
1626 downloaded. Instead, a matching symbolic link is created on the
1627 local filesystem. The pointed-to file will not be retrieved unless
1628 this recursive retrieval would have encountered it separately and
1629 downloaded it anyway. This option poses a security risk where a
1630 malicious FTP Server may cause Wget to write to files outside of
1631 the intended directories through a specially crafted .LISTING file.
1632 Note that when retrieving a file (not a directory) because it was
1634 specified on the command-line, rather than because it was recursed
1635 to, this option has no effect. Symbolic links are always traversed
1636 in this case.
1637 FTPS Options
1639 --ftps-implicit
1640 This option tells Wget to use FTPS implicitly. Implicit FTPS
1641 consists of initializing SSL/TLS from the very beginning of the
1642 control connection. This option does not send an "AUTH TLS"
1643 command: it assumes the server speaks FTPS and directly starts an
1644 SSL/TLS connection. If the attempt is successful, the session
1645 continues just like regular FTPS ("PBSZ" and "PROT" are sent,
1646 etc.). Implicit FTPS is no longer a requirement for FTPS
1647 implementations, and thus many servers may not support it. If
1648 --ftps-implicit is passed and no explicit port number specified,
1649 the default port for implicit FTPS, 990, will be used, instead of
1650 the default port for the "normal" (explicit) FTPS which is the same
1651 as that of FTP, 21.
1652 --no-ftps-resume-ssl
1654 Do not resume the SSL/TLS session in the data channel. When
1655 starting a data connection, Wget tries to resume the SSL/TLS
1656 session previously started in the control connection. SSL/TLS
1657 session resumption avoids performing an entirely new handshake by
1658 reusing the SSL/TLS parameters of a previous session. Typically,
1659 the FTPS servers want it that way, so Wget does this by default.
1660 Under rare circumstances however, one might want to start an
1661 entirely new SSL/TLS session in every data connection. This is
1662 what --no-ftps-resume-ssl is for.
1663 --ftps-clear-data-connection
1665 All the data connections will be in plain text. Only the control
1666 connection will be under SSL/TLS. Wget will send a "PROT C" command
1667 to achieve this, which must be approved by the server.
1668 --ftps-fallback-to-ftp
1670 Fall back to FTP if FTPS is not supported by the target server. For
1671 security reasons, this option is not asserted by default. The
1672 default behaviour is to exit with an error. If a server does not
1673 successfully reply to the initial "AUTH TLS" command, or in the
1674 case of implicit FTPS, if the initial SSL/TLS connection attempt is
1675 rejected, it is considered that such server does not support FTPS.
1676 Recursive Retrieval Options
1678 -r
1679 --recursive
1680 Turn on recursive retrieving. The default maximum depth is 5.
1681 -l depth
1683 --level=depth
1684 Specify recursion maximum depth level depth.
1685 --delete-after
1687 This option tells Wget to delete every single file it downloads,
1688 after having done so. It is useful for pre-fetching popular pages
1689 through a proxy, e.g.:
1690 wget -r -nd --delete-after http://whatever.com/~popular/page/
1692 The -r option is to retrieve recursively, and -nd to not create
1694 directories.
1695 Note that --delete-after deletes files on the local machine. It
1697 does not issue the DELE command to remote FTP sites, for instance.
1698 Also note that when --delete-after is specified, --convert-links is
1699 ignored, so .orig files are simply not created in the first place.
1700 -k
1702 --convert-links
1703 After the download is complete, convert the links in the document
1704 to make them suitable for local viewing. This affects not only the
1705 visible hyperlinks, but any part of the document that links to
1706 external content, such as embedded images, links to style sheets,
1707 hyperlinks to non-HTML content, etc.
1708 Each link will be changed in one of the two ways:
1710 · The links to files that have been downloaded by Wget will be
1712 changed to refer to the file they point to as a relative link.
1713 Example: if the downloaded file /foo/doc.html links to
1715 /bar/img.gif, also downloaded, then the link in doc.html will
1716 be modified to point to ../bar/img.gif. This kind of
1717 transformation works reliably for arbitrary combinations of
1718 directories.
1719 · The links to files that have not been downloaded by Wget will
1721 be changed to include host name and absolute path of the
1722 location they point to.
1723 Example: if the downloaded file /foo/doc.html links to
1725 /bar/img.gif (or to ../bar/img.gif), then the link in doc.html
1726 will be modified to point to http://hostname/bar/img.gif.
1727 Because of this, local browsing works reliably: if a linked file
1729 was downloaded, the link will refer to its local name; if it was
1730 not downloaded, the link will refer to its full Internet address
1731 rather than presenting a broken link. The fact that the former
1732 links are converted to relative links ensures that you can move the
1733 downloaded hierarchy to another directory.
1734 Note that only at the end of the download can Wget know which links
1736 have been downloaded. Because of that, the work done by -k will be
1737 performed at the end of all the downloads.
1738 --convert-file-only
1740 This option converts only the filename part of the URLs, leaving
1741 the rest of the URLs untouched. This filename part is sometimes
1742 referred to as the "basename", although we avoid that term here in
1743 order not to cause confusion.
1744 It works particularly well in conjunction with --adjust-extension,
1746 although this coupling is not enforced. It proves useful to
1747 populate Internet caches with files downloaded from different
1748 hosts.
1749 Example: if some link points to //foo.com/bar.cgi?xyz with
1751 --adjust-extension asserted and its local destination is intended
1752 to be ./foo.com/bar.cgi?xyz.css, then the link would be converted
1753 to //foo.com/bar.cgi?xyz.css. Note that only the filename part has
1754 been modified. The rest of the URL has been left untouched,
1755 including the net path ("//") which would otherwise be processed by
1756 Wget and converted to the effective scheme (ie. "http://").
1757 -K
1759 --backup-converted
1760 When converting a file, back up the original version with a .orig
1761 suffix. Affects the behavior of -N.
1762 -m
1764 --mirror
1765 Turn on options suitable for mirroring. This option turns on
1766 recursion and time-stamping, sets infinite recursion depth and
1767 keeps FTP directory listings. It is currently equivalent to -r -N
1768 -l inf --no-remove-listing.
1769 -p
1771 --page-requisites
1772 This option causes Wget to download all the files that are
1773 necessary to properly display a given HTML page. This includes
1774 such things as inlined images, sounds, and referenced stylesheets.
1775 Ordinarily, when downloading a single HTML page, any requisite
1777 documents that may be needed to display it properly are not
1778 downloaded. Using -r together with -l can help, but since Wget
1779 does not ordinarily distinguish between external and inlined
1780 documents, one is generally left with "leaf documents" that are
1781 missing their requisites.
1782 For instance, say document 1.html contains an "<IMG>" tag
1784 referencing 1.gif and an "<A>" tag pointing to external document
1785 2.html. Say that 2.html is similar but that its image is 2.gif and
1786 it links to 3.html. Say this continues up to some arbitrarily high
1787 number.
1788 If one executes the command:
1790 wget -r -l 2 http://<site>/1.html
1792 then 1.html, 1.gif, 2.html, 2.gif, and 3.html will be downloaded.
1794 As you can see, 3.html is without its requisite 3.gif because Wget
1795 is simply counting the number of hops (up to 2) away from 1.html in
1796 order to determine where to stop the recursion. However, with this
1797 command:
1798 wget -r -l 2 -p http://<site>/1.html
1800 all the above files and 3.html's requisite 3.gif will be
1802 downloaded. Similarly,
1803 wget -r -l 1 -p http://<site>/1.html
1805 will cause 1.html, 1.gif, 2.html, and 2.gif to be downloaded. One
1807 might think that:
1808 wget -r -l 0 -p http://<site>/1.html
1810 would download just 1.html and 1.gif, but unfortunately this is not
1812 the case, because -l 0 is equivalent to -l inf---that is, infinite
1813 recursion. To download a single HTML page (or a handful of them,
1814 all specified on the command-line or in a -i URL input file) and
1815 its (or their) requisites, simply leave off -r and -l:
1816 wget -p http://<site>/1.html
1818 Note that Wget will behave as if -r had been specified, but only
1820 that single page and its requisites will be downloaded. Links from
1821 that page to external documents will not be followed. Actually, to
1822 download a single page and all its requisites (even if they exist
1823 on separate websites), and make sure the lot displays properly
1824 locally, this author likes to use a few options in addition to -p:
1825 wget -E -H -k -K -p http://<site>/<document>
1827 To finish off this topic, it's worth knowing that Wget's idea of an
1829 external document link is any URL specified in an "<A>" tag, an
1830 "<AREA>" tag, or a "<LINK>" tag other than "<LINK
1831 REL="stylesheet">".
1832 --strict-comments
1834 Turn on strict parsing of HTML comments. The default is to
1835 terminate comments at the first occurrence of -->.
1836 According to specifications, HTML comments are expressed as SGML
1838 declarations. Declaration is special markup that begins with <!
1839 and ends with >, such as <!DOCTYPE ...>, that may contain comments
1840 between a pair of -- delimiters. HTML comments are "empty
1841 declarations", SGML declarations without any non-comment text.
1842 Therefore, <!--foo--> is a valid comment, and so is <!--one--
1843 --two-->, but <!--1--2--> is not.
1844 On the other hand, most HTML writers don't perceive comments as
1846 anything other than text delimited with <!-- and -->, which is not
1847 quite the same. For example, something like <!------------> works
1848 as a valid comment as long as the number of dashes is a multiple of
1849 four (!). If not, the comment technically lasts until the next --,
1850 which may be at the other end of the document. Because of this,
1851 many popular browsers completely ignore the specification and
1852 implement what users have come to expect: comments delimited with
1853 <!-- and -->.
1854 Until version 1.9, Wget interpreted comments strictly, which
1856 resulted in missing links in many web pages that displayed fine in
1857 browsers, but had the misfortune of containing non-compliant
1858 comments. Beginning with version 1.9, Wget has joined the ranks of
1859 clients that implements "naive" comments, terminating each comment
1860 at the first occurrence of -->.
1861 If, for whatever reason, you want strict comment parsing, use this
1863 option to turn it on.
1864 Recursive Accept/Reject Options
1866 -A acclist --accept acclist
1867 -R rejlist --reject rejlist
1868 Specify comma-separated lists of file name suffixes or patterns to
1869 accept or reject. Note that if any of the wildcard characters, *,
1870 ?, [ or ], appear in an element of acclist or rejlist, it will be
1871 treated as a pattern, rather than a suffix. In this case, you have
1872 to enclose the pattern into quotes to prevent your shell from
1873 expanding it, like in -A "*.mp3" or -A '*.mp3'.
1874 --accept-regex urlregex
1876 --reject-regex urlregex
1877 Specify a regular expression to accept or reject the complete URL.
1878 --regex-type regextype
1880 Specify the regular expression type. Possible types are posix or
1881 pcre. Note that to be able to use pcre type, wget has to be
1882 compiled with libpcre support.
1883 -D domain-list
1885 --domains=domain-list
1886 Set domains to be followed. domain-list is a comma-separated list
1887 of domains. Note that it does not turn on -H.
1888 --exclude-domains domain-list
1890 Specify the domains that are not to be followed.
1891 --follow-ftp
1893 Follow FTP links from HTML documents. Without this option, Wget
1894 will ignore all the FTP links.
1895 --follow-tags=list
1897 Wget has an internal table of HTML tag / attribute pairs that it
1898 considers when looking for linked documents during a recursive
1899 retrieval. If a user wants only a subset of those tags to be
1900 considered, however, he or she should be specify such tags in a
1901 comma-separated list with this option.
1902 --ignore-tags=list
1904 This is the opposite of the --follow-tags option. To skip certain
1905 HTML tags when recursively looking for documents to download,
1906 specify them in a comma-separated list.
1907 In the past, this option was the best bet for downloading a single
1909 page and its requisites, using a command-line like:
1910 wget --ignore-tags=a,area -H -k -K -r http://<site>/<document>
1912 However, the author of this option came across a page with tags
1914 like "<LINK REL="home" HREF="/">" and came to the realization that
1915 specifying tags to ignore was not enough. One can't just tell Wget
1916 to ignore "<LINK>", because then stylesheets will not be
1917 downloaded. Now the best bet for downloading a single page and its
1918 requisites is the dedicated --page-requisites option.
1919 --ignore-case
1921 Ignore case when matching files and directories. This influences
1922 the behavior of -R, -A, -I, and -X options, as well as globbing
1923 implemented when downloading from FTP sites. For example, with
1924 this option, -A "*.txt" will match file1.txt, but also file2.TXT,
1925 file3.TxT, and so on. The quotes in the example are to prevent the
1926 shell from expanding the pattern.
1927 -H
1929 --span-hosts
1930 Enable spanning across hosts when doing recursive retrieving.
1931 -L
1933 --relative
1934 Follow relative links only. Useful for retrieving a specific home
1935 page without any distractions, not even those from the same hosts.
1936 -I list
1938 --include-directories=list
1939 Specify a comma-separated list of directories you wish to follow
1940 when downloading. Elements of list may contain wildcards.
1941 -X list
1943 --exclude-directories=list
1944 Specify a comma-separated list of directories you wish to exclude
1945 from download. Elements of list may contain wildcards.
1946 -np
1948 --no-parent
1949 Do not ever ascend to the parent directory when retrieving
1950 recursively. This is a useful option, since it guarantees that
1951 only the files below a certain hierarchy will be downloaded.
1952
1954 Wget supports proxies for both HTTP and FTP retrievals. The standard
1955 way to specify proxy location, which Wget recognizes, is using the
1956 following environment variables:
1957 http_proxy
1959 https_proxy
1960 If set, the http_proxy and https_proxy variables should contain the
1961 URLs of the proxies for HTTP and HTTPS connections respectively.
1962 ftp_proxy
1964 This variable should contain the URL of the proxy for FTP
1965 connections. It is quite common that http_proxy and ftp_proxy are
1966 set to the same URL.
1967 no_proxy
1969 This variable should contain a comma-separated list of domain
1970 extensions proxy should not be used for. For instance, if the
1971 value of no_proxy is .mit.edu, proxy will not be used to retrieve
1972 documents from MIT.
1973
1975 Wget may return one of several error codes if it encounters problems.
1976 0 No problems occurred.
1978 1 Generic error code.
1980 2 Parse error---for instance, when parsing command-line options, the
1982 .wgetrc or .netrc...
1983 3 File I/O error.
1985 4 Network failure.
1987 5 SSL verification failure.
1989 6 Username/password authentication failure.
1991 7 Protocol errors.
1993 8 Server issued an error response.
1995 With the exceptions of 0 and 1, the lower-numbered exit codes take
1997 precedence over higher-numbered ones, when multiple types of errors are
1998 encountered.
1999 In versions of Wget prior to 1.12, Wget's exit status tended to be
2001 unhelpful and inconsistent. Recursive downloads would virtually always
2002 return 0 (success), regardless of any issues encountered, and non-
2003 recursive fetches only returned the status corresponding to the most
2004 recently-attempted download.
2005
2007 /etc/wgetrc
2008 Default location of the global startup file.
2009 .wgetrc
2011 User startup file.
2012
2014 You are welcome to submit bug reports via the GNU Wget bug tracker (see
2015 <https://savannah.gnu.org/bugs/?func=additem&group=wget>) or to our
2016 mailing list <bug-wget@gnu.org>.
2017 Visit <https://lists.gnu.org/mailman/listinfo/bug-wget> to get more
2019 info (how to subscribe, list archives, ...).
2020 Before actually submitting a bug report, please try to follow a few
2022 simple guidelines.
2023 1. Please try to ascertain that the behavior you see really is a bug.
2025 If Wget crashes, it's a bug. If Wget does not behave as
2026 documented, it's a bug. If things work strange, but you are not
2027 sure about the way they are supposed to work, it might well be a
2028 bug, but you might want to double-check the documentation and the
2029 mailing lists.
2030 2. Try to repeat the bug in as simple circumstances as possible. E.g.
2032 if Wget crashes while downloading wget -rl0 -kKE -t5 --no-proxy
2033 http://example.com -o /tmp/log, you should try to see if the crash
2034 is repeatable, and if will occur with a simpler set of options.
2035 You might even try to start the download at the page where the
2036 crash occurred to see if that page somehow triggered the crash.
2037 Also, while I will probably be interested to know the contents of
2039 your .wgetrc file, just dumping it into the debug message is
2040 probably a bad idea. Instead, you should first try to see if the
2041 bug repeats with .wgetrc moved out of the way. Only if it turns
2042 out that .wgetrc settings affect the bug, mail me the relevant
2043 parts of the file.
2044 3. Please start Wget with -d option and send us the resulting output
2046 (or relevant parts thereof). If Wget was compiled without debug
2047 support, recompile it---it is much easier to trace bugs with debug
2048 support on.
2049 Note: please make sure to remove any potentially sensitive
2051 information from the debug log before sending it to the bug
2052 address. The "-d" won't go out of its way to collect sensitive
2053 information, but the log will contain a fairly complete transcript
2054 of Wget's communication with the server, which may include
2055 passwords and pieces of downloaded data. Since the bug address is
2056 publically archived, you may assume that all bug reports are
2057 visible to the public.
2058 4. If Wget has crashed, try to run it in a debugger, e.g. "gdb `which
2060 wget` core" and type "where" to get the backtrace. This may not
2061 work if the system administrator has disabled core files, but it is
2062 safe to try.
2063
2065 This is not the complete manual for GNU Wget. For more complete
2066 information, including more detailed explanations of some of the
2067 options, and a number of commands available for use with .wgetrc files
2068 and the -e option, see the GNU Info entry for wget.
2069
2071 Originally written by Hrvoje Nikšić <hniksic@xemacs.org>.
2072
2074 Copyright (c) 1996-2011, 2015, 2018 Free Software Foundation, Inc.
2075 Permission is granted to copy, distribute and/or modify this document
2077 under the terms of the GNU Free Documentation License, Version 1.3 or
2078 any later version published by the Free Software Foundation; with no
2079 Invariant Sections, with no Front-Cover Texts, and with no Back-Cover
2080 Texts. A copy of the license is included in the section entitled "GNU
2081 Free Documentation License".
2082GNU Wget 1.19.5 2019-06-28 WGET(1)
