1WGET(1) GNU Wget WGET(1)
2
6 Wget - The non-interactive network downloader.
7
9 wget [option]... [URL]...
10
12 GNU Wget is a free utility for non-interactive download of files from
13 the Web. It supports HTTP, HTTPS, and FTP protocols, as well as
14 retrieval through HTTP proxies.
15 Wget is non-interactive, meaning that it can work in the background,
17 while the user is not logged on. This allows you to start a retrieval
18 and disconnect from the system, letting Wget finish the work. By
19 contrast, most of the Web browsers require constant user's presence,
20 which can be a great hindrance when transferring a lot of data.
21 Wget can follow links in HTML, XHTML, and CSS pages, to create local
23 versions of remote web sites, fully recreating the directory structure
24 of the original site. This is sometimes referred to as "recursive
25 downloading." While doing that, Wget respects the Robot Exclusion
26 Standard (/robots.txt). Wget can be instructed to convert the links in
27 downloaded files to point at the local files, for offline viewing.
28 Wget has been designed for robustness over slow or unstable network
30 connections; if a download fails due to a network problem, it will keep
31 retrying until the whole file has been retrieved. If the server
32 supports regetting, it will instruct the server to continue the
33 download from where it left off.
34
36 Option Syntax
37 Since Wget uses GNU getopt to process command-line arguments, every
38 option has a long form along with the short one. Long options are more
39 convenient to remember, but take time to type. You may freely mix
40 different option styles, or specify options after the command-line
41 arguments. Thus you may write:
42 wget -r --tries=10 http://fly.srk.fer.hr/ -o log
44 The space between the option accepting an argument and the argument may
46 be omitted. Instead of -o log you can write -olog.
47 You may put several options that do not require arguments together,
49 like:
50 wget -drc <URL>
52 This is completely equivalent to:
54 wget -d -r -c <URL>
56 Since the options can be specified after the arguments, you may
58 terminate them with --. So the following will try to download URL -x,
59 reporting failure to log:
60 wget -o log -- -x
62 The options that accept comma-separated lists all respect the
64 convention that specifying an empty list clears its value. This can be
65 useful to clear the .wgetrc settings. For instance, if your .wgetrc
66 sets "exclude_directories" to /cgi-bin, the following example will
67 first reset it, and then set it to exclude /~nobody and /~somebody.
68 You can also clear the lists in .wgetrc.
69 wget -X " -X /~nobody,/~somebody
71 Most options that do not accept arguments are boolean options, so named
73 because their state can be captured with a yes-or-no ("boolean")
74 variable. For example, --follow-ftp tells Wget to follow FTP links
75 from HTML files and, on the other hand, --no-glob tells it not to
76 perform file globbing on FTP URLs. A boolean option is either
77 affirmative or negative (beginning with --no). All such options share
78 several properties.
79 Unless stated otherwise, it is assumed that the default behavior is the
81 opposite of what the option accomplishes. For example, the documented
82 existence of --follow-ftp assumes that the default is to not follow FTP
83 links from HTML pages.
84 Affirmative options can be negated by prepending the --no- to the
86 option name; negative options can be negated by omitting the --no-
87 prefix. This might seem superfluous---if the default for an
88 affirmative option is to not do something, then why provide a way to
89 explicitly turn it off? But the startup file may in fact change the
90 default. For instance, using "follow_ftp = on" in .wgetrc makes Wget
91 follow FTP links by default, and using --no-follow-ftp is the only way
92 to restore the factory default from the command line.
93 Basic Startup Options
95 -V
96 --version
97 Display the version of Wget.
98 -h
100 --help
101 Print a help message describing all of Wget's command-line options.
102 -b
104 --background
105 Go to background immediately after startup. If no output file is
106 specified via the -o, output is redirected to wget-log.
107 -e command
109 --execute command
110 Execute command as if it were a part of .wgetrc. A command thus
111 invoked will be executed after the commands in .wgetrc, thus taking
112 precedence over them. If you need to specify more than one wgetrc
113 command, use multiple instances of -e.
114 Logging and Input File Options
116 -o logfile
117 --output-file=logfile
118 Log all messages to logfile. The messages are normally reported to
119 standard error.
120 -a logfile
122 --append-output=logfile
123 Append to logfile. This is the same as -o, only it appends to
124 logfile instead of overwriting the old log file. If logfile does
125 not exist, a new file is created.
126 -d
128 --debug
129 Turn on debug output, meaning various information important to the
130 developers of Wget if it does not work properly. Your system
131 administrator may have chosen to compile Wget without debug
132 support, in which case -d will not work. Please note that
133 compiling with debug support is always safe---Wget compiled with
134 the debug support will not print any debug info unless requested
135 with -d.
136 -q
138 --quiet
139 Turn off Wget's output.
140 -v
142 --verbose
143 Turn on verbose output, with all the available data. The default
144 output is verbose.
145 -nv
147 --no-verbose
148 Turn off verbose without being completely quiet (use -q for that),
149 which means that error messages and basic information still get
150 printed.
151 --report-speed=type
153 Output bandwidth as type. The only accepted value is bits.
154 -i file
156 --input-file=file
157 Read URLs from a local or external file. If - is specified as
158 file, URLs are read from the standard input. (Use ./- to read from
159 a file literally named -.)
160 If this function is used, no URLs need be present on the command
162 line. If there are URLs both on the command line and in an input
163 file, those on the command lines will be the first ones to be
164 retrieved. If --force-html is not specified, then file should
165 consist of a series of URLs, one per line.
166 However, if you specify --force-html, the document will be regarded
168 as html. In that case you may have problems with relative links,
169 which you can solve either by adding "<base href="url">" to the
170 documents or by specifying --base=url on the command line.
171 If the file is an external one, the document will be automatically
173 treated as html if the Content-Type matches text/html.
174 Furthermore, the file's location will be implicitly used as base
175 href if none was specified.
176 --input-metalink=file
178 Downloads files covered in local Metalink file. Metalink version 3
179 and 4 are supported.
180 --keep-badhash
182 Keeps downloaded Metalink's files with a bad hash. It appends
183 .badhash to the name of Metalink's files which have a checksum
184 mismatch, except without overwriting existing files.
185 --metalink-over-http
187 Issues HTTP HEAD request instead of GET and extracts Metalink
188 metadata from response headers. Then it switches to Metalink
189 download. If no valid Metalink metadata is found, it falls back to
190 ordinary HTTP download. Enables Content-Type:
191 application/metalink4+xml files download/processing.
192 --metalink-index=number
194 Set the Metalink application/metalink4+xml metaurl ordinal NUMBER.
195 From 1 to the total number of "application/metalink4+xml"
196 available. Specify 0 or inf to choose the first good one.
197 Metaurls, such as those from a --metalink-over-http, may have been
198 sorted by priority key's value; keep this in mind to choose the
199 right NUMBER.
200 --preferred-location
202 Set preferred location for Metalink resources. This has effect if
203 multiple resources with same priority are available.
204 --xattr
206 Enable use of file system's extended attributes to save the
207 original URL and the Referer HTTP header value if used.
208 Be aware that the URL might contain private information like access
210 tokens or credentials.
211 -F
213 --force-html
214 When input is read from a file, force it to be treated as an HTML
215 file. This enables you to retrieve relative links from existing
216 HTML files on your local disk, by adding "<base href="url">" to
217 HTML, or using the --base command-line option.
218 -B URL
220 --base=URL
221 Resolves relative links using URL as the point of reference, when
222 reading links from an HTML file specified via the -i/--input-file
223 option (together with --force-html, or when the input file was
224 fetched remotely from a server describing it as HTML). This is
225 equivalent to the presence of a "BASE" tag in the HTML input file,
226 with URL as the value for the "href" attribute.
227 For instance, if you specify http://foo/bar/a.html for URL, and
229 Wget reads ../baz/b.html from the input file, it would be resolved
230 to http://foo/baz/b.html.
231 --config=FILE
233 Specify the location of a startup file you wish to use instead of
234 the default one(s). Use --no-config to disable reading of config
235 files. If both --config and --no-config are given, --no-config is
236 ignored.
237 --rejected-log=logfile
239 Logs all URL rejections to logfile as comma separated values. The
240 values include the reason of rejection, the URL and the parent URL
241 it was found in.
242 Download Options
244 --bind-address=ADDRESS
245 When making client TCP/IP connections, bind to ADDRESS on the local
246 machine. ADDRESS may be specified as a hostname or IP address.
247 This option can be useful if your machine is bound to multiple IPs.
248 --bind-dns-address=ADDRESS
250 [libcares only] This address overrides the route for DNS requests.
251 If you ever need to circumvent the standard settings from
252 /etc/resolv.conf, this option together with --dns-servers is your
253 friend. ADDRESS must be specified either as IPv4 or IPv6 address.
254 Wget needs to be built with libcares for this option to be
255 available.
256 --dns-servers=ADDRESSES
258 [libcares only] The given address(es) override the standard
259 nameserver addresses, e.g. as configured in /etc/resolv.conf.
260 ADDRESSES may be specified either as IPv4 or IPv6 addresses, comma-
261 separated. Wget needs to be built with libcares for this option to
262 be available.
263 -t number
265 --tries=number
266 Set number of tries to number. Specify 0 or inf for infinite
267 retrying. The default is to retry 20 times, with the exception of
268 fatal errors like "connection refused" or "not found" (404), which
269 are not retried.
270 -O file
272 --output-document=file
273 The documents will not be written to the appropriate files, but all
274 will be concatenated together and written to file. If - is used as
275 file, documents will be printed to standard output, disabling link
276 conversion. (Use ./- to print to a file literally named -.)
277 Use of -O is not intended to mean simply "use the name file instead
279 of the one in the URL;" rather, it is analogous to shell
280 redirection: wget -O file http://foo is intended to work like wget
281 -O - http://foo > file; file will be truncated immediately, and all
282 downloaded content will be written there.
283 For this reason, -N (for timestamp-checking) is not supported in
285 combination with -O: since file is always newly created, it will
286 always have a very new timestamp. A warning will be issued if this
287 combination is used.
288 Similarly, using -r or -p with -O may not work as you expect: Wget
290 won't just download the first file to file and then download the
291 rest to their normal names: all downloaded content will be placed
292 in file. This was disabled in version 1.11, but has been reinstated
293 (with a warning) in 1.11.2, as there are some cases where this
294 behavior can actually have some use.
295 A combination with -nc is only accepted if the given output file
297 does not exist.
298 Note that a combination with -k is only permitted when downloading
300 a single document, as in that case it will just convert all
301 relative URIs to external ones; -k makes no sense for multiple URIs
302 when they're all being downloaded to a single file; -k can be used
303 only when the output is a regular file.
304 -nc
306 --no-clobber
307 If a file is downloaded more than once in the same directory,
308 Wget's behavior depends on a few options, including -nc. In
309 certain cases, the local file will be clobbered, or overwritten,
310 upon repeated download. In other cases it will be preserved.
311 When running Wget without -N, -nc, -r, or -p, downloading the same
313 file in the same directory will result in the original copy of file
314 being preserved and the second copy being named file.1. If that
315 file is downloaded yet again, the third copy will be named file.2,
316 and so on. (This is also the behavior with -nd, even if -r or -p
317 are in effect.) When -nc is specified, this behavior is
318 suppressed, and Wget will refuse to download newer copies of file.
319 Therefore, ""no-clobber"" is actually a misnomer in this
320 mode---it's not clobbering that's prevented (as the numeric
321 suffixes were already preventing clobbering), but rather the
322 multiple version saving that's prevented.
323 When running Wget with -r or -p, but without -N, -nd, or -nc, re-
325 downloading a file will result in the new copy simply overwriting
326 the old. Adding -nc will prevent this behavior, instead causing
327 the original version to be preserved and any newer copies on the
328 server to be ignored.
329 When running Wget with -N, with or without -r or -p, the decision
331 as to whether or not to download a newer copy of a file depends on
332 the local and remote timestamp and size of the file. -nc may not
333 be specified at the same time as -N.
334 A combination with -O/--output-document is only accepted if the
336 given output file does not exist.
337 Note that when -nc is specified, files with the suffixes .html or
339 .htm will be loaded from the local disk and parsed as if they had
340 been retrieved from the Web.
341 --backups=backups
343 Before (over)writing a file, back up an existing file by adding a
344 .1 suffix (_1 on VMS) to the file name. Such backup files are
345 rotated to .2, .3, and so on, up to backups (and lost beyond that).
346 --no-netrc
348 Do not try to obtain credentials from .netrc file. By default
349 .netrc file is searched for credentials in case none have been
350 passed on command line and authentication is required.
351 -c
353 --continue
354 Continue getting a partially-downloaded file. This is useful when
355 you want to finish up a download started by a previous instance of
356 Wget, or by another program. For instance:
357 wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z
359 If there is a file named ls-lR.Z in the current directory, Wget
361 will assume that it is the first portion of the remote file, and
362 will ask the server to continue the retrieval from an offset equal
363 to the length of the local file.
364 Note that you don't need to specify this option if you just want
366 the current invocation of Wget to retry downloading a file should
367 the connection be lost midway through. This is the default
368 behavior. -c only affects resumption of downloads started prior to
369 this invocation of Wget, and whose local files are still sitting
370 around.
371 Without -c, the previous example would just download the remote
373 file to ls-lR.Z.1, leaving the truncated ls-lR.Z file alone.
374 If you use -c on a non-empty file, and the server does not support
376 continued downloading, Wget will restart the download from scratch
377 and overwrite the existing file entirely.
378 Beginning with Wget 1.7, if you use -c on a file which is of equal
380 size as the one on the server, Wget will refuse to download the
381 file and print an explanatory message. The same happens when the
382 file is smaller on the server than locally (presumably because it
383 was changed on the server since your last download
384 attempt)---because "continuing" is not meaningful, no download
385 occurs.
386 On the other side of the coin, while using -c, any file that's
388 bigger on the server than locally will be considered an incomplete
389 download and only "(length(remote) - length(local))" bytes will be
390 downloaded and tacked onto the end of the local file. This
391 behavior can be desirable in certain cases---for instance, you can
392 use wget -c to download just the new portion that's been appended
393 to a data collection or log file.
394 However, if the file is bigger on the server because it's been
396 changed, as opposed to just appended to, you'll end up with a
397 garbled file. Wget has no way of verifying that the local file is
398 really a valid prefix of the remote file. You need to be
399 especially careful of this when using -c in conjunction with -r,
400 since every file will be considered as an "incomplete download"
401 candidate.
402 Another instance where you'll get a garbled file if you try to use
404 -c is if you have a lame HTTP proxy that inserts a "transfer
405 interrupted" string into the local file. In the future a
406 "rollback" option may be added to deal with this case.
407 Note that -c only works with FTP servers and with HTTP servers that
409 support the "Range" header.
410 --start-pos=OFFSET
412 Start downloading at zero-based position OFFSET. Offset may be
413 expressed in bytes, kilobytes with the `k' suffix, or megabytes
414 with the `m' suffix, etc.
415 --start-pos has higher precedence over --continue. When
417 --start-pos and --continue are both specified, wget will emit a
418 warning then proceed as if --continue was absent.
419 Server support for continued download is required, otherwise
421 --start-pos cannot help. See -c for details.
422 --progress=type
424 Select the type of the progress indicator you wish to use. Legal
425 indicators are "dot" and "bar".
426 The "bar" indicator is used by default. It draws an ASCII progress
428 bar graphics (a.k.a "thermometer" display) indicating the status of
429 retrieval. If the output is not a TTY, the "dot" bar will be used
430 by default.
431 Use --progress=dot to switch to the "dot" display. It traces the
433 retrieval by printing dots on the screen, each dot representing a
434 fixed amount of downloaded data.
435 The progress type can also take one or more parameters. The
437 parameters vary based on the type selected. Parameters to type are
438 passed by appending them to the type sperated by a colon (:) like
439 this: --progress=type:parameter1:parameter2.
440 When using the dotted retrieval, you may set the style by
442 specifying the type as dot:style. Different styles assign
443 different meaning to one dot. With the "default" style each dot
444 represents 1K, there are ten dots in a cluster and 50 dots in a
445 line. The "binary" style has a more "computer"-like
446 orientation---8K dots, 16-dots clusters and 48 dots per line (which
447 makes for 384K lines). The "mega" style is suitable for
448 downloading large files---each dot represents 64K retrieved, there
449 are eight dots in a cluster, and 48 dots on each line (so each line
450 contains 3M). If "mega" is not enough then you can use the "giga"
451 style---each dot represents 1M retrieved, there are eight dots in a
452 cluster, and 32 dots on each line (so each line contains 32M).
453 With --progress=bar, there are currently two possible parameters,
455 force and noscroll.
456 When the output is not a TTY, the progress bar always falls back to
458 "dot", even if --progress=bar was passed to Wget during invocation.
459 This behaviour can be overridden and the "bar" output forced by
460 using the "force" parameter as --progress=bar:force.
461 By default, the bar style progress bar scroll the name of the file
463 from left to right for the file being downloaded if the filename
464 exceeds the maximum length allotted for its display. In certain
465 cases, such as with --progress=bar:force, one may not want the
466 scrolling filename in the progress bar. By passing the "noscroll"
467 parameter, Wget can be forced to display as much of the filename as
468 possible without scrolling through it.
469 Note that you can set the default style using the "progress"
471 command in .wgetrc. That setting may be overridden from the
472 command line. For example, to force the bar output without
473 scrolling, use --progress=bar:force:noscroll.
474 --show-progress
476 Force wget to display the progress bar in any verbosity.
477 By default, wget only displays the progress bar in verbose mode.
479 One may however, want wget to display the progress bar on screen in
480 conjunction with any other verbosity modes like --no-verbose or
481 --quiet. This is often a desired a property when invoking wget to
482 download several small/large files. In such a case, wget could
483 simply be invoked with this parameter to get a much cleaner output
484 on the screen.
485 This option will also force the progress bar to be printed to
487 stderr when used alongside the --output-file option.
488 -N
490 --timestamping
491 Turn on time-stamping.
492 --no-if-modified-since
494 Do not send If-Modified-Since header in -N mode. Send preliminary
495 HEAD request instead. This has only effect in -N mode.
496 --no-use-server-timestamps
498 Don't set the local file's timestamp by the one on the server.
499 By default, when a file is downloaded, its timestamps are set to
501 match those from the remote file. This allows the use of
502 --timestamping on subsequent invocations of wget. However, it is
503 sometimes useful to base the local file's timestamp on when it was
504 actually downloaded; for that purpose, the
505 --no-use-server-timestamps option has been provided.
506 -S
508 --server-response
509 Print the headers sent by HTTP servers and responses sent by FTP
510 servers.
511 --spider
513 When invoked with this option, Wget will behave as a Web spider,
514 which means that it will not download the pages, just check that
515 they are there. For example, you can use Wget to check your
516 bookmarks:
517 wget --spider --force-html -i bookmarks.html
519 This feature needs much more work for Wget to get close to the
521 functionality of real web spiders.
522 -T seconds
524 --timeout=seconds
525 Set the network timeout to seconds seconds. This is equivalent to
526 specifying --dns-timeout, --connect-timeout, and --read-timeout,
527 all at the same time.
528 When interacting with the network, Wget can check for timeout and
530 abort the operation if it takes too long. This prevents anomalies
531 like hanging reads and infinite connects. The only timeout enabled
532 by default is a 900-second read timeout. Setting a timeout to 0
533 disables it altogether. Unless you know what you are doing, it is
534 best not to change the default timeout settings.
535 All timeout-related options accept decimal values, as well as
537 subsecond values. For example, 0.1 seconds is a legal (though
538 unwise) choice of timeout. Subsecond timeouts are useful for
539 checking server response times or for testing network latency.
540 --dns-timeout=seconds
542 Set the DNS lookup timeout to seconds seconds. DNS lookups that
543 don't complete within the specified time will fail. By default,
544 there is no timeout on DNS lookups, other than that implemented by
545 system libraries.
546 --connect-timeout=seconds
548 Set the connect timeout to seconds seconds. TCP connections that
549 take longer to establish will be aborted. By default, there is no
550 connect timeout, other than that implemented by system libraries.
551 --read-timeout=seconds
553 Set the read (and write) timeout to seconds seconds. The "time" of
554 this timeout refers to idle time: if, at any point in the download,
555 no data is received for more than the specified number of seconds,
556 reading fails and the download is restarted. This option does not
557 directly affect the duration of the entire download.
558 Of course, the remote server may choose to terminate the connection
560 sooner than this option requires. The default read timeout is 900
561 seconds.
562 --limit-rate=amount
564 Limit the download speed to amount bytes per second. Amount may be
565 expressed in bytes, kilobytes with the k suffix, or megabytes with
566 the m suffix. For example, --limit-rate=20k will limit the
567 retrieval rate to 20KB/s. This is useful when, for whatever
568 reason, you don't want Wget to consume the entire available
569 bandwidth.
570 This option allows the use of decimal numbers, usually in
572 conjunction with power suffixes; for example, --limit-rate=2.5k is
573 a legal value.
574 Note that Wget implements the limiting by sleeping the appropriate
576 amount of time after a network read that took less time than
577 specified by the rate. Eventually this strategy causes the TCP
578 transfer to slow down to approximately the specified rate.
579 However, it may take some time for this balance to be achieved, so
580 don't be surprised if limiting the rate doesn't work well with very
581 small files.
582 -w seconds
584 --wait=seconds
585 Wait the specified number of seconds between the retrievals. Use
586 of this option is recommended, as it lightens the server load by
587 making the requests less frequent. Instead of in seconds, the time
588 can be specified in minutes using the "m" suffix, in hours using
589 "h" suffix, or in days using "d" suffix.
590 Specifying a large value for this option is useful if the network
592 or the destination host is down, so that Wget can wait long enough
593 to reasonably expect the network error to be fixed before the
594 retry. The waiting interval specified by this function is
595 influenced by "--random-wait", which see.
596 --waitretry=seconds
598 If you don't want Wget to wait between every retrieval, but only
599 between retries of failed downloads, you can use this option. Wget
600 will use linear backoff, waiting 1 second after the first failure
601 on a given file, then waiting 2 seconds after the second failure on
602 that file, up to the maximum number of seconds you specify.
603 By default, Wget will assume a value of 10 seconds.
605 --random-wait
607 Some web sites may perform log analysis to identify retrieval
608 programs such as Wget by looking for statistically significant
609 similarities in the time between requests. This option causes the
610 time between requests to vary between 0.5 and 1.5 * wait seconds,
611 where wait was specified using the --wait option, in order to mask
612 Wget's presence from such analysis.
613 A 2001 article in a publication devoted to development on a popular
615 consumer platform provided code to perform this analysis on the
616 fly. Its author suggested blocking at the class C address level to
617 ensure automated retrieval programs were blocked despite changing
618 DHCP-supplied addresses.
619 The --random-wait option was inspired by this ill-advised
621 recommendation to block many unrelated users from a web site due to
622 the actions of one.
623 --no-proxy
625 Don't use proxies, even if the appropriate *_proxy environment
626 variable is defined.
627 -Q quota
629 --quota=quota
630 Specify download quota for automatic retrievals. The value can be
631 specified in bytes (default), kilobytes (with k suffix), or
632 megabytes (with m suffix).
633 Note that quota will never affect downloading a single file. So if
635 you specify wget -Q10k https://example.com/ls-lR.gz, all of the
636 ls-lR.gz will be downloaded. The same goes even when several URLs
637 are specified on the command-line. However, quota is respected
638 when retrieving either recursively, or from an input file. Thus
639 you may safely type wget -Q2m -i sites---download will be aborted
640 when the quota is exceeded.
641 Setting quota to 0 or to inf unlimits the download quota.
643 --no-dns-cache
645 Turn off caching of DNS lookups. Normally, Wget remembers the IP
646 addresses it looked up from DNS so it doesn't have to repeatedly
647 contact the DNS server for the same (typically small) set of hosts
648 it retrieves from. This cache exists in memory only; a new Wget
649 run will contact DNS again.
650 However, it has been reported that in some situations it is not
652 desirable to cache host names, even for the duration of a short-
653 running application like Wget. With this option Wget issues a new
654 DNS lookup (more precisely, a new call to "gethostbyname" or
655 "getaddrinfo") each time it makes a new connection. Please note
656 that this option will not affect caching that might be performed by
657 the resolving library or by an external caching layer, such as
658 NSCD.
659 If you don't understand exactly what this option does, you probably
661 won't need it.
662 --restrict-file-names=modes
664 Change which characters found in remote URLs must be escaped during
665 generation of local filenames. Characters that are restricted by
666 this option are escaped, i.e. replaced with %HH, where HH is the
667 hexadecimal number that corresponds to the restricted character.
668 This option may also be used to force all alphabetical cases to be
669 either lower- or uppercase.
670 By default, Wget escapes the characters that are not valid or safe
672 as part of file names on your operating system, as well as control
673 characters that are typically unprintable. This option is useful
674 for changing these defaults, perhaps because you are downloading to
675 a non-native partition, or because you want to disable escaping of
676 the control characters, or you want to further restrict characters
677 to only those in the ASCII range of values.
678 The modes are a comma-separated set of text values. The acceptable
680 values are unix, windows, nocontrol, ascii, lowercase, and
681 uppercase. The values unix and windows are mutually exclusive (one
682 will override the other), as are lowercase and uppercase. Those
683 last are special cases, as they do not change the set of characters
684 that would be escaped, but rather force local file paths to be
685 converted either to lower- or uppercase.
686 When "unix" is specified, Wget escapes the character / and the
688 control characters in the ranges 0--31 and 128--159. This is the
689 default on Unix-like operating systems.
690 When "windows" is given, Wget escapes the characters \, |, /, :, ?,
692 ", *, <, >, and the control characters in the ranges 0--31 and
693 128--159. In addition to this, Wget in Windows mode uses + instead
694 of : to separate host and port in local file names, and uses @
695 instead of ? to separate the query portion of the file name from
696 the rest. Therefore, a URL that would be saved as
697 www.xemacs.org:4300/search.pl?input=blah in Unix mode would be
698 saved as www.xemacs.org+4300/search.pl@input=blah in Windows mode.
699 This mode is the default on Windows.
700 If you specify nocontrol, then the escaping of the control
702 characters is also switched off. This option may make sense when
703 you are downloading URLs whose names contain UTF-8 characters, on a
704 system which can save and display filenames in UTF-8 (some possible
705 byte values used in UTF-8 byte sequences fall in the range of
706 values designated by Wget as "controls").
707 The ascii mode is used to specify that any bytes whose values are
709 outside the range of ASCII characters (that is, greater than 127)
710 shall be escaped. This can be useful when saving filenames whose
711 encoding does not match the one used locally.
712 -4
714 --inet4-only
715 -6
716 --inet6-only
717 Force connecting to IPv4 or IPv6 addresses. With --inet4-only or
718 -4, Wget will only connect to IPv4 hosts, ignoring AAAA records in
719 DNS, and refusing to connect to IPv6 addresses specified in URLs.
720 Conversely, with --inet6-only or -6, Wget will only connect to IPv6
721 hosts and ignore A records and IPv4 addresses.
722 Neither options should be needed normally. By default, an
724 IPv6-aware Wget will use the address family specified by the host's
725 DNS record. If the DNS responds with both IPv4 and IPv6 addresses,
726 Wget will try them in sequence until it finds one it can connect
727 to. (Also see "--prefer-family" option described below.)
728 These options can be used to deliberately force the use of IPv4 or
730 IPv6 address families on dual family systems, usually to aid
731 debugging or to deal with broken network configuration. Only one
732 of --inet6-only and --inet4-only may be specified at the same time.
733 Neither option is available in Wget compiled without IPv6 support.
734 --prefer-family=none/IPv4/IPv6
736 When given a choice of several addresses, connect to the addresses
737 with specified address family first. The address order returned by
738 DNS is used without change by default.
739 This avoids spurious errors and connect attempts when accessing
741 hosts that resolve to both IPv6 and IPv4 addresses from IPv4
742 networks. For example, www.kame.net resolves to
743 2001:200:0:8002:203:47ff:fea5:3085 and to 203.178.141.194. When
744 the preferred family is "IPv4", the IPv4 address is used first;
745 when the preferred family is "IPv6", the IPv6 address is used
746 first; if the specified value is "none", the address order returned
747 by DNS is used without change.
748 Unlike -4 and -6, this option doesn't inhibit access to any address
750 family, it only changes the order in which the addresses are
751 accessed. Also note that the reordering performed by this option
752 is stable---it doesn't affect order of addresses of the same
753 family. That is, the relative order of all IPv4 addresses and of
754 all IPv6 addresses remains intact in all cases.
755 --retry-connrefused
757 Consider "connection refused" a transient error and try again.
758 Normally Wget gives up on a URL when it is unable to connect to the
759 site because failure to connect is taken as a sign that the server
760 is not running at all and that retries would not help. This option
761 is for mirroring unreliable sites whose servers tend to disappear
762 for short periods of time.
763 --user=user
765 --password=password
766 Specify the username user and password password for both FTP and
767 HTTP file retrieval. These parameters can be overridden using the
768 --ftp-user and --ftp-password options for FTP connections and the
769 --http-user and --http-password options for HTTP connections.
770 --ask-password
772 Prompt for a password for each connection established. Cannot be
773 specified when --password is being used, because they are mutually
774 exclusive.
775 --use-askpass=command
777 Prompt for a user and password using the specified command. If no
778 command is specified then the command in the environment variable
779 WGET_ASKPASS is used. If WGET_ASKPASS is not set then the command
780 in the environment variable SSH_ASKPASS is used.
781 You can set the default command for use-askpass in the .wgetrc.
783 That setting may be overridden from the command line.
784 --no-iri
786 Turn off internationalized URI (IRI) support. Use --iri to turn it
787 on. IRI support is activated by default.
788 You can set the default state of IRI support using the "iri"
790 command in .wgetrc. That setting may be overridden from the command
791 line.
792 --local-encoding=encoding
794 Force Wget to use encoding as the default system encoding. That
795 affects how Wget converts URLs specified as arguments from locale
796 to UTF-8 for IRI support.
797 Wget use the function "nl_langinfo()" and then the "CHARSET"
799 environment variable to get the locale. If it fails, ASCII is used.
800 You can set the default local encoding using the "local_encoding"
802 command in .wgetrc. That setting may be overridden from the command
803 line.
804 --remote-encoding=encoding
806 Force Wget to use encoding as the default remote server encoding.
807 That affects how Wget converts URIs found in files from remote
808 encoding to UTF-8 during a recursive fetch. This options is only
809 useful for IRI support, for the interpretation of non-ASCII
810 characters.
811 For HTTP, remote encoding can be found in HTTP "Content-Type"
813 header and in HTML "Content-Type http-equiv" meta tag.
814 You can set the default encoding using the "remoteencoding" command
816 in .wgetrc. That setting may be overridden from the command line.
817 --unlink
819 Force Wget to unlink file instead of clobbering existing file. This
820 option is useful for downloading to the directory with hardlinks.
821 Directory Options
823 -nd
824 --no-directories
825 Do not create a hierarchy of directories when retrieving
826 recursively. With this option turned on, all files will get saved
827 to the current directory, without clobbering (if a name shows up
828 more than once, the filenames will get extensions .n).
829 -x
831 --force-directories
832 The opposite of -nd---create a hierarchy of directories, even if
833 one would not have been created otherwise. E.g. wget -x
834 http://fly.srk.fer.hr/robots.txt will save the downloaded file to
835 fly.srk.fer.hr/robots.txt.
836 -nH
838 --no-host-directories
839 Disable generation of host-prefixed directories. By default,
840 invoking Wget with -r http://fly.srk.fer.hr/ will create a
841 structure of directories beginning with fly.srk.fer.hr/. This
842 option disables such behavior.
843 --protocol-directories
845 Use the protocol name as a directory component of local file names.
846 For example, with this option, wget -r http://host will save to
847 http/host/... rather than just to host/....
848 --cut-dirs=number
850 Ignore number directory components. This is useful for getting a
851 fine-grained control over the directory where recursive retrieval
852 will be saved.
853 Take, for example, the directory at
855 ftp://ftp.xemacs.org/pub/xemacs/. If you retrieve it with -r, it
856 will be saved locally under ftp.xemacs.org/pub/xemacs/. While the
857 -nH option can remove the ftp.xemacs.org/ part, you are still stuck
858 with pub/xemacs. This is where --cut-dirs comes in handy; it makes
859 Wget not "see" number remote directory components. Here are
860 several examples of how --cut-dirs option works.
861 No options -> ftp.xemacs.org/pub/xemacs/
863 -nH -> pub/xemacs/
864 -nH --cut-dirs=1 -> xemacs/
865 -nH --cut-dirs=2 -> .
866 --cut-dirs=1 -> ftp.xemacs.org/xemacs/
868 ...
869 If you just want to get rid of the directory structure, this option
871 is similar to a combination of -nd and -P. However, unlike -nd,
872 --cut-dirs does not lose with subdirectories---for instance, with
873 -nH --cut-dirs=1, a beta/ subdirectory will be placed to
874 xemacs/beta, as one would expect.
875 -P prefix
877 --directory-prefix=prefix
878 Set directory prefix to prefix. The directory prefix is the
879 directory where all other files and subdirectories will be saved
880 to, i.e. the top of the retrieval tree. The default is . (the
881 current directory).
882 HTTP Options
884 --default-page=name
885 Use name as the default file name when it isn't known (i.e., for
886 URLs that end in a slash), instead of index.html.
887 -E
889 --adjust-extension
890 If a file of type application/xhtml+xml or text/html is downloaded
891 and the URL does not end with the regexp \.[Hh][Tt][Mm][Ll]?, this
892 option will cause the suffix .html to be appended to the local
893 filename. This is useful, for instance, when you're mirroring a
894 remote site that uses .asp pages, but you want the mirrored pages
895 to be viewable on your stock Apache server. Another good use for
896 this is when you're downloading CGI-generated materials. A URL
897 like http://site.com/article.cgi?25 will be saved as
898 article.cgi?25.html.
899 Note that filenames changed in this way will be re-downloaded every
901 time you re-mirror a site, because Wget can't tell that the local
902 X.html file corresponds to remote URL X (since it doesn't yet know
903 that the URL produces output of type text/html or
904 application/xhtml+xml.
905 As of version 1.12, Wget will also ensure that any downloaded files
907 of type text/css end in the suffix .css, and the option was renamed
908 from --html-extension, to better reflect its new behavior. The old
909 option name is still acceptable, but should now be considered
910 deprecated.
911 As of version 1.19.2, Wget will also ensure that any downloaded
913 files with a "Content-Encoding" of br, compress, deflate or gzip
914 end in the suffix .br, .Z, .zlib and .gz respectively.
915 At some point in the future, this option may well be expanded to
917 include suffixes for other types of content, including content
918 types that are not parsed by Wget.
919 --http-user=user
921 --http-password=password
922 Specify the username user and password password on an HTTP server.
923 According to the type of the challenge, Wget will encode them using
924 either the "basic" (insecure), the "digest", or the Windows "NTLM"
925 authentication scheme.
926 Another way to specify username and password is in the URL itself.
928 Either method reveals your password to anyone who bothers to run
929 "ps". To prevent the passwords from being seen, use the
930 --use-askpass or store them in .wgetrc or .netrc, and make sure to
931 protect those files from other users with "chmod". If the
932 passwords are really important, do not leave them lying in those
933 files either---edit the files and delete them after Wget has
934 started the download.
935 --no-http-keep-alive
937 Turn off the "keep-alive" feature for HTTP downloads. Normally,
938 Wget asks the server to keep the connection open so that, when you
939 download more than one document from the same server, they get
940 transferred over the same TCP connection. This saves time and at
941 the same time reduces the load on the server.
942 This option is useful when, for some reason, persistent (keep-
944 alive) connections don't work for you, for example due to a server
945 bug or due to the inability of server-side scripts to cope with the
946 connections.
947 --no-cache
949 Disable server-side cache. In this case, Wget will send the remote
950 server appropriate directives (Cache-Control: no-cache and Pragma:
951 no-cache) to get the file from the remote service, rather than
952 returning the cached version. This is especially useful for
953 retrieving and flushing out-of-date documents on proxy servers.
954 Caching is allowed by default.
956 --no-cookies
958 Disable the use of cookies. Cookies are a mechanism for
959 maintaining server-side state. The server sends the client a
960 cookie using the "Set-Cookie" header, and the client responds with
961 the same cookie upon further requests. Since cookies allow the
962 server owners to keep track of visitors and for sites to exchange
963 this information, some consider them a breach of privacy. The
964 default is to use cookies; however, storing cookies is not on by
965 default.
966 --load-cookies file
968 Load cookies from file before the first HTTP retrieval. file is a
969 textual file in the format originally used by Netscape's
970 cookies.txt file.
971 You will typically use this option when mirroring sites that
973 require that you be logged in to access some or all of their
974 content. The login process typically works by the web server
975 issuing an HTTP cookie upon receiving and verifying your
976 credentials. The cookie is then resent by the browser when
977 accessing that part of the site, and so proves your identity.
978 Mirroring such a site requires Wget to send the same cookies your
980 browser sends when communicating with the site. This is achieved
981 by --load-cookies---simply point Wget to the location of the
982 cookies.txt file, and it will send the same cookies your browser
983 would send in the same situation. Different browsers keep textual
984 cookie files in different locations:
985 "Netscape 4.x."
987 The cookies are in ~/.netscape/cookies.txt.
988 "Mozilla and Netscape 6.x."
990 Mozilla's cookie file is also named cookies.txt, located
991 somewhere under ~/.mozilla, in the directory of your profile.
992 The full path usually ends up looking somewhat like
993 ~/.mozilla/default/some-weird-string/cookies.txt.
994 "Internet Explorer."
996 You can produce a cookie file Wget can use by using the File
997 menu, Import and Export, Export Cookies. This has been tested
998 with Internet Explorer 5; it is not guaranteed to work with
999 earlier versions.
1000 "Other browsers."
1002 If you are using a different browser to create your cookies,
1003 --load-cookies will only work if you can locate or produce a
1004 cookie file in the Netscape format that Wget expects.
1005 If you cannot use --load-cookies, there might still be an
1007 alternative. If your browser supports a "cookie manager", you can
1008 use it to view the cookies used when accessing the site you're
1009 mirroring. Write down the name and value of the cookie, and
1010 manually instruct Wget to send those cookies, bypassing the
1011 "official" cookie support:
1012 wget --no-cookies --header "Cookie: <name>=<value>"
1014 --save-cookies file
1016 Save cookies to file before exiting. This will not save cookies
1017 that have expired or that have no expiry time (so-called "session
1018 cookies"), but also see --keep-session-cookies.
1019 --keep-session-cookies
1021 When specified, causes --save-cookies to also save session cookies.
1022 Session cookies are normally not saved because they are meant to be
1023 kept in memory and forgotten when you exit the browser. Saving
1024 them is useful on sites that require you to log in or to visit the
1025 home page before you can access some pages. With this option,
1026 multiple Wget runs are considered a single browser session as far
1027 as the site is concerned.
1028 Since the cookie file format does not normally carry session
1030 cookies, Wget marks them with an expiry timestamp of 0. Wget's
1031 --load-cookies recognizes those as session cookies, but it might
1032 confuse other browsers. Also note that cookies so loaded will be
1033 treated as other session cookies, which means that if you want
1034 --save-cookies to preserve them again, you must use
1035 --keep-session-cookies again.
1036 --ignore-length
1038 Unfortunately, some HTTP servers (CGI programs, to be more precise)
1039 send out bogus "Content-Length" headers, which makes Wget go wild,
1040 as it thinks not all the document was retrieved. You can spot this
1041 syndrome if Wget retries getting the same document again and again,
1042 each time claiming that the (otherwise normal) connection has
1043 closed on the very same byte.
1044 With this option, Wget will ignore the "Content-Length" header---as
1046 if it never existed.
1047 --header=header-line
1049 Send header-line along with the rest of the headers in each HTTP
1050 request. The supplied header is sent as-is, which means it must
1051 contain name and value separated by colon, and must not contain
1052 newlines.
1053 You may define more than one additional header by specifying
1055 --header more than once.
1056 wget --header='Accept-Charset: iso-8859-2' \
1058 --header='Accept-Language: hr' \
1059 http://fly.srk.fer.hr/
1060 Specification of an empty string as the header value will clear all
1062 previous user-defined headers.
1063 As of Wget 1.10, this option can be used to override headers
1065 otherwise generated automatically. This example instructs Wget to
1066 connect to localhost, but to specify foo.bar in the "Host" header:
1067 wget --header="Host: foo.bar" http://localhost/
1069 In versions of Wget prior to 1.10 such use of --header caused
1071 sending of duplicate headers.
1072 --compression=type
1074 Choose the type of compression to be used. Legal values are auto,
1075 gzip and none.
1076 If auto or gzip are specified, Wget asks the server to compress the
1078 file using the gzip compression format. If the server compresses
1079 the file and responds with the "Content-Encoding" header field set
1080 appropriately, the file will be decompressed automatically.
1081 If none is specified, wget will not ask the server to compress the
1083 file and will not decompress any server responses. This is the
1084 default.
1085 Compression support is currently experimental. In case it is turned
1087 on, please report any bugs to "bug-wget@gnu.org".
1088 --max-redirect=number
1090 Specifies the maximum number of redirections to follow for a
1091 resource. The default is 20, which is usually far more than
1092 necessary. However, on those occasions where you want to allow more
1093 (or fewer), this is the option to use.
1094 --proxy-user=user
1096 --proxy-password=password
1097 Specify the username user and password password for authentication
1098 on a proxy server. Wget will encode them using the "basic"
1099 authentication scheme.
1100 Security considerations similar to those with --http-password
1102 pertain here as well.
1103 --referer=url
1105 Include `Referer: url' header in HTTP request. Useful for
1106 retrieving documents with server-side processing that assume they
1107 are always being retrieved by interactive web browsers and only
1108 come out properly when Referer is set to one of the pages that
1109 point to them.
1110 --save-headers
1112 Save the headers sent by the HTTP server to the file, preceding the
1113 actual contents, with an empty line as the separator.
1114 -U agent-string
1116 --user-agent=agent-string
1117 Identify as agent-string to the HTTP server.
1118 The HTTP protocol allows the clients to identify themselves using a
1120 "User-Agent" header field. This enables distinguishing the WWW
1121 software, usually for statistical purposes or for tracing of
1122 protocol violations. Wget normally identifies as Wget/version,
1123 version being the current version number of Wget.
1124 However, some sites have been known to impose the policy of
1126 tailoring the output according to the "User-Agent"-supplied
1127 information. While this is not such a bad idea in theory, it has
1128 been abused by servers denying information to clients other than
1129 (historically) Netscape or, more frequently, Microsoft Internet
1130 Explorer. This option allows you to change the "User-Agent" line
1131 issued by Wget. Use of this option is discouraged, unless you
1132 really know what you are doing.
1133 Specifying empty user agent with --user-agent="" instructs Wget not
1135 to send the "User-Agent" header in HTTP requests.
1136 --post-data=string
1138 --post-file=file
1139 Use POST as the method for all HTTP requests and send the specified
1140 data in the request body. --post-data sends string as data,
1141 whereas --post-file sends the contents of file. Other than that,
1142 they work in exactly the same way. In particular, they both expect
1143 content of the form "key1=value1&key2=value2", with percent-
1144 encoding for special characters; the only difference is that one
1145 expects its content as a command-line parameter and the other
1146 accepts its content from a file. In particular, --post-file is not
1147 for transmitting files as form attachments: those must appear as
1148 "key=value" data (with appropriate percent-coding) just like
1149 everything else. Wget does not currently support
1150 "multipart/form-data" for transmitting POST data; only
1151 "application/x-www-form-urlencoded". Only one of --post-data and
1152 --post-file should be specified.
1153 Please note that wget does not require the content to be of the
1155 form "key1=value1&key2=value2", and neither does it test for it.
1156 Wget will simply transmit whatever data is provided to it. Most
1157 servers however expect the POST data to be in the above format when
1158 processing HTML Forms.
1159 When sending a POST request using the --post-file option, Wget
1161 treats the file as a binary file and will send every character in
1162 the POST request without stripping trailing newline or formfeed
1163 characters. Any other control characters in the text will also be
1164 sent as-is in the POST request.
1165 Please be aware that Wget needs to know the size of the POST data
1167 in advance. Therefore the argument to "--post-file" must be a
1168 regular file; specifying a FIFO or something like /dev/stdin won't
1169 work. It's not quite clear how to work around this limitation
1170 inherent in HTTP/1.0. Although HTTP/1.1 introduces chunked
1171 transfer that doesn't require knowing the request length in
1172 advance, a client can't use chunked unless it knows it's talking to
1173 an HTTP/1.1 server. And it can't know that until it receives a
1174 response, which in turn requires the request to have been completed
1175 -- a chicken-and-egg problem.
1176 Note: As of version 1.15 if Wget is redirected after the POST
1178 request is completed, its behaviour will depend on the response
1179 code returned by the server. In case of a 301 Moved Permanently,
1180 302 Moved Temporarily or 307 Temporary Redirect, Wget will, in
1181 accordance with RFC2616, continue to send a POST request. In case
1182 a server wants the client to change the Request method upon
1183 redirection, it should send a 303 See Other response code.
1184 This example shows how to log in to a server using POST and then
1186 proceed to download the desired pages, presumably only accessible
1187 to authorized users:
1188 # Log in to the server. This can be done only once.
1190 wget --save-cookies cookies.txt \
1191 --post-data 'user=foo&password=bar' \
1192 http://example.com/auth.php
1193 # Now grab the page or pages we care about.
1195 wget --load-cookies cookies.txt \
1196 -p http://example.com/interesting/article.php
1197 If the server is using session cookies to track user
1199 authentication, the above will not work because --save-cookies will
1200 not save them (and neither will browsers) and the cookies.txt file
1201 will be empty. In that case use --keep-session-cookies along with
1202 --save-cookies to force saving of session cookies.
1203 --method=HTTP-Method
1205 For the purpose of RESTful scripting, Wget allows sending of other
1206 HTTP Methods without the need to explicitly set them using
1207 --header=Header-Line. Wget will use whatever string is passed to
1208 it after --method as the HTTP Method to the server.
1209 --body-data=Data-String
1211 --body-file=Data-File
1212 Must be set when additional data needs to be sent to the server
1213 along with the Method specified using --method. --body-data sends
1214 string as data, whereas --body-file sends the contents of file.
1215 Other than that, they work in exactly the same way.
1216 Currently, --body-file is not for transmitting files as a whole.
1218 Wget does not currently support "multipart/form-data" for
1219 transmitting data; only "application/x-www-form-urlencoded". In the
1220 future, this may be changed so that wget sends the --body-file as a
1221 complete file instead of sending its contents to the server. Please
1222 be aware that Wget needs to know the contents of BODY Data in
1223 advance, and hence the argument to --body-file should be a regular
1224 file. See --post-file for a more detailed explanation. Only one of
1225 --body-data and --body-file should be specified.
1226 If Wget is redirected after the request is completed, Wget will
1228 suspend the current method and send a GET request till the
1229 redirection is completed. This is true for all redirection
1230 response codes except 307 Temporary Redirect which is used to
1231 explicitly specify that the request method should not change.
1232 Another exception is when the method is set to "POST", in which
1233 case the redirection rules specified under --post-data are
1234 followed.
1235 --content-disposition
1237 If this is set to on, experimental (not fully-functional) support
1238 for "Content-Disposition" headers is enabled. This can currently
1239 result in extra round-trips to the server for a "HEAD" request, and
1240 is known to suffer from a few bugs, which is why it is not
1241 currently enabled by default.
1242 This option is useful for some file-downloading CGI programs that
1244 use "Content-Disposition" headers to describe what the name of a
1245 downloaded file should be.
1246 When combined with --metalink-over-http and --trust-server-names, a
1248 Content-Type: application/metalink4+xml file is named using the
1249 "Content-Disposition" filename field, if available.
1250 --content-on-error
1252 If this is set to on, wget will not skip the content when the
1253 server responds with a http status code that indicates error.
1254 --trust-server-names
1256 If this is set, on a redirect, the local file name will be based on
1257 the redirection URL. By default the local file name is based on
1258 the original URL. When doing recursive retrieving this can be
1259 helpful because in many web sites redirected URLs correspond to an
1260 underlying file structure, while link URLs do not.
1261 --auth-no-challenge
1263 If this option is given, Wget will send Basic HTTP authentication
1264 information (plaintext username and password) for all requests,
1265 just like Wget 1.10.2 and prior did by default.
1266 Use of this option is not recommended, and is intended only to
1268 support some few obscure servers, which never send HTTP
1269 authentication challenges, but accept unsolicited auth info, say,
1270 in addition to form-based authentication.
1271 --retry-on-host-error
1273 Consider host errors, such as "Temporary failure in name
1274 resolution", as non-fatal, transient errors.
1275 --retry-on-http-error=code[,code,...]
1277 Consider given HTTP response codes as non-fatal, transient errors.
1278 Supply a comma-separated list of 3-digit HTTP response codes as
1279 argument. Useful to work around special circumstances where retries
1280 are required, but the server responds with an error code normally
1281 not retried by Wget. Such errors might be 503 (Service Unavailable)
1282 and 429 (Too Many Requests). Retries enabled by this option are
1283 performed subject to the normal retry timing and retry count
1284 limitations of Wget.
1285 Using this option is intended to support special use cases only and
1287 is generally not recommended, as it can force retries even in cases
1288 where the server is actually trying to decrease its load. Please
1289 use wisely and only if you know what you are doing.
1290 HTTPS (SSL/TLS) Options
1292 To support encrypted HTTP (HTTPS) downloads, Wget must be compiled with
1293 an external SSL library. The current default is GnuTLS. In addition,
1294 Wget also supports HSTS (HTTP Strict Transport Security). If Wget is
1295 compiled without SSL support, none of these options are available.
1296 --secure-protocol=protocol
1298 Choose the secure protocol to be used. Legal values are auto,
1299 SSLv2, SSLv3, TLSv1, TLSv1_1, TLSv1_2, TLSv1_3 and PFS. If auto is
1300 used, the SSL library is given the liberty of choosing the
1301 appropriate protocol automatically, which is achieved by sending a
1302 TLSv1 greeting. This is the default.
1303 Specifying SSLv2, SSLv3, TLSv1, TLSv1_1, TLSv1_2 or TLSv1_3 forces
1305 the use of the corresponding protocol. This is useful when talking
1306 to old and buggy SSL server implementations that make it hard for
1307 the underlying SSL library to choose the correct protocol version.
1308 Fortunately, such servers are quite rare.
1309 Specifying PFS enforces the use of the so-called Perfect Forward
1311 Security cipher suites. In short, PFS adds security by creating a
1312 one-time key for each SSL connection. It has a bit more CPU impact
1313 on client and server. We use known to be secure ciphers (e.g. no
1314 MD4) and the TLS protocol. This mode also explicitly excludes non-
1315 PFS key exchange methods, such as RSA.
1316 --https-only
1318 When in recursive mode, only HTTPS links are followed.
1319 --ciphers
1321 Set the cipher list string. Typically this string sets the cipher
1322 suites and other SSL/TLS options that the user wish should be used,
1323 in a set order of preference (GnuTLS calls it 'priority string').
1324 This string will be fed verbatim to the SSL/TLS engine (OpenSSL or
1325 GnuTLS) and hence its format and syntax is dependent on that. Wget
1326 will not process or manipulate it in any way. Refer to the OpenSSL
1327 or GnuTLS documentation for more information.
1328 --no-check-certificate
1330 Don't check the server certificate against the available
1331 certificate authorities. Also don't require the URL host name to
1332 match the common name presented by the certificate.
1333 As of Wget 1.10, the default is to verify the server's certificate
1335 against the recognized certificate authorities, breaking the SSL
1336 handshake and aborting the download if the verification fails.
1337 Although this provides more secure downloads, it does break
1338 interoperability with some sites that worked with previous Wget
1339 versions, particularly those using self-signed, expired, or
1340 otherwise invalid certificates. This option forces an "insecure"
1341 mode of operation that turns the certificate verification errors
1342 into warnings and allows you to proceed.
1343 If you encounter "certificate verification" errors or ones saying
1345 that "common name doesn't match requested host name", you can use
1346 this option to bypass the verification and proceed with the
1347 download. Only use this option if you are otherwise convinced of
1348 the site's authenticity, or if you really don't care about the
1349 validity of its certificate. It is almost always a bad idea not to
1350 check the certificates when transmitting confidential or important
1351 data. For self-signed/internal certificates, you should download
1352 the certificate and verify against that instead of forcing this
1353 insecure mode. If you are really sure of not desiring any
1354 certificate verification, you can specify --check-certificate=quiet
1355 to tell wget to not print any warning about invalid certificates,
1356 albeit in most cases this is the wrong thing to do.
1357 --certificate=file
1359 Use the client certificate stored in file. This is needed for
1360 servers that are configured to require certificates from the
1361 clients that connect to them. Normally a certificate is not
1362 required and this switch is optional.
1363 --certificate-type=type
1365 Specify the type of the client certificate. Legal values are PEM
1366 (assumed by default) and DER, also known as ASN1.
1367 --private-key=file
1369 Read the private key from file. This allows you to provide the
1370 private key in a file separate from the certificate.
1371 --private-key-type=type
1373 Specify the type of the private key. Accepted values are PEM (the
1374 default) and DER.
1375 --ca-certificate=file
1377 Use file as the file with the bundle of certificate authorities
1378 ("CA") to verify the peers. The certificates must be in PEM
1379 format.
1380 Without this option Wget looks for CA certificates at the system-
1382 specified locations, chosen at OpenSSL installation time.
1383 --ca-directory=directory
1385 Specifies directory containing CA certificates in PEM format. Each
1386 file contains one CA certificate, and the file name is based on a
1387 hash value derived from the certificate. This is achieved by
1388 processing a certificate directory with the "c_rehash" utility
1389 supplied with OpenSSL. Using --ca-directory is more efficient than
1390 --ca-certificate when many certificates are installed because it
1391 allows Wget to fetch certificates on demand.
1392 Without this option Wget looks for CA certificates at the system-
1394 specified locations, chosen at OpenSSL installation time.
1395 --crl-file=file
1397 Specifies a CRL file in file. This is needed for certificates that
1398 have been revocated by the CAs.
1399 --pinnedpubkey=file/hashes
1401 Tells wget to use the specified public key file (or hashes) to
1402 verify the peer. This can be a path to a file which contains a
1403 single public key in PEM or DER format, or any number of base64
1404 encoded sha256 hashes preceded by "sha256//" and separated by ";"
1405 When negotiating a TLS or SSL connection, the server sends a
1407 certificate indicating its identity. A public key is extracted from
1408 this certificate and if it does not exactly match the public key(s)
1409 provided to this option, wget will abort the connection before
1410 sending or receiving any data.
1411 --random-file=file
1413 [OpenSSL and LibreSSL only] Use file as the source of random data
1414 for seeding the pseudo-random number generator on systems without
1415 /dev/urandom.
1416 On such systems the SSL library needs an external source of
1418 randomness to initialize. Randomness may be provided by EGD (see
1419 --egd-file below) or read from an external source specified by the
1420 user. If this option is not specified, Wget looks for random data
1421 in $RANDFILE or, if that is unset, in $HOME/.rnd.
1422 If you're getting the "Could not seed OpenSSL PRNG; disabling SSL."
1424 error, you should provide random data using some of the methods
1425 described above.
1426 --egd-file=file
1428 [OpenSSL only] Use file as the EGD socket. EGD stands for Entropy
1429 Gathering Daemon, a user-space program that collects data from
1430 various unpredictable system sources and makes it available to
1431 other programs that might need it. Encryption software, such as
1432 the SSL library, needs sources of non-repeating randomness to seed
1433 the random number generator used to produce cryptographically
1434 strong keys.
1435 OpenSSL allows the user to specify his own source of entropy using
1437 the "RAND_FILE" environment variable. If this variable is unset,
1438 or if the specified file does not produce enough randomness,
1439 OpenSSL will read random data from EGD socket specified using this
1440 option.
1441 If this option is not specified (and the equivalent startup command
1443 is not used), EGD is never contacted. EGD is not needed on modern
1444 Unix systems that support /dev/urandom.
1445 --no-hsts
1447 Wget supports HSTS (HTTP Strict Transport Security, RFC 6797) by
1448 default. Use --no-hsts to make Wget act as a non-HSTS-compliant
1449 UA. As a consequence, Wget would ignore all the
1450 "Strict-Transport-Security" headers, and would not enforce any
1451 existing HSTS policy.
1452 --hsts-file=file
1454 By default, Wget stores its HSTS database in ~/.wget-hsts. You can
1455 use --hsts-file to override this. Wget will use the supplied file
1456 as the HSTS database. Such file must conform to the correct HSTS
1457 database format used by Wget. If Wget cannot parse the provided
1458 file, the behaviour is unspecified.
1459 The Wget's HSTS database is a plain text file. Each line contains
1461 an HSTS entry (ie. a site that has issued a
1462 "Strict-Transport-Security" header and that therefore has specified
1463 a concrete HSTS policy to be applied). Lines starting with a dash
1464 ("#") are ignored by Wget. Please note that in spite of this
1465 convenient human-readability hand-hacking the HSTS database is
1466 generally not a good idea.
1467 An HSTS entry line consists of several fields separated by one or
1469 more whitespace:
1470 "<hostname> SP [<port>] SP <include subdomains> SP <created> SP
1472 <max-age>"
1473 The hostname and port fields indicate the hostname and port to
1475 which the given HSTS policy applies. The port field may be zero,
1476 and it will, in most of the cases. That means that the port number
1477 will not be taken into account when deciding whether such HSTS
1478 policy should be applied on a given request (only the hostname will
1479 be evaluated). When port is different to zero, both the target
1480 hostname and the port will be evaluated and the HSTS policy will
1481 only be applied if both of them match. This feature has been
1482 included for testing/development purposes only. The Wget testsuite
1483 (in testenv/) creates HSTS databases with explicit ports with the
1484 purpose of ensuring Wget's correct behaviour. Applying HSTS
1485 policies to ports other than the default ones is discouraged by RFC
1486 6797 (see Appendix B "Differences between HSTS Policy and Same-
1487 Origin Policy"). Thus, this functionality should not be used in
1488 production environments and port will typically be zero. The last
1489 three fields do what they are expected to. The field
1490 include_subdomains can either be 1 or 0 and it signals whether the
1491 subdomains of the target domain should be part of the given HSTS
1492 policy as well. The created and max-age fields hold the timestamp
1493 values of when such entry was created (first seen by Wget) and the
1494 HSTS-defined value 'max-age', which states how long should that
1495 HSTS policy remain active, measured in seconds elapsed since the
1496 timestamp stored in created. Once that time has passed, that HSTS
1497 policy will no longer be valid and will eventually be removed from
1498 the database.
1499 If you supply your own HSTS database via --hsts-file, be aware that
1501 Wget may modify the provided file if any change occurs between the
1502 HSTS policies requested by the remote servers and those in the
1503 file. When Wget exists, it effectively updates the HSTS database by
1504 rewriting the database file with the new entries.
1505 If the supplied file does not exist, Wget will create one. This
1507 file will contain the new HSTS entries. If no HSTS entries were
1508 generated (no "Strict-Transport-Security" headers were sent by any
1509 of the servers) then no file will be created, not even an empty
1510 one. This behaviour applies to the default database file
1511 (~/.wget-hsts) as well: it will not be created until some server
1512 enforces an HSTS policy.
1513 Care is taken not to override possible changes made by other Wget
1515 processes at the same time over the HSTS database. Before dumping
1516 the updated HSTS entries on the file, Wget will re-read it and
1517 merge the changes.
1518 Using a custom HSTS database and/or modifying an existing one is
1520 discouraged. For more information about the potential security
1521 threats arose from such practice, see section 14 "Security
1522 Considerations" of RFC 6797, specially section 14.9 "Creative
1523 Manipulation of HSTS Policy Store".
1524 --warc-file=file
1526 Use file as the destination WARC file.
1527 --warc-header=string
1529 Use string into as the warcinfo record.
1530 --warc-max-size=size
1532 Set the maximum size of the WARC files to size.
1533 --warc-cdx
1535 Write CDX index files.
1536 --warc-dedup=file
1538 Do not store records listed in this CDX file.
1539 --no-warc-compression
1541 Do not compress WARC files with GZIP.
1542 --no-warc-digests
1544 Do not calculate SHA1 digests.
1545 --no-warc-keep-log
1547 Do not store the log file in a WARC record.
1548 --warc-tempdir=dir
1550 Specify the location for temporary files created by the WARC
1551 writer.
1552 FTP Options
1554 --ftp-user=user
1555 --ftp-password=password
1556 Specify the username user and password password on an FTP server.
1557 Without this, or the corresponding startup option, the password
1558 defaults to -wget@, normally used for anonymous FTP.
1559 Another way to specify username and password is in the URL itself.
1561 Either method reveals your password to anyone who bothers to run
1562 "ps". To prevent the passwords from being seen, store them in
1563 .wgetrc or .netrc, and make sure to protect those files from other
1564 users with "chmod". If the passwords are really important, do not
1565 leave them lying in those files either---edit the files and delete
1566 them after Wget has started the download.
1567 --no-remove-listing
1569 Don't remove the temporary .listing files generated by FTP
1570 retrievals. Normally, these files contain the raw directory
1571 listings received from FTP servers. Not removing them can be
1572 useful for debugging purposes, or when you want to be able to
1573 easily check on the contents of remote server directories (e.g. to
1574 verify that a mirror you're running is complete).
1575 Note that even though Wget writes to a known filename for this
1577 file, this is not a security hole in the scenario of a user making
1578 .listing a symbolic link to /etc/passwd or something and asking
1579 "root" to run Wget in his or her directory. Depending on the
1580 options used, either Wget will refuse to write to .listing, making
1581 the globbing/recursion/time-stamping operation fail, or the
1582 symbolic link will be deleted and replaced with the actual .listing
1583 file, or the listing will be written to a .listing.number file.
1584 Even though this situation isn't a problem, though, "root" should
1586 never run Wget in a non-trusted user's directory. A user could do
1587 something as simple as linking index.html to /etc/passwd and asking
1588 "root" to run Wget with -N or -r so the file will be overwritten.
1589 --no-glob
1591 Turn off FTP globbing. Globbing refers to the use of shell-like
1592 special characters (wildcards), like *, ?, [ and ] to retrieve more
1593 than one file from the same directory at once, like:
1594 wget ftp://gnjilux.srk.fer.hr/*.msg
1596 By default, globbing will be turned on if the URL contains a
1598 globbing character. This option may be used to turn globbing on or
1599 off permanently.
1600 You may have to quote the URL to protect it from being expanded by
1602 your shell. Globbing makes Wget look for a directory listing,
1603 which is system-specific. This is why it currently works only with
1604 Unix FTP servers (and the ones emulating Unix "ls" output).
1605 --no-passive-ftp
1607 Disable the use of the passive FTP transfer mode. Passive FTP
1608 mandates that the client connect to the server to establish the
1609 data connection rather than the other way around.
1610 If the machine is connected to the Internet directly, both passive
1612 and active FTP should work equally well. Behind most firewall and
1613 NAT configurations passive FTP has a better chance of working.
1614 However, in some rare firewall configurations, active FTP actually
1615 works when passive FTP doesn't. If you suspect this to be the
1616 case, use this option, or set "passive_ftp=off" in your init file.
1617 --preserve-permissions
1619 Preserve remote file permissions instead of permissions set by
1620 umask.
1621 --retr-symlinks
1623 By default, when retrieving FTP directories recursively and a
1624 symbolic link is encountered, the symbolic link is traversed and
1625 the pointed-to files are retrieved. Currently, Wget does not
1626 traverse symbolic links to directories to download them
1627 recursively, though this feature may be added in the future.
1628 When --retr-symlinks=no is specified, the linked-to file is not
1630 downloaded. Instead, a matching symbolic link is created on the
1631 local filesystem. The pointed-to file will not be retrieved unless
1632 this recursive retrieval would have encountered it separately and
1633 downloaded it anyway. This option poses a security risk where a
1634 malicious FTP Server may cause Wget to write to files outside of
1635 the intended directories through a specially crafted .LISTING file.
1636 Note that when retrieving a file (not a directory) because it was
1638 specified on the command-line, rather than because it was recursed
1639 to, this option has no effect. Symbolic links are always traversed
1640 in this case.
1641 FTPS Options
1643 --ftps-implicit
1644 This option tells Wget to use FTPS implicitly. Implicit FTPS
1645 consists of initializing SSL/TLS from the very beginning of the
1646 control connection. This option does not send an "AUTH TLS"
1647 command: it assumes the server speaks FTPS and directly starts an
1648 SSL/TLS connection. If the attempt is successful, the session
1649 continues just like regular FTPS ("PBSZ" and "PROT" are sent,
1650 etc.). Implicit FTPS is no longer a requirement for FTPS
1651 implementations, and thus many servers may not support it. If
1652 --ftps-implicit is passed and no explicit port number specified,
1653 the default port for implicit FTPS, 990, will be used, instead of
1654 the default port for the "normal" (explicit) FTPS which is the same
1655 as that of FTP, 21.
1656 --no-ftps-resume-ssl
1658 Do not resume the SSL/TLS session in the data channel. When
1659 starting a data connection, Wget tries to resume the SSL/TLS
1660 session previously started in the control connection. SSL/TLS
1661 session resumption avoids performing an entirely new handshake by
1662 reusing the SSL/TLS parameters of a previous session. Typically,
1663 the FTPS servers want it that way, so Wget does this by default.
1664 Under rare circumstances however, one might want to start an
1665 entirely new SSL/TLS session in every data connection. This is
1666 what --no-ftps-resume-ssl is for.
1667 --ftps-clear-data-connection
1669 All the data connections will be in plain text. Only the control
1670 connection will be under SSL/TLS. Wget will send a "PROT C" command
1671 to achieve this, which must be approved by the server.
1672 --ftps-fallback-to-ftp
1674 Fall back to FTP if FTPS is not supported by the target server. For
1675 security reasons, this option is not asserted by default. The
1676 default behaviour is to exit with an error. If a server does not
1677 successfully reply to the initial "AUTH TLS" command, or in the
1678 case of implicit FTPS, if the initial SSL/TLS connection attempt is
1679 rejected, it is considered that such server does not support FTPS.
1680 Recursive Retrieval Options
1682 -r
1683 --recursive
1684 Turn on recursive retrieving. The default maximum depth is 5.
1685 -l depth
1687 --level=depth
1688 Specify recursion maximum depth level depth.
1689 --delete-after
1691 This option tells Wget to delete every single file it downloads,
1692 after having done so. It is useful for pre-fetching popular pages
1693 through a proxy, e.g.:
1694 wget -r -nd --delete-after http://whatever.com/~popular/page/
1696 The -r option is to retrieve recursively, and -nd to not create
1698 directories.
1699 Note that --delete-after deletes files on the local machine. It
1701 does not issue the DELE command to remote FTP sites, for instance.
1702 Also note that when --delete-after is specified, --convert-links is
1703 ignored, so .orig files are simply not created in the first place.
1704 -k
1706 --convert-links
1707 After the download is complete, convert the links in the document
1708 to make them suitable for local viewing. This affects not only the
1709 visible hyperlinks, but any part of the document that links to
1710 external content, such as embedded images, links to style sheets,
1711 hyperlinks to non-HTML content, etc.
1712 Each link will be changed in one of the two ways:
1714 · The links to files that have been downloaded by Wget will be
1716 changed to refer to the file they point to as a relative link.
1717 Example: if the downloaded file /foo/doc.html links to
1719 /bar/img.gif, also downloaded, then the link in doc.html will
1720 be modified to point to ../bar/img.gif. This kind of
1721 transformation works reliably for arbitrary combinations of
1722 directories.
1723 · The links to files that have not been downloaded by Wget will
1725 be changed to include host name and absolute path of the
1726 location they point to.
1727 Example: if the downloaded file /foo/doc.html links to
1729 /bar/img.gif (or to ../bar/img.gif), then the link in doc.html
1730 will be modified to point to http://hostname/bar/img.gif.
1731 Because of this, local browsing works reliably: if a linked file
1733 was downloaded, the link will refer to its local name; if it was
1734 not downloaded, the link will refer to its full Internet address
1735 rather than presenting a broken link. The fact that the former
1736 links are converted to relative links ensures that you can move the
1737 downloaded hierarchy to another directory.
1738 Note that only at the end of the download can Wget know which links
1740 have been downloaded. Because of that, the work done by -k will be
1741 performed at the end of all the downloads.
1742 --convert-file-only
1744 This option converts only the filename part of the URLs, leaving
1745 the rest of the URLs untouched. This filename part is sometimes
1746 referred to as the "basename", although we avoid that term here in
1747 order not to cause confusion.
1748 It works particularly well in conjunction with --adjust-extension,
1750 although this coupling is not enforced. It proves useful to
1751 populate Internet caches with files downloaded from different
1752 hosts.
1753 Example: if some link points to //foo.com/bar.cgi?xyz with
1755 --adjust-extension asserted and its local destination is intended
1756 to be ./foo.com/bar.cgi?xyz.css, then the link would be converted
1757 to //foo.com/bar.cgi?xyz.css. Note that only the filename part has
1758 been modified. The rest of the URL has been left untouched,
1759 including the net path ("//") which would otherwise be processed by
1760 Wget and converted to the effective scheme (ie. "http://").
1761 -K
1763 --backup-converted
1764 When converting a file, back up the original version with a .orig
1765 suffix. Affects the behavior of -N.
1766 -m
1768 --mirror
1769 Turn on options suitable for mirroring. This option turns on
1770 recursion and time-stamping, sets infinite recursion depth and
1771 keeps FTP directory listings. It is currently equivalent to -r -N
1772 -l inf --no-remove-listing.
1773 -p
1775 --page-requisites
1776 This option causes Wget to download all the files that are
1777 necessary to properly display a given HTML page. This includes
1778 such things as inlined images, sounds, and referenced stylesheets.
1779 Ordinarily, when downloading a single HTML page, any requisite
1781 documents that may be needed to display it properly are not
1782 downloaded. Using -r together with -l can help, but since Wget
1783 does not ordinarily distinguish between external and inlined
1784 documents, one is generally left with "leaf documents" that are
1785 missing their requisites.
1786 For instance, say document 1.html contains an "<IMG>" tag
1788 referencing 1.gif and an "<A>" tag pointing to external document
1789 2.html. Say that 2.html is similar but that its image is 2.gif and
1790 it links to 3.html. Say this continues up to some arbitrarily high
1791 number.
1792 If one executes the command:
1794 wget -r -l 2 http://<site>/1.html
1796 then 1.html, 1.gif, 2.html, 2.gif, and 3.html will be downloaded.
1798 As you can see, 3.html is without its requisite 3.gif because Wget
1799 is simply counting the number of hops (up to 2) away from 1.html in
1800 order to determine where to stop the recursion. However, with this
1801 command:
1802 wget -r -l 2 -p http://<site>/1.html
1804 all the above files and 3.html's requisite 3.gif will be
1806 downloaded. Similarly,
1807 wget -r -l 1 -p http://<site>/1.html
1809 will cause 1.html, 1.gif, 2.html, and 2.gif to be downloaded. One
1811 might think that:
1812 wget -r -l 0 -p http://<site>/1.html
1814 would download just 1.html and 1.gif, but unfortunately this is not
1816 the case, because -l 0 is equivalent to -l inf---that is, infinite
1817 recursion. To download a single HTML page (or a handful of them,
1818 all specified on the command-line or in a -i URL input file) and
1819 its (or their) requisites, simply leave off -r and -l:
1820 wget -p http://<site>/1.html
1822 Note that Wget will behave as if -r had been specified, but only
1824 that single page and its requisites will be downloaded. Links from
1825 that page to external documents will not be followed. Actually, to
1826 download a single page and all its requisites (even if they exist
1827 on separate websites), and make sure the lot displays properly
1828 locally, this author likes to use a few options in addition to -p:
1829 wget -E -H -k -K -p http://<site>/<document>
1831 To finish off this topic, it's worth knowing that Wget's idea of an
1833 external document link is any URL specified in an "<A>" tag, an
1834 "<AREA>" tag, or a "<LINK>" tag other than "<LINK
1835 REL="stylesheet">".
1836 --strict-comments
1838 Turn on strict parsing of HTML comments. The default is to
1839 terminate comments at the first occurrence of -->.
1840 According to specifications, HTML comments are expressed as SGML
1842 declarations. Declaration is special markup that begins with <!
1843 and ends with >, such as <!DOCTYPE ...>, that may contain comments
1844 between a pair of -- delimiters. HTML comments are "empty
1845 declarations", SGML declarations without any non-comment text.
1846 Therefore, <!--foo--> is a valid comment, and so is <!--one--
1847 --two-->, but <!--1--2--> is not.
1848 On the other hand, most HTML writers don't perceive comments as
1850 anything other than text delimited with <!-- and -->, which is not
1851 quite the same. For example, something like <!------------> works
1852 as a valid comment as long as the number of dashes is a multiple of
1853 four (!). If not, the comment technically lasts until the next --,
1854 which may be at the other end of the document. Because of this,
1855 many popular browsers completely ignore the specification and
1856 implement what users have come to expect: comments delimited with
1857 <!-- and -->.
1858 Until version 1.9, Wget interpreted comments strictly, which
1860 resulted in missing links in many web pages that displayed fine in
1861 browsers, but had the misfortune of containing non-compliant
1862 comments. Beginning with version 1.9, Wget has joined the ranks of
1863 clients that implements "naive" comments, terminating each comment
1864 at the first occurrence of -->.
1865 If, for whatever reason, you want strict comment parsing, use this
1867 option to turn it on.
1868 Recursive Accept/Reject Options
1870 -A acclist --accept acclist
1871 -R rejlist --reject rejlist
1872 Specify comma-separated lists of file name suffixes or patterns to
1873 accept or reject. Note that if any of the wildcard characters, *,
1874 ?, [ or ], appear in an element of acclist or rejlist, it will be
1875 treated as a pattern, rather than a suffix. In this case, you have
1876 to enclose the pattern into quotes to prevent your shell from
1877 expanding it, like in -A "*.mp3" or -A '*.mp3'.
1878 --accept-regex urlregex
1880 --reject-regex urlregex
1881 Specify a regular expression to accept or reject the complete URL.
1882 --regex-type regextype
1884 Specify the regular expression type. Possible types are posix or
1885 pcre. Note that to be able to use pcre type, wget has to be
1886 compiled with libpcre support.
1887 -D domain-list
1889 --domains=domain-list
1890 Set domains to be followed. domain-list is a comma-separated list
1891 of domains. Note that it does not turn on -H.
1892 --exclude-domains domain-list
1894 Specify the domains that are not to be followed.
1895 --follow-ftp
1897 Follow FTP links from HTML documents. Without this option, Wget
1898 will ignore all the FTP links.
1899 --follow-tags=list
1901 Wget has an internal table of HTML tag / attribute pairs that it
1902 considers when looking for linked documents during a recursive
1903 retrieval. If a user wants only a subset of those tags to be
1904 considered, however, he or she should be specify such tags in a
1905 comma-separated list with this option.
1906 --ignore-tags=list
1908 This is the opposite of the --follow-tags option. To skip certain
1909 HTML tags when recursively looking for documents to download,
1910 specify them in a comma-separated list.
1911 In the past, this option was the best bet for downloading a single
1913 page and its requisites, using a command-line like:
1914 wget --ignore-tags=a,area -H -k -K -r http://<site>/<document>
1916 However, the author of this option came across a page with tags
1918 like "<LINK REL="home" HREF="/">" and came to the realization that
1919 specifying tags to ignore was not enough. One can't just tell Wget
1920 to ignore "<LINK>", because then stylesheets will not be
1921 downloaded. Now the best bet for downloading a single page and its
1922 requisites is the dedicated --page-requisites option.
1923 --ignore-case
1925 Ignore case when matching files and directories. This influences
1926 the behavior of -R, -A, -I, and -X options, as well as globbing
1927 implemented when downloading from FTP sites. For example, with
1928 this option, -A "*.txt" will match file1.txt, but also file2.TXT,
1929 file3.TxT, and so on. The quotes in the example are to prevent the
1930 shell from expanding the pattern.
1931 -H
1933 --span-hosts
1934 Enable spanning across hosts when doing recursive retrieving.
1935 -L
1937 --relative
1938 Follow relative links only. Useful for retrieving a specific home
1939 page without any distractions, not even those from the same hosts.
1940 -I list
1942 --include-directories=list
1943 Specify a comma-separated list of directories you wish to follow
1944 when downloading. Elements of list may contain wildcards.
1945 -X list
1947 --exclude-directories=list
1948 Specify a comma-separated list of directories you wish to exclude
1949 from download. Elements of list may contain wildcards.
1950 -np
1952 --no-parent
1953 Do not ever ascend to the parent directory when retrieving
1954 recursively. This is a useful option, since it guarantees that
1955 only the files below a certain hierarchy will be downloaded.
1956
1958 Wget supports proxies for both HTTP and FTP retrievals. The standard
1959 way to specify proxy location, which Wget recognizes, is using the
1960 following environment variables:
1961 http_proxy
1963 https_proxy
1964 If set, the http_proxy and https_proxy variables should contain the
1965 URLs of the proxies for HTTP and HTTPS connections respectively.
1966 ftp_proxy
1968 This variable should contain the URL of the proxy for FTP
1969 connections. It is quite common that http_proxy and ftp_proxy are
1970 set to the same URL.
1971 no_proxy
1973 This variable should contain a comma-separated list of domain
1974 extensions proxy should not be used for. For instance, if the
1975 value of no_proxy is .mit.edu, proxy will not be used to retrieve
1976 documents from MIT.
1977
1979 Wget may return one of several error codes if it encounters problems.
1980 0 No problems occurred.
1982 1 Generic error code.
1984 2 Parse error---for instance, when parsing command-line options, the
1986 .wgetrc or .netrc...
1987 3 File I/O error.
1989 4 Network failure.
1991 5 SSL verification failure.
1993 6 Username/password authentication failure.
1995 7 Protocol errors.
1997 8 Server issued an error response.
1999 With the exceptions of 0 and 1, the lower-numbered exit codes take
2001 precedence over higher-numbered ones, when multiple types of errors are
2002 encountered.
2003 In versions of Wget prior to 1.12, Wget's exit status tended to be
2005 unhelpful and inconsistent. Recursive downloads would virtually always
2006 return 0 (success), regardless of any issues encountered, and non-
2007 recursive fetches only returned the status corresponding to the most
2008 recently-attempted download.
2009
2011 /etc/wgetrc
2012 Default location of the global startup file.
2013 .wgetrc
2015 User startup file.
2016
2018 You are welcome to submit bug reports via the GNU Wget bug tracker (see
2019 <https://savannah.gnu.org/bugs/?func=additem&group=wget>) or to our
2020 mailing list <bug-wget@gnu.org>.
2021 Visit <https://lists.gnu.org/mailman/listinfo/bug-wget> to get more
2023 info (how to subscribe, list archives, ...).
2024 Before actually submitting a bug report, please try to follow a few
2026 simple guidelines.
2027 1. Please try to ascertain that the behavior you see really is a bug.
2029 If Wget crashes, it's a bug. If Wget does not behave as
2030 documented, it's a bug. If things work strange, but you are not
2031 sure about the way they are supposed to work, it might well be a
2032 bug, but you might want to double-check the documentation and the
2033 mailing lists.
2034 2. Try to repeat the bug in as simple circumstances as possible. E.g.
2036 if Wget crashes while downloading wget -rl0 -kKE -t5 --no-proxy
2037 http://example.com -o /tmp/log, you should try to see if the crash
2038 is repeatable, and if will occur with a simpler set of options.
2039 You might even try to start the download at the page where the
2040 crash occurred to see if that page somehow triggered the crash.
2041 Also, while I will probably be interested to know the contents of
2043 your .wgetrc file, just dumping it into the debug message is
2044 probably a bad idea. Instead, you should first try to see if the
2045 bug repeats with .wgetrc moved out of the way. Only if it turns
2046 out that .wgetrc settings affect the bug, mail me the relevant
2047 parts of the file.
2048 3. Please start Wget with -d option and send us the resulting output
2050 (or relevant parts thereof). If Wget was compiled without debug
2051 support, recompile it---it is much easier to trace bugs with debug
2052 support on.
2053 Note: please make sure to remove any potentially sensitive
2055 information from the debug log before sending it to the bug
2056 address. The "-d" won't go out of its way to collect sensitive
2057 information, but the log will contain a fairly complete transcript
2058 of Wget's communication with the server, which may include
2059 passwords and pieces of downloaded data. Since the bug address is
2060 publicly archived, you may assume that all bug reports are visible
2061 to the public.
2062 4. If Wget has crashed, try to run it in a debugger, e.g. "gdb `which
2064 wget` core" and type "where" to get the backtrace. This may not
2065 work if the system administrator has disabled core files, but it is
2066 safe to try.
2067
2069 This is not the complete manual for GNU Wget. For more complete
2070 information, including more detailed explanations of some of the
2071 options, and a number of commands available for use with .wgetrc files
2072 and the -e option, see the GNU Info entry for wget.
2073
2075 Originally written by Hrvoje Nikšić <hniksic@xemacs.org>.
2076
2078 Copyright (c) 1996-2011, 2015, 2018-2019 Free Software Foundation, Inc.
2079 Permission is granted to copy, distribute and/or modify this document
2081 under the terms of the GNU Free Documentation License, Version 1.3 or
2082 any later version published by the Free Software Foundation; with no
2083 Invariant Sections, with no Front-Cover Texts, and with no Back-Cover
2084 Texts. A copy of the license is included in the section entitled "GNU
2085 Free Documentation License".
2086GNU Wget 1.20.3 2019-04-05 WGET(1)
