1WGET(1) GNU Wget WGET(1)
2
6 Wget - The non-interactive network downloader.
7
9 wget [option]... [URL]...
10
12 GNU Wget is a free utility for non-interactive download of files from
13 the Web. It supports HTTP, HTTPS, and FTP protocols, as well as
14 retrieval through HTTP proxies.
15 Wget is non-interactive, meaning that it can work in the background,
17 while the user is not logged on. This allows you to start a retrieval
18 and disconnect from the system, letting Wget finish the work. By
19 contrast, most of the Web browsers require constant user's presence,
20 which can be a great hindrance when transferring a lot of data.
21 Wget can follow links in HTML, XHTML, and CSS pages, to create local
23 versions of remote web sites, fully recreating the directory structure
24 of the original site. This is sometimes referred to as "recursive
25 downloading." While doing that, Wget respects the Robot Exclusion
26 Standard (/robots.txt). Wget can be instructed to convert the links in
27 downloaded files to point at the local files, for offline viewing.
28 Wget has been designed for robustness over slow or unstable network
30 connections; if a download fails due to a network problem, it will keep
31 retrying until the whole file has been retrieved. If the server
32 supports regetting, it will instruct the server to continue the
33 download from where it left off.
34
36 Option Syntax
37 Since Wget uses GNU getopt to process command-line arguments, every
38 option has a long form along with the short one. Long options are more
39 convenient to remember, but take time to type. You may freely mix
40 different option styles, or specify options after the command-line
41 arguments. Thus you may write:
42 wget -r --tries=10 http://fly.srk.fer.hr/ -o log
44 The space between the option accepting an argument and the argument may
46 be omitted. Instead of -o log you can write -olog.
47 You may put several options that do not require arguments together,
49 like:
50 wget -drc <URL>
52 This is completely equivalent to:
54 wget -d -r -c <URL>
56 Since the options can be specified after the arguments, you may
58 terminate them with --. So the following will try to download URL -x,
59 reporting failure to log:
60 wget -o log -- -x
62 The options that accept comma-separated lists all respect the
64 convention that specifying an empty list clears its value. This can be
65 useful to clear the .wgetrc settings. For instance, if your .wgetrc
66 sets "exclude_directories" to /cgi-bin, the following example will
67 first reset it, and then set it to exclude /~nobody and /~somebody.
68 You can also clear the lists in .wgetrc.
69 wget -X "" -X /~nobody,/~somebody
71 Most options that do not accept arguments are boolean options, so named
73 because their state can be captured with a yes-or-no ("boolean")
74 variable. For example, --follow-ftp tells Wget to follow FTP links
75 from HTML files and, on the other hand, --no-glob tells it not to
76 perform file globbing on FTP URLs. A boolean option is either
77 affirmative or negative (beginning with --no). All such options share
78 several properties.
79 Unless stated otherwise, it is assumed that the default behavior is the
81 opposite of what the option accomplishes. For example, the documented
82 existence of --follow-ftp assumes that the default is to not follow FTP
83 links from HTML pages.
84 Affirmative options can be negated by prepending the --no- to the
86 option name; negative options can be negated by omitting the --no-
87 prefix. This might seem superfluous---if the default for an
88 affirmative option is to not do something, then why provide a way to
89 explicitly turn it off? But the startup file may in fact change the
90 default. For instance, using "follow_ftp = on" in .wgetrc makes Wget
91 follow FTP links by default, and using --no-follow-ftp is the only way
92 to restore the factory default from the command line.
93 Basic Startup Options
95 -V
96 --version
97 Display the version of Wget.
98 -h
100 --help
101 Print a help message describing all of Wget's command-line options.
102 -b
104 --background
105 Go to background immediately after startup. If no output file is
106 specified via the -o, output is redirected to wget-log.
107 -e command
109 --execute command
110 Execute command as if it were a part of .wgetrc. A command thus
111 invoked will be executed after the commands in .wgetrc, thus taking
112 precedence over them. If you need to specify more than one wgetrc
113 command, use multiple instances of -e.
114 Logging and Input File Options
116 -o logfile
117 --output-file=logfile
118 Log all messages to logfile. The messages are normally reported to
119 standard error.
120 -a logfile
122 --append-output=logfile
123 Append to logfile. This is the same as -o, only it appends to
124 logfile instead of overwriting the old log file. If logfile does
125 not exist, a new file is created.
126 -d
128 --debug
129 Turn on debug output, meaning various information important to the
130 developers of Wget if it does not work properly. Your system
131 administrator may have chosen to compile Wget without debug
132 support, in which case -d will not work. Please note that
133 compiling with debug support is always safe---Wget compiled with
134 the debug support will not print any debug info unless requested
135 with -d.
136 -q
138 --quiet
139 Turn off Wget's output.
140 -v
142 --verbose
143 Turn on verbose output, with all the available data. The default
144 output is verbose.
145 -nv
147 --no-verbose
148 Turn off verbose without being completely quiet (use -q for that),
149 which means that error messages and basic information still get
150 printed.
151 --report-speed=type
153 Output bandwidth as type. The only accepted value is bits.
154 -i file
156 --input-file=file
157 Read URLs from a local or external file. If - is specified as
158 file, URLs are read from the standard input. (Use ./- to read from
159 a file literally named -.)
160 If this function is used, no URLs need be present on the command
162 line. If there are URLs both on the command line and in an input
163 file, those on the command lines will be the first ones to be
164 retrieved. If --force-html is not specified, then file should
165 consist of a series of URLs, one per line.
166 However, if you specify --force-html, the document will be regarded
168 as html. In that case you may have problems with relative links,
169 which you can solve either by adding "<base href="url">" to the
170 documents or by specifying --base=url on the command line.
171 If the file is an external one, the document will be automatically
173 treated as html if the Content-Type matches text/html.
174 Furthermore, the file's location will be implicitly used as base
175 href if none was specified.
176 --input-metalink=file
178 Downloads files covered in local Metalink file. Metalink version 3
179 and 4 are supported.
180 --keep-badhash
182 Keeps downloaded Metalink's files with a bad hash. It appends
183 .badhash to the name of Metalink's files which have a checksum
184 mismatch, except without overwriting existing files.
185 --metalink-over-http
187 Issues HTTP HEAD request instead of GET and extracts Metalink
188 metadata from response headers. Then it switches to Metalink
189 download. If no valid Metalink metadata is found, it falls back to
190 ordinary HTTP download. Enables Content-Type:
191 application/metalink4+xml files download/processing.
192 --metalink-index=number
194 Set the Metalink application/metalink4+xml metaurl ordinal NUMBER.
195 From 1 to the total number of "application/metalink4+xml"
196 available. Specify 0 or inf to choose the first good one.
197 Metaurls, such as those from a --metalink-over-http, may have been
198 sorted by priority key's value; keep this in mind to choose the
199 right NUMBER.
200 --preferred-location
202 Set preferred location for Metalink resources. This has effect if
203 multiple resources with same priority are available.
204 --xattr
206 Enable use of file system's extended attributes to save the
207 original URL and the Referer HTTP header value if used.
208 Be aware that the URL might contain private information like access
210 tokens or credentials.
211 -F
213 --force-html
214 When input is read from a file, force it to be treated as an HTML
215 file. This enables you to retrieve relative links from existing
216 HTML files on your local disk, by adding "<base href="url">" to
217 HTML, or using the --base command-line option.
218 -B URL
220 --base=URL
221 Resolves relative links using URL as the point of reference, when
222 reading links from an HTML file specified via the -i/--input-file
223 option (together with --force-html, or when the input file was
224 fetched remotely from a server describing it as HTML). This is
225 equivalent to the presence of a "BASE" tag in the HTML input file,
226 with URL as the value for the "href" attribute.
227 For instance, if you specify http://foo/bar/a.html for URL, and
229 Wget reads ../baz/b.html from the input file, it would be resolved
230 to http://foo/baz/b.html.
231 --config=FILE
233 Specify the location of a startup file you wish to use instead of
234 the default one(s). Use --no-config to disable reading of config
235 files. If both --config and --no-config are given, --no-config is
236 ignored.
237 --rejected-log=logfile
239 Logs all URL rejections to logfile as comma separated values. The
240 values include the reason of rejection, the URL and the parent URL
241 it was found in.
242 Download Options
244 --bind-address=ADDRESS
245 When making client TCP/IP connections, bind to ADDRESS on the local
246 machine. ADDRESS may be specified as a hostname or IP address.
247 This option can be useful if your machine is bound to multiple IPs.
248 --bind-dns-address=ADDRESS
250 [libcares only] This address overrides the route for DNS requests.
251 If you ever need to circumvent the standard settings from
252 /etc/resolv.conf, this option together with --dns-servers is your
253 friend. ADDRESS must be specified either as IPv4 or IPv6 address.
254 Wget needs to be built with libcares for this option to be
255 available.
256 --dns-servers=ADDRESSES
258 [libcares only] The given address(es) override the standard
259 nameserver addresses, e.g. as configured in /etc/resolv.conf.
260 ADDRESSES may be specified either as IPv4 or IPv6 addresses, comma-
261 separated. Wget needs to be built with libcares for this option to
262 be available.
263 -t number
265 --tries=number
266 Set number of tries to number. Specify 0 or inf for infinite
267 retrying. The default is to retry 20 times, with the exception of
268 fatal errors like "connection refused" or "not found" (404), which
269 are not retried.
270 -O file
272 --output-document=file
273 The documents will not be written to the appropriate files, but all
274 will be concatenated together and written to file. If - is used as
275 file, documents will be printed to standard output, disabling link
276 conversion. (Use ./- to print to a file literally named -.)
277 Use of -O is not intended to mean simply "use the name file instead
279 of the one in the URL;" rather, it is analogous to shell
280 redirection: wget -O file http://foo is intended to work like wget
281 -O - http://foo > file; file will be truncated immediately, and all
282 downloaded content will be written there.
283 For this reason, -N (for timestamp-checking) is not supported in
285 combination with -O: since file is always newly created, it will
286 always have a very new timestamp. A warning will be issued if this
287 combination is used.
288 Similarly, using -r or -p with -O may not work as you expect: Wget
290 won't just download the first file to file and then download the
291 rest to their normal names: all downloaded content will be placed
292 in file. This was disabled in version 1.11, but has been reinstated
293 (with a warning) in 1.11.2, as there are some cases where this
294 behavior can actually have some use.
295 A combination with -nc is only accepted if the given output file
297 does not exist.
298 Note that a combination with -k is only permitted when downloading
300 a single document, as in that case it will just convert all
301 relative URIs to external ones; -k makes no sense for multiple URIs
302 when they're all being downloaded to a single file; -k can be used
303 only when the output is a regular file.
304 -nc
306 --no-clobber
307 If a file is downloaded more than once in the same directory,
308 Wget's behavior depends on a few options, including -nc. In
309 certain cases, the local file will be clobbered, or overwritten,
310 upon repeated download. In other cases it will be preserved.
311 When running Wget without -N, -nc, -r, or -p, downloading the same
313 file in the same directory will result in the original copy of file
314 being preserved and the second copy being named file.1. If that
315 file is downloaded yet again, the third copy will be named file.2,
316 and so on. (This is also the behavior with -nd, even if -r or -p
317 are in effect.) When -nc is specified, this behavior is
318 suppressed, and Wget will refuse to download newer copies of file.
319 Therefore, ""no-clobber"" is actually a misnomer in this
320 mode---it's not clobbering that's prevented (as the numeric
321 suffixes were already preventing clobbering), but rather the
322 multiple version saving that's prevented.
323 When running Wget with -r or -p, but without -N, -nd, or -nc, re-
325 downloading a file will result in the new copy simply overwriting
326 the old. Adding -nc will prevent this behavior, instead causing
327 the original version to be preserved and any newer copies on the
328 server to be ignored.
329 When running Wget with -N, with or without -r or -p, the decision
331 as to whether or not to download a newer copy of a file depends on
332 the local and remote timestamp and size of the file. -nc may not
333 be specified at the same time as -N.
334 A combination with -O/--output-document is only accepted if the
336 given output file does not exist.
337 Note that when -nc is specified, files with the suffixes .html or
339 .htm will be loaded from the local disk and parsed as if they had
340 been retrieved from the Web.
341 --backups=backups
343 Before (over)writing a file, back up an existing file by adding a
344 .1 suffix (_1 on VMS) to the file name. Such backup files are
345 rotated to .2, .3, and so on, up to backups (and lost beyond that).
346 --no-netrc
348 Do not try to obtain credentials from .netrc file. By default
349 .netrc file is searched for credentials in case none have been
350 passed on command line and authentication is required.
351 -c
353 --continue
354 Continue getting a partially-downloaded file. This is useful when
355 you want to finish up a download started by a previous instance of
356 Wget, or by another program. For instance:
357 wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z
359 If there is a file named ls-lR.Z in the current directory, Wget
361 will assume that it is the first portion of the remote file, and
362 will ask the server to continue the retrieval from an offset equal
363 to the length of the local file.
364 Note that you don't need to specify this option if you just want
366 the current invocation of Wget to retry downloading a file should
367 the connection be lost midway through. This is the default
368 behavior. -c only affects resumption of downloads started prior to
369 this invocation of Wget, and whose local files are still sitting
370 around.
371 Without -c, the previous example would just download the remote
373 file to ls-lR.Z.1, leaving the truncated ls-lR.Z file alone.
374 If you use -c on a non-empty file, and the server does not support
376 continued downloading, Wget will restart the download from scratch
377 and overwrite the existing file entirely.
378 Beginning with Wget 1.7, if you use -c on a file which is of equal
380 size as the one on the server, Wget will refuse to download the
381 file and print an explanatory message. The same happens when the
382 file is smaller on the server than locally (presumably because it
383 was changed on the server since your last download
384 attempt)---because "continuing" is not meaningful, no download
385 occurs.
386 On the other side of the coin, while using -c, any file that's
388 bigger on the server than locally will be considered an incomplete
389 download and only "(length(remote) - length(local))" bytes will be
390 downloaded and tacked onto the end of the local file. This
391 behavior can be desirable in certain cases---for instance, you can
392 use wget -c to download just the new portion that's been appended
393 to a data collection or log file.
394 However, if the file is bigger on the server because it's been
396 changed, as opposed to just appended to, you'll end up with a
397 garbled file. Wget has no way of verifying that the local file is
398 really a valid prefix of the remote file. You need to be
399 especially careful of this when using -c in conjunction with -r,
400 since every file will be considered as an "incomplete download"
401 candidate.
402 Another instance where you'll get a garbled file if you try to use
404 -c is if you have a lame HTTP proxy that inserts a "transfer
405 interrupted" string into the local file. In the future a
406 "rollback" option may be added to deal with this case.
407 Note that -c only works with FTP servers and with HTTP servers that
409 support the "Range" header.
410 --start-pos=OFFSET
412 Start downloading at zero-based position OFFSET. Offset may be
413 expressed in bytes, kilobytes with the `k' suffix, or megabytes
414 with the `m' suffix, etc.
415 --start-pos has higher precedence over --continue. When
417 --start-pos and --continue are both specified, wget will emit a
418 warning then proceed as if --continue was absent.
419 Server support for continued download is required, otherwise
421 --start-pos cannot help. See -c for details.
422 --progress=type
424 Select the type of the progress indicator you wish to use. Legal
425 indicators are "dot" and "bar".
426 The "bar" indicator is used by default. It draws an ASCII progress
428 bar graphics (a.k.a "thermometer" display) indicating the status of
429 retrieval. If the output is not a TTY, the "dot" bar will be used
430 by default.
431 Use --progress=dot to switch to the "dot" display. It traces the
433 retrieval by printing dots on the screen, each dot representing a
434 fixed amount of downloaded data.
435 The progress type can also take one or more parameters. The
437 parameters vary based on the type selected. Parameters to type are
438 passed by appending them to the type sperated by a colon (:) like
439 this: --progress=type:parameter1:parameter2.
440 When using the dotted retrieval, you may set the style by
442 specifying the type as dot:style. Different styles assign
443 different meaning to one dot. With the "default" style each dot
444 represents 1K, there are ten dots in a cluster and 50 dots in a
445 line. The "binary" style has a more "computer"-like
446 orientation---8K dots, 16-dots clusters and 48 dots per line (which
447 makes for 384K lines). The "mega" style is suitable for
448 downloading large files---each dot represents 64K retrieved, there
449 are eight dots in a cluster, and 48 dots on each line (so each line
450 contains 3M). If "mega" is not enough then you can use the "giga"
451 style---each dot represents 1M retrieved, there are eight dots in a
452 cluster, and 32 dots on each line (so each line contains 32M).
453 With --progress=bar, there are currently two possible parameters,
455 force and noscroll.
456 When the output is not a TTY, the progress bar always falls back to
458 "dot", even if --progress=bar was passed to Wget during invocation.
459 This behaviour can be overridden and the "bar" output forced by
460 using the "force" parameter as --progress=bar:force.
461 By default, the bar style progress bar scroll the name of the file
463 from left to right for the file being downloaded if the filename
464 exceeds the maximum length allotted for its display. In certain
465 cases, such as with --progress=bar:force, one may not want the
466 scrolling filename in the progress bar. By passing the "noscroll"
467 parameter, Wget can be forced to display as much of the filename as
468 possible without scrolling through it.
469 Note that you can set the default style using the "progress"
471 command in .wgetrc. That setting may be overridden from the
472 command line. For example, to force the bar output without
473 scrolling, use --progress=bar:force:noscroll.
474 --show-progress
476 Force wget to display the progress bar in any verbosity.
477 By default, wget only displays the progress bar in verbose mode.
479 One may however, want wget to display the progress bar on screen in
480 conjunction with any other verbosity modes like --no-verbose or
481 --quiet. This is often a desired a property when invoking wget to
482 download several small/large files. In such a case, wget could
483 simply be invoked with this parameter to get a much cleaner output
484 on the screen.
485 This option will also force the progress bar to be printed to
487 stderr when used alongside the --output-file option.
488 -N
490 --timestamping
491 Turn on time-stamping.
492 --no-if-modified-since
494 Do not send If-Modified-Since header in -N mode. Send preliminary
495 HEAD request instead. This has only effect in -N mode.
496 --no-use-server-timestamps
498 Don't set the local file's timestamp by the one on the server.
499 By default, when a file is downloaded, its timestamps are set to
501 match those from the remote file. This allows the use of
502 --timestamping on subsequent invocations of wget. However, it is
503 sometimes useful to base the local file's timestamp on when it was
504 actually downloaded; for that purpose, the
505 --no-use-server-timestamps option has been provided.
506 -S
508 --server-response
509 Print the headers sent by HTTP servers and responses sent by FTP
510 servers.
511 --spider
513 When invoked with this option, Wget will behave as a Web spider,
514 which means that it will not download the pages, just check that
515 they are there. For example, you can use Wget to check your
516 bookmarks:
517 wget --spider --force-html -i bookmarks.html
519 This feature needs much more work for Wget to get close to the
521 functionality of real web spiders.
522 -T seconds
524 --timeout=seconds
525 Set the network timeout to seconds seconds. This is equivalent to
526 specifying --dns-timeout, --connect-timeout, and --read-timeout,
527 all at the same time.
528 When interacting with the network, Wget can check for timeout and
530 abort the operation if it takes too long. This prevents anomalies
531 like hanging reads and infinite connects. The only timeout enabled
532 by default is a 900-second read timeout. Setting a timeout to 0
533 disables it altogether. Unless you know what you are doing, it is
534 best not to change the default timeout settings.
535 All timeout-related options accept decimal values, as well as
537 subsecond values. For example, 0.1 seconds is a legal (though
538 unwise) choice of timeout. Subsecond timeouts are useful for
539 checking server response times or for testing network latency.
540 --dns-timeout=seconds
542 Set the DNS lookup timeout to seconds seconds. DNS lookups that
543 don't complete within the specified time will fail. By default,
544 there is no timeout on DNS lookups, other than that implemented by
545 system libraries.
546 --connect-timeout=seconds
548 Set the connect timeout to seconds seconds. TCP connections that
549 take longer to establish will be aborted. By default, there is no
550 connect timeout, other than that implemented by system libraries.
551 --read-timeout=seconds
553 Set the read (and write) timeout to seconds seconds. The "time" of
554 this timeout refers to idle time: if, at any point in the download,
555 no data is received for more than the specified number of seconds,
556 reading fails and the download is restarted. This option does not
557 directly affect the duration of the entire download.
558 Of course, the remote server may choose to terminate the connection
560 sooner than this option requires. The default read timeout is 900
561 seconds.
562 --limit-rate=amount
564 Limit the download speed to amount bytes per second. Amount may be
565 expressed in bytes, kilobytes with the k suffix, or megabytes with
566 the m suffix. For example, --limit-rate=20k will limit the
567 retrieval rate to 20KB/s. This is useful when, for whatever
568 reason, you don't want Wget to consume the entire available
569 bandwidth.
570 This option allows the use of decimal numbers, usually in
572 conjunction with power suffixes; for example, --limit-rate=2.5k is
573 a legal value.
574 Note that Wget implements the limiting by sleeping the appropriate
576 amount of time after a network read that took less time than
577 specified by the rate. Eventually this strategy causes the TCP
578 transfer to slow down to approximately the specified rate.
579 However, it may take some time for this balance to be achieved, so
580 don't be surprised if limiting the rate doesn't work well with very
581 small files.
582 -w seconds
584 --wait=seconds
585 Wait the specified number of seconds between the retrievals. Use
586 of this option is recommended, as it lightens the server load by
587 making the requests less frequent. Instead of in seconds, the time
588 can be specified in minutes using the "m" suffix, in hours using
589 "h" suffix, or in days using "d" suffix.
590 Specifying a large value for this option is useful if the network
592 or the destination host is down, so that Wget can wait long enough
593 to reasonably expect the network error to be fixed before the
594 retry. The waiting interval specified by this function is
595 influenced by "--random-wait", which see.
596 --waitretry=seconds
598 If you don't want Wget to wait between every retrieval, but only
599 between retries of failed downloads, you can use this option. Wget
600 will use linear backoff, waiting 1 second after the first failure
601 on a given file, then waiting 2 seconds after the second failure on
602 that file, up to the maximum number of seconds you specify.
603 By default, Wget will assume a value of 10 seconds.
605 --random-wait
607 Some web sites may perform log analysis to identify retrieval
608 programs such as Wget by looking for statistically significant
609 similarities in the time between requests. This option causes the
610 time between requests to vary between 0.5 and 1.5 * wait seconds,
611 where wait was specified using the --wait option, in order to mask
612 Wget's presence from such analysis.
613 A 2001 article in a publication devoted to development on a popular
615 consumer platform provided code to perform this analysis on the
616 fly. Its author suggested blocking at the class C address level to
617 ensure automated retrieval programs were blocked despite changing
618 DHCP-supplied addresses.
619 The --random-wait option was inspired by this ill-advised
621 recommendation to block many unrelated users from a web site due to
622 the actions of one.
623 --no-proxy
625 Don't use proxies, even if the appropriate *_proxy environment
626 variable is defined.
627 -Q quota
629 --quota=quota
630 Specify download quota for automatic retrievals. The value can be
631 specified in bytes (default), kilobytes (with k suffix), or
632 megabytes (with m suffix).
633 Note that quota will never affect downloading a single file. So if
635 you specify wget -Q10k https://example.com/ls-lR.gz, all of the
636 ls-lR.gz will be downloaded. The same goes even when several URLs
637 are specified on the command-line. The quota is checked only at
638 the end of each downloaded file, so it will never result in a
639 partially downloaded file. Thus you may safely type wget -Q2m -i
640 sites---download will be aborted after the file that exhausts the
641 quota is completely downloaded.
642 Setting quota to 0 or to inf unlimits the download quota.
644 --no-dns-cache
646 Turn off caching of DNS lookups. Normally, Wget remembers the IP
647 addresses it looked up from DNS so it doesn't have to repeatedly
648 contact the DNS server for the same (typically small) set of hosts
649 it retrieves from. This cache exists in memory only; a new Wget
650 run will contact DNS again.
651 However, it has been reported that in some situations it is not
653 desirable to cache host names, even for the duration of a short-
654 running application like Wget. With this option Wget issues a new
655 DNS lookup (more precisely, a new call to "gethostbyname" or
656 "getaddrinfo") each time it makes a new connection. Please note
657 that this option will not affect caching that might be performed by
658 the resolving library or by an external caching layer, such as
659 NSCD.
660 If you don't understand exactly what this option does, you probably
662 won't need it.
663 --restrict-file-names=modes
665 Change which characters found in remote URLs must be escaped during
666 generation of local filenames. Characters that are restricted by
667 this option are escaped, i.e. replaced with %HH, where HH is the
668 hexadecimal number that corresponds to the restricted character.
669 This option may also be used to force all alphabetical cases to be
670 either lower- or uppercase.
671 By default, Wget escapes the characters that are not valid or safe
673 as part of file names on your operating system, as well as control
674 characters that are typically unprintable. This option is useful
675 for changing these defaults, perhaps because you are downloading to
676 a non-native partition, or because you want to disable escaping of
677 the control characters, or you want to further restrict characters
678 to only those in the ASCII range of values.
679 The modes are a comma-separated set of text values. The acceptable
681 values are unix, windows, nocontrol, ascii, lowercase, and
682 uppercase. The values unix and windows are mutually exclusive (one
683 will override the other), as are lowercase and uppercase. Those
684 last are special cases, as they do not change the set of characters
685 that would be escaped, but rather force local file paths to be
686 converted either to lower- or uppercase.
687 When "unix" is specified, Wget escapes the character / and the
689 control characters in the ranges 0--31 and 128--159. This is the
690 default on Unix-like operating systems.
691 When "windows" is given, Wget escapes the characters \, |, /, :, ?,
693 ", *, <, >, and the control characters in the ranges 0--31 and
694 128--159. In addition to this, Wget in Windows mode uses + instead
695 of : to separate host and port in local file names, and uses @
696 instead of ? to separate the query portion of the file name from
697 the rest. Therefore, a URL that would be saved as
698 www.xemacs.org:4300/search.pl?input=blah in Unix mode would be
699 saved as www.xemacs.org+4300/search.pl@input=blah in Windows mode.
700 This mode is the default on Windows.
701 If you specify nocontrol, then the escaping of the control
703 characters is also switched off. This option may make sense when
704 you are downloading URLs whose names contain UTF-8 characters, on a
705 system which can save and display filenames in UTF-8 (some possible
706 byte values used in UTF-8 byte sequences fall in the range of
707 values designated by Wget as "controls").
708 The ascii mode is used to specify that any bytes whose values are
710 outside the range of ASCII characters (that is, greater than 127)
711 shall be escaped. This can be useful when saving filenames whose
712 encoding does not match the one used locally.
713 -4
715 --inet4-only
716 -6
717 --inet6-only
718 Force connecting to IPv4 or IPv6 addresses. With --inet4-only or
719 -4, Wget will only connect to IPv4 hosts, ignoring AAAA records in
720 DNS, and refusing to connect to IPv6 addresses specified in URLs.
721 Conversely, with --inet6-only or -6, Wget will only connect to IPv6
722 hosts and ignore A records and IPv4 addresses.
723 Neither options should be needed normally. By default, an
725 IPv6-aware Wget will use the address family specified by the host's
726 DNS record. If the DNS responds with both IPv4 and IPv6 addresses,
727 Wget will try them in sequence until it finds one it can connect
728 to. (Also see "--prefer-family" option described below.)
729 These options can be used to deliberately force the use of IPv4 or
731 IPv6 address families on dual family systems, usually to aid
732 debugging or to deal with broken network configuration. Only one
733 of --inet6-only and --inet4-only may be specified at the same time.
734 Neither option is available in Wget compiled without IPv6 support.
735 --prefer-family=none/IPv4/IPv6
737 When given a choice of several addresses, connect to the addresses
738 with specified address family first. The address order returned by
739 DNS is used without change by default.
740 This avoids spurious errors and connect attempts when accessing
742 hosts that resolve to both IPv6 and IPv4 addresses from IPv4
743 networks. For example, www.kame.net resolves to
744 2001:200:0:8002:203:47ff:fea5:3085 and to 203.178.141.194. When
745 the preferred family is "IPv4", the IPv4 address is used first;
746 when the preferred family is "IPv6", the IPv6 address is used
747 first; if the specified value is "none", the address order returned
748 by DNS is used without change.
749 Unlike -4 and -6, this option doesn't inhibit access to any address
751 family, it only changes the order in which the addresses are
752 accessed. Also note that the reordering performed by this option
753 is stable---it doesn't affect order of addresses of the same
754 family. That is, the relative order of all IPv4 addresses and of
755 all IPv6 addresses remains intact in all cases.
756 --retry-connrefused
758 Consider "connection refused" a transient error and try again.
759 Normally Wget gives up on a URL when it is unable to connect to the
760 site because failure to connect is taken as a sign that the server
761 is not running at all and that retries would not help. This option
762 is for mirroring unreliable sites whose servers tend to disappear
763 for short periods of time.
764 --user=user
766 --password=password
767 Specify the username user and password password for both FTP and
768 HTTP file retrieval. These parameters can be overridden using the
769 --ftp-user and --ftp-password options for FTP connections and the
770 --http-user and --http-password options for HTTP connections.
771 --ask-password
773 Prompt for a password for each connection established. Cannot be
774 specified when --password is being used, because they are mutually
775 exclusive.
776 --use-askpass=command
778 Prompt for a user and password using the specified command. If no
779 command is specified then the command in the environment variable
780 WGET_ASKPASS is used. If WGET_ASKPASS is not set then the command
781 in the environment variable SSH_ASKPASS is used.
782 You can set the default command for use-askpass in the .wgetrc.
784 That setting may be overridden from the command line.
785 --no-iri
787 Turn off internationalized URI (IRI) support. Use --iri to turn it
788 on. IRI support is activated by default.
789 You can set the default state of IRI support using the "iri"
791 command in .wgetrc. That setting may be overridden from the command
792 line.
793 --local-encoding=encoding
795 Force Wget to use encoding as the default system encoding. That
796 affects how Wget converts URLs specified as arguments from locale
797 to UTF-8 for IRI support.
798 Wget use the function nl_langinfo() and then the "CHARSET"
800 environment variable to get the locale. If it fails, ASCII is used.
801 You can set the default local encoding using the "local_encoding"
803 command in .wgetrc. That setting may be overridden from the command
804 line.
805 --remote-encoding=encoding
807 Force Wget to use encoding as the default remote server encoding.
808 That affects how Wget converts URIs found in files from remote
809 encoding to UTF-8 during a recursive fetch. This options is only
810 useful for IRI support, for the interpretation of non-ASCII
811 characters.
812 For HTTP, remote encoding can be found in HTTP "Content-Type"
814 header and in HTML "Content-Type http-equiv" meta tag.
815 You can set the default encoding using the "remoteencoding" command
817 in .wgetrc. That setting may be overridden from the command line.
818 --unlink
820 Force Wget to unlink file instead of clobbering existing file. This
821 option is useful for downloading to the directory with hardlinks.
822 Directory Options
824 -nd
825 --no-directories
826 Do not create a hierarchy of directories when retrieving
827 recursively. With this option turned on, all files will get saved
828 to the current directory, without clobbering (if a name shows up
829 more than once, the filenames will get extensions .n).
830 -x
832 --force-directories
833 The opposite of -nd---create a hierarchy of directories, even if
834 one would not have been created otherwise. E.g. wget -x
835 http://fly.srk.fer.hr/robots.txt will save the downloaded file to
836 fly.srk.fer.hr/robots.txt.
837 -nH
839 --no-host-directories
840 Disable generation of host-prefixed directories. By default,
841 invoking Wget with -r http://fly.srk.fer.hr/ will create a
842 structure of directories beginning with fly.srk.fer.hr/. This
843 option disables such behavior.
844 --protocol-directories
846 Use the protocol name as a directory component of local file names.
847 For example, with this option, wget -r http://host will save to
848 http/host/... rather than just to host/....
849 --cut-dirs=number
851 Ignore number directory components. This is useful for getting a
852 fine-grained control over the directory where recursive retrieval
853 will be saved.
854 Take, for example, the directory at
856 ftp://ftp.xemacs.org/pub/xemacs/. If you retrieve it with -r, it
857 will be saved locally under ftp.xemacs.org/pub/xemacs/. While the
858 -nH option can remove the ftp.xemacs.org/ part, you are still stuck
859 with pub/xemacs. This is where --cut-dirs comes in handy; it makes
860 Wget not "see" number remote directory components. Here are
861 several examples of how --cut-dirs option works.
862 No options -> ftp.xemacs.org/pub/xemacs/
864 -nH -> pub/xemacs/
865 -nH --cut-dirs=1 -> xemacs/
866 -nH --cut-dirs=2 -> .
867 --cut-dirs=1 -> ftp.xemacs.org/xemacs/
869 ...
870 If you just want to get rid of the directory structure, this option
872 is similar to a combination of -nd and -P. However, unlike -nd,
873 --cut-dirs does not lose with subdirectories---for instance, with
874 -nH --cut-dirs=1, a beta/ subdirectory will be placed to
875 xemacs/beta, as one would expect.
876 -P prefix
878 --directory-prefix=prefix
879 Set directory prefix to prefix. The directory prefix is the
880 directory where all other files and subdirectories will be saved
881 to, i.e. the top of the retrieval tree. The default is . (the
882 current directory).
883 HTTP Options
885 --default-page=name
886 Use name as the default file name when it isn't known (i.e., for
887 URLs that end in a slash), instead of index.html.
888 -E
890 --adjust-extension
891 If a file of type application/xhtml+xml or text/html is downloaded
892 and the URL does not end with the regexp \.[Hh][Tt][Mm][Ll]?, this
893 option will cause the suffix .html to be appended to the local
894 filename. This is useful, for instance, when you're mirroring a
895 remote site that uses .asp pages, but you want the mirrored pages
896 to be viewable on your stock Apache server. Another good use for
897 this is when you're downloading CGI-generated materials. A URL
898 like http://site.com/article.cgi?25 will be saved as
899 article.cgi?25.html.
900 Note that filenames changed in this way will be re-downloaded every
902 time you re-mirror a site, because Wget can't tell that the local
903 X.html file corresponds to remote URL X (since it doesn't yet know
904 that the URL produces output of type text/html or
905 application/xhtml+xml.
906 As of version 1.12, Wget will also ensure that any downloaded files
908 of type text/css end in the suffix .css, and the option was renamed
909 from --html-extension, to better reflect its new behavior. The old
910 option name is still acceptable, but should now be considered
911 deprecated.
912 As of version 1.19.2, Wget will also ensure that any downloaded
914 files with a "Content-Encoding" of br, compress, deflate or gzip
915 end in the suffix .br, .Z, .zlib and .gz respectively.
916 At some point in the future, this option may well be expanded to
918 include suffixes for other types of content, including content
919 types that are not parsed by Wget.
920 --http-user=user
922 --http-password=password
923 Specify the username user and password password on an HTTP server.
924 According to the type of the challenge, Wget will encode them using
925 either the "basic" (insecure), the "digest", or the Windows "NTLM"
926 authentication scheme.
927 Another way to specify username and password is in the URL itself.
929 Either method reveals your password to anyone who bothers to run
930 "ps". To prevent the passwords from being seen, use the
931 --use-askpass or store them in .wgetrc or .netrc, and make sure to
932 protect those files from other users with "chmod". If the
933 passwords are really important, do not leave them lying in those
934 files either---edit the files and delete them after Wget has
935 started the download.
936 --no-http-keep-alive
938 Turn off the "keep-alive" feature for HTTP downloads. Normally,
939 Wget asks the server to keep the connection open so that, when you
940 download more than one document from the same server, they get
941 transferred over the same TCP connection. This saves time and at
942 the same time reduces the load on the server.
943 This option is useful when, for some reason, persistent (keep-
945 alive) connections don't work for you, for example due to a server
946 bug or due to the inability of server-side scripts to cope with the
947 connections.
948 --no-cache
950 Disable server-side cache. In this case, Wget will send the remote
951 server appropriate directives (Cache-Control: no-cache and Pragma:
952 no-cache) to get the file from the remote service, rather than
953 returning the cached version. This is especially useful for
954 retrieving and flushing out-of-date documents on proxy servers.
955 Caching is allowed by default.
957 --no-cookies
959 Disable the use of cookies. Cookies are a mechanism for
960 maintaining server-side state. The server sends the client a
961 cookie using the "Set-Cookie" header, and the client responds with
962 the same cookie upon further requests. Since cookies allow the
963 server owners to keep track of visitors and for sites to exchange
964 this information, some consider them a breach of privacy. The
965 default is to use cookies; however, storing cookies is not on by
966 default.
967 --load-cookies file
969 Load cookies from file before the first HTTP retrieval. file is a
970 textual file in the format originally used by Netscape's
971 cookies.txt file.
972 You will typically use this option when mirroring sites that
974 require that you be logged in to access some or all of their
975 content. The login process typically works by the web server
976 issuing an HTTP cookie upon receiving and verifying your
977 credentials. The cookie is then resent by the browser when
978 accessing that part of the site, and so proves your identity.
979 Mirroring such a site requires Wget to send the same cookies your
981 browser sends when communicating with the site. This is achieved
982 by --load-cookies---simply point Wget to the location of the
983 cookies.txt file, and it will send the same cookies your browser
984 would send in the same situation. Different browsers keep textual
985 cookie files in different locations:
986 "Netscape 4.x."
988 The cookies are in ~/.netscape/cookies.txt.
989 "Mozilla and Netscape 6.x."
991 Mozilla's cookie file is also named cookies.txt, located
992 somewhere under ~/.mozilla, in the directory of your profile.
993 The full path usually ends up looking somewhat like
994 ~/.mozilla/default/some-weird-string/cookies.txt.
995 "Internet Explorer."
997 You can produce a cookie file Wget can use by using the File
998 menu, Import and Export, Export Cookies. This has been tested
999 with Internet Explorer 5; it is not guaranteed to work with
1000 earlier versions.
1001 "Other browsers."
1003 If you are using a different browser to create your cookies,
1004 --load-cookies will only work if you can locate or produce a
1005 cookie file in the Netscape format that Wget expects.
1006 If you cannot use --load-cookies, there might still be an
1008 alternative. If your browser supports a "cookie manager", you can
1009 use it to view the cookies used when accessing the site you're
1010 mirroring. Write down the name and value of the cookie, and
1011 manually instruct Wget to send those cookies, bypassing the
1012 "official" cookie support:
1013 wget --no-cookies --header "Cookie: <name>=<value>"
1015 --save-cookies file
1017 Save cookies to file before exiting. This will not save cookies
1018 that have expired or that have no expiry time (so-called "session
1019 cookies"), but also see --keep-session-cookies.
1020 --keep-session-cookies
1022 When specified, causes --save-cookies to also save session cookies.
1023 Session cookies are normally not saved because they are meant to be
1024 kept in memory and forgotten when you exit the browser. Saving
1025 them is useful on sites that require you to log in or to visit the
1026 home page before you can access some pages. With this option,
1027 multiple Wget runs are considered a single browser session as far
1028 as the site is concerned.
1029 Since the cookie file format does not normally carry session
1031 cookies, Wget marks them with an expiry timestamp of 0. Wget's
1032 --load-cookies recognizes those as session cookies, but it might
1033 confuse other browsers. Also note that cookies so loaded will be
1034 treated as other session cookies, which means that if you want
1035 --save-cookies to preserve them again, you must use
1036 --keep-session-cookies again.
1037 --ignore-length
1039 Unfortunately, some HTTP servers (CGI programs, to be more precise)
1040 send out bogus "Content-Length" headers, which makes Wget go wild,
1041 as it thinks not all the document was retrieved. You can spot this
1042 syndrome if Wget retries getting the same document again and again,
1043 each time claiming that the (otherwise normal) connection has
1044 closed on the very same byte.
1045 With this option, Wget will ignore the "Content-Length" header---as
1047 if it never existed.
1048 --header=header-line
1050 Send header-line along with the rest of the headers in each HTTP
1051 request. The supplied header is sent as-is, which means it must
1052 contain name and value separated by colon, and must not contain
1053 newlines.
1054 You may define more than one additional header by specifying
1056 --header more than once.
1057 wget --header='Accept-Charset: iso-8859-2' \
1059 --header='Accept-Language: hr' \
1060 http://fly.srk.fer.hr/
1061 Specification of an empty string as the header value will clear all
1063 previous user-defined headers.
1064 As of Wget 1.10, this option can be used to override headers
1066 otherwise generated automatically. This example instructs Wget to
1067 connect to localhost, but to specify foo.bar in the "Host" header:
1068 wget --header="Host: foo.bar" http://localhost/
1070 In versions of Wget prior to 1.10 such use of --header caused
1072 sending of duplicate headers.
1073 --compression=type
1075 Choose the type of compression to be used. Legal values are auto,
1076 gzip and none.
1077 If auto or gzip are specified, Wget asks the server to compress the
1079 file using the gzip compression format. If the server compresses
1080 the file and responds with the "Content-Encoding" header field set
1081 appropriately, the file will be decompressed automatically.
1082 If none is specified, wget will not ask the server to compress the
1084 file and will not decompress any server responses. This is the
1085 default.
1086 Compression support is currently experimental. In case it is turned
1088 on, please report any bugs to "bug-wget@gnu.org".
1089 --max-redirect=number
1091 Specifies the maximum number of redirections to follow for a
1092 resource. The default is 20, which is usually far more than
1093 necessary. However, on those occasions where you want to allow more
1094 (or fewer), this is the option to use.
1095 --proxy-user=user
1097 --proxy-password=password
1098 Specify the username user and password password for authentication
1099 on a proxy server. Wget will encode them using the "basic"
1100 authentication scheme.
1101 Security considerations similar to those with --http-password
1103 pertain here as well.
1104 --referer=url
1106 Include `Referer: url' header in HTTP request. Useful for
1107 retrieving documents with server-side processing that assume they
1108 are always being retrieved by interactive web browsers and only
1109 come out properly when Referer is set to one of the pages that
1110 point to them.
1111 --save-headers
1113 Save the headers sent by the HTTP server to the file, preceding the
1114 actual contents, with an empty line as the separator.
1115 -U agent-string
1117 --user-agent=agent-string
1118 Identify as agent-string to the HTTP server.
1119 The HTTP protocol allows the clients to identify themselves using a
1121 "User-Agent" header field. This enables distinguishing the WWW
1122 software, usually for statistical purposes or for tracing of
1123 protocol violations. Wget normally identifies as Wget/version,
1124 version being the current version number of Wget.
1125 However, some sites have been known to impose the policy of
1127 tailoring the output according to the "User-Agent"-supplied
1128 information. While this is not such a bad idea in theory, it has
1129 been abused by servers denying information to clients other than
1130 (historically) Netscape or, more frequently, Microsoft Internet
1131 Explorer. This option allows you to change the "User-Agent" line
1132 issued by Wget. Use of this option is discouraged, unless you
1133 really know what you are doing.
1134 Specifying empty user agent with --user-agent="" instructs Wget not
1136 to send the "User-Agent" header in HTTP requests.
1137 --post-data=string
1139 --post-file=file
1140 Use POST as the method for all HTTP requests and send the specified
1141 data in the request body. --post-data sends string as data,
1142 whereas --post-file sends the contents of file. Other than that,
1143 they work in exactly the same way. In particular, they both expect
1144 content of the form "key1=value1&key2=value2", with percent-
1145 encoding for special characters; the only difference is that one
1146 expects its content as a command-line parameter and the other
1147 accepts its content from a file. In particular, --post-file is not
1148 for transmitting files as form attachments: those must appear as
1149 "key=value" data (with appropriate percent-coding) just like
1150 everything else. Wget does not currently support
1151 "multipart/form-data" for transmitting POST data; only
1152 "application/x-www-form-urlencoded". Only one of --post-data and
1153 --post-file should be specified.
1154 Please note that wget does not require the content to be of the
1156 form "key1=value1&key2=value2", and neither does it test for it.
1157 Wget will simply transmit whatever data is provided to it. Most
1158 servers however expect the POST data to be in the above format when
1159 processing HTML Forms.
1160 When sending a POST request using the --post-file option, Wget
1162 treats the file as a binary file and will send every character in
1163 the POST request without stripping trailing newline or formfeed
1164 characters. Any other control characters in the text will also be
1165 sent as-is in the POST request.
1166 Please be aware that Wget needs to know the size of the POST data
1168 in advance. Therefore the argument to "--post-file" must be a
1169 regular file; specifying a FIFO or something like /dev/stdin won't
1170 work. It's not quite clear how to work around this limitation
1171 inherent in HTTP/1.0. Although HTTP/1.1 introduces chunked
1172 transfer that doesn't require knowing the request length in
1173 advance, a client can't use chunked unless it knows it's talking to
1174 an HTTP/1.1 server. And it can't know that until it receives a
1175 response, which in turn requires the request to have been completed
1176 -- a chicken-and-egg problem.
1177 Note: As of version 1.15 if Wget is redirected after the POST
1179 request is completed, its behaviour will depend on the response
1180 code returned by the server. In case of a 301 Moved Permanently,
1181 302 Moved Temporarily or 307 Temporary Redirect, Wget will, in
1182 accordance with RFC2616, continue to send a POST request. In case
1183 a server wants the client to change the Request method upon
1184 redirection, it should send a 303 See Other response code.
1185 This example shows how to log in to a server using POST and then
1187 proceed to download the desired pages, presumably only accessible
1188 to authorized users:
1189 # Log in to the server. This can be done only once.
1191 wget --save-cookies cookies.txt \
1192 --post-data 'user=foo&password=bar' \
1193 http://example.com/auth.php
1194 # Now grab the page or pages we care about.
1196 wget --load-cookies cookies.txt \
1197 -p http://example.com/interesting/article.php
1198 If the server is using session cookies to track user
1200 authentication, the above will not work because --save-cookies will
1201 not save them (and neither will browsers) and the cookies.txt file
1202 will be empty. In that case use --keep-session-cookies along with
1203 --save-cookies to force saving of session cookies.
1204 --method=HTTP-Method
1206 For the purpose of RESTful scripting, Wget allows sending of other
1207 HTTP Methods without the need to explicitly set them using
1208 --header=Header-Line. Wget will use whatever string is passed to
1209 it after --method as the HTTP Method to the server.
1210 --body-data=Data-String
1212 --body-file=Data-File
1213 Must be set when additional data needs to be sent to the server
1214 along with the Method specified using --method. --body-data sends
1215 string as data, whereas --body-file sends the contents of file.
1216 Other than that, they work in exactly the same way.
1217 Currently, --body-file is not for transmitting files as a whole.
1219 Wget does not currently support "multipart/form-data" for
1220 transmitting data; only "application/x-www-form-urlencoded". In the
1221 future, this may be changed so that wget sends the --body-file as a
1222 complete file instead of sending its contents to the server. Please
1223 be aware that Wget needs to know the contents of BODY Data in
1224 advance, and hence the argument to --body-file should be a regular
1225 file. See --post-file for a more detailed explanation. Only one of
1226 --body-data and --body-file should be specified.
1227 If Wget is redirected after the request is completed, Wget will
1229 suspend the current method and send a GET request till the
1230 redirection is completed. This is true for all redirection
1231 response codes except 307 Temporary Redirect which is used to
1232 explicitly specify that the request method should not change.
1233 Another exception is when the method is set to "POST", in which
1234 case the redirection rules specified under --post-data are
1235 followed.
1236 --content-disposition
1238 If this is set to on, experimental (not fully-functional) support
1239 for "Content-Disposition" headers is enabled. This can currently
1240 result in extra round-trips to the server for a "HEAD" request, and
1241 is known to suffer from a few bugs, which is why it is not
1242 currently enabled by default.
1243 This option is useful for some file-downloading CGI programs that
1245 use "Content-Disposition" headers to describe what the name of a
1246 downloaded file should be.
1247 When combined with --metalink-over-http and --trust-server-names, a
1249 Content-Type: application/metalink4+xml file is named using the
1250 "Content-Disposition" filename field, if available.
1251 --content-on-error
1253 If this is set to on, wget will not skip the content when the
1254 server responds with a http status code that indicates error.
1255 --trust-server-names
1257 If this is set, on a redirect, the local file name will be based on
1258 the redirection URL. By default the local file name is based on
1259 the original URL. When doing recursive retrieving this can be
1260 helpful because in many web sites redirected URLs correspond to an
1261 underlying file structure, while link URLs do not.
1262 --auth-no-challenge
1264 If this option is given, Wget will send Basic HTTP authentication
1265 information (plaintext username and password) for all requests,
1266 just like Wget 1.10.2 and prior did by default.
1267 Use of this option is not recommended, and is intended only to
1269 support some few obscure servers, which never send HTTP
1270 authentication challenges, but accept unsolicited auth info, say,
1271 in addition to form-based authentication.
1272 --retry-on-host-error
1274 Consider host errors, such as "Temporary failure in name
1275 resolution", as non-fatal, transient errors.
1276 --retry-on-http-error=code[,code,...]
1278 Consider given HTTP response codes as non-fatal, transient errors.
1279 Supply a comma-separated list of 3-digit HTTP response codes as
1280 argument. Useful to work around special circumstances where retries
1281 are required, but the server responds with an error code normally
1282 not retried by Wget. Such errors might be 503 (Service Unavailable)
1283 and 429 (Too Many Requests). Retries enabled by this option are
1284 performed subject to the normal retry timing and retry count
1285 limitations of Wget.
1286 Using this option is intended to support special use cases only and
1288 is generally not recommended, as it can force retries even in cases
1289 where the server is actually trying to decrease its load. Please
1290 use wisely and only if you know what you are doing.
1291 HTTPS (SSL/TLS) Options
1293 To support encrypted HTTP (HTTPS) downloads, Wget must be compiled with
1294 an external SSL library. The current default is GnuTLS. In addition,
1295 Wget also supports HSTS (HTTP Strict Transport Security). If Wget is
1296 compiled without SSL support, none of these options are available.
1297 --secure-protocol=protocol
1299 Choose the secure protocol to be used. Legal values are auto,
1300 SSLv2, SSLv3, TLSv1, TLSv1_1, TLSv1_2, TLSv1_3 and PFS. If auto is
1301 used, the SSL library is given the liberty of choosing the
1302 appropriate protocol automatically, which is achieved by sending a
1303 TLSv1 greeting. This is the default.
1304 Specifying SSLv2, SSLv3, TLSv1, TLSv1_1, TLSv1_2 or TLSv1_3 forces
1306 the use of the corresponding protocol. This is useful when talking
1307 to old and buggy SSL server implementations that make it hard for
1308 the underlying SSL library to choose the correct protocol version.
1309 Fortunately, such servers are quite rare.
1310 Specifying PFS enforces the use of the so-called Perfect Forward
1312 Security cipher suites. In short, PFS adds security by creating a
1313 one-time key for each SSL connection. It has a bit more CPU impact
1314 on client and server. We use known to be secure ciphers (e.g. no
1315 MD4) and the TLS protocol. This mode also explicitly excludes non-
1316 PFS key exchange methods, such as RSA.
1317 --https-only
1319 When in recursive mode, only HTTPS links are followed.
1320 --ciphers
1322 Set the cipher list string. Typically this string sets the cipher
1323 suites and other SSL/TLS options that the user wish should be used,
1324 in a set order of preference (GnuTLS calls it 'priority string').
1325 This string will be fed verbatim to the SSL/TLS engine (OpenSSL or
1326 GnuTLS) and hence its format and syntax is dependent on that. Wget
1327 will not process or manipulate it in any way. Refer to the OpenSSL
1328 or GnuTLS documentation for more information.
1329 --no-check-certificate
1331 Don't check the server certificate against the available
1332 certificate authorities. Also don't require the URL host name to
1333 match the common name presented by the certificate.
1334 As of Wget 1.10, the default is to verify the server's certificate
1336 against the recognized certificate authorities, breaking the SSL
1337 handshake and aborting the download if the verification fails.
1338 Although this provides more secure downloads, it does break
1339 interoperability with some sites that worked with previous Wget
1340 versions, particularly those using self-signed, expired, or
1341 otherwise invalid certificates. This option forces an "insecure"
1342 mode of operation that turns the certificate verification errors
1343 into warnings and allows you to proceed.
1344 If you encounter "certificate verification" errors or ones saying
1346 that "common name doesn't match requested host name", you can use
1347 this option to bypass the verification and proceed with the
1348 download. Only use this option if you are otherwise convinced of
1349 the site's authenticity, or if you really don't care about the
1350 validity of its certificate. It is almost always a bad idea not to
1351 check the certificates when transmitting confidential or important
1352 data. For self-signed/internal certificates, you should download
1353 the certificate and verify against that instead of forcing this
1354 insecure mode. If you are really sure of not desiring any
1355 certificate verification, you can specify --check-certificate=quiet
1356 to tell wget to not print any warning about invalid certificates,
1357 albeit in most cases this is the wrong thing to do.
1358 --certificate=file
1360 Use the client certificate stored in file. This is needed for
1361 servers that are configured to require certificates from the
1362 clients that connect to them. Normally a certificate is not
1363 required and this switch is optional.
1364 --certificate-type=type
1366 Specify the type of the client certificate. Legal values are PEM
1367 (assumed by default) and DER, also known as ASN1.
1368 --private-key=file
1370 Read the private key from file. This allows you to provide the
1371 private key in a file separate from the certificate.
1372 --private-key-type=type
1374 Specify the type of the private key. Accepted values are PEM (the
1375 default) and DER.
1376 --ca-certificate=file
1378 Use file as the file with the bundle of certificate authorities
1379 ("CA") to verify the peers. The certificates must be in PEM
1380 format.
1381 Without this option Wget looks for CA certificates at the system-
1383 specified locations, chosen at OpenSSL installation time.
1384 --ca-directory=directory
1386 Specifies directory containing CA certificates in PEM format. Each
1387 file contains one CA certificate, and the file name is based on a
1388 hash value derived from the certificate. This is achieved by
1389 processing a certificate directory with the "c_rehash" utility
1390 supplied with OpenSSL. Using --ca-directory is more efficient than
1391 --ca-certificate when many certificates are installed because it
1392 allows Wget to fetch certificates on demand.
1393 Without this option Wget looks for CA certificates at the system-
1395 specified locations, chosen at OpenSSL installation time.
1396 --crl-file=file
1398 Specifies a CRL file in file. This is needed for certificates that
1399 have been revocated by the CAs.
1400 --pinnedpubkey=file/hashes
1402 Tells wget to use the specified public key file (or hashes) to
1403 verify the peer. This can be a path to a file which contains a
1404 single public key in PEM or DER format, or any number of base64
1405 encoded sha256 hashes preceded by "sha256//" and separated by ";"
1406 When negotiating a TLS or SSL connection, the server sends a
1408 certificate indicating its identity. A public key is extracted from
1409 this certificate and if it does not exactly match the public key(s)
1410 provided to this option, wget will abort the connection before
1411 sending or receiving any data.
1412 --random-file=file
1414 [OpenSSL and LibreSSL only] Use file as the source of random data
1415 for seeding the pseudo-random number generator on systems without
1416 /dev/urandom.
1417 On such systems the SSL library needs an external source of
1419 randomness to initialize. Randomness may be provided by EGD (see
1420 --egd-file below) or read from an external source specified by the
1421 user. If this option is not specified, Wget looks for random data
1422 in $RANDFILE or, if that is unset, in $HOME/.rnd.
1423 If you're getting the "Could not seed OpenSSL PRNG; disabling SSL."
1425 error, you should provide random data using some of the methods
1426 described above.
1427 --egd-file=file
1429 [OpenSSL only] Use file as the EGD socket. EGD stands for Entropy
1430 Gathering Daemon, a user-space program that collects data from
1431 various unpredictable system sources and makes it available to
1432 other programs that might need it. Encryption software, such as
1433 the SSL library, needs sources of non-repeating randomness to seed
1434 the random number generator used to produce cryptographically
1435 strong keys.
1436 OpenSSL allows the user to specify his own source of entropy using
1438 the "RAND_FILE" environment variable. If this variable is unset,
1439 or if the specified file does not produce enough randomness,
1440 OpenSSL will read random data from EGD socket specified using this
1441 option.
1442 If this option is not specified (and the equivalent startup command
1444 is not used), EGD is never contacted. EGD is not needed on modern
1445 Unix systems that support /dev/urandom.
1446 --no-hsts
1448 Wget supports HSTS (HTTP Strict Transport Security, RFC 6797) by
1449 default. Use --no-hsts to make Wget act as a non-HSTS-compliant
1450 UA. As a consequence, Wget would ignore all the
1451 "Strict-Transport-Security" headers, and would not enforce any
1452 existing HSTS policy.
1453 --hsts-file=file
1455 By default, Wget stores its HSTS database in ~/.wget-hsts. You can
1456 use --hsts-file to override this. Wget will use the supplied file
1457 as the HSTS database. Such file must conform to the correct HSTS
1458 database format used by Wget. If Wget cannot parse the provided
1459 file, the behaviour is unspecified.
1460 The Wget's HSTS database is a plain text file. Each line contains
1462 an HSTS entry (ie. a site that has issued a
1463 "Strict-Transport-Security" header and that therefore has specified
1464 a concrete HSTS policy to be applied). Lines starting with a dash
1465 ("#") are ignored by Wget. Please note that in spite of this
1466 convenient human-readability hand-hacking the HSTS database is
1467 generally not a good idea.
1468 An HSTS entry line consists of several fields separated by one or
1470 more whitespace:
1471 "<hostname> SP [<port>] SP <include subdomains> SP <created> SP
1473 <max-age>"
1474 The hostname and port fields indicate the hostname and port to
1476 which the given HSTS policy applies. The port field may be zero,
1477 and it will, in most of the cases. That means that the port number
1478 will not be taken into account when deciding whether such HSTS
1479 policy should be applied on a given request (only the hostname will
1480 be evaluated). When port is different to zero, both the target
1481 hostname and the port will be evaluated and the HSTS policy will
1482 only be applied if both of them match. This feature has been
1483 included for testing/development purposes only. The Wget testsuite
1484 (in testenv/) creates HSTS databases with explicit ports with the
1485 purpose of ensuring Wget's correct behaviour. Applying HSTS
1486 policies to ports other than the default ones is discouraged by RFC
1487 6797 (see Appendix B "Differences between HSTS Policy and Same-
1488 Origin Policy"). Thus, this functionality should not be used in
1489 production environments and port will typically be zero. The last
1490 three fields do what they are expected to. The field
1491 include_subdomains can either be 1 or 0 and it signals whether the
1492 subdomains of the target domain should be part of the given HSTS
1493 policy as well. The created and max-age fields hold the timestamp
1494 values of when such entry was created (first seen by Wget) and the
1495 HSTS-defined value 'max-age', which states how long should that
1496 HSTS policy remain active, measured in seconds elapsed since the
1497 timestamp stored in created. Once that time has passed, that HSTS
1498 policy will no longer be valid and will eventually be removed from
1499 the database.
1500 If you supply your own HSTS database via --hsts-file, be aware that
1502 Wget may modify the provided file if any change occurs between the
1503 HSTS policies requested by the remote servers and those in the
1504 file. When Wget exits, it effectively updates the HSTS database by
1505 rewriting the database file with the new entries.
1506 If the supplied file does not exist, Wget will create one. This
1508 file will contain the new HSTS entries. If no HSTS entries were
1509 generated (no "Strict-Transport-Security" headers were sent by any
1510 of the servers) then no file will be created, not even an empty
1511 one. This behaviour applies to the default database file
1512 (~/.wget-hsts) as well: it will not be created until some server
1513 enforces an HSTS policy.
1514 Care is taken not to override possible changes made by other Wget
1516 processes at the same time over the HSTS database. Before dumping
1517 the updated HSTS entries on the file, Wget will re-read it and
1518 merge the changes.
1519 Using a custom HSTS database and/or modifying an existing one is
1521 discouraged. For more information about the potential security
1522 threats arose from such practice, see section 14 "Security
1523 Considerations" of RFC 6797, specially section 14.9 "Creative
1524 Manipulation of HSTS Policy Store".
1525 --warc-file=file
1527 Use file as the destination WARC file.
1528 --warc-header=string
1530 Use string into as the warcinfo record.
1531 --warc-max-size=size
1533 Set the maximum size of the WARC files to size.
1534 --warc-cdx
1536 Write CDX index files.
1537 --warc-dedup=file
1539 Do not store records listed in this CDX file.
1540 --no-warc-compression
1542 Do not compress WARC files with GZIP.
1543 --no-warc-digests
1545 Do not calculate SHA1 digests.
1546 --no-warc-keep-log
1548 Do not store the log file in a WARC record.
1549 --warc-tempdir=dir
1551 Specify the location for temporary files created by the WARC
1552 writer.
1553 FTP Options
1555 --ftp-user=user
1556 --ftp-password=password
1557 Specify the username user and password password on an FTP server.
1558 Without this, or the corresponding startup option, the password
1559 defaults to -wget@, normally used for anonymous FTP.
1560 Another way to specify username and password is in the URL itself.
1562 Either method reveals your password to anyone who bothers to run
1563 "ps". To prevent the passwords from being seen, store them in
1564 .wgetrc or .netrc, and make sure to protect those files from other
1565 users with "chmod". If the passwords are really important, do not
1566 leave them lying in those files either---edit the files and delete
1567 them after Wget has started the download.
1568 --no-remove-listing
1570 Don't remove the temporary .listing files generated by FTP
1571 retrievals. Normally, these files contain the raw directory
1572 listings received from FTP servers. Not removing them can be
1573 useful for debugging purposes, or when you want to be able to
1574 easily check on the contents of remote server directories (e.g. to
1575 verify that a mirror you're running is complete).
1576 Note that even though Wget writes to a known filename for this
1578 file, this is not a security hole in the scenario of a user making
1579 .listing a symbolic link to /etc/passwd or something and asking
1580 "root" to run Wget in his or her directory. Depending on the
1581 options used, either Wget will refuse to write to .listing, making
1582 the globbing/recursion/time-stamping operation fail, or the
1583 symbolic link will be deleted and replaced with the actual .listing
1584 file, or the listing will be written to a .listing.number file.
1585 Even though this situation isn't a problem, though, "root" should
1587 never run Wget in a non-trusted user's directory. A user could do
1588 something as simple as linking index.html to /etc/passwd and asking
1589 "root" to run Wget with -N or -r so the file will be overwritten.
1590 --no-glob
1592 Turn off FTP globbing. Globbing refers to the use of shell-like
1593 special characters (wildcards), like *, ?, [ and ] to retrieve more
1594 than one file from the same directory at once, like:
1595 wget ftp://gnjilux.srk.fer.hr/*.msg
1597 By default, globbing will be turned on if the URL contains a
1599 globbing character. This option may be used to turn globbing on or
1600 off permanently.
1601 You may have to quote the URL to protect it from being expanded by
1603 your shell. Globbing makes Wget look for a directory listing,
1604 which is system-specific. This is why it currently works only with
1605 Unix FTP servers (and the ones emulating Unix "ls" output).
1606 --no-passive-ftp
1608 Disable the use of the passive FTP transfer mode. Passive FTP
1609 mandates that the client connect to the server to establish the
1610 data connection rather than the other way around.
1611 If the machine is connected to the Internet directly, both passive
1613 and active FTP should work equally well. Behind most firewall and
1614 NAT configurations passive FTP has a better chance of working.
1615 However, in some rare firewall configurations, active FTP actually
1616 works when passive FTP doesn't. If you suspect this to be the
1617 case, use this option, or set "passive_ftp=off" in your init file.
1618 --preserve-permissions
1620 Preserve remote file permissions instead of permissions set by
1621 umask.
1622 --retr-symlinks
1624 By default, when retrieving FTP directories recursively and a
1625 symbolic link is encountered, the symbolic link is traversed and
1626 the pointed-to files are retrieved. Currently, Wget does not
1627 traverse symbolic links to directories to download them
1628 recursively, though this feature may be added in the future.
1629 When --retr-symlinks=no is specified, the linked-to file is not
1631 downloaded. Instead, a matching symbolic link is created on the
1632 local filesystem. The pointed-to file will not be retrieved unless
1633 this recursive retrieval would have encountered it separately and
1634 downloaded it anyway. This option poses a security risk where a
1635 malicious FTP Server may cause Wget to write to files outside of
1636 the intended directories through a specially crafted .LISTING file.
1637 Note that when retrieving a file (not a directory) because it was
1639 specified on the command-line, rather than because it was recursed
1640 to, this option has no effect. Symbolic links are always traversed
1641 in this case.
1642 FTPS Options
1644 --ftps-implicit
1645 This option tells Wget to use FTPS implicitly. Implicit FTPS
1646 consists of initializing SSL/TLS from the very beginning of the
1647 control connection. This option does not send an "AUTH TLS"
1648 command: it assumes the server speaks FTPS and directly starts an
1649 SSL/TLS connection. If the attempt is successful, the session
1650 continues just like regular FTPS ("PBSZ" and "PROT" are sent,
1651 etc.). Implicit FTPS is no longer a requirement for FTPS
1652 implementations, and thus many servers may not support it. If
1653 --ftps-implicit is passed and no explicit port number specified,
1654 the default port for implicit FTPS, 990, will be used, instead of
1655 the default port for the "normal" (explicit) FTPS which is the same
1656 as that of FTP, 21.
1657 --no-ftps-resume-ssl
1659 Do not resume the SSL/TLS session in the data channel. When
1660 starting a data connection, Wget tries to resume the SSL/TLS
1661 session previously started in the control connection. SSL/TLS
1662 session resumption avoids performing an entirely new handshake by
1663 reusing the SSL/TLS parameters of a previous session. Typically,
1664 the FTPS servers want it that way, so Wget does this by default.
1665 Under rare circumstances however, one might want to start an
1666 entirely new SSL/TLS session in every data connection. This is
1667 what --no-ftps-resume-ssl is for.
1668 --ftps-clear-data-connection
1670 All the data connections will be in plain text. Only the control
1671 connection will be under SSL/TLS. Wget will send a "PROT C" command
1672 to achieve this, which must be approved by the server.
1673 --ftps-fallback-to-ftp
1675 Fall back to FTP if FTPS is not supported by the target server. For
1676 security reasons, this option is not asserted by default. The
1677 default behaviour is to exit with an error. If a server does not
1678 successfully reply to the initial "AUTH TLS" command, or in the
1679 case of implicit FTPS, if the initial SSL/TLS connection attempt is
1680 rejected, it is considered that such server does not support FTPS.
1681 Recursive Retrieval Options
1683 -r
1684 --recursive
1685 Turn on recursive retrieving. The default maximum depth is 5.
1686 -l depth
1688 --level=depth
1689 Set the maximum number of subdirectories that Wget will recurse
1690 into to depth. In order to prevent one from accidentally
1691 downloading very large websites when using recursion this is
1692 limited to a depth of 5 by default, i.e., it will traverse at most
1693 5 directories deep starting from the provided URL. Set -l 0 or -l
1694 inf for infinite recursion depth.
1695 wget -r -l 0 http://<site>/1.html
1697 Ideally, one would expect this to download just 1.html. but
1699 unfortunately this is not the case, because -l 0 is equivalent to
1700 -l inf---that is, infinite recursion. To download a single HTML
1701 page (or a handful of them), specify them all on the command line
1702 and leave away -r and -l. To download the essential items to view a
1703 single HTML page, see page requisites.
1704 --delete-after
1706 This option tells Wget to delete every single file it downloads,
1707 after having done so. It is useful for pre-fetching popular pages
1708 through a proxy, e.g.:
1709 wget -r -nd --delete-after http://whatever.com/~popular/page/
1711 The -r option is to retrieve recursively, and -nd to not create
1713 directories.
1714 Note that --delete-after deletes files on the local machine. It
1716 does not issue the DELE command to remote FTP sites, for instance.
1717 Also note that when --delete-after is specified, --convert-links is
1718 ignored, so .orig files are simply not created in the first place.
1719 -k
1721 --convert-links
1722 After the download is complete, convert the links in the document
1723 to make them suitable for local viewing. This affects not only the
1724 visible hyperlinks, but any part of the document that links to
1725 external content, such as embedded images, links to style sheets,
1726 hyperlinks to non-HTML content, etc.
1727 Each link will be changed in one of the two ways:
1729 • The links to files that have been downloaded by Wget will be
1731 changed to refer to the file they point to as a relative link.
1732 Example: if the downloaded file /foo/doc.html links to
1734 /bar/img.gif, also downloaded, then the link in doc.html will
1735 be modified to point to ../bar/img.gif. This kind of
1736 transformation works reliably for arbitrary combinations of
1737 directories.
1738 • The links to files that have not been downloaded by Wget will
1740 be changed to include host name and absolute path of the
1741 location they point to.
1742 Example: if the downloaded file /foo/doc.html links to
1744 /bar/img.gif (or to ../bar/img.gif), then the link in doc.html
1745 will be modified to point to http://hostname/bar/img.gif.
1746 Because of this, local browsing works reliably: if a linked file
1748 was downloaded, the link will refer to its local name; if it was
1749 not downloaded, the link will refer to its full Internet address
1750 rather than presenting a broken link. The fact that the former
1751 links are converted to relative links ensures that you can move the
1752 downloaded hierarchy to another directory.
1753 Note that only at the end of the download can Wget know which links
1755 have been downloaded. Because of that, the work done by -k will be
1756 performed at the end of all the downloads.
1757 --convert-file-only
1759 This option converts only the filename part of the URLs, leaving
1760 the rest of the URLs untouched. This filename part is sometimes
1761 referred to as the "basename", although we avoid that term here in
1762 order not to cause confusion.
1763 It works particularly well in conjunction with --adjust-extension,
1765 although this coupling is not enforced. It proves useful to
1766 populate Internet caches with files downloaded from different
1767 hosts.
1768 Example: if some link points to //foo.com/bar.cgi?xyz with
1770 --adjust-extension asserted and its local destination is intended
1771 to be ./foo.com/bar.cgi?xyz.css, then the link would be converted
1772 to //foo.com/bar.cgi?xyz.css. Note that only the filename part has
1773 been modified. The rest of the URL has been left untouched,
1774 including the net path ("//") which would otherwise be processed by
1775 Wget and converted to the effective scheme (ie. "http://").
1776 -K
1778 --backup-converted
1779 When converting a file, back up the original version with a .orig
1780 suffix. Affects the behavior of -N.
1781 -m
1783 --mirror
1784 Turn on options suitable for mirroring. This option turns on
1785 recursion and time-stamping, sets infinite recursion depth and
1786 keeps FTP directory listings. It is currently equivalent to -r -N
1787 -l inf --no-remove-listing.
1788 -p
1790 --page-requisites
1791 This option causes Wget to download all the files that are
1792 necessary to properly display a given HTML page. This includes
1793 such things as inlined images, sounds, and referenced stylesheets.
1794 Ordinarily, when downloading a single HTML page, any requisite
1796 documents that may be needed to display it properly are not
1797 downloaded. Using -r together with -l can help, but since Wget
1798 does not ordinarily distinguish between external and inlined
1799 documents, one is generally left with "leaf documents" that are
1800 missing their requisites.
1801 For instance, say document 1.html contains an "<IMG>" tag
1803 referencing 1.gif and an "<A>" tag pointing to external document
1804 2.html. Say that 2.html is similar but that its image is 2.gif and
1805 it links to 3.html. Say this continues up to some arbitrarily high
1806 number.
1807 If one executes the command:
1809 wget -r -l 2 http://<site>/1.html
1811 then 1.html, 1.gif, 2.html, 2.gif, and 3.html will be downloaded.
1813 As you can see, 3.html is without its requisite 3.gif because Wget
1814 is simply counting the number of hops (up to 2) away from 1.html in
1815 order to determine where to stop the recursion. However, with this
1816 command:
1817 wget -r -l 2 -p http://<site>/1.html
1819 all the above files and 3.html's requisite 3.gif will be
1821 downloaded. Similarly,
1822 wget -r -l 1 -p http://<site>/1.html
1824 will cause 1.html, 1.gif, 2.html, and 2.gif to be downloaded. One
1826 might think that:
1827 wget -r -l 0 -p http://<site>/1.html
1829 would download just 1.html and 1.gif, but unfortunately this is not
1831 the case, because -l 0 is equivalent to -l inf---that is, infinite
1832 recursion. To download a single HTML page (or a handful of them,
1833 all specified on the command-line or in a -i URL input file) and
1834 its (or their) requisites, simply leave off -r and -l:
1835 wget -p http://<site>/1.html
1837 Note that Wget will behave as if -r had been specified, but only
1839 that single page and its requisites will be downloaded. Links from
1840 that page to external documents will not be followed. Actually, to
1841 download a single page and all its requisites (even if they exist
1842 on separate websites), and make sure the lot displays properly
1843 locally, this author likes to use a few options in addition to -p:
1844 wget -E -H -k -K -p http://<site>/<document>
1846 To finish off this topic, it's worth knowing that Wget's idea of an
1848 external document link is any URL specified in an "<A>" tag, an
1849 "<AREA>" tag, or a "<LINK>" tag other than "<LINK
1850 REL="stylesheet">".
1851 --strict-comments
1853 Turn on strict parsing of HTML comments. The default is to
1854 terminate comments at the first occurrence of -->.
1855 According to specifications, HTML comments are expressed as SGML
1857 declarations. Declaration is special markup that begins with <!
1858 and ends with >, such as <!DOCTYPE ...>, that may contain comments
1859 between a pair of -- delimiters. HTML comments are "empty
1860 declarations", SGML declarations without any non-comment text.
1861 Therefore, <!--foo--> is a valid comment, and so is <!--one--
1862 --two-->, but <!--1--2--> is not.
1863 On the other hand, most HTML writers don't perceive comments as
1865 anything other than text delimited with <!-- and -->, which is not
1866 quite the same. For example, something like <!------------> works
1867 as a valid comment as long as the number of dashes is a multiple of
1868 four (!). If not, the comment technically lasts until the next --,
1869 which may be at the other end of the document. Because of this,
1870 many popular browsers completely ignore the specification and
1871 implement what users have come to expect: comments delimited with
1872 <!-- and -->.
1873 Until version 1.9, Wget interpreted comments strictly, which
1875 resulted in missing links in many web pages that displayed fine in
1876 browsers, but had the misfortune of containing non-compliant
1877 comments. Beginning with version 1.9, Wget has joined the ranks of
1878 clients that implements "naive" comments, terminating each comment
1879 at the first occurrence of -->.
1880 If, for whatever reason, you want strict comment parsing, use this
1882 option to turn it on.
1883 Recursive Accept/Reject Options
1885 -A acclist --accept acclist
1886 -R rejlist --reject rejlist
1887 Specify comma-separated lists of file name suffixes or patterns to
1888 accept or reject. Note that if any of the wildcard characters, *,
1889 ?, [ or ], appear in an element of acclist or rejlist, it will be
1890 treated as a pattern, rather than a suffix. In this case, you have
1891 to enclose the pattern into quotes to prevent your shell from
1892 expanding it, like in -A "*.mp3" or -A '*.mp3'.
1893 --accept-regex urlregex
1895 --reject-regex urlregex
1896 Specify a regular expression to accept or reject the complete URL.
1897 --regex-type regextype
1899 Specify the regular expression type. Possible types are posix or
1900 pcre. Note that to be able to use pcre type, wget has to be
1901 compiled with libpcre support.
1902 -D domain-list
1904 --domains=domain-list
1905 Set domains to be followed. domain-list is a comma-separated list
1906 of domains. Note that it does not turn on -H.
1907 --exclude-domains domain-list
1909 Specify the domains that are not to be followed.
1910 --follow-ftp
1912 Follow FTP links from HTML documents. Without this option, Wget
1913 will ignore all the FTP links.
1914 --follow-tags=list
1916 Wget has an internal table of HTML tag / attribute pairs that it
1917 considers when looking for linked documents during a recursive
1918 retrieval. If a user wants only a subset of those tags to be
1919 considered, however, he or she should be specify such tags in a
1920 comma-separated list with this option.
1921 --ignore-tags=list
1923 This is the opposite of the --follow-tags option. To skip certain
1924 HTML tags when recursively looking for documents to download,
1925 specify them in a comma-separated list.
1926 In the past, this option was the best bet for downloading a single
1928 page and its requisites, using a command-line like:
1929 wget --ignore-tags=a,area -H -k -K -r http://<site>/<document>
1931 However, the author of this option came across a page with tags
1933 like "<LINK REL="home" HREF="/">" and came to the realization that
1934 specifying tags to ignore was not enough. One can't just tell Wget
1935 to ignore "<LINK>", because then stylesheets will not be
1936 downloaded. Now the best bet for downloading a single page and its
1937 requisites is the dedicated --page-requisites option.
1938 --ignore-case
1940 Ignore case when matching files and directories. This influences
1941 the behavior of -R, -A, -I, and -X options, as well as globbing
1942 implemented when downloading from FTP sites. For example, with
1943 this option, -A "*.txt" will match file1.txt, but also file2.TXT,
1944 file3.TxT, and so on. The quotes in the example are to prevent the
1945 shell from expanding the pattern.
1946 -H
1948 --span-hosts
1949 Enable spanning across hosts when doing recursive retrieving.
1950 -L
1952 --relative
1953 Follow relative links only. Useful for retrieving a specific home
1954 page without any distractions, not even those from the same hosts.
1955 -I list
1957 --include-directories=list
1958 Specify a comma-separated list of directories you wish to follow
1959 when downloading. Elements of list may contain wildcards.
1960 -X list
1962 --exclude-directories=list
1963 Specify a comma-separated list of directories you wish to exclude
1964 from download. Elements of list may contain wildcards.
1965 -np
1967 --no-parent
1968 Do not ever ascend to the parent directory when retrieving
1969 recursively. This is a useful option, since it guarantees that
1970 only the files below a certain hierarchy will be downloaded.
1971
1973 Wget supports proxies for both HTTP and FTP retrievals. The standard
1974 way to specify proxy location, which Wget recognizes, is using the
1975 following environment variables:
1976 http_proxy
1978 https_proxy
1979 If set, the http_proxy and https_proxy variables should contain the
1980 URLs of the proxies for HTTP and HTTPS connections respectively.
1981 ftp_proxy
1983 This variable should contain the URL of the proxy for FTP
1984 connections. It is quite common that http_proxy and ftp_proxy are
1985 set to the same URL.
1986 no_proxy
1988 This variable should contain a comma-separated list of domain
1989 extensions proxy should not be used for. For instance, if the
1990 value of no_proxy is .mit.edu, proxy will not be used to retrieve
1991 documents from MIT.
1992
1994 Wget may return one of several error codes if it encounters problems.
1995 0 No problems occurred.
1997 1 Generic error code.
1999 2 Parse error---for instance, when parsing command-line options, the
2001 .wgetrc or .netrc...
2002 3 File I/O error.
2004 4 Network failure.
2006 5 SSL verification failure.
2008 6 Username/password authentication failure.
2010 7 Protocol errors.
2012 8 Server issued an error response.
2014 With the exceptions of 0 and 1, the lower-numbered exit codes take
2016 precedence over higher-numbered ones, when multiple types of errors are
2017 encountered.
2018 In versions of Wget prior to 1.12, Wget's exit status tended to be
2020 unhelpful and inconsistent. Recursive downloads would virtually always
2021 return 0 (success), regardless of any issues encountered, and non-
2022 recursive fetches only returned the status corresponding to the most
2023 recently-attempted download.
2024
2026 /etc/wgetrc
2027 Default location of the global startup file.
2028 .wgetrc
2030 User startup file.
2031
2033 You are welcome to submit bug reports via the GNU Wget bug tracker (see
2034 <https://savannah.gnu.org/bugs/?func=additem&group=wget>) or to our
2035 mailing list <bug-wget@gnu.org>.
2036 Visit <https://lists.gnu.org/mailman/listinfo/bug-wget> to get more
2038 info (how to subscribe, list archives, ...).
2039 Before actually submitting a bug report, please try to follow a few
2041 simple guidelines.
2042 1. Please try to ascertain that the behavior you see really is a bug.
2044 If Wget crashes, it's a bug. If Wget does not behave as
2045 documented, it's a bug. If things work strange, but you are not
2046 sure about the way they are supposed to work, it might well be a
2047 bug, but you might want to double-check the documentation and the
2048 mailing lists.
2049 2. Try to repeat the bug in as simple circumstances as possible. E.g.
2051 if Wget crashes while downloading wget -rl0 -kKE -t5 --no-proxy
2052 http://example.com -o /tmp/log, you should try to see if the crash
2053 is repeatable, and if will occur with a simpler set of options.
2054 You might even try to start the download at the page where the
2055 crash occurred to see if that page somehow triggered the crash.
2056 Also, while I will probably be interested to know the contents of
2058 your .wgetrc file, just dumping it into the debug message is
2059 probably a bad idea. Instead, you should first try to see if the
2060 bug repeats with .wgetrc moved out of the way. Only if it turns
2061 out that .wgetrc settings affect the bug, mail me the relevant
2062 parts of the file.
2063 3. Please start Wget with -d option and send us the resulting output
2065 (or relevant parts thereof). If Wget was compiled without debug
2066 support, recompile it---it is much easier to trace bugs with debug
2067 support on.
2068 Note: please make sure to remove any potentially sensitive
2070 information from the debug log before sending it to the bug
2071 address. The "-d" won't go out of its way to collect sensitive
2072 information, but the log will contain a fairly complete transcript
2073 of Wget's communication with the server, which may include
2074 passwords and pieces of downloaded data. Since the bug address is
2075 publicly archived, you may assume that all bug reports are visible
2076 to the public.
2077 4. If Wget has crashed, try to run it in a debugger, e.g. "gdb `which
2079 wget` core" and type "where" to get the backtrace. This may not
2080 work if the system administrator has disabled core files, but it is
2081 safe to try.
2082
2084 This is not the complete manual for GNU Wget. For more complete
2085 information, including more detailed explanations of some of the
2086 options, and a number of commands available for use with .wgetrc files
2087 and the -e option, see the GNU Info entry for wget.
2088 Also see wget2(1), the updated version of GNU Wget with even better
2090 support for recursive downloading and modern protocols like HTTP/2.
2091
2093 Originally written by Hrvoje Nikšić <hniksic@xemacs.org>. Currently
2094 maintained by Darshit Shah <darnir@gnu.org> and Tim Rühsen
2095 <tim.ruehsen@gmx.de>.
2096
2098 Copyright (c) 1996--2011, 2015, 2018--2022 Free Software Foundation,
2099 Inc.
2100 Permission is granted to copy, distribute and/or modify this document
2102 under the terms of the GNU Free Documentation License, Version 1.3 or
2103 any later version published by the Free Software Foundation; with no
2104 Invariant Sections, with no Front-Cover Texts, and with no Back-Cover
2105 Texts. A copy of the license is included in the section entitled "GNU
2106 Free Documentation License".
2107GNU Wget 1.21.3 2023-01-21 WGET(1)
