1DEBMIRROR(1) User Contributed Perl Documentation DEBMIRROR(1)
2
6 debmirror - Debian partial mirror script, with ftp, http or rsync and
7 package pool support
8
10 debmirror [options] mirrordir
11
13 This program downloads and maintains a partial local Debian mirror. It
14 can mirror any combination of architectures, distributions, and
15 sections. Files are transferred by ftp, and package pools are fully
16 supported. It also does locking and updates trace files.
17 The partial mirror created by this program is not suitable to be used
19 as a public Debian mirror. If that is your aim, you should instead
20 follow the instructions at <http://www.debian.org/mirrors/ftpmirror>.
21 This program mirrors in three steps.
23 1. download Packages and Sources files
25 First it downloads all Packages and Sources files for the subset of
26 Debian it was instructed to get.
27 2. download everything else
29 The Packages and Sources files are scanned, to build up a list of
30 all the files they refer to. A few other miscellaneous files are
31 added to the list. Then the program makes sure that each file in
32 the list is present on the local mirror and is up-to-date, using
33 file size (and optionally checksum) checks. Any necessary files
34 are downloaded.
35 3. clean up unknown files
37 Any files and directories on the local mirror that are not in the
38 list are removed.
39
41 mirrordir
42 This required (unless defined in a configuration file) parameter
43 specifies where the local mirror directory is. If the directory
44 does not exist, it will be created. Be careful; telling this
45 program that your home directory is the mirrordir is guaranteed to
46 replace your home directory with a Debian mirror!
47 -p, --progress
49 Displays progress bars as files are downloaded.
50 -v, --verbose
52 Displays progress between file downloads.
53 --debug
55 Enables verbose debug output, including ftp protocol dump.
56 --dry-run
58 Simulate a mirror run. This will still download the meta files to
59 the ./.temp working directory, but won't replace the old meta
60 files, won't download debs and source files and only simulates
61 cleanup.
62 --help
64 Display a usage summary.
65 --version
67 Print version number and exit.
68 -h, --host=remotehost
70 Specify the remote host to mirror from. Defaults to ftp.debian.org,
71 you are strongly encouraged to find a closer mirror.
72 -r, --root=directory
74 Specifies the directory on the remote host that is the root of the
75 Debian archive. Defaults to debian, which will work for most
76 mirrors. The root directory has a dists subdirectory.
77 --method=method
79 Specify the method to download files. Currently, supported methods
80 are ftp, http, https, and rsync. The file method is experimentally
81 supported.
82 --passive
84 Download in passive mode when using ftp.
85 -u, --user=remoteusername
87 Specify the remote user name to use to log into the remote host.
88 Defaults to "anonymous".
89 --passwd=remoteuserpassword
91 Specify the remote user password to use to log into the remote ftp
92 host. It is used with --user and defaults to "anonymous@".
93 --proxy=http://user:pass@url:port/
95 Specifies the http proxy (like Squid) to use for http or ftp
96 methods.
97 -d, --dist=foo[,bar,..]
99 Specify the distribution (etch, lenny, squeeze, sid) of Debian to
100 mirror. This switch may be used multiple times, and multiple
101 distributions may be specified at once, separated by commas.
102 You may also use the stable, testing, unstable, names.
104 --omit-suite-symlinks
106 With this option set, debmirror will not create the symlink from
107 suite to codename. This is needed for example when mirroring
108 archived Debian releases as they will all have either "stable" or
109 "oldstable" as suite in their Release files.
110 -s, --section=foo[,bar,..]
112 Specify the section of Debian to mirror. Defaults to
113 "main,contrib,non-free,main/debian-installer".
114 -a, --arch=foo[,bar,..]
116 Specify the architectures to mirror. The default is --arch=i386.
117 Specifying --arch=none will mirror no archs.
118 --rsync-extra=foo[,bar,..]
120 Allows you to also mirror files from a number of directories that
121 are not part of the package archive itself.
122 Debmirror will always use rsync for the transfer of these files,
124 irrespective of what transfer method is specified in the --method
125 option. This will therefore not work if your remote mirror does
126 not support rsync, or if the mirror needs a different --root option
127 for rsync than for the main transfer method specified with
128 --method.
129 Note that excluding individual files in the directories is not
131 supported.
132 The following values are supported.
134 doc
136 Download all files and subdirectories in doc directory, and all
137 README files in the root directory of the archive.
138 indices
140 Download all files and subdirectories in indices directory. Note
141 that this directory can contain some rather large files; don't
142 include this type unless you know you need these files.
143 tools
145 Download all files and subdirectories in tools directory.
146 trace
148 Download the remote mirror's trace files for the archive
149 (project/trace/*). This is enabled by default.
150 none
152 This can be used to disable getting extra files with rsync.
153 If specified, the update of trace files will be done at the
155 beginning of the mirror run; the other types are done near the end.
156 This switch may be used multiple times, and multiple values may be
158 specified at once, separated by commas; unknown values are ignored.
159 --di-dist=dists | foo[,bar,..]
161 Mirror current Debian Installer images for the specified dists.
162 See further the section "Mirroring Debian Installer images" below.
163 --di-arch=arches | foo[,bar,..]
165 Mirror current Debian Installer images for the specified
166 architectures. See further the section "Mirroring Debian Installer
167 images" below.
168 --source
170 Include source in the mirror (default).
171 --nosource
173 Do not include source.
174 --i18n
176 Additionally download Translation-<lang>.* files, which contain
177 translations of package descriptions. Selection of specific
178 translations is possible using the --include and --exclude options.
179 The default is to download only the English file.
180 --getcontents
182 Additionally download Contents.<arch>.gz files. Note that these
183 files can be relatively big and can change frequently, especially
184 for the testing and unstable suites. Use of the available diff
185 files is strongly recommended (see the --diff option).
186 --checksums
188 Use checksums to determine if files on the local mirror that are
189 the correct size actually have the correct content. Not enabled by
190 default, because it is too paranoid, and too slow.
191 When the state cache is used, debmirror will only check checksums
193 during runs where the cache has expired or been invalidated, so it
194 is worth considering to use these two options together.
195 --ignore-missing-release
197 Don't fail if the Release file is missing.
198 --check-gpg, --no-check-gpg
200 Controls whether gpg signatures from the Release.gpg file should be
201 checked. The default is to check signatures.
202 --keyring=file
204 Use file as an additional gpg-format keyring. May be given
205 multiple times.
206 Note that these will be used in addition to
208 $GNUPGHOME/trustedkeys.gpg. The latter can be removed from the set
209 of keyrings by setting $GNUPGHOME to something non-existent when
210 using this option.
211 On a typical Debian system, the Debian archive keyring can be used
213 directly with this option:
214 debmirror --keyring /usr/share/keyrings/debian-archive-keyring.gpg ...
216 --ignore-release-gpg
218 Don't fail if the Release.gpg file is missing. If the file does
219 exist, it is mirrored and verified, but any errors are ignored.
220 --ignore=regex
222 Never delete any files whose filenames match the regex. May be used
223 multiple times.
224 --exclude=regex
226 Never download any files whose filenames match the regex. May be
227 used multiple times.
228 --include=regex
230 Don't exclude any files whose filenames match the regex. May be
231 used multiple times.
232 --exclude-deb-section=regex
234 Never download any files whose Debian Section (games, doc, oldlibs,
235 science, ...) match the regex. May be used multiple times.
236 --limit-priority=regex
238 Limit download to files whose Debian Priority (required, extra,
239 optional, ...) match the regex. May be used multiple times.
240 --exclude-field=fieldname=regex
242 Never download any binary packages where the contents of fieldname
243 match the regex. May be used multiple times. If this option is used
244 and the mirror includes source packages, only those source packages
245 corresponding to included binary packages will be downloaded.
246 --include-field=fieldname=regex
248 Don't exclude any binary packages where the contents of fieldname
249 match the regex. May be used multiple times. If this option is used
250 and the mirror includes source packages, only those source packages
251 corresponding to included binary packages will be downloaded.
252 -t, --timeout=seconds
254 Specifies the timeout to use for network operations (either FTP or
255 rsync). Set this to a higher value if you experience failed
256 downloads. Defaults to 300 seconds.
257 --max-batch=number
259 Download at most max-batch number of files (and ignore rest).
260 --rsync-batch=number
262 Download at most number of files with each rsync call and then
263 loop.
264 --rsync-options=options
266 Specify alternative rsync options to be used. Default options are
267 "-aIL --partial". Care must be taken when specifying alternative
268 options not to disrupt operations, it's best to only add to those
269 options.
270 The most likely option to add is "--bwlimit=x" to avoid saturating
272 the bandwidth of your link.
273 --postcleanup
275 Clean up the local mirror but only after mirroring is complete and
276 only if there was no error.
277 This is the default, because it ensures that the mirror is
279 consistent at all times.
280 --precleanup
282 Clean up the local mirror before starting mirroring.
283 This option may be useful if you have limited disk space, but it
285 will result in an inconsistent mirror when debmirror is running.
286 The deprecated --cleanup option also enables this mode.
288 --nocleanup
290 Do not clean up the local mirror.
291 --skippackages
293 Don't re-download Packages and Sources files. Useful if you know
294 they are up-to-date.
295 --diff=use|mirror|none
297 If --diff=use is specified and the Release file contains entries
298 for diff files, then debmirror will attempt to use them to update
299 Packages, Sources, and Contents files (which can significantly
300 reduce the download size for meta files), but will not include them
301 in the mirror. This is the default behavior and avoids having time
302 consuming diff files for a fast local mirror.
303 Specifying --diff=mirror does the same as use, but will also
305 include the downloaded diff files in the local mirror. Specify
306 --diff=none to completely ignore diff files.
307 Note that if rsync is used as method to download files and the
309 archive being mirrored has "rsyncable" gzipped meta files, then
310 using --diff=none may be the most efficient way to download them.
311 See the gzip(1) man page for information about its rsyncable
312 option.
313 --gzip-options=options
315 Specify alternative options to be used when calling gzip(1) to
316 compress meta files after applying diffs. The default options are
317 "-9 -n --rsyncable" which corresponds with the options used to gzip
318 meta files for the main Debian archive.
319 These options may need to be modified if the checksum of the file
321 as gzipped by debmirror does not match the checksum listed in the
322 Release file (which will result in the gzipped file being
323 downloaded unnecessarily after diffs were successfully applied).
324 --slow-cpu
326 By default debmirror saves some bandwidth by performing cpu-
327 intensive tasks, such as compressing files to generate .gz and .xz
328 files. Use this mode if the computer's CPU is slow, and it makes
329 more sense to use more bandwidth and less CPU.
330 This option implies --diff=none.
332 --state-cache-days=number
334 Save the state of the mirror in a cache file between runs. The
335 cache will expire after the specified number of days, at which time
336 a full check and cleanup of the mirror will be done. While the
337 cache is valid, debmirror will trust that the mirror is consistent
338 with this cache.
339 The cache is only used for files that have a unique name, i.e.
341 binary packages and source files. If a mirror update fails for any
342 reason, the cache will be invalidated and the next run will include
343 a full check.
344 Main advantage of using the state cache is that it avoids a large
346 amount of disk access while checking which files need to be
347 fetched. It may also reduce the time required for mirror updates.
348 --ignore-small-errors
350 Normally debmirror will report an error if any deb files or sources
351 fail to download and refuse to update the meta data to an
352 inconsistent mirror. Normally this is a good things as it indicates
353 something went wrong during download and should be retried. But
354 sometimes the upstream mirror actually is broken. Specifying
355 --ignore-small-errors causes debmirror to ignore missing or broken
356 deb and source files but still be pedantic about checking meta
357 files.
358 --allow-dist-rename
360 The directory name for a dist should be equal to its Codename and
361 not to a Suite. If the local mirror currently has directories named
362 after Suites, debmirror can rename them automatically. An existing
363 symlink from codename to suite will be removed, but debmirror will
364 automatically create a new symlink suite -> codename (immediately
365 after moving meta files in place). This conversion should only be
366 needed once.
367 --disable-ssl-verification
369 When https is used, debmirror checks that the SSL certificate is
370 value.
371 If the server has a self-signed certificate, the check can be
373 disabled with this option.
374 --debmarshal
376 On each pull, keep the repository meta data from dists/* in a
377 numbered subdirectory, and maintain a symlink latest to the most
378 recent pull. This is similar to Debmarshal in tracking mode, see
379 debmarshal.debian.net for examples and use. debmirror cleanup is
380 disabled when this flag is specified. Separate pool and snapshot
381 cleanup utilities are available at
382 http://code.google.com/p/debmarshal/source/browse/#svn/trunk/repository2
383 --config-file=file
385 Specify a configuration file. This option may be repeated to read
386 multiple configuration files. By default debmirror reads
387 /etc/debmirror.conf and ~/.debmirror.conf (see section FILES).
388 Experimental options
390 --retry-rsync-packages=number
391 While downloading Packages and related files via rsync, try up to
392 this many times if rsync fails to connect. Defaults to 1, to try
393 only once. (A typical nondefault value is 10. To try an unlimited
394 number of times, use -1 or 0.)
395
397 Using regular expressions in options
398 Various options accept regular expressions that can be used to tune
399 what is included in the mirror. They can be any regular expression
400 valid in perl, which also means that extended syntax is standard. Make
401 sure to anchor regular expressions appropriately: this is not done by
402 debmirror.
403 The --include and --exclude options can be combined. This combination
405 for example will, if the --i18n option is used, exclude all Translation
406 files, except for the ones for Portuguese (pt) and Brazillian (pt_BR):
407 --exclude='/Translation-.*' --include='/Translation-pt.*'
409 Mirroring Debian Installer images
411 Debmirror will only mirror the "current" images that are on the remote
412 mirror. At least one of the options --di-dist or --di-arch must be
413 passed to enable mirroring of the images.
414 The special values "dists" and "arches" can be used to tell debmirror
416 to use the same dists and architectures for D-I images as for the
417 archive, but it is also possible to specify different values. If either
418 option is not set, it will default to the same values as for the
419 archive.
420 If you wish to create custom CD images using for example debian-cd, you
422 will probably also want add the option "--rsync-extra=doc,tools".
423 Limitations
425 There are no progress updates displayed for D-I images.
427 Archive size
429 The tables in the file /usr/share/doc/debmirror/mirror_size give an
430 indication of the space needed to mirror the Debian archive. They are
431 particularly useful if you wish to set up a partial mirror. Only the
432 size of source and binary packages is included. You should allow for
433 around 1-4 GB of meta data (in ./dists/<dist>) per suite (depending in
434 your settings). Plus whatever space is needed for extra directories
435 (e.g. tools, doc) you wish to mirror.
436 The tables also show how much additional space is required if you add a
438 release on top of its predecessor. Note that the additional space
439 needed for testing and (to a lesser extent) unstable varies during the
440 development cycle of a release. The additional space needed for testing
441 is zero immediately after a stable release and grows from that time
442 onwards.
443 Note Debmirror keeps an extra copy of all meta data. This is necessary
445 to guarantee that the local mirror stays consistent while debmirror is
446 running.
447
449 Simply make a mirror in /srv/mirror/debian, using all defaults (or the
450 settings defined in debmirror.conf):
451 debmirror /srv/mirror/debian
453 Make a mirror of i386 and sparc binaries, main only, and include both
455 unstable and testing versions of Debian; download from
456 'ftp.kernel.org':
457 debmirror -a i386,sparc -d sid -d etch -s main --nosource \
459 -h ftp.nl.debian.org --progress $HOME/mirror/debian
460 Make a mirror using rsync (rsync server is 'ftp.debian.org::debian'),
462 excluding the section 'debug' and the package 'foo-doc':
463 debmirror -e rsync $HOME/mirror/debian --exclude='/foo-doc_' \
465 --exclude-deb-section='^debug$'
466
468 /etc/debmirror.conf
469 ~/.debmirror.conf
470 Debmirror will look for the presence of these files and load them
472 in the indicated order if they exist.
473 See the example in /usr/share/doc/debmirror/examples for syntax.
474 ~/.gnupg/trustedkeys.gpg
476 When gpg checking is enabled,
478 debmirror uses gpgv to verify Release and Release.gpg using the
479 default keying ~/.gnupg/trustedkeys.gpg. This can be changed by
480 exporting GNUPGHOME resulting in $GNUPGHOME/trustedkeys.gpg being
481 used. (Note that keyring files can also be specified directly
482 with debmirror's --keyring option -- see above).
483 To add the right key to this keyring you can import it from the
485 debian keyring (in case of the debian archive) using:
486 gpg --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export \
488 | gpg --no-default-keyring --keyring trustedkeys.gpg --import
489 or download the key from a keyserver:
491 gpg --no-default-keyring --keyring trustedkeys.gpg \
493 --keyserver keyring.debian.org --recv-keys <key ID>
494 The <key ID> can be found in the gpgv error message in debmirror:
496 gpgv: Signature made Tue Jan 23 09:07:53 2007 CET using DSA key ID 2D230C5F
497
499 This program is copyright 2000-2001, 2010-2014 by Joey Hess
500 <joeyh@debian.org>, under the terms of the GNU GPL (either version 2 of
501 the licence or, at your option, any later version), copyright 2001-2002
502 by Joerg Wendland <joergland@debian.org>, copyright 2003-2007 by Goswin
503 von Brederlow <goswin-v-b@web.de>, copyright 2009-2010 by Frans Pop
504 <fjp@debian.org>, copyright 2015 by Thaddeus H. Black <thb@debian.org>,
505 and copyright 2016 by Colin Watson <cjwatson@debian.org>.
506 The author disclaims any responsibility for any mangling of your
508 system, unexpected bandwidth usage bills, meltdown of the Debian mirror
509 network, etc, that this script may cause. See NO WARRANTY section of
510 GPL.
511
513 Author:
514 Joey Hess <joeyh@debian.org>
515 Previous maintainers:
517 Joerg Wendland <joergland@debian.org>
518 Goswin von Brederlow <goswin-v-b@web.de>
519 Frans Pop <fjp@debian.org>
520 Joey Hess <joeyh@debian.org>
521 Thaddeus H. Black <thb@debian.org>
522 Current maintainer:
524 Colin Watson <cjwatson@debian.org>
525
527 Waste bandwidth -- put a partial mirror on your laptop today!
528perl v5.28.1 2018-08-06 DEBMIRROR(1)