1MIRMON(1)             User Contributed Perl Documentation            MIRMON(1)
2
3
4

NAME

6       mirmon - monitor the state of mirrors
7

SYNOPSIS

9         mirmon [-v] [-q] [-t timeout] [-c conf] [-get all⎪update⎪url url]
10

OPTIONS

12       -v  Be verbose ; mirmon normally only reports errors and changes in the
13           mirror list.
14
15       -q  Be quiet.
16
17       -t timeout
18           Set the timeout ; the default is 300.
19
20       -get all ⎪ update ⎪ url <url>
21           With all, probe all sites.  With update, probe a selection of the
22           sites ; see option "max_poll" below.  With url, probe only the
23           given url, which must appear in the mirror-list.
24
25       -c name
26           Use config file name. The default list is
27
28             ./mirmon.conf $HOME/.mirmon.conf /etc/mirmon.conf
29

USAGE

31       The program is intended to be run by cron every hour.
32
33         42 * * * * perl /path/to/mirmon -get update
34
35       It quietly probes a subset of the sites in a given list, writes the
36       results in the 'state' file and generates a web page with the results.
37       The subset contains the sites that are new, bad and/or not probed for a
38       specified time.
39
40       When no 'get' option is specified, the program just generates a new web
41       page from the last known state.
42
43       The program checks the mirrors by running a (user specified) program on
44       a pipe. A (user specified) number of probes is run in parallel using
45       nonblocking IO. When something can be read from the pipe, it switches
46       the pipe to blocking IO and reads one line from the pipe. Then it
47       flushes and closes the pipe. No attempt is made to kill the probe.
48
49       The probe should return something that looks like
50
51         1043625600 ...
52
53       that is, a line of text starting with a timestamp. The exit status of
54       the probe is ignored.
55

CONFIG FILE

57       location
58
59       A config file can be specified with the -c option.  If -c is not used,
60       the program looks for a config file in
61
62       * ./mirmon.conf
63       * $HOME/.mirmon.conf
64       * /etc/mirmon.conf
65
66       syntax
67
68       A config file looks like this :
69
70         +--------------------------------------------------
71         ⎪# lines that start with '#' are comment
72         ⎪# blank lines are ignored too
73         ⎪# tabs are replaced by a space
74
75         ⎪# the config entries are 'key' and 'value' pairs
76         ⎪# a 'key' begins in column 1
77         ⎪# the 'value' is the rest of the line
78         ⎪somekey  A_val B_val ...
79         ⎪otherkey X_val Y_val ...
80
81         ⎪# indented lines are glued
82         ⎪# the next three lines mean 'somekey part1 part2 part3'
83         ⎪somekey part1
84         ⎪  part2
85         ⎪  part3
86
87         ⎪# lines starting with a '+' are concatenated
88         ⎪# the next three lines mean 'somekey part1part2part3'
89         ⎪somekey part1
90         ⎪+ part2
91         ⎪+ part3
92
93         ⎪# lines starting with a '.' are glued too
94         ⎪# don't use a '.' on a line by itself
95         ⎪# 'somekey' gets the value "part1\n part2\n part3"
96         ⎪somekey part1
97         ⎪. part2
98         ⎪. part3
99         +--------------------------------------------------
100
101       required entries
102
103       project_name name
104           Specify a short plaintext name for the project.
105
106             project_name Apache
107             project_name CTAN
108
109       project_url url
110           Specify an url pointing to the 'home' of the project.
111
112             project_url http://www.apache.org/
113
114       mirror_list file-name
115           Specify the file containing the mirrors to probe.
116
117             mirror_list /path/to/mirror-list
118
119           If your mirror list is generated by a program, use
120
121             mirror_list /path/to/program arg1 ... ⎪
122
123           Two formats are supported :
124
125           * plain : lines like
126                 us http://www.tux.org/ [email] ...
127                 nl http://apache.cs.uu.nl/dist/ [email] ...
128                 nl rsync://archive.cs.uu.nl/apache-dist/ [email] ...
129
130           * apache : lines like those in the apache mirrors.list
131                 ftp  us ftp://ftp.tux.org/pub/net/apache/dist/ user@tux.org ...
132                 http nl http://apache.cs.uu.nl/dist/ user@cs.uu.nl ...
133
134           Note that in style 'plain' the third item is reserved for an
135           optional email address : the site's contact address.
136
137           Specify the required format with option "list_style" (see below).
138           The default style is 'plain'.
139
140       web_page file-name
141           Specify where the html report page is written.
142
143       icons directory-name
144           Specify the directory where the icons can be found, relative to the
145           web_page, or relative to the DOCUMENTROOT of the web server.
146
147           If/when the web_page lives in directory ".../mirmon/" and the icons
148           live in directory ".../mirmon/icons/", specify
149
150             icons icons
151
152           If/when the icons live in "/path/to/DOCUMENTROOT/icons/mirmon/",
153           specify
154
155             icons /icons/mirmon
156
157       probe program + arguments
158           Specify the program+args to probe the mirrors. Example:
159
160             probe /usr/bin/wget -q -O - -T %TIMEOUT% -t 1 %URL%TIME.txt
161
162           Before the program is started, %TIMEOUT% and %URL% are substituted
163           with the proper timeout and url values.
164
165           Here it is assumed that each hour the root server writes a time‐
166           stamp in /path/to/archive/TIME.txt, for instance with a crontab
167           entry like
168
169             42 * * * * perl -e 'print time, "\n"' > /path/to/archive/TIME.txt
170
171           Mirmon reads one line of output from the probe and interprets the
172           first word on that line as a timestamp ; for example :
173
174             1043625600
175             1043625600 Mon Jan 27 00:00:00 2003
176             1043625600 www.apache.org Mon Jan 27 00:00:00 2003
177
178           Mirmon is distributed with a program "probe" that handles ftp, http
179           and rsync urls.
180
181       state file-name
182           Specify where the file containing the state is written.
183
184           The program reads this file on startup and writes the file when
185           mirrors are probed (-get is specified).
186
187       countries file-name
188           Specify the file containing the country codes; The file should con‐
189           tain lines like
190
191             us - United States
192             nl - Netherlands
193
194           The mirmon package contains a recent ISO list.
195
196           Fake domains like Backup, Master are allowed, and are listed first
197           in the report ; lowercase-first fake domains (like backup) are
198           listed last.
199
200       optional entries
201
202       max_probes number
203           Optionally specify the number of parallel probes (default 25).
204
205       timeout seconds
206           Optionally specify the timeout for the probes (default 300).
207
208           After the last probe is started, the program waits for <timeout> +
209           10 seconds, cleans up and exits.
210
211       project_logo logo
212           Optionally specify (the SRC of the IMG of) a logo to be placed top
213           right on the page.
214
215             project_logo /icons/apache.gif
216             project_logo http://www.apache.org/icons/...
217
218       htm_head html
219           Optionally specify some HTML to be placed before </HEAD>.
220
221             htm_head
222               <link REL=StyleSheet HREF="/style.css" TYPE="text/css">
223
224       htm_top html
225           Optionally specify some HTML to be placed near the top of the page.
226
227             htm_top testing 1, 2, 3
228
229       htm_foot html
230           Optionally specify HTML to be placed near the bottom of the page.
231
232             htm_foot
233               <HR>
234               <A HREF="..."><IMG SRC="..." BORDER=0></A>
235               <HR>
236
237       put_histo top⎪bottom⎪nowhere
238           Optionally specify where the age histogram must be placed.  The
239           default is 'top'.
240
241       min_poll time-spec
242           For 'min_poll' see next item. A time-spec is a number followed by a
243           unit 's' (seconds), or 'm' (minutes), or 'h' (hours), or 'd'
244           (days).  For example '3d' (three days) or '36h' (36 hours).
245
246       max_poll time-spec
247           Optionally specify the maximum probe interval. When the program is
248           called with option '-get update', all sites are probed which are :
249
250           * new
251               the site appears in the list, but there is no known state
252
253           * bad
254               the last probe of the site was unsuccessful
255
256           * old
257               the last probe was more than 'max_poll' ago.
258
259           Sites are not probed if the last probe was less than 'min_poll'
260           ago.  So, if you specify
261
262             min_poll 4h
263             max_poll 12h
264
265           the 'reachable' sites are probed twice daily and the 'unreachable'
266           sites are probed at most six times a day.
267
268           The default 'min_poll' is '1h' (1 hour).  The default 'max_poll' is
269           '4h' (4 hours).
270
271       min_sync time-spec
272           Optionally specify how often the mirrors are required to make an
273           update.
274
275           The default 'min_sync' is '1d' (1 day).
276
277       max_sync time-spec
278           Optionally specify the maximum allowable sync interval.
279
280           Sites exceeding the limit will be considered 'old'.  The default
281           'max_sync' is '2d' (2 days).
282
283       always_get region ...
284           Optionally specify a list of regions that must be probed always.
285
286             always_get Master Tier1
287
288           This is intended for fake regions like Master etc.
289
290       no_randomize
291           Mirmon tries to balance the probe load over the hourly mirmon runs.
292           If the current run has a below average number of mirrors to probe,
293           mirmon probes a few extra, randomly chosen mirrors, picked from the
294           runs that have the highest load.
295
296           If you don't want this behaviour, use no_randomize.
297
298       no_add_slash
299           If the url part of a line in the mirror_list doesn't end in a slash
300           ('/'), mirmon adds a slash and issues a warning unless it is in
301           quiet mode.
302
303           If you don't want this behaviour, use no_add_slash.
304
305       list_style plain⎪apache
306           Optionally specify the format ('plain' or 'apache') of the mir‐
307           ror-list.
308
309           See the description of 'mirror_list' above.  The default list_style
310           is 'plain'.
311
312       site_url site url
313           Optionally specify a substitute url for a site.
314
315           When access to a site is restricted (in Australia, for instance),
316           another (sometimes secret) url can be used to probe the site.  The
317           <site> of an url is the part between '://' and the first '/'.
318
319       env key value
320           Optionally specify an environment variable.
321
322       include file-name
323           Optionally specify a file to include.
324
325           The specified file is processed 'in situ'. After the specified file
326           is read and processed, config processing is resumed in the file
327           where the "include" was encountered.  The include depth is unlim‐
328           ited. However, it is a fatal error to include a file twice under
329           the same name.
330
331       show
332           When the config processor encounters the 'show' command, it dumps
333           the content of the current config to standout, if option "-v" is
334           specified. This is intented for debugging.
335
336       exit
337           When the config processor encounters the 'exit' command, it termi‐
338           nates the program. This is intented for debugging.
339

STATE FILE FORMAT

341       The state file consists of lines; one line per site.  Each line con‐
342       sists of white space separated fields.  The seven fields are :
343
344       * field 1 : url
345           The url as given in the mirror list.
346
347       * field 2 : age
348           The mirror's timestamp found by the last successful probe, or
349           'undef' if no probe was ever successful.
350
351       * field 3 : status last probe
352           The status of the last probe, or 'undef' if the mirror was never
353           probed.
354
355       * field 4 : time last successful probe
356           The timestamp of the last successful probe or 'undef' if the mirror
357           was never successfully probed.
358
359       * field 5 : probe history
360           The probe history is a list of 's' (for success) and 'f' (for fail‐
361           ure) characters indicating the result of the probe. New results are
362           appended whenever the mirror is probed.
363
364       * field 6 : state history
365           The state history consists of a timestamp, a '-' char, and a list
366           of chars indicating a past status: 's' (fresh), 'b' (oldish), 'f'
367           (old), 'z' (bad) or 'x' (skip).  The timestamp indicates when the
368           state history was last updated.  The current status of the mirror
369           is determined by the mirror's age and a few configuration parame‐
370           ters (min_sync, max_sync, max_poll).  The state history is updated
371           when the mirror is probed.  If the last update of the history was
372           less than 24 hours ago, the last status is replaced by the current
373           status.  If the last update of the history was more than 24 hours
374           ago, the current status is appended to the history.  One or more
375           'skip's is inserted, if the timestamp is two or more days old (when
376           mirmon hasn't run for more than two days).
377
378       * field 7 : last probe
379           The timestamp of the last probe, or 'undef' if the mirror was never
380           probed.
381

INSTALLATION

383       general
384
385       * Note: The (empty) state file must exist before mirmon runs.
386       * The mirmon repository is here :
387             https://svn.science.uu.nl/repos/project.mirmon/trunk/
388
389       * The mirmon tarball is here :
390             http://www.staff.science.uu.nl/~penni101/mirmon/mirmon.tar.gz
391
392       installation suggestions
393
394       To install and configure mirmon, take the following steps :
395
396       * First, make the webdir :
397           cd DOCUMENTROOT
398           mkdir mirmon
399
400         For DOCUMENTROOT, substitute the full pathname of the document root
401         of your webserver.
402
403       * Check out the mirmon repository :
404           cd /usr/local/src
405           svn checkout REPO mirmon
406
407         where
408
409           REPO = https://svn.science.uu.nl/repos/project.mirmon/trunk/
410
411         or download the package and unpack it.
412
413       * Chdir to directory mirmon :
414           cd mirmon
415
416       * Create the (empty) state file :
417           touch state.txt
418
419       * Install the icons in the webdir :
420           mkdir DOCUMENTROOT/mirmon/icons
421           cp icons/* DOCUMENTROOT/mirmon/icons
422
423       * Create a mirror list "mirror_list" ;
424         Use your favorite editor, or genererate the list from an existing
425         database.
426
427           nl http://archive.cs.uu.nl/your-project/ contact@cs.uu.nl
428           uk http://mirrors.this.org/your-project/ mirrors@this.org
429           us http://mirrors.that.org/your-project/ mirrors@that.org
430
431         The email addresses are optional.
432
433       * Create a mirmon config file "mirmon.conf" with your favorite editor.
434           # lines must start in the first column ; no leading white space
435           project_name ....
436           project_url  ....
437           mirror_list mirror_list
438           state state.txt
439           countries countries.list
440           web_page DOCUMENTROOT/mirmon/index.html
441           icons /mirmon/icons
442           probe /usr/bin/wget -q -O - -T %TIMEOUT% -t 1 %URL%TIME.txt
443
444         This assumes the project's timestamp is in file "TIME.txt".
445
446       * If you have rsync urls, change the probe line to :
447           probe perl /usr/local/src/mirmon/probe -t %TIMEOUT% %URL%TIME.txt
448
449       * Run mirmon :
450           perl mirmon -v -get all
451
452         The mirmon report should now be in 'DOCUMENTROOT/mirmon/index.html'
453
454           http://www.your.project.org/mirmon/
455
456       * If/when, at a later date, you want to upgrade mirmon :
457           cd /usr/local/src/mirmon
458           svn status -u
459           svn up
460

SEE ALSO

462       mirmon.pm(3)
463
464

AUTHOR

466         (c) 2003-2016 Henk P. Penning
467         Faculty of Science, Utrecht University
468         http://www.staff.science.uu.nl/~penni101/ -- penning@uu.nl
469         mirmon-2.11 - Sat Jul 23 09:12:31 2016 ; henkp
470
471
472
473perl v5.8.8                       2016-07-23                         MIRMON(1)
Impressum