1MAKEHISTORY(8) InterNetNews Documentation MAKEHISTORY(8)
2
6 makehistory - Initialize or rebuild INN history database
7
9 makehistory [-abFIOSx] [-f filename] [-l count] [-L load-average] [-s
10 size] [-T tmpdir]
11
13 makehistory rebuilds the history(5) text file, which contains a list of
14 message-IDs of articles already seen by the server. It can also be
15 used to rebuild the overview database. Note that even though the dbz
16 indices for the history file are also rebuilt by makehistory, it is
17 useful to run makedbz(8) after makehistory(8) in order to improve the
18 efficiency of the indices (makehistory does not know how large to make
19 the hash table at first run, unless the size is given by the -s flag).
20 The default location of the history text file is pathdb/history; to
22 specify an alternate location, use the -f flag.
23 By default, makehistory will scan the entire spool, using the storage
25 manager, and write a history line for every article. To also generate
26 overview information, use the -O flag.
27
29 WARNING: If you're trying to rebuild the overview database, be sure to
30 stop innd(8) and delete or zero out the existing database before you
31 start for the best results. An overview rebuild should not be done
32 while the server is running. Unless the existing overview is deleted,
33 you may end up with problems like out-of-order overview entries,
34 excessively large overview buffers, and the like.
35 If ovmethod in inn.conf is "ovdb", you must have the ovdb processes
37 running while rebuilding overview. ovdb needs them available while
38 writing overview entries. You can start them by hand separate from the
39 rest of the server by running ovdb_init; see ovdb_init(8) for more
40 details.
41 Similarly, if ovmethod in inn.conf is "ovsqlite", you must have the
43 ovsqlite-server process running while rebuilding overview. See
44 ovsqlite-server(8) for more details and how to start it by hand.
45 Rebuilding overview data is as straight-forward as:
47 1. Checking that the configuration file of the new overview storage
49 method is present in pathetc and fits your needs (buffindexed.conf,
50 ovdb.conf or ovsqlite.conf). Note that the tradindexed overview
51 storage method does not have a dedicated configuration file.
52 2. Making sure that INN is stopped ("rc.news stop" as the news user,
54 or whichever command you're using).
55 3. Setting the new overview storage method in the ovmethod parameter
57 in inn.conf.
58 4. Making sure that the directory specified by the pathoverview
60 parameter in inn.conf exists and is empty (or contains freshly
61 created buffindexed buffers, if using that overview storage
62 method). Otherwise, rename the current directory (to backup
63 existing overview data) and re-create pathoverview as the news
64 user.
65 5. Starting ovdb_init or ovsqlite-server as the news user if the new
67 overview storage method is respectively ovdb or ovsqlite.
68 6. Running "makehistory -O -x -F" and waiting for the command to
70 finish. (You may notice a few logs about articles for which
71 overview data cannot be inserted into the new overview storage
72 method. As long as there aren't tons of them, it is normal,
73 notably because there is an internal limit in the length of
74 overview data generated by makehistory, contrary to innd.
75 Unfortunately, these rare articles won't be present in the new
76 overview.)
77 7. Stopping ovdb or ovsqlite helper programs if you started them
79 during the previous steps (running "rc.news stop" as the news user
80 will stop them; do not mind the messages related to the fact that
81 the news server was not running).
82 8. Starting INN and checking the logs to make sure everything is fine.
84 You will normally notice that the active file is renumbered
85 (rc.news takes care of that when run after an overview rebuild;
86 otherwise, manually run "ctlinnd renumber ''").
87
89 -a Append to the history file rather than generating a new one. If
90 you append to the main history file, make sure innd(8) is throttled
91 or not running, or you can corrupt the history.
92 -b Delete any messages found in the spool that do not have valid
94 Message-ID header fields in them.
95 -F Fork a separate process to flush overview data to disk rather than
97 doing it directly. The advantage of this is that it allows
98 makehistory to continue to collect more data from the spool while
99 the first batch of data is being written to the overview database.
100 The disadvantage is that up to twice as much temporary disk space
101 will be used for the generated overview data. This option only
102 makes sense in combination with -O. With buffindexed, the overchan
103 program is invoked to write overview.
104 -f filename
106 Rather than writing directly to pathdb/history, instead write to
107 filename, also in pathdb.
108 -I Don't store overview data for articles numbered lower than the
110 lowest article number in active. This is useful if there are for
111 whatever reason old articles on disk that shouldn't be available to
112 readers or put into the overview database.
113 -l count
115 This option specifies how many articles to process before writing
116 the accumulated overview information out to the overview database.
117 The default is 10000. Since overview write performance is faster
118 with sorted data, each "batch" gets sorted. Increasing the batch
119 size with this option may further improve write performance, at the
120 cost of longer sort times. Also, temporary space will be needed to
121 store the overview batches. At a rough estimate, about 300 * count
122 bytes of temporary space will be required (not counting temp files
123 created by sort(1)). See the description of the -T option for how
124 to specify the temporary storage location. This option has no
125 effect with buffindexed, because buffindexed does not need sorted
126 overview and no batching is done.
127 -L load-average
129 Temporarily pause activities if the system load average exceeds the
130 specified level load-average. This allows makehistory to run on a
131 system being used for other purposes without monopolizing system
132 resources and thus making the response time for other applications
133 unacceptably slow. Using nice(1) does not help much for that
134 because the problem comes from disk I/O usage, and ionice(1) is not
135 always available or efficient.
136 -O Create the overview database as well as the history file. Overview
138 information is only required if the server supports readers; it is
139 not needed for a transit-only server (see enableoverview in
140 inn.conf(5)). If you are using the buffindexed overview storage
141 method, erase all of your overview buffers before running
142 makehistory with -O.
143 -S Rather than storing the overview data into the overview database,
145 just write it to standard output in a form suitable for feeding to
146 overchan later if wished. When this option is used, -F, -I, -l,
147 and -T are ignored. This option only makes sense in combination
148 with -O.
149 -s size
151 Size the history database for approximately size pairs. Accurately
152 specifying the size is an optimization that will create a more
153 efficient database. (The size should be the estimated eventual
154 size of the history file, typically the size of the old file, in
155 lines.)
156 -T tmpdir
158 If -O is given, makehistory needs a location to write temporary
159 overview data. By default, it uses pathtmp, set in inn.conf, but
160 if this option is given, the provided tmpdir is used instead. This
161 is also used for temporary files created by sort(1) (which is
162 invoked in the process of writing overview information since sorted
163 overview information writes faster). By default, sort usually uses
164 your system temporary directory; see the sort(1) man page on your
165 system to be sure.
166 -x If this option is given, makehistory won't write out history file
168 entries. This is useful mostly for building overview without
169 generating a new history file.
170
172 Here's a typical example of rebuilding the entire history and overview
173 database, removing broken articles in the news spool. This uses the
174 default temporary file locations and should be done while innd isn't
175 running (or is throttled).
176 makehistory -b -f history.n -O -l 30000 -I
178 This will rebuild the overview (if using buffindexed, erase the
180 existing overview buffers before running this command) and leave a new
181 history file as "history.n" in pathdb. To preserve all of the history
182 entries from the old history file that correspond to rejected articles
183 or expired articles, follow the above command with:
184 cd <pathdb>
186 awk 'NF == 2 { print }' < history >> history.n
187 (replacing the path with your pathdb, if it isn't the default). Then
189 look over the new history file for problems and run:
190 makedbz -s `wc -l < history.n` -f history.n
192 Then rename all of the files matching "history.n.*" to "history.*",
194 replacing the current history database and indices. After that, it's
195 safe to unthrottle innd.
196 For a simpler example:
198 makehistory -b -f history.n -I -O
200 will scan the spool, removing broken articles and generating history
202 and overview entries for articles missing from history.
203 To just rebuild overview:
205 makehistory -O -x -F
207
209 pathdb/history
210 This is the default output file for makehistory.
211 pathtmp
213 Where temporary files are written unless -T is given.
214
216 Originally written by Rich $alz <rsalz@uunet.uu.net> for InterNetNews
217 and updated by various other people since.
218
220 active(5), ctlinnd(8), history(5), inn.conf(5), innd(8), libinn_dbz(3),
221 makedbz(8), ovdb_init(8), overchan(8), ovsqlite-server(8).
222INN 2.7.1 2023-03-07 MAKEHISTORY(8)