1OVDB(5) InterNetNews Documentation OVDB(5)
2
6 ovdb - Overview storage method for INN
7
9 Ovdb is a storage method that uses the BerkeleyDB library to store
10 overview data. It requires version 2.6.x or later of the BerkeleyDB
11 library, but has mostly been tested with version 3 and 4.
12 Ovdb makes use of the full transaction/logging/locking functionality of
14 the BerkeleyDB environment. BerkeleyDB may be downloaded from
15 <http://www.sleepycat.com> and is needed to build the ovdb backend.
16
18 This is version 2 of ovdb. If you have a database created with a pre‐
19 vious version of ovdb (such as the one shipped with INN 2.3.0) your
20 database will need to be upgraded using ovdb_init(8). See the man page
21 ovdb_init(8) for upgrade instructions.
22
24 To build ovdb support into INN, specify the option "--with-berkeleydb"
25 when running the configure script. By default, configure will search
26 for a BerkeleyDB tree in several likely locations, and choose the high‐
27 est version (based on the name of the directory, e.g., BerkeleyDB.3.0)
28 that it finds. There will be a message in the configure output indi‐
29 cating the chosen pathname.
30 You can override this pathname by adding a path to the option, e.g.,
32 "--with-berkeleydb=/usr/BerkeleyDB.3.1". This directory is expected to
33 have subdirectories include and lib, containing db.h, and the library
34 itself, respectively.
35 The ovdb database will take up more disk space for a given spool than
37 the other overview methods. Plan on needing at least 1.1 KB for every
38 article in your spool (not counting crossposts). So, if you have 5
39 million articles, you'll need at least 5.5 GB of disk space for ovdb.
40 With BerkeleyDB 2.x, the db files are 'grow only'; the library will not
41 shrink them, even if data is removed. So, reserving extra space above
42 the estimate is a good idea. Plus, you'll need additional space for
43 transaction logs: at least 100 MB. By default the transaction logs go
44 in the same directory as the database. To improve performance, they
45 can be placed on a different disk -- see the DB_CONFIG section.
46
48 To enable ovdb, set the ovmethod parameter in inn.conf to "ovdb". The
49 ovdb database is stored in the directory specified by the pathoverview
50 paramter in inn.conf. This is the "DB_HOME" directory. To start out,
51 this directory should be empty (other than an optional DB_CONFIG file;
52 see DB_CONFIG for details) and innd (or makehistory) will create the
53 files as necessary in that directory. Make sure the directory is owned
54 by the news user.
55 Other parameters for configuring ovdb are in the ovdb.conf(5) configu‐
57 ration file. See also the sample ovdb.conf.
58 cachesize
60 Size of the memory pool cache, in kilobytes. The cache will have a
61 backing store file in the DB directory which will be at least as
62 big. In general, the bigger the cache, the better. Use "ovdb_stat
63 -m" to see cache hit percentages. To make a change of this parame‐
64 ter take effect, shut down and restart INN (be sure to kill all of
65 the nnrpds when shutting down). Default is 8000, which is adequate
66 for small to medium sized servers. Large servers will probably
67 need at least 20000.
68 numdbfiles
70 Overview data is split between this many files. Currently, innd
71 will keep all of the files open, so don't set this too high or innd
72 may run out of file descriptors. nnrpd only opens one at a time,
73 regardless. May be set to one, or just a few, but only do that if
74 your OS supports large (>2G) files. Changing this parameter has no
75 effect on an already-established database. Default is 32.
76 txn_nosync
78 If txn_nosync is set to false, BerkeleyDB flushes the log after
79 every transaction. This minimizes the number of transactions that
80 may be lost in the event of a crash, but results in significantly
81 degraded performance. Default is true.
82 useshm
84 If useshm is set to true, BerkeleyDB will use shared memory instead
85 of mmap for its environment regions (cache, lock, etc). With some
86 platforms, this may improve performance. Default is false. This
87 parameter is ignored if you have BerkeleyDB 2.x
88 shmkey
90 Sets the shared memory key used by BerkeleyDB when 'useshm' is
91 true. BerkeleyDB will create several (usually 5) shared memory
92 segments, using sequentially numbered keys starting with 'shmkey'.
93 Choose a key that does not conflict with any existing shared memory
94 segments on your system. Default is 6400. This parameter is only
95 used with BerkeleyDB 3.1 or newer.
96 pagesize
98 Sets the page size for the DB files (in bytes). Must be a power of
99 2. Best choices are 4096 or 8192. The default is 8192. Changing
100 this parameter has no effect on an already-established database.
101 minkey
103 Sets the minimum number of keys per page. See the BerkeleyDB docu‐
104 mentation for more info. Default is based on page size:
105 default_minkey = MAX(2, pagesize / 2048 - 1)
107 The lowest allowed minkey is 2. Setting minkey higher than the
109 default is not recommended, as it will cause the databases to have
110 a lot of overflow pages. Changing this parameter has no effect on
111 an already-established database.
112 maxlocks
114 Sets the BerkeleyDB "lk_max" parameter, which is the maxmium number
115 of locks that can exist in the database at the same time. Default
116 is 4000.
117 nocompact
119 The nocompact parameter affects expireover's behavior. The expire‐
120 over function in ovdb can do its job in one of two ways: by simply
121 deleting expired records from the database, or by re-writing the
122 overview records into a different location leaving out the expired
123 records. The first method is faster, but it leaves 'holes' that
124 result in space that can not immediately be reused. The second
125 method 'compacts' the records by rewriting them.
126 If this parameter is set to 0, expireover will compact all news‐
128 groups; if set to 1, expireover will not compact any newsgroups;
129 and if set to a value greater than one, expireover will only com‐
130 pact groups that have less than that number of articles.
131 Experience has shown that compacting has minimal effect (other than
133 making expireover take longer) so the default is now 1. This
134 parameter will probably be removed in the future.
135 readserver
137 Normally, each nnrpd process directly accesses the BerkeleyDB envi‐
138 ronment. The process of attaching to the database (and detaching
139 when finished) is fairly expensive, and can result in high loads in
140 situations when there are lots of reader connections of relatively
141 short duration.
142 When the readserver parameter is "true", the nnrpds will access
144 overview via a helper server (ovdb_server -- which is started by
145 ovdb_init). This can also result in cleaner shutdowns for the
146 database, improving stability and avoiding deadlocks and corrupted
147 databases. If you are experiencing any instability in ovdb, try
148 setting this parameter to true. Default is false.
149 numrsprocs
151 This parameter is only used when readserver is true. It sets the
152 number of ovdb_server processes. As each ovdb_server can process
153 only one transaction at a time, running more servers can improve
154 reader response times. Default is 5.
155 maxrsconn
157 This parameter is only used when readserver is true. It sets a
158 maximum number of readers that a given ovdb_server process will
159 serve at one time. This means the maximum number of readers for
160 all of the ovdb_server processes is (numrsprocs * maxrsconn).
161 Default is 0, which means an umlimited number of connections is
162 allowed.
163
165 A file called DB_CONFIG may be placed in the database directory to cus‐
166 tomize where the various database files and transaction logs are writ‐
167 ten. By default, all of the files are written in the "DB_HOME" direc‐
168 tory. One way to improve performance is to put the transaction logs on
169 a different disk. To do this, put:
170 DB_LOG_DIR /path/to/logs
172 in the DB_CONFIG file. If the pathname you give starts with a /, it is
174 treated as an absolute path; otherwise, it is relative to the "DB_HOME"
175 directory. Make sure that any directories you specify exist and have
176 proper ownership/mode before starting INN, because they won't be cre‐
177 ated automatically. Also, don't change the DB_CONFIG file while any‐
178 thing that uses ovdb is running.
179 Another thing that you can do with this file is to split the overview
181 database across multiple disks. In the DB_CONFIG file, you can list
182 directories that BerkeleyDB will search when it goes to open a data‐
183 base.
184 For example, let's say that you have pathoverview set to /mnt/overview
186 and you have four additional file systems created on /mnt/ov?. You
187 would create a file "/mnt/overview/DB_CONFIG" containing the following
188 lines:
189 set_data_dir /mnt/overview
191 set_data_dir /mnt/ov1
192 set_data_dir /mnt/ov2
193 set_data_dir /mnt/ov3
194 set_data_dir /mnt/ov4
195 (For BerkeleyDB 2.x, replace "set_data_dir" with "DB_DATA_DIR".)
197 Distribute your ovNNNNN files into the four filesystems. (say, 8
199 each). When called upon to open a database file, the db library will
200 look for it in each of the specified directories (in order). If said
201 file is not found, one will be created in the first of those directo‐
202 ries.
203 Whenever you change DB_CONFIG or move database files around, make sure
205 all news processes that use the database are shut down first (including
206 nnrpds).
207 The DB_CONFIG functionality is part of BerkeleyDB itself, rather than
209 something provided by ovdb. See the BerkeleyDB documentation for com‐
210 plete details for the version of BerkeleyDB that you're running.
211
213 When starting the news system, rc.news will invoke ovdb_init.
214 ovdb_init must be run before using the database. It performs the fol‐
215 lowing tasks:
216 · Creates the database environment, if necessary.
218 · If the database is idle, it performs a normal recovery. The recov‐
220 ery will remove stale locks, recreate the memory pool cache, and
221 repair any damage caused by a system crash or improper shutdown.
222 · Starts the DB housekeeping processes (ovdb_monitor) if they're not
224 already running.
225 And when stopping INN, rc.news kills the ovdb_monitor processes after
227 the other INN processes have been shut down.
228
230 Problems relating to ovdb are logged to news.err with "OVDB" in the
231 error message.
232 INN programs that use overview will fail to start up if the ovdb_moni‐
234 tor processes aren't running. Be sure to run ovdb_init before running
235 anything that accesses overview.
236 Also, INN programs that use overview will fail to start up if the user
238 running them is not the "news" user.
239 If a program accessing the database crashes, or otherwise exits
241 uncleanly, it might leave a stale lock in the database. This lock
242 could cause other processes to deadlock on that stale lock. To fix
243 this, shut down all news processes (using "kill -9" if necessary) and
244 then restart. ovdb_init should perform a recovery operation which will
245 remove the locks and repair damage caused by killing the deadlocked
246 processes.
247
249 inn.conf
250 The ovmethod and pathoverview parameters are relevant to ovdb.
251 ovdb.conf
253 Optional configuration file for tuning. See CONFIGURATION above.
254 pathoverview
256 Directory where the database goes. BerkeleyDB calls it the
257 'DB_HOME' directory.
258 pathoverview/DB_CONFIG
260 Optional file to configure the layout of the database files.
261 pathrun/ovdb.sem
263 A file that gets locked by every process that is accessing the
264 database. This is used by ovdb_init to determine whether the data‐
265 base is active or quiescent.
266 pathrun/ovdb_monitor.pid
268 Contains the process ID of ovdb_monitor.
269
271 Implement a way to limit how many databases can be open at once (to
272 reduce file descriptor usage); maybe using something similar to the
273 cache code in ov3.c
274
276 Written by Heath Kehoe <hakehoe@avalon.net> for InterNetNews
277
279 inn.conf(5), innd(8), nnrpd(8), ovdb_init(8), ovdb_monitor(8),
280 ovdb_stat(8)
281 BerkeleyDB documentation: in the docs directory of the BerkeleyDB
283 source distribution, or on the Sleepycat web page: <http://www.sleepy‐
284 cat.com/>.
285INN 2.4.2 2004-06-08 OVDB(5)