1OVDB(5) InterNetNews Documentation OVDB(5)
2
6 ovdb - Overview storage method for INN
7
9 Ovdb is a storage method that uses the Berkeley DB library to store
10 overview data. It requires version 4.4 or later of the Berkeley DB
11 library (4.7+ is recommended because older versions suffer from various
12 issues).
13 Ovdb makes use of the full transaction/logging/locking functionality of
15 the Berkeley DB environment. Berkeley DB may be downloaded from
16 <http://www.sleepycat.com> and is needed to build the ovdb backend.
17
19 This is version 2 of ovdb. If you have a database created with a
20 previous version of ovdb (such as the one shipped with INN 2.3.0) your
21 database will need to be upgraded using ovdb_init(8). See the man page
22 ovdb_init(8) for upgrade instructions.
23
25 To build ovdb support into INN, specify the option --with-berkeleydb
26 when running the configure script. By default, configure will search
27 for a Berkeley DB tree in several likely locations, and choose the
28 highest version (based on the name of the directory, e.g.,
29 BerkeleyDB.4.4) that it finds. There will be a message in the
30 configure output indicating the chosen pathname.
31 You can override this pathname by adding a path to the option, e.g.,
33 --with-berkeleydb=/usr/BerkeleyDB.4.4. This directory is expected to
34 have subdirectories include and lib, containing db.h, and the library
35 itself, respectively.
36 The ovdb database may take up more disk space for a given spool than
38 the other overview methods. Plan on needing at least 1.1 KB for every
39 article in your spool (not counting crossposts). So, if you have 5
40 million articles, you'll need at least 5.5 GB of disk space for ovdb.
41 With compression enabled, this estimate changes to 0.7 KB per article.
42 See the COMPRESSION section below. Plus, you'll need additional space
43 for transaction logs: at least 100 MB. By default the transaction logs
44 go in the same directory as the database. To improve performance, they
45 can be placed on a different disk -- see the DB_CONFIG section.
46
48 To enable ovdb, set the ovmethod parameter in inn.conf to "ovdb". The
49 ovdb database is stored in the directory specified by the pathoverview
50 paramter in inn.conf. This is the "DB_HOME" directory. To start out,
51 this directory should be empty (other than an optional DB_CONFIG file;
52 see DB_CONFIG for details) and innd (or makehistory) will create the
53 files as necessary in that directory. Make sure the directory is owned
54 by the news user.
55 Other parameters for configuring ovdb are in the ovdb.conf(5)
57 configuration file. See also the sample ovdb.conf.
58 cachesize
60 Size of the memory pool cache, in kilobytes. The cache will have a
61 backing store file in the DB directory which will be at least as
62 big. In general, the bigger the cache, the better. Use "ovdb_stat
63 -m" to see cache hit percentages. To make a change of this
64 parameter take effect, shut down and restart INN (be sure to kill
65 all of the nnrpds when shutting down). Default is 8000, which is
66 adequate for small to medium sized servers. Large servers will
67 probably need at least 20000.
68 compress
70 If INN was compiled with zlib, and this compress parameter is true,
71 OVDB will compress overview records that are longer than 600 bytes.
72 See the COMPRESSION section below.
73 numdbfiles
75 Overview data is split between this many files. Currently, innd
76 will keep all of the files open, so don't set this too high or innd
77 may run out of file descriptors. nnrpd only opens one at a time,
78 regardless. May be set to one, or just a few, but only do that if
79 your OS supports large (>2G) files. Changing this parameter has no
80 effect on an already-established database. Default is 32.
81 txn_nosync
83 If txn_nosync is set to false, Berkeley DB flushes the log after
84 every transaction. This minimizes the number of transactions that
85 may be lost in the event of a crash, but results in significantly
86 degraded performance. Default is true.
87 useshm
89 If useshm is set to true, Berkeley DB will use shared memory
90 instead of mmap for its environment regions (cache, lock, etc).
91 With some platforms, this may improve performance. Default is
92 false.
93 shmkey
95 Sets the shared memory key used by Berkeley DB when 'useshm' is
96 true. Berkeley DB will create several (usually 5) shared memory
97 segments, using sequentially numbered keys starting with 'shmkey'.
98 Choose a key that does not conflict with any existing shared memory
99 segments on your system. Default is 6400.
100 pagesize
102 Sets the page size for the DB files (in bytes). Must be a power of
103 2. Best choices are 4096 or 8192. The default is 8192. Changing
104 this parameter has no effect on an already-established database.
105 minkey
107 Sets the minimum number of keys per page. See the Berkeley DB
108 documentation for more info. Default is based on page size and
109 whether compression is enabled:
110 default_minkey = MAX(2, pagesize / 2600) if compress is false
112 default_minkey = MAX(2, pagesize / 1500) if compress is true
113 The lowest allowed minkey is 2. Setting minkey higher than the
115 default is not recommended, as it will cause the databases to have
116 a lot of overflow pages. Changing this parameter has no effect on
117 an already-established database.
118 maxlocks
120 Sets the Berkeley DB "lk_max" parameter, which is the maximum
121 number of locks that can exist in the database at the same time.
122 Default is 4000.
123 nocompact
125 The nocompact parameter affects expireover's behavior. The
126 expireover function in ovdb can do its job in one of two ways: by
127 simply deleting expired records from the database, or by re-writing
128 the overview records into a different location leaving out the
129 expired records. The first method is faster, but it leaves 'holes'
130 that result in space that can not immediately be reused. The
131 second method 'compacts' the records by rewriting them.
132 If this parameter is set to 0, expireover will compact all
134 newsgroups; if set to 1, expireover will not compact any
135 newsgroups; and if set to a value greater than one, expireover will
136 only compact groups that have less than that number of articles.
137 Experience has shown that compacting has minimal effect (other than
139 making expireover take longer) so the default is now 1. This
140 parameter will probably be removed in the future.
141 readserver
143 Normally, each nnrpd process directly accesses the Berkeley DB
144 environment. The process of attaching to the database (and
145 detaching when finished) is fairly expensive, and can result in
146 high loads in situations when there are lots of reader connections
147 of relatively short duration.
148 When the readserver parameter is true, the nnrpds will access
150 overview via a helper server (ovdb_server -- which is started by
151 ovdb_init). This can also result in cleaner shutdowns for the
152 database, improving stability and avoiding deadlocks and corrupted
153 databases. If you are experiencing any instability in ovdb, try
154 setting this parameter to true. Default is false.
155 numrsprocs
157 This parameter is only used when readserver is true. It sets the
158 number of ovdb_server processes. As each ovdb_server can process
159 only one transaction at a time, running more servers can improve
160 reader response times. Default is 5.
161 maxrsconn
163 This parameter is only used when readserver is true. It sets a
164 maximum number of readers that a given ovdb_server process will
165 serve at one time. This means the maximum number of readers for
166 all of the ovdb_server processes is (numrsprocs * maxrsconn). This
167 does not limit the actual number of readers, since nnrpd will fall
168 back to opening the database directly if it can't connect to a
169 readserver. Default is 0, which means an umlimited number of
170 connections is allowed.
171
173 New in this version of OVDB is the ability to compress overview data
174 before it is stored into the database. In addition to consuming less
175 disk space, compression keeps the average size of the database keys
176 smaller. This in turn increases the average number of keys per page,
177 which can significantly improve performance and also helps keep the
178 database more compact. This feature requires that INN be built with
179 zlib. Only records larger than 600 bytes get compressed, because that
180 is the point at which compression starts to become significant.
181 If compression is not enabled (either from the "compress" option in
183 ovdb.conf or INN was not built from zlib), the database will be
184 backward compatible with older versions of OVDB. However, if
185 compression is enabled, the database is marked with a newer version
186 that will prevent older versions of OVDB from opening the database.
187 You can upgrade an existing database to use compression simply by
189 setting compress to true in ovdb.conf. Note that existing records in
190 the database will remain uncompressed; only new records added after
191 enabling compression will be compressed.
192 If you disable compression on a database that previously had it
194 enabled, new records will be stored uncompressed, but the database will
195 still be incompatible with older versions of OVDB (and will also be
196 incompatible with this version of OVDB if it was not built with zlib).
197 So to downgrade to a completely uncompressed database you will have to
198 rebuild the database using makehistory.
199
201 A file called DB_CONFIG may be placed in the database directory to
202 customize where the various database files and transaction logs are
203 written. By default, all of the files are written in the "DB_HOME"
204 directory. One way to improve performance is to put the transaction
205 logs on a different disk. To do this, put:
206 DB_LOG_DIR /path/to/logs
208 in the DB_CONFIG file. If the pathname you give starts with a /, it is
210 treated as an absolute path; otherwise, it is relative to the "DB_HOME"
211 directory. Make sure that any directories you specify exist and have
212 proper ownership/mode before starting INN, because they won't be
213 created automatically. Also, don't change the DB_CONFIG file while
214 anything that uses ovdb is running.
215 Another thing that you can do with this file is to split the overview
217 database across multiple disks. In the DB_CONFIG file, you can list
218 directories that Berkeley DB will search when it goes to open a
219 database.
220 For example, let's say that you have pathoverview set to /mnt/overview
222 and you have four additional file systems created on /mnt/ov?. You
223 would create a file "/mnt/overview/DB_CONFIG" containing the following
224 lines:
225 set_data_dir /mnt/overview
227 set_data_dir /mnt/ov1
228 set_data_dir /mnt/ov2
229 set_data_dir /mnt/ov3
230 set_data_dir /mnt/ov4
231 Distribute your ovNNNNN files into the four filesystems. (say, 8
233 each). When called upon to open a database file, the db library will
234 look for it in each of the specified directories (in order). If said
235 file is not found, one will be created in the first of those
236 directories.
237 Whenever you change DB_CONFIG or move database files around, make sure
239 all news processes that use the database are shut down first (including
240 nnrpds).
241 The DB_CONFIG functionality is part of Berkeley DB itself, rather than
243 something provided by ovdb. See the Berkeley DB documentation for
244 complete details for the version of Berkeley DB that you're running.
245
247 When starting the news system, rc.news will invoke ovdb_init.
248 ovdb_init must be run before using the database. It performs the
249 following tasks:
250 · Creates the database environment, if necessary.
252 · If the database is idle, it performs a normal recovery. The
254 recovery will remove stale locks, recreate the memory pool cache,
255 and repair any damage caused by a system crash or improper
256 shutdown.
257 · Starts the DB housekeeping processes (ovdb_monitor) if they're not
259 already running.
260 And when stopping INN, rc.news kills the ovdb_monitor processes after
262 the other INN processes have been shut down.
263
265 Problems relating to ovdb are logged to news.err with "OVDB" in the
266 error message.
267 INN programs that use overview will fail to start up if the
269 ovdb_monitor processes aren't running. Be sure to run ovdb_init before
270 running anything that accesses overview.
271 Also, INN programs that use overview will fail to start up if the user
273 running them is not the "news" user.
274 If a program accessing the database crashes, or otherwise exits
276 uncleanly, it might leave a stale lock in the database. This lock
277 could cause other processes to deadlock on that stale lock. To fix
278 this, shut down all news processes (using "kill -9" if necessary) and
279 then restart. ovdb_init should perform a recovery operation which will
280 remove the locks and repair damage caused by killing the deadlocked
281 processes.
282
284 inn.conf
285 The ovmethod and pathoverview parameters are relevant to ovdb.
286 ovdb.conf
288 Optional configuration file for tuning. See CONFIGURATION above.
289 pathoverview
291 Directory where the database goes. Berkeley DB calls it the
292 'DB_HOME' directory.
293 pathoverview/DB_CONFIG
295 Optional file to configure the layout of the database files.
296 pathrun/ovdb.sem
298 A file that gets locked by every process that is accessing the
299 database. This is used by ovdb_init to determine whether the
300 database is active or quiescent.
301 pathrun/ovdb_monitor.pid
303 Contains the process ID of ovdb_monitor.
304
306 Implement a way to limit how many databases can be open at once (to
307 reduce file descriptor usage); maybe using something similar to the
308 cache code in ov3.c
309
311 Written by Heath Kehoe <hakehoe@avalon.net> for InterNetNews
312 $Id: ovdb.pod 8334 2009-02-20 20:18:37Z iulius $
314
316 inn.conf(5), innd(8), nnrpd(8), ovdb_init(8), ovdb_monitor(8),
317 ovdb_stat(8)
318 Berkeley DB documentation: in the docs directory of the Berkeley DB
320 source distribution, or on the Sleepycat web page:
321 <http://www.sleepycat.com/>.
322INN 2.5.2 2010-08-11 OVDB(5)