1OVDB(5)                   InterNetNews Documentation                   OVDB(5)
2
3
4

NAME

6       ovdb - Overview storage method for INN
7

DESCRIPTION

9       The ovdb overview is a storage method that uses the Berkeley DB library
10       to store overview data.  It requires version 4.4 or later of the
11       Berkeley DB library (4.7+ is recommended because older versions suffer
12       from various issues).
13
14       The ovdb overview method makes use of the full
15       transaction/logging/locking functionality of the Berkeley DB
16       environment.  Berkeley DB may be downloaded from
17       <http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/overview/index.html>
18       and is needed to build the ovdb backend.
19

UPGRADING

21       There are several versions of the ovdb storage method:
22
23       • Version 1, the initial version shipped with INN 2.3.0 up to
24         INN 2.3.5.
25
26       • Version 2, with improved performance, since INN 2.4.0.
27
28       • Version 3, corresponding to version 2 with compression enabled,
29         starting with INN 2.5.0.
30
31       If you have a database created with a previous version of ovdb, your
32       database will need to be upgraded using ovdb_init.  See the
33       ovdb_init(8) man page for upgrade instructions, as well as the
34       COMPRESSION section below.
35
36       Note that when the Berkeley DB library is updated to a newer version,
37       the ovdb database also needs being upgraded.
38

INSTALLATION

40       If the Berkeley DB library is found at configure time, INN will be
41       built with Berkeley DB support unless the --without-bdb flag is
42       explicitly passed to configure.  By default, configure will search for
43       Berkeley DB in standard locations; there will be a message in the
44       configure output indicating the pathname that will be used.
45
46       You can override this pathname by adding a path to the option, for
47       instance --with-bdb=/usr/BerkeleyDB.4.4.  This directory is expected to
48       have subdirectories include and lib (lib32 and lib64 are also checked),
49       containing respectively db.h, and the library itself.  In case non-
50       standard paths to the Berkeley DB libraries are used, one or both of
51       the options --with-bdb-include and --with-bdb-lib can be given to
52       configure with a path.
53
54       The ovdb database may take up more disk space for a given spool than
55       the other overview methods.  Plan on needing at least 1.1 KB for every
56       article in your spool (not counting crossposts).  So, if you have 5
57       million articles, you'll need at least 5.5 GB of disk space for ovdb.
58       With compression enabled, this estimate changes to 0.9 KB per article,
59       so you'll need at least 4.5 GB of disk space for 5 million articles.
60       See the COMPRESSION section below.  Plus, you'll need additional space
61       for transaction logs: at least 100 MB.  By default, the transaction
62       logs go in the same directory as the database.  To improve performance,
63       they can be placed on a different disk -- see the DB_CONFIG section.
64

CONFIGURATION

66       To enable the ovdb overview method, set the ovmethod parameter in
67       inn.conf to "ovdb".  The ovdb database is stored in the directory
68       specified by the pathoverview parameter in inn.conf.  This is the
69       "DB_HOME" directory.  To start out, this directory should be empty
70       (other than an optional DB_CONFIG file; see DB_CONFIG for details), and
71       innd (or makehistory) will create the files as necessary in that
72       directory.  Also, make sure the directory is owned by the news user.
73
74       Other parameters for configuring ovdb are in the ovdb.conf
75       configuration file.  The following parameters can be set in that file:
76
77       compress
78           If INN was compiled with zlib, and this compress parameter is true,
79           ovdb will compress overview records that are longer than 600 bytes.
80           See the COMPRESSION section below.
81
82       cachesize
83           Size of the memory pool cache, in kilobytes.  The cache will have a
84           backing store file in the DB directory which will be at least as
85           big.  In general, the bigger the cache, the better.  Use "ovdb_stat
86           -m" to see cache hit percentages.  To make a change of this
87           parameter take effect, shut down and restart INN (be sure to kill
88           all of the nnrpd processes when shutting down).  Default is 8000
89           (KB), which is adequate for small to medium-sized servers.  Large
90           servers will probably need at least 20000 (KB).
91
92       ncache
93           Number of regions across which to split the cache.  The region size
94           is equal to cachesize divided by ncache.  Default is 1 for ncache,
95           that is to say the cache will be allocated contiguously in memory.
96
97       numdbfiles
98           Overview data is split between this many files.  Currently, innd
99           will keep all of the files open, so don't set this too high or innd
100           may run out of file descriptors.  nnrpd only opens one at a time,
101           regardless.  May be set to one, or just a few, but only do that if
102           your OS supports large (> 2 GB) files.  Changing this parameter has
103           no effect on an already-established database.  Default is 32.
104
105       txn_nosync
106           If txn_nosync is set to false, Berkeley DB flushes the log after
107           every transaction.  This minimizes the number of transactions that
108           may be lost in the event of a crash, but results in significantly
109           degraded performance.  Default is true.
110
111       useshm
112           If useshm is set to true, Berkeley DB will use shared memory
113           instead of mmap for its environment regions (cache, lock, etc).
114           With some platforms, this may improve performance.  Default is
115           false.
116
117       shmkey
118           Sets the shared memory key used by Berkeley DB when useshm is true.
119           Berkeley DB will create several (usually 5) shared memory segments,
120           using sequentially numbered keys starting with "shmkey".  Choose a
121           key that does not conflict with any existing shared memory segments
122           on your system.  Default is 6400.
123
124       pagesize
125           Sets the page size for the DB files (in bytes).  Must be a power of
126           2.  Best choices are 4096 or 8192.  The default is 8192.  Changing
127           this parameter has no effect on an already-established database.
128
129       minkey
130           Sets the minimum number of keys per page.  See the Berkeley DB
131           documentation for more information.  Default is based on page size
132           and whether compression is enabled:
133
134              default_minkey = MAX(2, pagesize / 2600) if compress is false
135              default_minkey = MAX(2, pagesize / 1500) if compress is true
136
137           The lowest allowed minkey is 2.  Setting minkey higher than the
138           default is not recommended, as it will cause the databases to have
139           a lot of overflow pages.  Changing this parameter has no effect on
140           an already-established database.
141
142       maxlocks
143           Sets the Berkeley DB lk_max parameter, which is the maximum number
144           of locks that can exist in the database at the same time.  Default
145           is 4000.
146
147       nocompact
148           The nocompact parameter affects the behaviour of expireover.  The
149           expireover function in ovdb can do its job in one of two ways:  by
150           simply deleting expired records from the database; or by re-writing
151           the overview records into a different location leaving out the
152           expired records.  The first method is faster, but it leaves 'holes'
153           that result in space that can not immediately be reused.  The
154           second method 'compacts' the records by rewriting them.
155
156           If this parameter is set to 0, expireover will compact all
157           newsgroups; if set to 1, expireover will not compact any
158           newsgroups; and if set to a value greater than one, expireover will
159           only compact groups that have less than that number of articles.
160
161           Experience has shown that compacting has minimal effect (other than
162           making expireover take longer) so the default is 1.  This parameter
163           will probably be removed in the future.
164
165       readserver
166           When the readserver parameter is set to false, each nnrpd process
167           directly accesses the Berkeley DB environment.  The process of
168           attaching to the database (and detaching when finished) is fairly
169           expensive, and can result in high loads in situations when there
170           are lots of reader connections of relatively short duration.
171
172           When the readserver parameter is set to true, the nnrpd processes
173           will access overview via a helper server (ovdb_server -- which is
174           started by ovdb_init).  All ovdb reads will then be funnelled
175           through a single process with a cleaner interface to the underlying
176           Berkeley DB database.  This will result in cleaner shutdowns for
177           the database, improving stability and avoiding deadlocks, timing
178           issues and corrupted databases.  That's why you should try to set
179           this parameter to true if you are experiencing any instability in
180           the ovdb overview method.
181
182           Default value is true.
183
184       numrsprocs
185           This parameter is only used when readserver is true.  It sets the
186           number of ovdb_server processes.  As each ovdb_server can process
187           only one transaction at a time, running more servers can improve
188           reader response times.  Default is 5.
189
190       maxrsconn
191           This parameter is only used when readserver is true.  It sets a
192           maximum number of readers that a given ovdb_server process will
193           serve at one time.  This means the maximum number of readers for
194           all of the ovdb_server processes is (numrsprocs * maxrsconn).  This
195           does not limit the actual number of readers, since nnrpd will fall
196           back to opening the database directly if it can't connect to an
197           ovdb_server.  Default is 0, which means an unlimited number of
198           connections is allowed.
199

COMPRESSION

201       The ovdb storage method has the ability to compress overview data
202       before it is stored into the database.  In addition to consuming less
203       disk space, compression keeps the average size of the database keys
204       smaller.  This in turn increases the average number of keys per page,
205       which can significantly improve performance and also helps keep the
206       database more compact.  This feature requires that INN be built with
207       zlib.  Only records larger than 600 bytes get compressed, because that
208       is the point at which compression starts to become significant.
209
210       If compression is not enabled (either from the compress option in
211       ovdb.conf or INN was not built with zlib support), the database will be
212       backward compatible with older versions of ovdb.  However, if
213       compression is enabled, the database is marked with a newer version
214       that will prevent older versions of ovdb from opening the database.
215
216       You can upgrade an existing database to use compression simply by
217       setting compress to true in ovdb.conf.  Note that existing records in
218       the database will remain uncompressed; only new records added after
219       enabling compression will be compressed.
220
221       If you disable compression on a database that previously had it
222       enabled, new records will be stored uncompressed, but the database will
223       still be incompatible with older versions of ovdb (and will also be
224       incompatible with this version of ovdb if INN was not built with zlib
225       support).  So to downgrade to a completely uncompressed database, you
226       will have to rebuild the database using makehistory.
227

DB_CONFIG

229       A file called DB_CONFIG may be placed in the database directory
230       (pathoverview in inn.conf) to customize where the various database
231       files and transaction logs are written.  By default, all of the files
232       are written in the "DB_HOME" directory.  One way to improve performance
233       is to put the transaction logs on a different disk.  To do this, put:
234
235           DB_LOG_DIR /path/to/logs
236
237       in the DB_CONFIG file.  If the pathname you give starts with a "/", it
238       is treated as an absolute path; otherwise, it is relative to the
239       "DB_HOME" directory.  Make sure that any directories you specify exist
240       and have proper ownership/mode before starting INN, because they won't
241       be created automatically.  Also, don't change the DB_CONFIG file while
242       anything that uses ovdb is running.
243
244       Another thing that you can do with this file is to split the overview
245       database across multiple disks.  In the DB_CONFIG file, you can list
246       directories that Berkeley DB will search when it goes to open a
247       database.
248
249       For example, let's say that you have pathoverview set to /mnt/overview
250       and you have four additional file systems created on /mnt/ovX.  You
251       would create a file /mnt/overview/DB_CONFIG containing the following
252       lines:
253
254           set_data_dir /mnt/overview
255           set_data_dir /mnt/ov1
256           set_data_dir /mnt/ov2
257           set_data_dir /mnt/ov3
258           set_data_dir /mnt/ov4
259
260       Distribute your ovNNNNN files into the four filesystems (say, 8 each).
261       When called upon to open a database file, the db library will look for
262       it in each of the specified directories (in order).  If said file is
263       not found, one will be created in the first of those directories.
264
265       Whenever you change DB_CONFIG or move database files around, make sure
266       all news processes that use the database are shut down first (including
267       nnrpd processes).
268
269       The DB_CONFIG functionality is part of Berkeley DB itself, rather than
270       something provided by ovdb.  See the Berkeley DB documentation for
271       complete details for the version of Berkeley DB that you're running.
272

RUNNING

274       When starting the news system, rc.news will invoke the ovdb_init
275       program.  See the ovdb_init(8) man page for information about the tasks
276       it performs.  ovdb_init must be run before using the database.
277
278       And when stopping INN, rc.news kills the ovdb_monitor processes after
279       the other INN processes have been shut down.
280

DIAGNOSTICS

282       Problems relating to ovdb are logged to news.err with "OVDB" in the
283       error message.
284
285       INN programs that use overview will fail to start up if the
286       ovdb_monitor processes aren't running.  Be sure to run ovdb_init before
287       running anything that accesses overview.
288
289       Also, INN programs that use overview will fail to start up if the user
290       running them is not the news user.
291
292       If a program accessing the database crashes, or otherwise exits
293       uncleanly, it might leave a stale lock in the database.  This lock
294       could cause other processes to deadlock on that stale lock.  To fix
295       this, shut down all news processes (using "kill -9" if necessary) and
296       then restart.  ovdb_init should perform a recovery operation which will
297       remove the locks and repair damage caused by killing the deadlocked
298       processes.
299

FILES

301       pathetc/inn.conf
302           The ovmethod and pathoverview parameters are relevant to ovdb.
303
304       pathetc/ovdb.conf
305           Optional configuration file for tuning.  See CONFIGURATION above.
306
307       pathoverview
308           Directory where the database goes.  Berkeley DB calls it the
309           "DB_HOME" directory.
310
311       pathoverview/DB_CONFIG
312           Optional file to configure the layout of the database files.
313
314       pathrun/ovdb.sem
315           A file that gets locked by every process that is accessing the
316           database.  This is used by ovdb_init to determine whether the
317           database is active or quiescent.
318
319       pathrun/ovdb_monitor.pid
320           Contains the process ID of ovdb_monitor.
321

TO DO

323       Implement a way to limit how many databases can be open at once (to
324       reduce file descriptor usage); maybe using something similar to the
325       cache code in legacy ov3.c file.
326

HISTORY

328       Written by Heath Kehoe <hakehoe@avalon.net> for InterNetNews.
329
330       $Id: ovdb.pod 10525 2021-01-20 11:51:15Z iulius $
331

SEE ALSO

333       inn.conf(5), innd(8), makehistory(8), nnrpd(8), ovdb_init(8),
334       ovdb_monitor(8), ovdb_stat(8).
335
336       Berkeley DB documentation:  in the docs directory of the Berkeley DB
337       source distribution, or on the Oracle Berkeley DB web page
338       (<http://www.oracle.com/technetwork/database/database-technologies/berkeleydb/overview/index.html>).
339
340
341
342INN 2.6.4                         2021-01-21                           OVDB(5)
Impressum