1BUFFINDEXED.CONF(5) InterNetNews Documentation BUFFINDEXED.CONF(5)
2
6 buffindexed.conf - Configuration for the buffindexed overview method
7
9 buffindexed.conf, found in pathetc, specifies the buffers that the
10 buffindexed overview method should use. It is required if the server
11 uses buffindexed (as configured by the ovmethod parameter in inn.conf).
12 Buffindexed uses pre-built buffer files to store overview data and
14 indexes to that data. The buffers are divided into 8 KB internally,
15 and a given block is used either for overview data or for index data.
16 A block is always allocated to a single newsgroup and is never shared
17 among newsgroups. It also means that overview data is limited to 8 KB
18 per article, which may lead to the lack of integration of a few
19 articles with headers of unusual length into the overview database.
20 In addition to the buffers, buffindexed also stores information in a
22 file named group.index in pathdb. (This file should not be mistaken
23 for the one named group.index in pathoverview which is used by the
24 tradindexed overview method.) It contains information about each
25 newsgroup: the pointer to the index block for the newsgroup, the high
26 mark, the low mark, the flag of the group, the number of articles, and
27 so forth. This file is created automatically when all buffers are
28 initialized and should not be manually edited.
29 Buffindexed buffers are of fixed size, so buffindexed will never use
31 more space than what is available in those buffers. If all buffers are
32 full, innd will throttle when it attempts to store overview information
33 for any additional articles until space is freed (with expireover, for
34 instance) or another buffer is added. This is unlike the CNFS storage
35 method.
36 You can see the current usage of the buffers with the -o option to
38 inndf.
39 In the buffindexed.conf file, blank lines and lines beginning with a
41 number sign ("#") are ignored. All other lines must be of the format:
42 <index>:<filename>:<size>
44 The order of lines is not significant.
46 <index> is the index of this overview buffer and must be unique. Other
48 than that constraint, it can be any number between 0 and 65535.
49 <filename> is the path to the buffer. The length of the path should
51 not be longer than 63 characters.
52 <size> is the length of the buffer in kilobytes (1 KB = 1024 bytes).
54 If <filename> does not specify a special device, the file size of the
55 buffer must be <size> * 1024 bytes. If it does specify a special
56 device, that device must have at least <size> space available. For
57 more information on setting up the buffers, see "CREATING BUFFERS".
58 An example of buffindexed.conf file can be:
60 0:<pathoverview in inn.conf>/OV1:1536000
62 1:<pathoverview in inn.conf>/OV2:1536000
63 When you first start innd with everything configured properly, you
65 should see messages like this in pathlog/news.notice:
66 Aug 27 00:00:00 kevlar innd: buffindexed: no magic cookie found
68 for ovbuff 0, initializing
69 You MUST recreate overview completely using makehistory if you remove
71 or replace buffers. However, new buffers can be added without any
72 special care (other than restarting innd after modifying
73 buffindexed.conf). If you need to rebuild overview, you should zero
74 all of the buffers first.
75 We recommend not to reserve too much spare space in existing buffers,
77 so that to minimize the duration of the expireover process, and to just
78 add new buffers when space left is low (see the result of "inndf -no").
79 Plan on needing at least 0.65 KB for every article in your spool (not
80 counting crossposts). So, if you have 5 million articles, you'll need
81 at least 3.25 GB of disk space for buffindexed.
82
84 There are two methods to create a new buffindexed buffer:
85 1. Create a large file on top of a regular file system. The easiest
87 way to do this is probably with dd(1), using a command like:
88 dd if=/dev/zero of=/path/to/cycbuff bs=1024 count=<size>
90 where <size> is the size from the relevant line in
92 buffindexed.conf.
93 This is the simplest method, but has the disadvantage that very
95 large files on regular file systems can be fairly slow to access,
96 particularly at the end of the file, and INN incurs unnecessary
97 file system overhead when accessing the buffer.
98 2. Use block devices directly. If your operating system allows you to
100 call mmap() on block devices (Solaris and recent versions of Linux
101 do, FreeBSD at last report does not), this method can avoid all of
102 the native file system overhead. Note, however, that Solaris has
103 problems with byte range locking on block devices, and therefore
104 this method should not be used on Solaris.
105 Partition the disk. If you're using Solaris, set up your
107 partitions to avoid the first cylinder of the disk (or otherwise
108 the buffindexed header will overwrite the disk partition table and
109 render the buffers inaccessible). Then, create device files for
110 each block device you're going to use.
111 It's not recommended to use the block device files in /dev, since
113 the news system doesn't have permission to write to them and
114 changing the permissions of the system device files may affect
115 something else. Instead, use mknod(1) to create a new set of block
116 devices (in somewhere like pathspool/overview that's only writable
117 by the news user). To do this, run "ls -Ll" on the devices in /dev
118 that correspond to the block devices that you want to use. The
119 major and minor device numbers are in the fifth and sixth columns
120 (right before the date), respectively. Then run mknod like:
121 mknod <filename> b <major> <minor>
123 where <filename> is the path to the device to create (matching the
125 <filename> part of the buffindexed configuration line) and <major>
126 and <minor> are the major and minor device numbers as discovered
127 above.
128 Here's a short script to do this when given the path to the system
130 device file as an argument:
131 #!/bin/sh
133 base=`echo "$1" | sed 's%.*/%%'`
134 major=`ls -Ll "$1" | awk '{print $5}' | tr -d ,`
135 minor=`ls -Ll "$1" | awk '{print $6}`
136 mkdir -p <pathoverview in inn.conf>
137 mknod <pathoverview>/"$base" b "$major" "$minor"
138 chown news:news <pathoverview>/"$base"
139 chmod 644 <pathoverview>/"$base"
140 Make sure that the created files are owned by the news user and
142 news group, as specified at configure time (the default being
143 "news" for both). Also make sure that the permissions on the
144 devices allow the news user to read and write.
145
147 Written by Katsuhiro Kondou <kondou@nec.co.jp> for InterNetNews.
148 Converted to POD by Russ Allbery <eagle@eyrie.org>.
149
151 expireover(8), inn.conf(5), inndf(8), makehistory(8).
152INN 2.7.1 2023-04-16 BUFFINDEXED.CONF(5)