1CYCBUFF.CONF(5) InterNetNews Documentation CYCBUFF.CONF(5)
2
6 cycbuff.conf - Configuration file for INN CNFS storage method
7
9 This file defines the cyclical buffers that make up the storage pools
10 for CNFS (Cyclic News File System). Some options controlling the
11 behavior of the CNFS storage system can also be set here. cycbuff.conf
12 is required if the CNFS (Cyclic News File System) storage method is
13 used. INN will look for it in pathetc (as set in inn.conf).
14 CNFS stores articles in logical objects called metacycbuffs. Each
16 metacycbuff is in turn composed of one or more physical buffers called
17 cycbuffs. As articles are written to the metacycbuff, each article is
18 written to the next cycbuff in the list in a round-robin fashion
19 (unless "sequential" mode is specified, in which case each cycbuff is
20 filled before moving on to the next). This is so that you can
21 distribute the individual cycbuffs across multiple physical disks and
22 balance the load between them. Note that in order to use any cycbuff
23 larger than 2 GB on 32-bit platforms (and some very rare 64-bit
24 platforms that aren't Linux), you need to build INN with the
25 --enable-largefiles option.
26 For information about how to configure INN to use CNFS, see
28 storage.conf(5).
29 Blank lines and lines beginning with a hash sign ("#") are ignored.
31 All other lines must be of one of the following forms:
32 cycbuffupdate:<interval>
34 refreshinterval:<interval>
35 cycbuff:<name>:<file>:<size>
36 metacycbuff:<name>:<buffer>[,<buffer>,...][:<mode>]
37 (where items enclosed in [] are optional). Order is mostly not
39 significant, but all cycbuff lines must occur before all metacycbuff
40 lines. Long lines can be continued on the next line by ending the line
41 with a backslash ("\").
42 cycbuffupdate:<interval>
44 Sets the number of articles written before the cycbuff header is
45 written back to disk to <interval>. Under most operating systems,
46 the header doesn't have to be written to disk for the updated data
47 to be available to other processes on the same system that are
48 reading articles out of CNFS, but any accesses to the CNFS cycbuffs
49 over NFS will only see the data present at the last write of the
50 header. After a system crash, all updates since the last write of
51 the CNFS header may be lost. The default value, if this line is
52 omitted, is 25, meaning that the header is written to disk after
53 every 25 articles stored in that cycbuff.
54 refreshinterval:<interval>
56 Sets the interval (in seconds) between re-reads of the cycbuff
57 header to <interval>. This primarily affects nnrpd and controls
58 the frequency with which it updates its knowledge of the current
59 contents of the CNFS cycbuffs. The default value, if this line is
60 omitted, is 30.
61 cycbuff:<name>:<file>:<size>
63 Configures a particular CNFS cycbuff. <name> is a symbolic name
64 for the buffer, to be used later in a metacycbuff line. It must be
65 no longer than seven characters. <file> is the full path to the
66 buffer file or block device, and must be no longer than 63
67 characters. <size> is the length of the buffer in kilobytes (1 KB
68 is 1024 bytes). If <file> is not a block device, it should be
69 <size> * 1024 bytes long.
70 If you're trying to stay under 2 GB, keep your sizes below 2097152.
72 Note that CNFSv4 supports files and partitions up to 16 TB.
74 metacycbuff:<name>:<buffer>[,<buffer>,...][:<mode>]
76 Specifies a collection of CNFS buffers that make up a single
77 logical storage location from the perspective of INN. Metacycbuffs
78 are referred to in storage.conf as storage locations for articles,
79 so in order to actually put articles in a cycbuff, it has to be
80 listed as part of some metacycbuff which is then referenced in
81 storage.conf.
82 <name> is the symbolic name of the metacycbuff, referred to in the
84 options: field of "cnfs" entries in storage.conf. It must be no
85 longer than eight characters. <buffer> is the name of a cycbuff
86 (the <name> part of a cycbuff line), and any number of cycbuffs may
87 be specified, separated by commas.
88 If there is more than one cycbuff in a metacycbuff, there are two
90 ways that INN can distribute articles between the cycbuffs. The
91 default mode, "INTERLEAVE", stores the articles in each cycbuff in
92 a round-robin fashion, one article per cycbuff in the order listed.
93 If the cycbuffs are of wildly different sizes, this can cause some
94 of them to roll over much faster than others, and it may not give
95 the best performance depending on your disk layout. The other
96 storage mode, "SEQUENTIAL", instead writes to each cycbuff in turn
97 until that cycbuff is full and then moves on to the next one,
98 returning to the first and starting a new cycle when the last one
99 is full. To specify a mode rather than leaving it at the default,
100 add a colon and the mode ("INTERLEAVE" or "SEQUENTIAL") at the end
101 of the metacycbuff line.
102 innd only reads cycbuff.conf on startup, so if you change anything in
104 this file and want innd to pick up the changes, you have to use
105 "ctlinnd xexec innd"; "ctlinnd reload all ''" is not sufficient.
106 When articles are stored, the cycbuff into which they're stored is
108 saved as part of the article token. In order for INN to retrieve
109 articles from a cycbuff, that cycbuff must be listed in cycbuff.conf.
110 However, if INN should not write to a cycbuff, it doesn't need to be
111 (and shouldn't be) listed in a metacycbuff.
112 This provides an easy way to retire a cycbuff. Just remove it from its
114 metacycbuff, leaving in the cycbuff line, and restart innd (with, for
115 example, "ctlinnd xexec innd"). No new articles will be put into the
116 cycbuff, but neither will any articles expire from it. After you no
117 longer need the articles in the cycbuff, just remove it entirely from
118 cycbuff.conf. Then all of the articles will appear to have been
119 deleted to INN, and the next nightly expire run will clean up any
120 remaining references to them.
121 Adding a new cycbuff just requires creating it (see below), adding a
123 cycbuff line, adding it to a metacycbuff, and then restarting innd.
124 Similarly, changing the allocation of an existing cycbuff from a
125 metacycbuff to another one just requires modifying the two metacycbuffs
126 accordingly, and then restarting innd. (Only the cycbuff is noted
127 noted in the storage API tokens.)
128
130 When creating a new cycbuff, there are two different methods for
131 creating the buffers in which the articles will be stored.
132 1. Create a large file on top of a regular file system. The easiest
134 way to do this is probably with dd(1), using a command like:
135 dd if=/dev/zero of=/path/to/cycbuff bs=1024 count=<size>
137 where <size> is the size from the cycbuff line in cycbuff.conf.
139 INSTALL contains a script that will generate these commands for you
140 from your cycbuff.conf file.
141 This is the simplest method, but has the disadvantage that very
143 large files on regular file systems can be fairly slow to access,
144 particularly at the end of the file, and INN incurs unnecessary
145 file system overhead when accessing the cycbuff.
146 2. Use block devices directly. If your operating system allows you to
148 call mmap() on block devices (Solaris and recent versions of Linux
149 do, FreeBSD at last report does not), this is the recommended
150 method since you can avoid all of the native file system overhead.
151 Note that some OSes do not support files larger than 2 GB, which
153 will limit the size you can make a single cycbuff, but you can
154 still combine many cycbuffs into each metacycbuff. Very old
155 versions of Linux (before 2.4 kernels, that raised the limit to
156 2 TB) are known to have this limitation; FreeBSD does not. Some
157 OSes that support large files don't support direct access to block
158 devices for large partitions (Solaris prior to Solaris 7, or not
159 running in 64-bit mode, is in this category); on those OSes, if you
160 want cycbuffs over 2 GB, you'll have to use regular files. If in
161 doubt, keep your cycbuffs smaller than 2 GB.
162 Partition the disk to make each partition equal to or smaller than
164 2 GB. If you're using Solaris, set up your partitions to avoid the
165 first cylinder of the disk (or otherwise the cycbuff header will
166 overwrite the disk partition table and render the cycbuffs
167 inaccessible). Then, create device files for each block device
168 you're going to use.
169 It's not recommended to use the block device files in /dev, since
171 the news system doesn't have permission to write to them and
172 changing the permissions of the system device files may affect
173 something else. Instead, use mknod(1) to create a new set of block
174 devices (in somewhere like pathspool/cycbuffs that's only writable
175 by the news user). To do this, run "ls -Ll" on the devices in /dev
176 that correspond to the block devices that you want to use. The
177 major and minor device numbers are in the fifth and sixth columns
178 (right before the date), respectively. Then run mknod like:
179 mknod <file> b <major> <minor>
181 where <file> is the path to the device to create (matching the
183 <file> part of the cycbuff line) and <major> and <minor> are the
184 major and minor device numbers as discovered above.
185 Here's a short script to do this when given the path to the system
187 device file as an argument:
188 #!/bin/sh
190 base=`echo "$1" | sed 's%.*/%%'`
191 major=`ls -Ll "$1" | awk '{print $5}' | tr -d ,`
192 minor=`ls -Ll "$1" | awk '{print $6}`
193 mkdir -p <pathspool in inn.conf>/cycbuffs
194 mknod <pathspool>/cycbuffs/"$base" b "$major" "$minor"
195 chown news:news <pathspool>/cycbuffs/"$base"
196 chmod 644 <pathspool>/cycbuffs/"$base"
197 Make sure that the created files are owned by the news user and
199 news group, as specified at configure time (the default being
200 "news" for both). Also make sure that the permissions on the
201 devices allow the news user to read and write, and if you want
202 other users on the system to be able to use sm to retrieve
203 articles, make sure they're world-readable.
204 Once you have everything configured properly and you start innd, you
206 should see messages in news.notice that look like:
207 innd: CNFS: no magic cookie found for cycbuff ONE, initializing
209 where "ONE" will be whatever you called your cycbuff.
211
213 Written by Katsuhiro Kondou <kondou@nec.co.jp> for InterNetNews.
214 Rewritten into POD by Russ Allbery <eagle@eyrie.org>.
215
217 ctlinnd(8), innd(8), nnrpd(8), sm(1), storage.conf(5).
218INN 2.6.5 2022-01-23 CYCBUFF.CONF(5)