1STORAGE.CONF(5) File Formats Manual STORAGE.CONF(5)
2
6 storage.conf - configuration file for storage manager
7
9 The storage manager is a unified interface between INN and a variety of
10 different storage method, allowing the news administrator to choose
11 between different storage methods with different tradeoffs (or even use
12 several at the same time for different newsgroups, or articles of dif‐
13 ferent sizes). The rest of INN need not care what type of storage
14 method was used for a given article; the storage manager will figure
15 this out automatically when that article is retrieved via the storage
16 API.
17 The <pathetc in inn.conf>/storage.conf file contains the rules to be
19 used in assigning articles to different storage methods.
20 The file consists of a series of storage method entries. Blank lines
22 and lines beginning with a number sign (``#'') are ignored. The maxi‐
23 mum number of characters in each line is 255. The order of entries in
24 this file is important, see below.
25 Each entry specifies a storage method and a set of rules. Articles
27 that match all of the rules of a storage method entry will be stored
28 using that storage method; if an article matches multiple storage
29 method entries, the first will be used. Each entry is formatted as
30 follows:
31 method <methodname> {
33 class: <storage_class>
34 newsgroups: <wildmat>
35 size: <minsize>[,<maxsize>]
36 expires: <mintime>[,<maxtime>]
37 options: <options>
38 exactmatch: <bool>
39 }
40 If spaces or tabs are included in a value, that value must be quoted
42 with ``"''. If either ``#'' or ``"'' are meant to be included verbatim
43 in a value, they should be escaped with ``\''.
44 <methodname> is the name of a storage method to use for articles that
46 match the rules of this entry. The currently available storage methods
47 are ``timecaf'', ``timehash'', ``cnfs'', ``tradspool'' and ``trash''.
48 See the STORAGE METHODS section below for more details.
49 The meanings of the keys in each entry are as follows:
51 class An identifier for this storage method entry. <storage_class>
53 should be a number and should be unique across all of the
54 entries in this file. It's used mainly for specifying expira‐
55 tion times by storage class as described in expire.ctl(5).
56 newsgroups
58 What newsgroups are stored using this storage method. <wildmat>
59 is a uwildmat(3) pattern that is matched against the newsgroups
60 an article is posted to. If ``storeonxref'' in inn.conf is
61 ``true'', this pattern will be matched against the newsgroup
62 names in the ``Xref'' header; otherwise, it will be matched
63 against newsgroup names in the ``Newsgroups'' header (see
64 inn.conf(5) for discussion of the differences between these pos‐
65 sibilities). Poison wildmat expressions (expressions starting
66 with ``@'') are allowed and can be used to exclude certain group
67 patterns. ``!'' cannot be used, however. The <wildmat> pattern
68 is matched in order. There is no default newsgroups pattern; if
69 an entry should match all newsgroups, use an explicit ``news‐
70 groups: *''.
71 size A range of article sizes (in bytes) that should be stored using
73 this storage method. If <maxsize> is ``0'' or not given, the
74 upper size of articles is limited only by ``maxartsize'' in
75 inn.conf. The ``size'' field is optional and may be omitted
76 entirely if you want articles of any size (that otherwise ful‐
77 fill the requirements of this storage method entry) to be stored
78 in this storage method.
79 expires
81 A range of article expiration times that should be stored using
82 this storage method. Be careful; this is less useful than it
83 may appear at first. This is based only on the ``Expires''
84 header of the article, not on any local expiration policies or
85 anything in expire.ctl! If <mintime> is non-zero, then this
86 entry will not match any article without an ``Expires'' header.
87 This key is therefore only really useful for assigning articles
88 with requested longer expire times to a separate storage method.
89 Articles only match if the time until expiration (that is, the
90 amount of time into the future that the ``Expires'' header of
91 the article requests that it remain around) falls in the inter‐
92 val specified by <mintime> and <maxtime>. The format of these
93 parameters is 0d0h0m0s (days, hours, minutes, and seconds into
94 the future). If <maxtime> is ``0s'' or is not specified, there
95 is no upper bound on expire times falling into this entry (note
96 that this key has no effect on when the article will actually be
97 expired, only on whether or not the article will be stored using
98 this storage method). This field is also optional and may be
99 omitted entirely if all articles with or without an ``Expires''
100 header (that otherwise fulfill the requirements of this storage
101 method entry) should be stored according to it.
102 options
104 This key is for passing special options to storage methods that
105 require them (currently only ``cnfs''). See the STORAGE METHODS
106 section below for a description of its use.
107 exactmatch
109 If this key is set to ``true'', all newsgroups will be examined
110 to see if they match newsgroups patterns. (Normally, any
111 nonzero number of matching newsgroups is sufficient, provided no
112 newsgroup matches a poison wildmat as described above.) This is
113 a boolan value and ``true'', ``yes'' and ``on'' are usable to
114 enable this key. The case of these values is not significant.
115 The default is false.
116 If an article matches all of the constraints of an entry, it is stored
118 via that storage method and is associated with that <storage_class>.
119 This file is scanned in order and the first matching entry is used to
120 store the article.
121 If an article doesn't match any entry, either by being posted to a
123 newsgroup that doesn't match any of the <wildmat> patterns or by being
124 outside the size and expires ranges of all entries whose newsgroups
125 pattern it does match, the article is not stored and is rejected by
126 innd(8). When this happens, the error message
127 cant store article: no matching entry in storage.conf
129 is logged to syslog. If you want to silently drop articles matching
131 certain newsgroup patterns or size or expires ranges, assign them to
132 the ``trash'' storage method rather than having them not match any
133 storage method entry.
134
136 Currently, there are four storage methods available. Each method has
137 its pros and cons; you can choose any mixture of them as is suitable
138 for your environment. Note that each method has an attribute ``EXPEN‐
139 SIVESTAT'' which indicates whether checking the existence of an arti‐
140 cle is expensive or not. This is used to run expireover(8).
141 cnfs The ``cnfs'' storage method stores articles in large cyclic buf‐
143 fers (CNFS stands for Cyclic News File System). It's by far the
144 fastest of all storage methods (except for ``trash''), since it
145 eliminates the overhead of dealing with a file system and creat‐
146 ing new files. Articles are stored in CNFS buffers in arrival
147 order, and when the buffer fills, it wraps around to the begin‐
148 ning and stores new articles over top of the oldest articles in
149 the buffer. The expire time of articles stored in CNFS buffers
150 is therefore entirely determined by how long it takes the buffer
151 to wrap around, which depends on how quickly data is being
152 stored in it. (This method is therefore said to have self-
153 expire functionality.) ``EXPENSIVESTAT'' is ``false'' for this
154 method. CNFS has its own configuration file, cycbuff.conf,
155 which describes some subtlties to the basic description given
156 above. Storage method entries for the ``cnfs'' storage method
157 must have an ``options'' field specifying the metacycbuff into
158 which articles matching that entry should be stored; see
159 cycbuff.conf(5) for details on metacycbuffs.
160 timecaf
162 This method stores multiple articles in one file, whose name is
163 based on the article's arrival time and the storage class. The
164 file name will be <patharticles in inn.conf>/timecaf-
165 nn/bb/aacc.CF, where ``nn'' is the hexadecimal value of <stor‐
166 age_class>, ``bb'' and ``aacc'' are hexadecimal components of
167 the arrival time, and ``CF'' is a hardcoded extension. (The
168 arrival time, in seconds since the epoch, is converted to hexa‐
169 decimal and interpreted as 0xaabbccdd, with ``aa'', ``bb'', and
170 ``cc'' used to build the path.) This method does not have self-
171 expire functionality (meaning expire(8) has to run periodically
172 to delete old articles). ``EXPENSIVESTAT'' is ``false'' for
173 this method.
174 timehash
176 This method is very similar to ``timecaf'' except that each
177 article is stored in a separate file. The name of the file for
178 a given article will be <patharticles in inn.conf>/time-
179 nn/bb/cc/yyyy-aadd, where ``nn'' is the hexadecimal value of
180 <storage_class>, ``yyyy'' is a hexadecimal sequence number, and
181 ``bb'', ``cc'', and ``aadd'' are components of the arrival time
182 in hexadecimal (the arrival time is interpreted as documented
183 above under ``timecaf''). This method does not have self-expire
184 functionality. ``EXPENSIVESTAT'' is ``true'' for this method.
185 tradspool
187 Traditional spool, or ``tradspool'', is the traditional news
188 article storage format. Each article is stored in a file named:
189 <patharticles in inn.conf>/news/group/name/nnnnn, where
190 ``news/group/name'' is the name of the newsgroup to which the
191 article was posted with each period changed to a slash, and
192 ``nnnnn'' is the sequence number of the article in that news‐
193 group. For crossposted articles, the article is linked into
194 each newsgroup to which it is crossposted (using either hard or
195 symbolic links). This is the way versions of INN prior to 2.0
196 stored all articles, as well as being the article storage format
197 used by C News and earlier news systems. This method does not
198 have self-expire functionality. ``EXPENSIVESTAT'' is ``true''
199 for this method.
200 trash This method silently discards all articles stored in it. Its
202 only real uses are for testing and for silently discarding arti‐
203 cles matching a particular storage method entry (for whatever
204 reason). Articles stored in this method take up no disk space
205 and can never be retrieved, so this method has self-expire func‐
206 tionality of a sort. ``EXPENSIVESTAT'' is ``false'' for this
207 method.
208
210 The following sample storage.conf file would store all articles posted
211 to alt.binaries.* in the ``BINARIES'' CNFS metacycbuff, all articles
212 over roughly 50 KB in any other hierarchy in the ``LARGE'' CNFS metacy‐
213 cbuff, all other articles in alt.* in one timehash class, and all other
214 articles in any newsgroups in a second timehash class, except for the
215 internal.* hierarchy which is stored in traditional spool format.
216 method tradspool {
218 class: 1
219 newsgroups: internal.*
220 }
221 method cnfs {
223 class: 2
224 newsgroups: alt.binaries.*
225 options: BINARIES
226 }
227 method cnfs {
229 class: 3
230 newsgroups: *
231 size: 50000
232 options: LARGE
233 }
234 method timehash {
236 class: 4
237 newsgroups: alt.*
238 }
239 method timehash {
241 class: 5
242 newsgroups: *
243 }
244 Notice that the last storage method entry will catch everything. This
246 is a good habit to get into; make sure that you have at least one
247 catch-all entry just in case something you didn't expect falls through
248 the cracks. Notice also that the special rule for the internal.* hier‐
249 archy is first, so it will catch even articles crossposted to alt.bina‐
250 ries.* or over 50 KB in size.
251
253 Written by Katsuhiro Kondou <kondou@nec.co.jp> for InterNetNews. This
254 is revision 6124, dated 2003-01-14.
255
257 cycbuff.conf(5), expire.ctl(5), inn.conf(5), innd(8), newsfeeds(5),
258 uwildmat(3).
259 STORAGE.CONF(5)