1SQUATTER(8) Cyrus IMAP SQUATTER(8)
2
6 squatter - Cyrus IMAP documentation
7 Create SQUAT and Xapian indexes for mailboxes
9
11 general:
12 squatter [ -C config-file ] [mode] [options] [source]
13 i.e.:
15 squatter [ -C config-file ] [ -v ] [ -a ] [ -S seconds ] [ -Z ]
16 squatter [ -C config-file ] [ -v ] [ -a ] [ -i ] [ -N name ] [ -S seconds ] [ -r ] [ -Z ] mailbox...
17 squatter [ -C config-file ] [ -v ] [ -a ] [ -i ] [ -N name ] [ -S seconds ] [ -r ] [ -Z ] -u user...
18 squatter [ -C config-file ] [ -v ] [ -a ] -R [ -n channel ] [ -d ] [ -S seconds ] [ -Z ]
19 squatter [ -C config-file ] [ -v ] [ -a ] -f synclogfile [ -S seconds ] [ -Z ]
20 squatter [ -C config-file ] [ -v ] -t srctier(s)... -z desttier [ -F ] [ -U ] [ -T dir ] [ -X ] [ -o ] [ -S seconds ] [ -u user... ]
21
23 NOTE:
24 The name “squatter” once referred both to the SQUAT indexing engine
25 and to the command used to create indexes. Now that Cyrus supports
26 more than one index type – SQUAT and Xapian, as of this writing –
27 the name “squatter” refers to the command used to control index cre‐
28 ation. The terms “SQUAT” or “SQUAT index(es)” refers to the indexes
29 used by the older SQUAT indexing engine. Post v3 the search_engine
30 setting in imapd.conf determines which search engine is used.
31 squatter creates a new text index for one or more IMAP mailboxes. The
33 index is a unified index of all of the header and body text of each
34 message in a given mailbox. This index is used to significantly reduce
35 IMAP SEARCH times on a mailbox.
36 mode is one of indexer, search, rolling, synclog, compact or audit.
38 By default, squatter creates an index of ALL messages in the mailbox,
40 not just those since the last time that it was run. The -i option is
41 used to select incremental updates. Any messages appended to the mail‐
42 box after squatter is run, will NOT be included in the index. To in‐
43 clude new messages in the index, squatter must be run again, or on a
44 regular basis via crontab, an entry in the EVENTS section of
45 cyrus.conf(5) or use rolling mode (-R).
46 In the first synopsis, squatter indexes all mailboxes.
48 In the second synopsis, squatter indexes the specified mailbox(es).
50 The mailboxes are space-separated.
51 In the third synopsis, squatter indexes the specified user(s) mail‐
53 box(es).
54 For the latter two index modes (mailbox, user) one may optionally spec‐
56 ify -r to recurse from the specified start, or -a to limit action only
57 to mailboxes which have the shared /vendor/cmu/cyrus-imapd/squat anno‐
58 tation set to “true”.
59 In the fourth synopsis, squatter runs in rolling mode. In this mode
61 squatter backgrounds itself and runs as a daemon (unless -d is set),
62 listening to a sync log channel chosen using the -n option, and set up
63 using the sync_log_channels setting in imapd.conf(5). Very soon after
64 messages are delivered or uploaded to mailboxes squatter will incremen‐
65 tally index the affected mailbox (see notes, below).
66 In the fifth synopsis, squatter reads a single sync log file and per‐
68 forms incremental indexing on the mailbox(es) listed therein. This is
69 sometimes useful for cleaning up after problems with rolling mode.
70 In the sixth synopsis, squatter will compact indices from srctier(s) to
72 desttier, optionally reindexing (-X) or filtering expunged records (-F)
73 in the process. The optional -T flag may be used to specify a direc‐
74 tory to use for temporary files. These files are eventually copied
75 with rsync -a and then removed by rm. rsync can increase the load av‐
76 erage of the system, especially when the temporary directory is on
77 tmpfs. To throttle rsync it is possible to modify the call in
78 imap/search_xapian.c and pass --bwlimit=<number> as further parameter.
79 The -o flag may be used to direct that a single index be copied, rather
80 than compacted, from srctier to desttier. The -u flag may be used to
81 restrict operation to the specified user(s).
82 For all modes, the -S option may be specified, causing squatter to
84 pause seconds seconds after each mailbox, to smooth loads.
85 When using the Xapian engine the -Z option may be specified, for the
87 indexing modes. This tells squatter to consult the Xapian internally
88 indexed GUIDs, rather than relying on what’s stored in cyrus.in‐
89 dexed.db, allowing for recovery from broken cyrus.indexed.db at the
90 sacrifice of efficiency.
91 NOTE:
93 Incremental updates are very inefficient with the SQUAT search en‐
94 gine. If using SQUAT for large and active mailboxes, you should run
95 squatter periodically as an EVENT in cyrus.conf(5).
96 NOTE:
98 Messages and mailboxes that have not been indexed CAN still be
99 SEARCHed, just not as quickly as those with an index.
100 squatter reads its configuration options out of the imapd.conf(5) file
102 unless specified otherwise by -C.
103
105 -C config-file
106 Use the specified configuration file config-file rather than the
107 default imapd.conf(5).
108 -a Only create indexes for mailboxes which have the shared /ven‐
110 dor/cmu/cyrus-imapd/squat annotation set to “true”.
111 The value of the /vendor/cmu/cyrus-imapd/squat annotation is in‐
113 herited by all children of the given mailbox, so an entire mail‐
114 box tree can be indexed (or not indexed) by setting a single an‐
115 notation on the root of that tree with a value of “true” (or
116 “false”). If a mailbox does not have a /ven‐
117 dor/cmu/cyrus-imapd/squat annotation set on it (or does not in‐
118 herit one), then the mailbox is not indexed. In other words, the
119 implicit value of /vendor/cmu/cyrus-imapd/squat is “false”.
120 -A Missing documentation, starts audit mode. This feature is only
122 available on the master branch.
123 -d In rolling mode, don’t background and do emit log messages on
125 standard error. Useful for debugging. This feature was intro‐
126 duced in version 3.0.
127 -F In compact mode, filter the resulting database to only include
129 messages which are not expunged in mailboxes with existing
130 name/uidvalidity. This feature was introduced in version 3.0.
131 -f synclogfile
133 Read the synclogfile and incrementally index all the mailboxes
134 listed therein, then exit. This feature was introduced in ver‐
135 sion 3.0.
136 -h Display this usage information.
138 -i Incremental updates where indexes already exist.
140 -N name
142 Only index mailboxes beginning with name while iterating through
143 the mailbox list derived from other options.
144 -n channel
146 In rolling mode, specify the name of the sync log channel that
147 squatter will listen to. The default is “squatter”. This chan‐
148 nel must be defined in imapd.conf(5) before being used. This
149 feature was introduced in version 3.0.
150 -o In compact mode, if only one source database is selected, just
152 copy it to the destination rather than compacting. This feature
153 was introduced in version 3.0.
154 -P Missing documentation, reindex parts mode. This feature is only
156 available on the master branch.
157 -R Run in rolling mode; squatter runs as a daemon listening to a
159 sync log channel and continuously incrementally indexing mail‐
160 boxes. See also -d and -n. This feature was introduced in ver‐
161 sion 3.0.
162 -r Recursively create indexes for all sub-mailboxes of the user,
164 mailboxes or mailbox prefixes given as arguments.
165 -S seconds
167 After processing each mailbox, sleep for “seconds” before con‐
168 tinuing. Can be used to provide some load balancing. Accepts
169 fractional amounts. This feature was introduced in version 3.0.
170 -T directory
172 When indexing, work on a temporary copy of the search engine
173 databases in directory. That directory would typically be on
174 some very fast filesystem, like an SSD or tmpfs. This option
175 may not work with all search engines, but it’s only effect is to
176 speed up initial indexing. Xapian only. This feature was in‐
177 troduced in version 3.0.
178 -t srctier...
180 In compact mode, the comma-separated source tier(s) for the com‐
181 pacted indices. At least one source tier must be specified in
182 compact mode. Xapian only. This feature was introduced in ver‐
183 sion 3.0.
184 -u Extra options refer to usernames (e.g. foo@bar.com) rather than
186 mailbox names. Usernames are space-separated. This feature was
187 introduced in version 3.0.
188 -U In compact mode, only compact if re-indexing. Xapian only.
190 This feature is only available on the master branch.
191 -v Increase the verbosity of progress/status messages. Sometimes
193 additional messages are emitted on the terminal with this option
194 and the messages are unconditionally sent to syslog. Sometimes
195 messages are sent to syslog, only if -v is provided. In rolling
196 and synclog modes, -vv sends even more messages to syslog.
197 -X Reindex all the messages before compacting. This mode reads all
199 the lists of messages indexed by the listed tiers, and re-in‐
200 dexes them into a temporary database before compacting that into
201 place. Xapian only. This feature was introduced in version
202 3.0.
203 -z desttier
205 In compact mode, the destination tier for the compacted indices.
206 This must be specified in compact mode. Xapian only. This fea‐
207 ture was introduced in version 3.0.
208 -Z When indexing messages, use the Xapian internal cyrusid rather
210 than referencing the ranges of already indexed messages to know
211 if a particular message is indexed. Useful if the ranges get
212 out of sync with the actual messages (e.g. if files on a tier
213 are lost) Xapian only. This feature is only available on the
214 master branch.
215
217 squatter is typically deployed via entries in cyrus.conf(5), in either
218 the DAEMON or EVENTS sections.
219 For the older SQUAT search engine, which offers poor performance in
221 rolling mode (-R) we recommend triggering periodic runs via entries in
222 the EVENTS section, as follows:
223 Sample entries from the EVENTS section of cyrus.conf(5) for periodic
225 squatter runs:
226 EVENTS {
228 # reindex changed mailboxes (fulltext) approximately every three hours
229 squatter1 cmd="/usr/bin/ionice -c idle /usr/lib/cyrus/bin/squatter -i" period=180
230 # reindex all mailboxes (fulltext) daily
232 squattera cmd="/usr/lib/cyrus/bin/squatter" at=0117
233 }
234 For the newer Xapian search engine, and with sufficiently fast storage,
236 the rolling mode (-R) offers advantages. Use of rolling mode requires
237 that squatter be invoked in the DAEMON section.
238 Sample entries for the DAEMON section of cyrus.conf(5) for rolling
240 squatter operation:
241 DAEMON {
243 # run a rolling squatter using the default sync_log channel "squatter"
244 squatter cmd="squatter -R"
245 # run a rolling squatter using a specific sync_log channel
247 squatter cmd="squatter -R -n indexer"
248 }
249 NOTE:
251 When using the -R rolling mode, you MUST enable sync_log operation
252 in imapd.conf(5) via the sync_log: on setting, and MUST define a
253 sync_log channel via the sync_log_channels: setting. If also using
254 replication, you must either explicitly specify your replication
255 sync_log channel via the sync_log_channels directive with a name, or
256 specify the default empty name with “” (the two-character string
257 U+22 U+22). [Please see imapd.conf(5) for details].
258 NOTE:
260 When configuring rolling search indexing on a replica, one must con‐
261 sider whether sync_logs will be written at all. In this case,
262 please consider the setting sync_log_unsuppressable_channels to en‐
263 sure that the sync_log channel upon which one’s squatter instance
264 depends will continue to be written. See imapd.conf(5) for details.
265 NOTE:
267 When using the Xapian search engine, you must define various set‐
268 tings in imapd.conf(5). Please read all relevant Xapian documenta‐
269 tion in this release before using Xapian.
270 [NB: More examples needed]
272
274 Support for additional search engines was added in version 3.0.
275 The following command-line switches were added in version 3.0:
277 -F -R -X -d -f -o -u
279 The following command-line settings were added in version 3.0:
281 -S <seconds>, -T <directory>, -f <synclogfile>, -n <channel>, -t srctier..., -z desttier
283
285 /etc/imapd.conf, /etc/cyrus.conf
286
288 imapd.conf(5), cyrus.conf(5)
289
291 The Cyrus Team, Nic Bernstein (Onlight)
292
294 1993-2018, The Cyrus Team
2953.2.8 September 01, 2021 SQUATTER(8)