1SQUATTER(8) Cyrus IMAP SQUATTER(8)
2
6 squatter - Cyrus IMAP documentation
7 Create SQUAT and Xapian indexes for mailboxes
9
11 general:
12 squatter [ -C config-file ] [mode] [options] [source]
13 i.e.:
15 squatter [ -C config-file ] [ -v ] [ -a ] [ -S seconds ]
16 squatter [ -C config-file ] [ -v ] [ -a ] [ -i ] [ -N name ] [ -S seconds ] [ -r ] mailbox...
17 squatter [ -C config-file ] [ -v ] [ -a ] [ -i ] [ -N name ] [ -S seconds ] [ -r ] -u user...
18 squatter [ -C config-file ] [ -v ] [ -a ] -R [ -n channel ] [ -d ] [ -S seconds ]
19 squatter [ -C config-file ] [ -v ] [ -a ] -f synclogfile [ -S seconds ]
20 squatter [ -C config-file ] [ -v ] -I file
21 squatter [ -C config-file ] [ -v ] -t srctier(s)... -z desttier [ -F ] [ -T dir ] [ -X ] [ -o ] [ -S seconds ] [ -u user... ]
22
24 NOTE:
25 The name "squatter" once referred both to the SQUAT indexing engine
26 and to the command used to create indexes. Now that Cyrus supports
27 more than one index type -- SQUAT and Xapian, as of this writing --
28 the name "squatter" refers to the command used to control index cre‐
29 ation. The terms "SQUAT" or "SQUAT index(es)" refers to the indexes
30 used by the older SQUAT indexing engine. Post v3 the search_engine
31 setting in imapd.conf determines which search engine is used.
32 squatter creates a new text index for one or more IMAP mailboxes. The
34 index is a unified index of all of the header and body text of each
35 message in a given mailbox. This index is used to significantly reduce
36 IMAP SEARCH times on a mailbox.
37 mode is one of indexer, indexfrom (-I), search, rolling, synclog or
39 compact.
40 By default, squatter creates an index of ALL messages in the mailbox,
42 not just those since the last time that it was run. The -i option is
43 used to select incremental updates. Any messages appended to the mail‐
44 box after squatter is run, will NOT be included in the index. To
45 include new messages in the index, squatter must be run again, or on a
46 regular basis via crontab, an entry in the EVENTS section of
47 cyrus.conf(5) or use rolling mode (-R).
48 In the first synopsis, squatter indexes all mailboxes.
50 In the second synopsis, squatter indexes the specified mailbox(es).
52 The mailboxes are space-separated.
53 In the third synopsis, squatter indexes the specified user(s) mail‐
55 box(es).
56 For the latter two index modes (mailbox, user) one may optionally spec‐
58 ify -r to recurse from the specified start, or -a to limit action only
59 to mailboxes which have the shared /vendor/cmu/cyrus-imapd/squat anno‐
60 tation set to "true".
61 In the fourth synopsis, squatter runs in rolling mode. In this mode
63 squatter backgrounds itself and runs as a daemon (unless -d is set),
64 listening to a sync log channel chosen using the -n option, and set up
65 using the sync_log_channels setting in imapd.conf(5). Very soon after
66 messages are delivered or uploaded to mailboxes squatter will incremen‐
67 tally index the affected mailbox (see notes, below).
68 In the fifth synopsis, squatter reads a single sync log file and per‐
70 forms incremental indexing on the mailbox(es) listed therein. This is
71 sometimes useful for cleaning up after problems with rolling mode.
72 In the sixth synopsis, squatter reads file containing mailbox uid
74 tuples and performs indexing on the specified messages.
75 In the seventh synopsis, squatter will compact indices from srctier(s)
77 to desttier, optionally reindexing (-X) or filtering expunged records
78 (-F) in the process. The optional -T flag may be used to specify a
79 directory to use for temporary files. The -o flag may be used to
80 direct that a single index be copied, rather than compacted, from src‐
81 tier to desttier. The -u flag may be used to restrict operation to the
82 specified user(s).
83 For all modes, the -S option may be specified, causing squatter to
85 pause seconds seconds after each mailbox, to smooth loads.
86 NOTE:
88 Incremental updates are very inefficient with the SQUAT search
89 engine. If using SQUAT for large and active mailboxes, you should
90 run squatter periodically as an EVENT in cyrus.conf(5).
91 NOTE:
93 Messages and mailboxes that have not been indexed CAN still be
94 SEARCHed, just not as quickly as those with an index.
95 squatter reads its configuration options out of the imapd.conf(5) file
97 unless specified otherwise by -C.
98
100 -C config-file
101 Use the specified configuration file config-file rather than the
102 default imapd.conf(5).
103 -a Only create indexes for mailboxes which have the shared /ven‐
105 dor/cmu/cyrus-imapd/squat annotation set to "true".
106 The value of the /vendor/cmu/cyrus-imapd/squat annotation is
108 inherited by all children of the given mailbox, so an entire
109 mailbox tree can be indexed (or not indexed) by setting a single
110 annotation on the root of that tree with a value of "true" (or
111 "false"). If a mailbox does not have a /ven‐
112 dor/cmu/cyrus-imapd/squat annotation set on it (or does not
113 inherit one), then the mailbox is not indexed. In other words,
114 the implicit value of /vendor/cmu/cyrus-imapd/squat is "false".
115 -d In rolling mode, don't background and do emit log messages on
117 standard error. Useful for debugging. This feature was intro‐
118 duced in version 3.0.
119 -F In compact mode, filter the resulting database to only include
121 messages which are not expunged in mailboxes with existing
122 name/uidvalidity. This feature was introduced in version 3.0.
123 -f synclogfile
125 Read the synclogfile and incrementally index all the mailboxes
126 listed therein, then exit. This feature was introduced in ver‐
127 sion 3.0.
128 -h Display this usage information.
130 -I file
132 Read from file and index individual messages described by mail‐
133 box/uid tuples contained therein.
134 -i Incremental updates where indexes already exist.
136 -N name
138 Only index mailboxes beginning with name while iterating through
139 the mailbox list derived from other options.
140 -n channel
142 In rolling mode, specify the name of the sync log channel that
143 squatter will listen to. The default is "squatter". This chan‐
144 nel must be defined in imapd.conf(5) before being used. This
145 feature was introduced in version 3.0.
146 -o In compact mode, if only one source database is selected, just
148 copy it to the destination rather than compacting. This feature
149 was introduced in version 3.0.
150 -R Run in rolling mode; squatter runs as a daemon listening to a
152 sync log channel and continuously incrementally indexing mail‐
153 boxes. See also -d and -n. This feature was introduced in ver‐
154 sion 3.0.
155 -r Recursively create indexes for all sub-mailboxes of the user,
157 mailboxes or mailbox prefixes given as arguments.
158 -S seconds
160 After processing each mailbox, sleep for "seconds" before con‐
161 tinuing. Can be used to provide some load balancing. Accepts
162 fractional amounts. This feature was introduced in version 3.0.
163 -T directory
165 When indexing, work on a temporary copy of the search engine
166 databases in directory. That directory would typically be on
167 some very fast filesystem, like an SSD or tmpfs. This option
168 may not work with all search engines, but it's only effect is to
169 speed up initial indexing. Xapian only. This feature was
170 introduced in version 3.0.
171 -t srctier...
173 In compact mode, the comma separated source tier(s) for the com‐
174 pacted indices. At least one source tier must be specified in
175 compact mode. Xapian only. This feature was introduced in ver‐
176 sion 3.0.
177 -u Extra options refer to usernames (e.g. foo@bar.com) rather than
179 mailbox names. Usernames are space-separated. This feature was
180 introduced in version 3.0.
181 -v Increase the verbosity of progress/status messages. Sometimes
183 additional messages are emitted on the terminal with this option
184 and the messages are unconditionally sent to syslog. Sometimes
185 messages are sent to syslog, only if -v is provided. In rolling
186 and synclog modes, -vv sends even more messages to syslog.
187 -X Reindex all the messages before compacting. This mode reads all
189 the lists of messages indexed by the listed tiers, and
190 re-indexes them into a temporary database before compacting that
191 into place. Xapian only. This feature was introduced in ver‐
192 sion 3.0.
193 -z desttier
195 In compact mode, the destination tier for the compacted indices.
196 This must be specified in compact mode. Xapian only. This fea‐
197 ture was introduced in version 3.0.
198
200 squatter is typically deployed via entries in cyrus.conf(5), in either
201 the DAEMON or EVENTS sections.
202 For the older SQUAT search engine, which offers poor performance in
204 rolling mode (-R) we recommend triggering periodic runs via entries in
205 the EVENTS section, as follows:
206 Sample entries from the EVENTS section of cyrus.conf(5) for periodic
208 squatter runs:
209 EVENTS {
211 # reindex changed mailboxes (fulltext) approximately every three hours
212 squatter1 cmd="/usr/bin/ionice -c idle /usr/lib/cyrus/bin/squatter -i" period=180
213 # reindex all mailboxes (fulltext) daily
215 squattera cmd="/usr/lib/cyrus/bin/squatter" at=0117
216 }
217 For the newer Xapian search engine, and with sufficiently fast storage,
219 the rolling mode (-R) offers advantages. Use of rolling mode requires
220 that squatter be invoked in the DAEMON section.
221 Sample entries for the DAEMON section of cyrus.conf(5) for rolling
223 squatter operation:
224 DAEMON {
226 # run a rolling squatter using the default sync_log channel "squatter"
227 squatter cmd="squatter -R"
228 # run a rolling squatter using a specific sync_log channel
230 squatter cmd="squatter -R -n indexer"
231 }
232 NOTE:
234 When using the -R rolling mode, you MUST enable sync_log operation
235 in imapd.conf(5) via the sync_log: on setting, and MUST define a
236 sync_log channel via the sync_log_channels: setting. If also using
237 replication, you must either explicitly specify your replication
238 sync_log channel via the sync_log_channels directive with a name, or
239 specify the default empty name with "" (the two-character string
240 U+22 U+22). [Please see imapd.conf(5) for details].
241 NOTE:
243 When configuring rolling search indexing on a replica, one must con‐
244 sider whether sync_logs will be written at all. In this case,
245 please consider the setting sync_log_unsuppressable_channels to
246 ensure that the sync_log channel upon which one's squatter instance
247 depends will continue to be written. See imapd.conf(5) for details.
248 NOTE:
250 When using the Xapian search engine, you must define various set‐
251 tings in imapd.conf(5). Please read all relevant Xapian documenta‐
252 tion in this release before using Xapian.
253 [NB: More examples needed]
255
257 Support for additional search engines was added in version 3.0.
258 The following command-line switches were added in version 3.0:
260 -F -R -X -d -f -o -u
262 The following command-line settings were added in version 3.0:
264 -S <seconds>, -T <directory>, -f <synclogfile>, -n <channel>, -t srctier..., -z desttier
266
268 /etc/imapd.conf, /etc/cyrus.conf
269
271 imapd.conf(5), cyrus.conf(5)
272
274 The Cyrus Team, Nic Bernstein (Onlight)
275
277 1993-2017, The Cyrus Team
2783.0.13 December 16, 2019 SQUATTER(8)