1SQUATTER(8) Cyrus IMAP SQUATTER(8)
2
6 squatter - Cyrus IMAP documentation
7 Create SQUAT indexes for mailboxes
9
11 general:
12 squatter [ -C config-file ] [mode] [options] [source]
13 i.e.:
15 squatter [ -C config-file ] [-v]
16 squatter [ -C config-file ] [ -a ] [ -i ] [-N name] [-S *seconds*] [ **-r ] mailbox...
17 squatter [ -C config-file ] [ -a ] [ -i ] [-N name] [-S *seconds*] [ **-r ] -u user...
18 squatter [ -C config-file ] -R [ -n channel ] [ -d ]
19 squatter [ -C config-file ] -f synclogfile
20 squatter [ -C config-file ] -I file
21 squatter [ -C config-file ] -t srctier... -z desttier [ -F ] [ -T dir ] [ -X ] [ -o ]
22
24 NOTE:
25 The name "squatter" once referred both to the SQUAT indexing engine
26 and to the command used to create indexes. Now that Cyrus supports
27 more than one index type -- SQUAT and Xapian, as of this writing --
28 the name "squatter" refers to the command used to control index cre‐
29 ation. The terms "SQUAT" or "SQUAT index(es)" refers to the indexes
30 used by the older SQUAT indexing engine. Post v3 the search_engine
31 setting in imapd.conf determines which search engine is used.
32 squatter creates a new text index for one or more IMAP mailboxes. The
34 index is a unified index of all of the header and body text of each
35 message in a given mailbox. This index is used to significantly reduce
36 IMAP SEARCH times on a mailbox.
37 By default, squatter creates an index of ALL messages in the mailbox,
39 not just those since the last time that it was run. The -i option is
40 used to select incremental updates. Any messages appended to the mail‐
41 box after squatter is run, will NOT be included in the index. To
42 include new messages in the index, squatter must be run again.
43 In the first synopsis, squatter indexes all mailboxes.
45 In the second synopsis, squatter indexes the specified mailbox(es).
47 In the third synopsis, squatter indexes the specified user(s) mail‐
49 box(es).
50 For any of those three source modes (default=all, mailbox, user) one
52 may optionally specify -r to recurse from the specified start, or -a to
53 limit action only to mailboxes which have the shared /ven‐
54 dor/cmu/cyrus-imapd/squat annotation set to "true".
55 In the fourth synopsis, squatter runs in rolling mode. In this mode
57 squatter backgrounds itself and runs as a daemon (unless -d is set),
58 listening to a sync log channel chosen using the -n option, and set up
59 using the sync_log_channels setting in imapd.conf(5). Very soon after
60 messages are delivered or uploaded to mailboxes squatter will incremen‐
61 tally index the affected mailbox (see notes, below).
62 In the fifth synopsis, squatter reads a single sync log file and per‐
64 forms incremental indexing on the mailbox(es) listed therein. This is
65 sometimes useful for cleaning up after problems with rolling mode.
66 In the sixth synopsis, squatter reads file containing mailbox uid
68 tuples and performs indexing on the specified messages.
69 In the seventh synopsis, squatter will compact indices from srctier(s)
71 to desttier, optionally reindexing (-X) or filtering expunged records
72 (-F) in the process. The optional -T flag may be used to specify a
73 directory to use for temporary files. The -o flag may be used to
74 direct that a single index be copied, rather than compressed, from src‐
75 tier to desttier.
76 For all modes, the -S option may be specified, causing squatter to
78 pause seconds seconds after each mailbox, to smooth loads.
79 NOTE:
81 Incremental updates are very inefficient with the SQUAT search
82 engine. If using SQUAT for large and active mailboxes, you should
83 run squatter periodically as an EVENT in cyrus.conf(5).
84 NOTE:
86 Messages and mailboxes that have not been indexed CAN still be
87 SEARCHed, just not as quickly as those with an index.
88 squatter reads its configuration options out of the imapd.conf(5) file
90 unless specified otherwise by -C.
91
93 -C config-file
94 Use the specified configuration file config-file rather than the
95 default imapd.conf(5).
96 -a Only create indexes for mailboxes which have the shared /ven‐
98 dor/cmu/cyrus-imapd/squat annotation set to "true".
99 The value of the /vendor/cmu/cyrus-imapd/squat annotation is
101 inherited by all children of the given mailbox, so an entire
102 mailbox tree can be indexed (or not indexed) by setting a single
103 annotation on the root of that tree with a value of "true" (or
104 "false"). If a mailbox does not have a /ven‐
105 dor/cmu/cyrus-imapd/squat annotation set on it (or does not
106 inherit one), then the mailbox is not indexed. In other words,
107 the implicit value of /vendor/cmu/cyrus-imapd/squat is "false".
108 -d In rolling mode, don't background and do emit log messages on
110 standard error. Useful for debugging. This feature was intro‐
111 duced in version 3.0.
112 -F In compact mode, filter the resulting database to only include
114 messages which are not expunged in mailboxes with existing
115 name/uidvalidity. This feature was introduced in version 3.0.
116 -f synclogfile
118 Read the synclogfile and incrementally index all the mailboxes
119 listed therein, then exit. This feature was introduced in ver‐
120 sion 3.0.
121 -h Display this usage information.
123 -I file
125 Read from file and index individual messages described by mail‐
126 box/uid tuples contained therein.
127 -i Incremental updates where indexes already exist.
129 -N name
131 Only index mailboxes beginning with name while iterating through
132 the mailbox list derived from other options.
133 -n channel
135 In rolling mode, specify the name of the sync log channel that
136 squatter will listen to. The default is "squatter". This chan‐
137 nel must be defined in imapd.conf(5) before being used. This
138 feature was introduced in version 3.0.
139 -o In compact mode, if only one source database is selected, just
141 copy it to the destination rather than compacting. This feature
142 was introduced in version 3.0.
143 -R Run in rolling mode; squatter runs as a daemon listening to a
145 sync log channel and continuously incrementally indexing mail‐
146 boxes. See also -d and -n. This feature was introduced in ver‐
147 sion 3.0.
148 -r Recursively create indexes for all sub-mailboxes of the user,
150 mailboxes or mailbox prefixes given as arguments.
151 -S seconds
153 After processing each mailbox, sleep for "seconds" before con‐
154 tinuing. Can be used to provide some load balancing. Accepts
155 fractional amounts. This feature was introduced in version 3.0.
156 -T directory
158 When indexing, work on a temporary copy of the search engine
159 databases in directory. That directory would typically be on
160 some very fast filesystem, like an SSD or tmpfs. This option
161 may not work with all search engines, but it's only effect is to
162 speed up initial indexing. This feature was introduced in ver‐
163 sion 3.0.
164 -t srctier...
166 In compact mode, the source tier(s) for the compacted indices.
167 At least one source tier must be specified in compact mode.
168 This feature was introduced in version 3.0.
169 -u Extra options refer to usernames (e.g. foo@bar.com) rather than
171 mailbox names. This feature was introduced in version 3.0.
172 -v Increase the verbosity of progress/status messages.
174 -X Reindex all the messages before compacting. This mode reads all
176 the lists of messages indexed by the listed tiers, and
177 re-indexes them into a temporary database before compacting that
178 into place.
179 -z desttier
181 In compact mode, the destination tier for the compacted indices.
182 This must be specified in compact mode. This feature was intro‐
183 duced in version 3.0.
184
186 squatter is typically deployed via entries in cyrus.conf(5), in either
187 the DAEMON or EVENTS sections.
188 For the older SQUAT search engine, which offers poor performance in
190 rolling mode (-R) we recommend triggering periodic runs via entries in
191 the EVENTS section, as follows:
192 Sample entries from the EVENTS section of cyrus.conf(5) for periodic
194 squatter runs:
195 EVENTS {
197 # reindex changed mailboxes (fulltext) approximately every three hours
198 squatter1 cmd="/usr/bin/ionice -c idle /usr/lib/cyrus/bin/squatter -i" period=180
199 # reindex all mailboxes (fulltext) daily
201 squattera cmd="/usr/lib/cyrus/bin/squatter" at=0117
202 }
203 For the newer Xapian search engine, and with sufficiently fast storage,
205 the rolling mode (-R) offers advantages. Use of rolling mode requires
206 that squatter be invoked in the DAEMON section.
207 Sample entries for the DAEMON section of cyrus.conf(5) for rolling
209 squatter operation:
210 DAEMON {
212 # run a rolling squatter using the default sync_log channel "squatter"
213 squatter cmd="squatter -R"
214 # run a rolling squatter using a specific sync_log channel
216 squatter cmd="squatter -R -n indexer"
217 }
218 NOTE:
220 When using the -R rolling mode, you MUST enable sync_log operation
221 in imapd.conf(5) via the sync_log: on setting, and MUST define a
222 sync_log channel via the sync_log_channels: setting. If also using
223 replication, you must either explicitly specify your replication
224 sync_log channel via the sync_log_channels directive with a name, or
225 specify the default empty name with "" (the two-character string
226 U+22 U+22). [Please see imapd.conf(5) for details].
227 NOTE:
229 When configuring rolling search indexing on a replica, one must con‐
230 sider whether sync_logs will be written at all. In this case,
231 please consider the setting sync_log_unsuppressable_channels to
232 ensure that the sync_log channel upon which one's squatter instance
233 depends will continue to be written. See imapd.conf(5) for details.
234 NOTE:
236 When using the Xapian search engine, you must define various set‐
237 tings in imapd.conf(5). Please read all relevant Xapian documenta‐
238 tion in this release before using Xapian.
239 [NB: More examples needed]
241
243 Support for additional search enginges was added in version 3.0.
244 The following command-line switches were added in version 3.0:
246 -R -u -d -O -F -A
248 The following command-line settings were added in version 3.0:
250 -S <seconds>, -T <directory>, -f <synclogfile>, -n <channel>, -t srctier..., -z desttier
252
254 /etc/imapd.conf, /etc/cyrus.conf
255
257 imapd.conf(5), cyrus.conf(5)
258
260 The Cyrus Team, Nic Bernstein (Onlight)
261
263 1993-2017, The Cyrus Team
2643.0.7 May 18, 2018 SQUATTER(8)