1SQUATTER(8)                       Cyrus IMAP                       SQUATTER(8)
2
3
4

NAME

6       squatter - Cyrus IMAP documentation
7
8       Create SQUAT and Xapian indexes for mailboxes
9

SYNOPSIS

11       general:
12       squatter [ -C config-file ] [mode] [options] [source]
13
14       i.e.:
15       squatter [ -C config-file ] [ -v ] [ -a ] [ -S seconds ]
16       squatter [ -C config-file ] [ -v ] [ -a ] [ -i ] [ -N name ] [ -S seconds ] [ -r ]  mailbox...
17       squatter [ -C config-file ] [ -v ] [ -a ] [ -i ] [ -N name ] [ -S seconds ] [ -r ]  -u user...
18       squatter [ -C config-file ] [ -v ] [ -a ] -R [ -n channel ] [ -d ] [ -S seconds ]
19       squatter [ -C config-file ] [ -v ] [ -a ] -f synclogfile [ -S seconds ]
20       squatter [ -C config-file ] [ -v ] -I file
21       squatter [ -C config-file ] [ -v ] -t srctier(s)... -z desttier [ -F ] [ -T dir ] [ -X ] [ -o ]  [ -S seconds ] [ -u user... ]
22

DESCRIPTION

24       NOTE:
25          The  name "squatter" once referred both to the SQUAT indexing engine
26          and to the command used to create indexes.  Now that Cyrus  supports
27          more  than one index type -- SQUAT and Xapian, as of this writing --
28          the name "squatter" refers to the command used to control index cre‐
29          ation.  The terms "SQUAT" or "SQUAT index(es)" refers to the indexes
30          used by the older SQUAT indexing engine.  Post v3 the  search_engine
31          setting in imapd.conf determines which search engine is used.
32
33       squatter  creates a new text index for one or more IMAP mailboxes.  The
34       index is a unified index of all of the header and  body  text  of  each
35       message in a given mailbox.  This index is used to significantly reduce
36       IMAP SEARCH times on a mailbox.
37
38       mode is one of indexer, indexfrom (-I),  search,  rolling,  synclog  or
39       compact.
40
41       By  default,  squatter creates an index of ALL messages in the mailbox,
42       not just those since the last time that it was run.  The -i  option  is
43       used to select incremental updates.  Any messages appended to the mail‐
44       box after squatter is run, will NOT  be  included  in  the  index.   To
45       include  new messages in the index, squatter must be run again, or on a
46       regular  basis  via  crontab,  an  entry  in  the  EVENTS  section   of
47       cyrus.conf(5) or use rolling mode (-R).
48
49       In the first synopsis, squatter indexes all mailboxes.
50
51       In  the  second  synopsis,  squatter indexes the specified mailbox(es).
52       The mailboxes are space-separated.
53
54       In the third synopsis, squatter indexes  the  specified  user(s)  mail‐
55       box(es).
56
57       For the latter two index modes (mailbox, user) one may optionally spec‐
58       ify -r to recurse from the specified start, or -a to limit action  only
59       to  mailboxes which have the shared /vendor/cmu/cyrus-imapd/squat anno‐
60       tation set to "true".
61
62       In the fourth synopsis, squatter runs in rolling mode.   In  this  mode
63       squatter  backgrounds  itself  and runs as a daemon (unless -d is set),
64       listening to a sync log channel chosen using the -n option, and set  up
65       using  the sync_log_channels setting in imapd.conf(5).  Very soon after
66       messages are delivered or uploaded to mailboxes squatter will incremen‐
67       tally index the affected mailbox (see notes, below).
68
69       In  the  fifth synopsis, squatter reads a single sync log file and per‐
70       forms incremental indexing on the mailbox(es) listed therein.  This  is
71       sometimes useful for cleaning up after problems with rolling mode.
72
73       In  the  sixth  synopsis,  squatter  reads  file containing mailbox uid
74       tuples and performs indexing on the specified messages.
75
76       In the seventh synopsis, squatter will compact indices from  srctier(s)
77       to  desttier,  optionally reindexing (-X) or filtering expunged records
78       (-F) in the process.  The optional -T flag may be  used  to  specify  a
79       directory  to  use  for  temporary  files.   The -o flag may be used to
80       direct that a single index be copied, rather than compacted, from  src‐
81       tier to desttier.  The -u flag may be used to restrict operation to the
82       specified user(s).
83
84       For all modes, the -S option may  be  specified,  causing  squatter  to
85       pause seconds seconds after each mailbox, to smooth loads.
86
87       NOTE:
88          Incremental  updates  are  very  inefficient  with  the SQUAT search
89          engine.  If using SQUAT for large and active mailboxes,  you  should
90          run squatter periodically as an EVENT in cyrus.conf(5).
91
92       NOTE:
93          Messages  and  mailboxes  that  have  not  been indexed CAN still be
94          SEARCHed, just not as quickly as those with an index.
95
96       squatter reads its configuration options out of the imapd.conf(5)  file
97       unless specified otherwise by -C.
98

OPTIONS

100       -C config-file
101              Use the specified configuration file config-file rather than the
102              default imapd.conf(5).
103
104       -a     Only create indexes for mailboxes which have  the  shared  /ven‐
105              dor/cmu/cyrus-imapd/squat annotation set to "true".
106
107              The  value  of  the  /vendor/cmu/cyrus-imapd/squat annotation is
108              inherited by all children of the given  mailbox,  so  an  entire
109              mailbox tree can be indexed (or not indexed) by setting a single
110              annotation on the root of that tree with a value of  "true"  (or
111              "false").     If    a    mailbox   does   not   have   a   /ven‐
112              dor/cmu/cyrus-imapd/squat annotation set  on  it  (or  does  not
113              inherit  one),  then the mailbox is not indexed. In other words,
114              the implicit value of /vendor/cmu/cyrus-imapd/squat is "false".
115
116       -d     In rolling mode, don't background and do emit  log  messages  on
117              standard  error.  Useful for debugging.  This feature was intro‐
118              duced in version 3.0.
119
120       -F     In compact mode, filter the resulting database to  only  include
121              messages  which  are  not  expunged  in  mailboxes with existing
122              name/uidvalidity.  This feature was introduced in version 3.0.
123
124       -f synclogfile
125              Read the synclogfile and incrementally index all  the  mailboxes
126              listed  therein, then exit.  This feature was introduced in ver‐
127              sion 3.0.
128
129       -h     Display this usage information.
130
131       -I file
132              Read from file and index individual messages described by  mail‐
133              box/uid tuples contained therein.
134
135       -i     Incremental updates where indexes already exist.
136
137       -N name
138              Only index mailboxes beginning with name while iterating through
139              the mailbox list derived from other options.
140
141       -n channel
142              In rolling mode, specify the name of the sync log  channel  that
143              squatter will listen to.  The default is "squatter".  This chan‐
144              nel must be defined in imapd.conf(5) before  being  used.   This
145              feature was introduced in version 3.0.
146
147       -o     In  compact  mode, if only one source database is selected, just
148              copy it to the destination rather than compacting.  This feature
149              was introduced in version 3.0.
150
151       -R     Run  in  rolling  mode; squatter runs as a daemon listening to a
152              sync log channel and continuously incrementally  indexing  mail‐
153              boxes.  See also -d and -n.  This feature was introduced in ver‐
154              sion 3.0.
155
156       -r     Recursively create indexes for all sub-mailboxes  of  the  user,
157              mailboxes or mailbox prefixes given as arguments.
158
159       -S seconds
160              After  processing  each mailbox, sleep for "seconds" before con‐
161              tinuing. Can be used to provide some  load  balancing.   Accepts
162              fractional amounts. This feature was introduced in version 3.0.
163
164       -T directory
165              When  indexing,  work  on  a temporary copy of the search engine
166              databases in directory.  That directory would  typically  be  on
167              some  very  fast  filesystem, like an SSD or tmpfs.  This option
168              may not work with all search engines, but it's only effect is to
169              speed  up  initial  indexing.   Xapian  only.   This feature was
170              introduced in version 3.0.
171
172       -t srctier...
173              In compact mode, the comma separated source tier(s) for the com‐
174              pacted  indices.   At least one source tier must be specified in
175              compact mode.  Xapian only.  This feature was introduced in ver‐
176              sion 3.0.
177
178       -u     Extra  options refer to usernames (e.g. foo@bar.com) rather than
179              mailbox names.  Usernames are space-separated.  This feature was
180              introduced in version 3.0.
181
182       -v     Increase  the  verbosity of progress/status messages.  Sometimes
183              additional messages are emitted on the terminal with this option
184              and  the messages are unconditionally sent to syslog.  Sometimes
185              messages are sent to syslog, only if -v is provided.  In rolling
186              and synclog modes, -vv sends even more messages to syslog.
187
188       -X     Reindex all the messages before compacting.  This mode reads all
189              the  lists  of  messages  indexed  by  the  listed  tiers,   and
190              re-indexes them into a temporary database before compacting that
191              into place.  Xapian only.  This feature was introduced  in  ver‐
192              sion 3.0.
193
194       -z desttier
195              In compact mode, the destination tier for the compacted indices.
196              This must be specified in compact mode.  Xapian only.  This fea‐
197              ture was introduced in version 3.0.
198

EXAMPLES

200       squatter  is typically deployed via entries in cyrus.conf(5), in either
201       the DAEMON or EVENTS sections.
202
203       For the older SQUAT search engine, which  offers  poor  performance  in
204       rolling  mode (-R) we recommend triggering periodic runs via entries in
205       the EVENTS section, as follows:
206
207       Sample entries from the EVENTS section of  cyrus.conf(5)  for  periodic
208       squatter runs:
209
210          EVENTS {
211              # reindex changed mailboxes (fulltext) approximately every three hours
212              squatter1   cmd="/usr/bin/ionice -c idle /usr/lib/cyrus/bin/squatter -i" period=180
213
214              # reindex all mailboxes (fulltext) daily
215              squattera   cmd="/usr/lib/cyrus/bin/squatter" at=0117
216          }
217
218       For the newer Xapian search engine, and with sufficiently fast storage,
219       the rolling mode (-R) offers advantages.  Use of rolling mode  requires
220       that squatter be invoked in the DAEMON section.
221
222       Sample  entries  for  the  DAEMON  section of cyrus.conf(5) for rolling
223       squatter operation:
224
225          DAEMON {
226            # run a rolling squatter using the default sync_log channel "squatter"
227            squatter cmd="squatter -R"
228
229            # run a rolling squatter using a specific sync_log channel
230            squatter cmd="squatter -R -n indexer"
231          }
232
233       NOTE:
234          When using the -R rolling mode, you MUST enable  sync_log  operation
235          in  imapd.conf(5)  via  the  sync_log: on setting, and MUST define a
236          sync_log channel via the sync_log_channels: setting.  If also  using
237          replication,  you  must  either  explicitly specify your replication
238          sync_log channel via the sync_log_channels directive with a name, or
239          specify  the  default  empty  name with "" (the two-character string
240          U+22 U+22).  [Please see imapd.conf(5) for details].
241
242       NOTE:
243          When configuring rolling search indexing on a replica, one must con‐
244          sider  whether  sync_logs  will  be  written  at all.  In this case,
245          please  consider  the  setting  sync_log_unsuppressable_channels  to
246          ensure  that the sync_log channel upon which one's squatter instance
247          depends will continue to be written.  See imapd.conf(5) for details.
248
249       NOTE:
250          When using the Xapian search engine, you must  define  various  set‐
251          tings  in imapd.conf(5).  Please read all relevant Xapian documenta‐
252          tion in this release before using Xapian.
253
254       [NB: More examples needed]
255

HISTORY

257       Support for additional search engines was added in version 3.0.
258
259       The following command-line switches were added in version 3.0:
260
261          -F -R -X -d -f -o -u
262
263       The following command-line settings were added in version 3.0:
264
265          -S <seconds>, -T <directory>, -f <synclogfile>, -n <channel>, -t srctier..., -z desttier
266

FILES

268       /etc/imapd.conf, /etc/cyrus.conf
269

SEE ALSO

271       imapd.conf(5), cyrus.conf(5)
272

AUTHOR

274       The Cyrus Team, Nic Bernstein (Onlight)
275
277       1993-2017, The Cyrus Team
278
279
280
281
2823.0.13                         December 16, 2019                   SQUATTER(8)
Impressum