1SQUATTER(8)                       Cyrus IMAP                       SQUATTER(8)
2
3
4

NAME

6       squatter - Cyrus IMAP documentation
7
8       Create SQUAT indexes for mailboxes
9

SYNOPSIS

11       general:
12       squatter [ -C config-file ] [mode] [options] [source]
13
14       i.e.:
15       squatter [ -C config-file ] [-v]
16       squatter [ -C config-file ] [ -a ] [ -i ] [-N name] [-S seconds] [ -r ]  mailbox...
17       squatter [ -C config-file ] [ -a ] [ -i ] [-N name] [-S seconds] [ -r ]  -u user...
18       squatter [ -C config-file ] -R [ -n channel ] [ -d ]
19       squatter [ -C config-file ] -f synclogfile
20       squatter [ -C config-file ] -I file
21       squatter [ -C config-file ] -t srctier... -z desttier [ -F ] [ -T dir ] [ -X ] [ -o ]
22

DESCRIPTION

24       NOTE:
25          The  name "squatter" once referred both to the SQUAT indexing engine
26          and to the command used to create indexes.  Now that Cyrus  supports
27          more  than one index type -- SQUAT and Xapian, as of this writing --
28          the name "squatter" refers to the command used to control index cre‐
29          ation.  The terms "SQUAT" or "SQUAT index(es)" refers to the indexes
30          used by the older SQUAT indexing engine.  Post v3 the  search_engine
31          setting in imapd.conf determines which search engine is used.
32
33       squatter  creates a new text index for one or more IMAP mailboxes.  The
34       index is a unified index of all of the header and  body  text  of  each
35       message in a given mailbox.  This index is used to significantly reduce
36       IMAP SEARCH times on a mailbox.
37
38       By default, squatter creates an index of ALL messages in  the  mailbox,
39       not  just  those since the last time that it was run.  The -i option is
40       used to select incremental updates.  Any messages appended to the mail‐
41       box  after  squatter  is  run,  will  NOT be included in the index.  To
42       include new messages in the index, squatter must be run again.
43
44       In the first synopsis, squatter indexes all mailboxes.
45
46       In the second synopsis, squatter indexes the specified mailbox(es).
47
48       In the third synopsis, squatter indexes  the  specified  user(s)  mail‐
49       box(es).
50
51       For  any  of  those three source modes (default=all, mailbox, user) one
52       may optionally specify -r to recurse from the specified start, or -a to
53       limit   action   only   to   mailboxes  which  have  the  shared  /ven‐
54       dor/cmu/cyrus-imapd/squat annotation set to "true".
55
56       In the fourth synopsis, squatter runs in rolling mode.   In  this  mode
57       squatter  backgrounds  itself  and runs as a daemon (unless -d is set),
58       listening to a sync log channel chosen using the -n option, and set  up
59       using  the sync_log_channels setting in imapd.conf(5).  Very soon after
60       messages are delivered or uploaded to mailboxes squatter will incremen‐
61       tally index the affected mailbox (see notes, below).
62
63       In  the  fifth synopsis, squatter reads a single sync log file and per‐
64       forms incremental indexing on the mailbox(es) listed therein.  This  is
65       sometimes useful for cleaning up after problems with rolling mode.
66
67       In  the  sixth  synopsis,  squatter  reads  file containing mailbox uid
68       tuples and performs indexing on the specified messages.
69
70       In the seventh synopsis, squatter will compact indices from  srctier(s)
71       to  desttier,  optionally reindexing (-X) or filtering expunged records
72       (-F) in the process.  The optional -T flag may be  used  to  specify  a
73       directory  to  use  for  temporary  files.   The -o flag may be used to
74       direct that a single index be copied, rather than compressed, from src‐
75       tier to desttier.
76
77       For  all  modes,  the  -S  option may be specified, causing squatter to
78       pause seconds seconds after each mailbox, to smooth loads.
79
80       NOTE:
81          Incremental updates are  very  inefficient  with  the  SQUAT  search
82          engine.   If  using SQUAT for large and active mailboxes, you should
83          run squatter periodically as an EVENT in cyrus.conf(5).
84
85       NOTE:
86          Messages and mailboxes that have  not  been  indexed  CAN  still  be
87          SEARCHed, just not as quickly as those with an index.
88
89       squatter  reads its configuration options out of the imapd.conf(5) file
90       unless specified otherwise by -C.
91

OPTIONS

93       -C config-file
94              Use the specified configuration file config-file rather than the
95              default imapd.conf(5).
96
97       -a     Only  create  indexes  for mailboxes which have the shared /ven‐
98              dor/cmu/cyrus-imapd/squat annotation set to "true".
99
100              The value of  the  /vendor/cmu/cyrus-imapd/squat  annotation  is
101              inherited  by  all  children  of the given mailbox, so an entire
102              mailbox tree can be indexed (or not indexed) by setting a single
103              annotation  on  the root of that tree with a value of "true" (or
104              "false").    If   a   mailbox   does   not    have    a    /ven‐
105              dor/cmu/cyrus-imapd/squat  annotation  set  on  it  (or does not
106              inherit one), then the mailbox is not indexed. In  other  words,
107              the implicit value of /vendor/cmu/cyrus-imapd/squat is "false".
108
109       -d     In  rolling  mode,  don't background and do emit log messages on
110              standard error.  Useful for debugging.  This feature was  intro‐
111              duced in version 3.0.
112
113       -F     In  compact  mode, filter the resulting database to only include
114              messages which are  not  expunged  in  mailboxes  with  existing
115              name/uidvalidity.  This feature was introduced in version 3.0.
116
117       -f synclogfile
118              Read  the  synclogfile and incrementally index all the mailboxes
119              listed therein, then exit.  This feature was introduced in  ver‐
120              sion 3.0.
121
122       -h     Display this usage information.
123
124       -I file
125              Read  from file and index individual messages described by mail‐
126              box/uid tuples contained therein.
127
128       -i     Incremental updates where indexes already exist.
129
130       -N name
131              Only index mailboxes beginning with name while iterating through
132              the mailbox list derived from other options.
133
134       -n channel
135              In  rolling  mode, specify the name of the sync log channel that
136              squatter will listen to.  The default is "squatter".  This chan‐
137              nel  must  be  defined in imapd.conf(5) before being used.  This
138              feature was introduced in version 3.0.
139
140       -o     In compact mode, if only one source database is  selected,  just
141              copy it to the destination rather than compacting.  This feature
142              was introduced in version 3.0.
143
144       -R     Run in rolling mode; squatter runs as a daemon  listening  to  a
145              sync  log  channel and continuously incrementally indexing mail‐
146              boxes.  See also -d and -n.  This feature was introduced in ver‐
147              sion 3.0.
148
149       -r     Recursively  create  indexes  for all sub-mailboxes of the user,
150              mailboxes or mailbox prefixes given as arguments.
151
152       -S seconds
153              After processing each mailbox, sleep for "seconds"  before  con‐
154              tinuing.  Can  be  used to provide some load balancing.  Accepts
155              fractional amounts. This feature was introduced in version 3.0.
156
157       -T directory
158              When indexing, work on a temporary copy  of  the  search  engine
159              databases  in  directory.   That directory would typically be on
160              some very fast filesystem, like an SSD or  tmpfs.   This  option
161              may not work with all search engines, but it's only effect is to
162              speed up initial indexing.  This feature was introduced in  ver‐
163              sion 3.0.
164
165       -t srctier...
166              In  compact  mode, the source tier(s) for the compacted indices.
167              At least one source tier must  be  specified  in  compact  mode.
168              This feature was introduced in version 3.0.
169
170       -u     Extra  options refer to usernames (e.g. foo@bar.com) rather than
171              mailbox names.  This feature was introduced in version 3.0.
172
173       -v     Increase the verbosity of progress/status messages.
174
175       -X     Reindex all the messages before compacting.  This mode reads all
176              the   lists  of  messages  indexed  by  the  listed  tiers,  and
177              re-indexes them into a temporary database before compacting that
178              into place.
179
180       -z desttier
181              In compact mode, the destination tier for the compacted indices.
182              This must be specified in compact mode.  This feature was intro‐
183              duced in version 3.0.
184

EXAMPLES

186       squatter  is typically deployed via entries in cyrus.conf(5), in either
187       the DAEMON or EVENTS sections.
188
189       For the older SQUAT search engine, which  offers  poor  performance  in
190       rolling  mode (-R) we recommend triggering periodic runs via entries in
191       the EVENTS section, as follows:
192
193       Sample entries from the EVENTS section of  cyrus.conf(5)  for  periodic
194       squatter runs:
195
196          EVENTS {
197              # reindex changed mailboxes (fulltext) approximately every three hours
198              squatter1   cmd="/usr/bin/ionice -c idle /usr/lib/cyrus/bin/squatter -i" period=180
199
200              # reindex all mailboxes (fulltext) daily
201              squattera   cmd="/usr/lib/cyrus/bin/squatter" at=0117
202          }
203
204       For the newer Xapian search engine, and with sufficiently fast storage,
205       the rolling mode (-R) offers advantages.  Use of rolling mode  requires
206       that squatter be invoked in the DAEMON section.
207
208       Sample  entries  for  the  DAEMON  section of cyrus.conf(5) for rolling
209       squatter operation:
210
211          DAEMON {
212            # run a rolling squatter using the default sync_log channel "squatter"
213            squatter cmd="squatter -R"
214
215            # run a rolling squatter using a specific sync_log channel
216            squatter cmd="squatter -R -n indexer"
217          }
218
219       NOTE:
220          When using the -R rolling mode, you MUST enable  sync_log  operation
221          in  imapd.conf(5)  via  the  sync_log: on setting, and MUST define a
222          sync_log channel via the sync_log_channels: setting.  If also  using
223          replication,  you  must  either  explicitly specify your replication
224          sync_log channel via the sync_log_channels directive with a name, or
225          specify  the  default  empty  name with "" (the two-character string
226          U+22 U+22).  [Please see imapd.conf(5) for details].
227
228       NOTE:
229          When configuring rolling search indexing on a replica, one must con‐
230          sider  whether  sync_logs  will  be  written  at all.  In this case,
231          please  consider  the  setting  sync_log_unsuppressable_channels  to
232          ensure  that the sync_log channel upon which one's squatter instance
233          depends will continue to be written.  See imapd.conf(5) for details.
234
235       NOTE:
236          When using the Xapian search engine, you must  define  various  set‐
237          tings  in imapd.conf(5).  Please read all relevant Xapian documenta‐
238          tion in this release before using Xapian.
239
240       [NB: More examples needed]
241

HISTORY

243       Support for additional search enginges was added in version 3.0.
244
245       The following command-line switches were added in version 3.0:
246
247          -R -u -d -O -F -A
248
249       The following command-line settings were added in version 3.0:
250
251          -S <seconds>, -T <directory>, -f <synclogfile>, -n <channel>, -t srctier..., -z desttier
252

FILES

254       /etc/imapd.conf, /etc/cyrus.conf
255

SEE ALSO

257       imapd.conf(5), cyrus.conf(5)
258

AUTHOR

260       The Cyrus Team, Nic Bernstein (Onlight)
261
263       1993-2017, The Cyrus Team
264
265
266
267
2683.0.10                           May 27, 2019                      SQUATTER(8)
Impressum