1SQUATTER(8)                       Cyrus IMAP                       SQUATTER(8)
2
3
4

NAME

6       squatter - Cyrus IMAP documentation
7
8       Create SQUAT and Xapian indexes for mailboxes
9

SYNOPSIS

11       general:
12       squatter [ -C config-file ] [mode] [options] [source]
13
14       i.e.:
15       squatter [ -C config-file ] [ -v ] [ -a ] [ -S seconds ] [ -Z ]
16       squatter [ -C config-file ] [ -v ] [ -a ] [ -i ] [ -N name ] [ -S seconds ] [ -r ] [ -Z ] mailbox...
17       squatter [ -C config-file ] [ -v ] [ -a ] [ -i ] [ -N name ] [ -S seconds ] [ -r ] [ -Z ] -u user...
18       squatter [ -C config-file ] [ -v ] [ -a ] -R [ -n channel ] [ -d ] [ -S seconds ] [ -Z ]
19       squatter [ -C config-file ] [ -v ] [ -a ] -f synclogfile [ -S seconds ] [ -Z ]
20       squatter [ -C config-file ] [ -v ] -t srctier(s)... -z desttier [ -F ] [ -U ] [ -T dir ] [ -X ] [ -o ] [ -S seconds ] [ -u user... ]
21

DESCRIPTION

23       NOTE:
24          The  name "squatter" once referred both to the SQUAT indexing engine
25          and to the command used to create indexes.  Now that Cyrus  supports
26          more  than one index type -- SQUAT and Xapian, as of this writing --
27          the name "squatter" refers to the command used to control index cre‐
28          ation.  The terms "SQUAT" or "SQUAT index(es)" refers to the indexes
29          used by the older SQUAT indexing engine.  Post v3 the  search_engine
30          setting in imapd.conf determines which search engine is used.
31
32       squatter  creates a new text index for one or more IMAP mailboxes.  The
33       index is a unified index of all of the header and  body  text  of  each
34       message in a given mailbox.  This index is used to significantly reduce
35       IMAP SEARCH times on a mailbox.
36
37       mode is one of indexer, search, rolling, synclog, compact or audit.
38
39       By default, squatter creates an index of ALL messages in  the  mailbox,
40       not  just  those since the last time that it was run.  The -i option is
41       used to select incremental updates.  Any messages appended to the mail‐
42       box  after  squatter is run, will NOT be included in the index.  To in‐
43       clude new messages in the index, squatter must be run again,  or  on  a
44       regular   basis  via  crontab,  an  entry  in  the  EVENTS  section  of
45       cyrus.conf(5) or use rolling mode (-R).
46
47       In the first synopsis, squatter indexes all mailboxes.
48
49       In the second synopsis, squatter  indexes  the  specified  mailbox(es).
50       The mailboxes are space-separated.
51
52       In  the  third  synopsis,  squatter indexes the specified user(s) mail‐
53       box(es).
54
55       For the latter two index modes (mailbox, user) one may optionally spec‐
56       ify  -r to recurse from the specified start, or -a to limit action only
57       to mailboxes which have the shared /vendor/cmu/cyrus-imapd/squat  anno‐
58       tation set to "true".
59
60       In  the  fourth  synopsis, squatter runs in rolling mode.  In this mode
61       squatter backgrounds itself and runs as a daemon (unless  -d  is  set),
62       listening  to a sync log channel chosen using the -n option, and set up
63       using the sync_log_channels setting in imapd.conf(5).  Very soon  after
64       messages are delivered or uploaded to mailboxes squatter will incremen‐
65       tally index the affected mailbox (see notes, below).
66
67       In the fifth synopsis, squatter reads a single sync log file  and  per‐
68       forms  incremental indexing on the mailbox(es) listed therein.  This is
69       sometimes useful for cleaning up after problems with rolling mode.
70
71       In the sixth synopsis, squatter will compact indices from srctier(s) to
72       desttier, optionally reindexing (-X) or filtering expunged records (-F)
73       in the process.  The optional -T flag may be used to specify  a  direc‐
74       tory  to  use  for  temporary files.  These files are eventually copied
75       with rsync -a and then removed by rm.  rsync can increase the load  av‐
76       erage  of  the  system,  especially  when the temporary directory is on
77       tmpfs.  To throttle  rsync  it  is  possible  to  modify  the  call  in
78       imap/search_xapian.c and pass -\-bwlimit=<number> as further parameter.
79       The -o flag may be used to direct that a single index be copied, rather
80       than  compacted,  from srctier to desttier.  The -u flag may be used to
81       restrict operation to the specified user(s).
82
83       For all modes, the -S option may  be  specified,  causing  squatter  to
84       pause seconds seconds after each mailbox, to smooth loads.
85
86       When  using  the  Xapian engine the -Z option may be specified, for the
87       indexing modes.  This tells squatter to consult the  Xapian  internally
88       indexed  GUIDs,  rather  than  relying  on  what's  stored in cyrus.in‐
89       dexed.db, allowing for recovery from  broken  cyrus.indexed.db  at  the
90       sacrifice of efficiency.
91
92       NOTE:
93          Incremental  updates  are very inefficient with the SQUAT search en‐
94          gine.  If using SQUAT for large and active mailboxes, you should run
95          squatter periodically as an EVENT in cyrus.conf(5).
96
97       NOTE:
98          Messages  and  mailboxes  that  have  not  been indexed CAN still be
99          SEARCHed, just not as quickly as those with an index.
100
101       squatter reads its configuration options out of the imapd.conf(5)  file
102       unless specified otherwise by -C.
103

OPTIONS

105       -C config-file
106              Use the specified configuration file config-file rather than the
107              default imapd.conf(5).
108
109       -a     Only create indexes for mailboxes which have  the  shared  /ven‐
110              dor/cmu/cyrus-imapd/squat annotation set to "true".
111
112              The value of the /vendor/cmu/cyrus-imapd/squat annotation is in‐
113              herited by all children of the given mailbox, so an entire mail‐
114              box tree can be indexed (or not indexed) by setting a single an‐
115              notation on the root of that tree with a  value  of  "true"  (or
116              "false").     If    a    mailbox   does   not   have   a   /ven‐
117              dor/cmu/cyrus-imapd/squat annotation set on it (or does not  in‐
118              herit one), then the mailbox is not indexed. In other words, the
119              implicit value of /vendor/cmu/cyrus-imapd/squat is "false".
120
121       -A     Missing documentation, starts audit mode.  This feature is  only
122              available on the master branch.
123
124       -d     In  rolling  mode,  don't background and do emit log messages on
125              standard error.  Useful for debugging.  This feature was  intro‐
126              duced in version 3.0.
127
128       -F     In  compact  mode, filter the resulting database to only include
129              messages which are  not  expunged  in  mailboxes  with  existing
130              name/uidvalidity.  This feature was introduced in version 3.0.
131
132       -f synclogfile
133              Read  the  synclogfile and incrementally index all the mailboxes
134              listed therein, then exit.  This feature was introduced in  ver‐
135              sion 3.0.
136
137       -h     Display this usage information.
138
139       -i     Incremental updates where indexes already exist.
140
141       -N name
142              Only index mailboxes beginning with name while iterating through
143              the mailbox list derived from other options.
144
145       -n channel
146              In rolling mode, specify the name of the sync log  channel  that
147              squatter will listen to.  The default is "squatter".  This chan‐
148              nel must be defined in imapd.conf(5) before  being  used.   This
149              feature was introduced in version 3.0.
150
151       -o     In  compact  mode, if only one source database is selected, just
152              copy it to the destination rather than compacting.  This feature
153              was introduced in version 3.0.
154
155       -P     Missing documentation, reindex parts mode.  This feature is only
156              available on the master branch.
157
158       -R     Run in rolling mode; squatter runs as a daemon  listening  to  a
159              sync  log  channel and continuously incrementally indexing mail‐
160              boxes.  See also -d and -n.  This feature was introduced in ver‐
161              sion 3.0.
162
163       -r     Recursively  create  indexes  for all sub-mailboxes of the user,
164              mailboxes or mailbox prefixes given as arguments.
165
166       -S seconds
167              After processing each mailbox, sleep for "seconds"  before  con‐
168              tinuing.  Can  be  used to provide some load balancing.  Accepts
169              fractional amounts. This feature was introduced in version 3.0.
170
171       -T directory
172              When indexing, work on a temporary copy  of  the  search  engine
173              databases  in  directory.   That directory would typically be on
174              some very fast filesystem, like an SSD or  tmpfs.   This  option
175              may not work with all search engines, but it's only effect is to
176              speed up initial indexing.  Xapian only.  This feature  was  in‐
177              troduced in version 3.0.
178
179       -t srctier...
180              In compact mode, the comma-separated source tier(s) for the com‐
181              pacted indices.  At least one source tier must be  specified  in
182              compact mode.  Xapian only.  This feature was introduced in ver‐
183              sion 3.0.
184
185       -u     Extra options refer to usernames (e.g. foo@bar.com) rather  than
186              mailbox names.  Usernames are space-separated.  This feature was
187              introduced in version 3.0.
188
189       -U     In compact mode, only  compact  if  re-indexing.   Xapian  only.
190              This feature is only available on the master branch.
191
192       -v     Increase  the  verbosity of progress/status messages.  Sometimes
193              additional messages are emitted on the terminal with this option
194              and  the messages are unconditionally sent to syslog.  Sometimes
195              messages are sent to syslog, only if -v is provided.  In rolling
196              and synclog modes, -vv sends even more messages to syslog.
197
198       -X     Reindex all the messages before compacting.  This mode reads all
199              the lists of messages indexed by the listed  tiers,  and  re-in‐
200              dexes them into a temporary database before compacting that into
201              place.  Xapian only.  This feature  was  introduced  in  version
202              3.0.
203
204       -z desttier
205              In compact mode, the destination tier for the compacted indices.
206              This must be specified in compact mode.  Xapian only.  This fea‐
207              ture was introduced in version 3.0.
208
209       -Z     When  indexing  messages, use the Xapian internal cyrusid rather
210              than referencing the ranges of already indexed messages to  know
211              if  a  particular  message is indexed.  Useful if the ranges get
212              out of sync with the actual messages (e.g. if files  on  a  tier
213              are  lost)  Xapian  only.  This feature is only available on the
214              master branch.
215

EXAMPLES

217       squatter is typically deployed via entries in cyrus.conf(5), in  either
218       the DAEMON or EVENTS sections.
219
220       For  the  older  SQUAT  search engine, which offers poor performance in
221       rolling mode (-R) we recommend triggering periodic runs via entries  in
222       the EVENTS section, as follows:
223
224       Sample  entries  from  the EVENTS section of cyrus.conf(5) for periodic
225       squatter runs:
226
227          EVENTS {
228              # reindex changed mailboxes (fulltext) approximately every three hours
229              squatter1   cmd="/usr/bin/ionice -c idle /usr/lib/cyrus/bin/squatter -i" period=180
230
231              # reindex all mailboxes (fulltext) daily
232              squattera   cmd="/usr/lib/cyrus/bin/squatter" at=0117
233          }
234
235       For the newer Xapian search engine, and with sufficiently fast storage,
236       the  rolling mode (-R) offers advantages.  Use of rolling mode requires
237       that squatter be invoked in the DAEMON section.
238
239       Sample entries for the DAEMON  section  of  cyrus.conf(5)  for  rolling
240       squatter operation:
241
242          DAEMON {
243            # run a rolling squatter using the default sync_log channel "squatter"
244            squatter cmd="squatter -R"
245
246            # run a rolling squatter using a specific sync_log channel
247            squatter cmd="squatter -R -n indexer"
248          }
249
250       NOTE:
251          When  using  the -R rolling mode, you MUST enable sync_log operation
252          in imapd.conf(5) via the sync_log: on setting,  and  MUST  define  a
253          sync_log  channel via the sync_log_channels: setting.  If also using
254          replication, you must either  explicitly  specify  your  replication
255          sync_log channel via the sync_log_channels directive with a name, or
256          specify the default empty name with  ""  (the  two-character  string
257          U+22 U+22).  [Please see imapd.conf(5) for details].
258
259       NOTE:
260          When configuring rolling search indexing on a replica, one must con‐
261          sider whether sync_logs will be  written  at  all.   In  this  case,
262          please  consider the setting sync_log_unsuppressable_channels to en‐
263          sure that the sync_log channel upon which  one's  squatter  instance
264          depends will continue to be written.  See imapd.conf(5) for details.
265
266       NOTE:
267          When  using  the  Xapian search engine, you must define various set‐
268          tings in imapd.conf(5).  Please read all relevant Xapian  documenta‐
269          tion in this release before using Xapian.
270
271       [NB: More examples needed]
272

HISTORY

274       Support for additional search engines was added in version 3.0.
275
276       The following command-line switches were added in version 3.0:
277
278          -F -R -X -d -f -o -u
279
280       The following command-line settings were added in version 3.0:
281
282          -S <seconds>, -T <directory>, -f <synclogfile>, -n <channel>, -t srctier..., -z desttier
283

FILES

285       /etc/imapd.conf, /etc/cyrus.conf
286

SEE ALSO

288       imapd.conf(5), cyrus.conf(5)
289

AUTHOR

291       The Cyrus Team, Nic Bernstein (Onlight)
292
294       1993-2018, The Cyrus Team
295
296
297
298
2993.2.4                          October 05, 2020                    SQUATTER(8)
Impressum