public-inbox-extindex(1)

1PUBLIC-INBOX-EXTINDEX(1)   public-inbox user manual   PUBLIC-INBOX-EXTINDEX(1)
2
3
4

NAME

6       public-inbox-extindex - create and update external search indices
7

SYNOPSIS

9       public-inbox-extindex [OPTIONS] EXTINDEX_DIR INBOX_DIR...
10
11       public-inbox-extindex [OPTIONS] [EXTINDEX_DIR] --all
12

DESCRIPTION

14       public-inbox-extindex creates and updates an external search and
15       overview database used by the read-only public-inbox PSGI (HTTP), NNTP,
16       and IMAP interfaces.  This requires either the Search::Xapian XS
17       bindings OR the Xapian SWIG bindings, along with DBD::SQLite and DBI
18       Perl modules.
19

OPTIONS

21       -j JOBS
22       --jobs=JOBS
23       --no-fsync
24       --dangerous
25       --rethread
26       --max-size SIZE
27       --batch-size SIZE
28           These switches behave as they do for public-inbox-index(1)
29
30       --all
31           Index all "publicinbox" entries in "PI_CONFIG".
32
33           "publicinbox" entries indexed by "public-inbox-extindex" can have
34           full Xapian searching abilities with the per-"publicinbox"
35           "indexlevel" set to "basic" and their respective Xapian ("xap15" or
36           "xapian15") directories removed.  For multiple public-inboxes where
37           cross-posting is common, this allows significant space savings on
38           Xapian indices.
39
40       --gc
41           Perform garbage collection instead of indexing.  Use this if
42           inboxes are removed from the extindex, or if messages are purged or
43           removed from some inboxes.
44
45       --reindex
46           Forces a re-index of all messages in the extindex.  This can be
47           used for in-place upgrades and bugfixes while read-only server
48           processes are utilizing the index.  Keep in mind this roughly
49           doubles the size of the already-large Xapian database.
50
51           The extindex locks will be released roughly every 10s to allow
52           public-inbox-mda(1) and public-inbox-watch(1) processes to write to
53           the extindex.
54
55       --fast
56           Used with "--reindex", it will only look for new and stale entries
57           and not touch already-indexed messages.
58

FILES

60       public-inbox-extindex-format(5)
61

CONFIGURATION

63       public-inbox-extindex does not write to the public-inbox-config(5)
64       file, it must be entered manually.  The extindex name of "all" is a
65       special case which corresponds to indexing "--all" inboxes.  An example
66       for "--all" is as follows:
67
68               [extindex "all"]
69                       topdir = /path/to/extindex_dir
70                       url = all
71                       coderepo = foo
72                       coderepo = bar
73
74       Putting an "extindex" entry in the config allows PublicInbox::WWW.  You
75       can have any number of "extentry.$NAME" sections where $NAME is
76       something other than "all" to display a union of several inboxes.
77
78       See public-inbox-config(5) for more details.
79

ENVIRONMENT

81       PI_CONFIG
82               Used to override the default "~/.public-inbox/config" value.
83
84       XAPIAN_FLUSH_THRESHOLD
85               The number of documents to update before committing changes to
86               disk.  This environment is handled directly by Xapian, refer to
87               Xapian API documentation for more details.
88
89               Setting "XAPIAN_FLUSH_THRESHOLD" or
90               "publicinbox.indexBatchSize" for a large "--reindex" may cause
91               public-inbox-mda(1), public-inbox-learn(1) and
92               public-inbox-watch(1) tasks to wait long and unpredictable
93               periods of time during "--reindex".
94
95               Default: none, uses "publicinbox.indexBatchSize"
96

UPGRADING

98       Occasionally, public-inbox will update it's schema version and require
99       a full index by running this command.
100

CONTACT

102       Feedback welcome via plain-text mail to <mailto:meta@public-inbox.org>
103
104       The mail archives are hosted at <https://public-inbox.org/meta/> and
105       <http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
106

COPYRIGHT

108       Copyright all contributors <mailto:meta@public-inbox.org>
109
110       License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
111