public-inbox-extindex(1)

1PUBLIC-INBOX-EXTINDEX(1)   public-inbox user manual   PUBLIC-INBOX-EXTINDEX(1)
2
3
4

NAME

6       public-inbox-extindex - create and update external search indices
7

SYNOPSIS

9       public-inbox-extindex [OPTIONS] EXTINDEX_DIR INBOX_DIR...
10
11       public-inbox-extindex [OPTIONS] [EXTINDEX_DIR] --all
12

DESCRIPTION

14       public-inbox-extindex creates and updates an external search and
15       overview database used by the read-only public-inbox PSGI (HTTP), NNTP,
16       and IMAP interfaces.  This requires either the Search::Xapian XS
17       bindings OR the Xapian SWIG bindings, along with DBD::SQLite and DBI
18       Perl modules.
19

OPTIONS

21       -j JOBS
22       --jobs=JOBS
23           ... TODO, see public-inbox-index(5)
24
25       --all
26           Index all "publicinbox" entries in "PI_CONFIG".
27
28           "publicinbox" entries indexed by "public-inbox-extindex" can have
29           full Xapian searching abilities with the per-"publicinbox"
30           "indexlevel" set to "basic" and their respective Xapian ("xap15" or
31           "xapian15") directories removed.  For multiple public-inboxes where
32           cross-posting is common, this allows significant space savings on
33           Xapian indices.
34
35       --gc
36           Perform garbage collection instead of indexing.  Use this if
37           inboxes are removed from the extindex, or if messages are purged or
38           removed from some inboxes.
39
40       --reindex
41           Forces a re-index of all messages in the extindex.  This can be
42           used for in-place upgrades and bugfixes while read-only server
43           processes are utilizing the index.  Keep in mind this roughly
44           doubles the size of the already-large Xapian database.
45
46           The extindex locks will be released roughly every 10s to allow
47           public-inbox-mda(1) and public-inbox-watch(1) processes to write to
48           the extindex.
49
50       --fast
51           Used with "--reindex", it will only look for new and stale entries
52           and not touch already-indexed messages.
53

FILES

55       public-inbox-extindex-format(5)
56

CONFIGURATION

58       public-inbox-extindex does not currently write to the
59       public-inbox-config(5) file, configuration may be entered manually.
60       The extindex name of "all" is a special case which corresponds to
61       indexing "--all" inboxes.  An example for "--all" is as follows:
62
63               [extindex "all"]
64                       topdir = /path/to/extindex_dir
65                       url = all
66                       coderepo = foo
67                       coderepo = bar
68
69       See public-inbox-config(5) for more details.
70

ENVIRONMENT

72       PI_CONFIG
73               Used to override the default "~/.public-inbox/config" value.
74
75       XAPIAN_FLUSH_THRESHOLD
76               The number of documents to update before committing changes to
77               disk.  This environment is handled directly by Xapian, refer to
78               Xapian API documentation for more details.
79
80               Setting "XAPIAN_FLUSH_THRESHOLD" or
81               "publicinbox.indexBatchSize" for a large "--reindex" may cause
82               public-inbox-mda(1), public-inbox-learn(1) and
83               public-inbox-watch(1) tasks to wait long and unpredictable
84               periods of time during "--reindex".
85
86               Default: none, uses "publicinbox.indexBatchSize"
87

UPGRADING

89       Occasionally, public-inbox will update it's schema version and require
90       a full index by running this command.
91

CONTACT

93       Feedback welcome via plain-text mail to <mailto:meta@public-inbox.org>
94
95       The mail archives are hosted at <https://public-inbox.org/meta/> and
96       <http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>
97

COPYRIGHT

99       Copyright 2021 all contributors <mailto:meta@public-inbox.org>
100
101       License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
102