1Lucy::Docs::FileFormat(U3s)er Contributed Perl DocumentatLiuocny::Docs::FileFormat(3)
2
6 Lucy::Docs::FileFormat - Overview of index file format
7
9 It is not necessary to understand the current implementation details of
10 the index file format in order to use Apache Lucy effectively, but it
11 may be helpful if you are interested in tweaking for high performance,
12 exotic usage, or debugging and development.
13 On a file system, an index is a directory. The files inside have a
15 hierarchical relationship: an index is made up of XsegmentsX, each of
16 which is an independent inverted index with its own subdirectory; each
17 segment is made up of several component parts.
18 [index]--|
20 |--snapshot_XXX.json
21 |--schema_XXX.json
22 |--write.lock
23 |
24 |--seg_1--|
25 | |--segmeta.json
26 | |--cfmeta.json
27 | |--cf.dat-------|
28 | |--[lexicon]
29 | |--[postings]
30 | |--[documents]
31 | |--[highlight]
32 | |--[deletions]
33 |
34 |--seg_2--|
35 | |--segmeta.json
36 | |--cfmeta.json
37 | |--cf.dat-------|
38 | |--[lexicon]
39 | |--[postings]
40 | |--[documents]
41 | |--[highlight]
42 | |--[deletions]
43 |
44 |--[...]--|
45 Write-once philosophy
47 All segment directory names consist of the string Xseg_X followed by a
48 number in base 36: seg_1, seg_5m, seg_p9s2 and so on, with higher
49 numbers indicating more recent segments. Once a segment is finished
50 and committed, its name is never re-used and its files are never
51 modified.
52 Old segments become obsolete and can be removed when their data has
54 been consolidated into new segments during the process of segment
55 merging and optimization. A fully-optimized index has only one
56 segment.
57 Top-level entries
59 There are a handful of Xtop-levelX files and directories which belong
60 to the entire index rather than to a particular segment.
61 snapshot_XXX.json
63 A XsnapshotX file, e.g. "snapshot_m7p.json", is list of index files and
65 directories. Because index files, once written, are never modified,
66 the list of entries in a snapshot defines a point-in-time view of the
67 data in an index.
68 Like segment directories, snapshot files also utilize the
70 unique-base-36-number naming convention; the higher the number, the
71 more recent the file. The appearance of a new snapshot file within the
72 index directory constitutes an index update. While a new segment is
73 being written new files may be added to the index directory, but until
74 a new snapshot file gets written, a Searcher opening the index for
75 reading wonXt know about them.
76 schema_XXX.json
78 The schema file is a Schema object describing the indexXs format,
80 serialized as JSON. It, too, is versioned, and a given snapshot file
81 will reference one and only one schema file.
82 locks
84 By default, only one indexing process may safely modify the index at
86 any given time. Processes reserve an index by laying claim to the
87 "write.lock" file within the "locks/" directory. A smattering of other
88 lock files may be used from time to time, as well.
89 A segmentXs component parts
91 By default, each segment has up to five logical components: lexicon,
92 postings, document storage, highlight data, and deletions. Binary data
93 from these components gets stored in virtual files within the Xcf.datX
94 compound file; metadata is stored in a shared Xsegmeta.jsonX file.
95 segmeta.json
97 The segmeta.json file is a central repository for segment metadata. In
99 addition to information such as document counts and field numbers, it
100 also warehouses arbitrary metadata on behalf of individual index
101 components.
102 Lexicon
104 Each indexed field gets its own lexicon in each segment. The exact
106 files involved depend on the fieldXs type, but generally speaking there
107 will be two parts. First, thereXs a primary "lexicon-XXX.dat" file
108 which houses a complete term list associating terms with corpus
109 frequency statistics, postings file locations, etc. Second, one or
110 more Xlexicon indexX files may be present which contain periodic
111 samples from the primary lexicon file to facilitate fast lookups.
112 Postings
114 XPostingX is a technical term from the field of information retrieval,
116 defined as a single instance of a one term indexing one document. If
117 you are looking at the index in the back of a book, and you see that
118 XfreedomX is referenced on pages 8, 86, and 240, that would be three
119 postings, which taken together form a Xposting listX. The same
120 terminology applies to an index in electronic form.
121 Each segment has one postings file per indexed field. When a search is
123 performed for a single term, first that term is looked up in the
124 lexicon. If the term exists in the segment, the record in the lexicon
125 will contain information about which postings file to look at and where
126 to look.
127 The first thing any posting record tells you is a document id. By
129 iterating over all the postings associated with a term, you can find
130 all the documents that match that term, a process which is analogous to
131 looking up page numbers in a bookXs index. However, each posting
132 record typically contains other information in addition to document id,
133 e.g. the positions at which the term occurs within the field.
134 Documents
136 The document storage section is a simple database, organized into two
138 files:
139 • documents.dat - Serialized documents.
141 • documents.ix - Document storage index, a solid array of 64-bit
143 integers where each integer location corresponds to a document id,
144 and the value at that location points at a file position in the
145 documents.dat file.
146 Highlight data
148 The files which store data used for excerpting and highlighting are
150 organized similarly to the files used to store documents.
151 • highlight.dat - Chunks of serialized highlight data, one per doc
153 id.
154 • highlight.ix - Highlight data index X as with the "documents.ix"
156 file, a solid array of 64-bit file pointers.
157 Deletions
159 When a document is XdeletedX from a segment, it is not actually purged
161 right away; it is merely marked as XdeletedX via a deletions file.
162 Deletions files contains bit vectors with one bit for each document in
163 the segment; if bit #254 is set then document 254 is deleted, and if
164 that document turns up in a search it will be masked out.
165 It is only when a segmentXs contents are rewritten to a new segment
167 during the segment-merging process that deleted documents truly go
168 away.
169 Compound Files
171 If you peer inside an index directory, you wonXt actually find any
172 files named Xdocuments.datX, Xhighlight.ixX, etc. unless there is an
173 indexing process underway. What you will find instead is one Xcf.datX
174 and one Xcfmeta.jsonX file per segment.
175 To minimize the need for file descriptors at search-time, all per-
177 segment binary data files are concatenated together in Xcf.datX at the
178 close of each indexing session. Information about where each file
179 begins and ends is stored in "cfmeta.json". When the segment is opened
180 for reading, a single file descriptor per Xcf.datX file can be shared
181 among several readers.
182 A Typical Search
184 HereXs a simplified narrative, dramatizing how a search for XfreedomX
185 against a given segment plays out:
186 • The searcher asks the relevant Lexicon Index, XDo you know anything
188 about XfreedomX?X Lexicon Index replies, XCanXt say for sure, but
189 if the main Lexicon file does, XfreedomX is probably somewhere
190 around byte 21008X.
191 • The main Lexicon tells the searcher XOne moment, let me scan our
193 recordsX Yes, we have 2 documents which contain XfreedomX. YouXll
194 find them in seg_6/postings-4.dat starting at byte 66991.X
195 • The Postings file says XYep, we have XfreedomX, all right!
197 Document id 40 has 1 XfreedomX, and document 44 has 8. If you need
198 to know more, like if any XfreedomX is part of the phrase Xfreedom
199 of speechX, ask me about positions!
200 • If the searcher is only looking for XfreedomX in isolation, thatXs
202 where it stops. It now knows enough to assign the documents scores
203 against XfreedomX, with the 8-freedom document likely ranking
204 higher than the single-freedom document.
205perl v5.34.0 2021-07-22 Lucy::Docs::FileFormat(3)