1Fsdb::Filter::dbmerge(3U)ser Contributed Perl DocumentatiFosndb::Filter::dbmerge(3)
2
6 dbmerge - merge all inputs in sorted order based on the the specified
7 columns
8
10 dbmerge --input A.fsdb --input B.fsdb [-T TemporaryDirectory] [-nNrR] column [column...]
11 or
13 cat A.fsdb | dbmerge --input - --input B.fsdb [-T
14 TemporaryDirectory] [-nNrR] column [column...]
15 or
17 dbmerge [-T TemporaryDirectory] [-nNrR] column [column...] --inputs
18 A.fsdb [B.fsdb ...]
19 or
21 { echo "A.fsdb"; echo "B.fsdb" } | dbmerge --xargs column
22 [column...]
23
25 Merge all provided, pre-sorted input files, producing one sorted
26 result. Inputs can both be specified with "--input", or one can come
27 from standard input and the other from "--input". With "--xargs", each
28 line of standard input is a filename for input.
29 Inputs must have identical schemas (columns, column order, and field
31 separators).
32 Unlike dbmerge2, dbmerge supports an arbitrary number of input files.
34 Because this program is intended to merge multiple sources, it does not
36 default to reading from standard input. If you wish to list - as an
37 explicit input source.
38 Also, because we deal with multiple input files, this module doesn't
40 output anything until it's run.
41 dbmerge consumes a fixed amount of memory regardless of input size. It
43 therefore buffers output on disk as necessary. (Merging is implemented
44 a series of two-way merges, so disk space is O(number of records).)
45 dbmerge will merge data in parallel, if possible. The "--parallelism"
47 option can control the degree of parallelism, if desired.
48
50 General option:
51 --xargs
53 Expect that input filenames are given, one-per-line, on standard
54 input. (In this case, merging can start incrementally.)
55 --removeinputs
57 Delete the source files after they have been consumed. (Defaults
58 off, leaving the inputs in place.)
59 -T TmpDir
61 where to put tmp files. Also uses environment variable TMPDIR, if
62 -T is not specified. Default is /tmp.
63 --parallelism N or -j N
65 Allow up to N merges to happen in parallel. Default is the number
66 of CPUs in the machine.
67 --endgame (or --noendgame)
69 Enable endgame mode, extra parallelism when finishing up. (On by
70 default.)
71 Sort specification options (can be interspersed with column names):
73 -r or --descending
75 sort in reverse order (high to low)
76 -R or --ascending
78 sort in normal order (low to high)
79 -n or --numeric
81 sort numerically
82 -N or --lexical
84 sort lexicographically
85 This module also supports the standard fsdb options:
87 -d Enable debugging output.
89 -i or --input InputSource
91 Read from InputSource, typically a file name, or "-" for standard
92 input, or (if in Perl) a IO::Handle, Fsdb::IO or Fsdb::BoundedQueue
93 objects.
94 -o or --output OutputDestination
96 Write to OutputDestination, typically a file name, or "-" for
97 standard output, or (if in Perl) a IO::Handle, Fsdb::IO or
98 Fsdb::BoundedQueue objects.
99 --autorun or --noautorun
101 By default, programs process automatically, but Fsdb::Filter
102 objects in Perl do not run until you invoke the run() method. The
103 "--(no)autorun" option controls that behavior within Perl.
104 --header H
106 Use H as the full Fsdb header, rather than reading a header from
107 then input.
108 --help
110 Show help.
111 --man
113 Show full manual.
114
116 Input:
117 File a.fsdb:
118 #fsdb cid cname
120 11 numanal
121 10 pascal
122 File b.fsdb:
124 #fsdb cid cname
126 12 os
127 13 statistics
128 These two files are both sorted by "cname", and they have identical
130 schemas.
131 Command:
133 dbmerge --input a.fsdb --input b.fsdb cname
134 or
136 cat a.fsdb | dbmerge --input b.fsdb cname
138 Output:
140 #fsdb cid cname
141 11 numanal
142 12 os
143 10 pascal
144 13 statistics
145 # | dbmerge --input a.fsdb --input b.fsdb cname
146
148 dbmerge2(1), dbsort(1), Fsdb(3)
149
151 new
152 $filter = new Fsdb::Filter::dbmerge(@arguments);
153 Create a new object, taking command-line arguments.
155 set_defaults
157 $filter->set_defaults();
158 Internal: set up defaults.
160 parse_options
162 $filter->parse_options(@ARGV);
163 Internal: parse command-line arguments.
165 _pretty_fn
167 _pretty_fn($fn)
168 Internal: pretty-print a filename or Fsdb::BoundedQueue.
170 segment_next_output
172 $out = $self->segment_next_output($output_type)
173 Internal: return a Fsdb::IO::Writer as $OUT that either points to our
175 output or a temporary file, depending on how things are going.
176 The $OUTPUT_TYPE can be 'final' or 'ipc' or 'file'.
178 segment_cleanup
180 $out = $self->segment_cleanup($file);
181 Internal: Clean up a file, if necessary. (Sigh, used to be function
183 pointers, but not clear how they would interact with threads.)
184 _unique_id
186 $id = $self->_unique_id()
187 Generate a sequence number for debugging.
189 segments_merge2_run
191 $out = $self->segments_merge2_run($out_fn, $is_final_output,
192 $in0, $in1);
193 Internal: do the actual merge2 work (maybe our parent put us in a
195 thread, maybe not).
196 enqueue_work
198 $self->enqueue_work($depth, $work);
199 Internal: put $WORK on the queue at $DEPTH, updating the max count.
201 segments_merge_one_depth
203 $self->segments_merge_one_depth($depth);
204 Merge queued files, if any.
206 Also release any queued threads.
208 segments_xargs
210 $self->segments_xargs();
211 Internal: read new filenames to process (from stdin) and send them to
213 the work queue.
214 Making a separate Fred to handle xargs is a lot of work, but it
216 guarantees it comes in on an IO::Handle that is selectable.
217 segments_merge_all
219 $self->segments_merge_all()
220 Internal: Merge queued files, if any. Iterates over all depths of the
222 merge tree, and handles any forked threads.
223 Merging Strategy
225 Merging is done in a binary tree is managed through the "_work" queue.
227 It has an array of "depth" entries, one for each level of the tree.
228 Items are processed in order at each level of the tree, and only level-
230 by-level, so the sort is stable.
231 Parallelism Model
233 Parallelism is also managed through the "_work" queue, each element of
235 which consists of one file or stream suitable for merging. The work
236 queue contains both ready output (files or BoundedQueue streams) that
237 can be immediately handled, and pairs of semaphore/pending output for
238 work that is not yet started. All manipulation of the work queue
239 happens in the main thread (with "segments_merge_all" and
240 "segments_merge_one_depth").
241 We start a thread to handle each item in the work queue, and limit
243 parallelism to the "_max_parallelism", defaulting to the number of
244 available processors.
245 There two two kinds of parallelism, regular and endgame. For regular
247 parallelism we pick two items off the work queue, merge them, and put
248 the result back on the queue as a new file. Items in the work queue
249 may not be ready. For in-progress items we wait until they are done.
250 For not-yet-started items we start them, then wait until they are done.
251 Endgame parallelism handles the final stages of a large merge. When
253 there are enough processors that we can start a merge jobs for all
254 remaining levels of the merge tree. At this point we switch from
255 merging to files to merging into "Fsdb::BoundedQueue" pipelines that
256 connect merge processes which start and run concurrently.
257 The final merge is done in the main thread so that that the main thread
259 can handle the output stream and recording the merge action.
260 setup
262 $filter->setup();
263 Internal: setup, parse headers.
265 run
267 $filter->run();
268 Internal: run over each rows.
270
272 Copyright (C) 1991-2019 by John Heidemann <johnh@isi.edu>
273 This program is distributed under terms of the GNU general public
275 license, version 2. See the file COPYING with the distribution for
276 details.
277perl v5.30.1 2020-01-30 Fsdb::Filter::dbmerge(3)