1Fsdb::Filter::dbpipelinUes(e3r)Contributed Perl DocumentFastdibo:n:Filter::dbpipeline(3)
2
6 dbpipeline - allow db commands to be assembled as pipelines in Perl
7
9 use Fsdb::Filter::dbpipeline qw(:all);
10 dbpipeline(
11 dbrow(qw(name test1)),
12 dbroweval('_test1 += 5;')
13 );
14 Or for more customized versions, see "dbpipeline_filter",
16 "dbpipeline_sink", "dbpipeline_open2", and "dbpipeline_close2_hash".
17
19 This module makes it easy to create pipelines in Perl using separate
20 processes. (In the past we used to use perl threads.)
21 By default (as with all Fsdb modules), input is from STDIN and output
23 to STDOUT. Two helper functions, fromfile and tofile can grab data
24 from files.
25 Dbpipeline differs in several ways from all other Fsdb::Filter modules:
27 it does not have a corresponding Unix command (it is used only from
28 within Perl). It does not log its presence to the output stream (this
29 is arguably a bug, but it doesn't actually do anything).
30
32 Unlike most Fsdb modules, dbpipeline defaults to "--autorun".
33 This module also supports the standard fsdb options:
35 -d Enable debugging output.
37 -i or --input InputSource
39 Read from InputSource, typically a file name, or "-" for standard
40 input, or (if in Perl) a IO::Handle, Fsdb::IO or Fsdb::BoundedQueue
41 objects.
42 -o or --output OutputDestination
44 Write to OutputDestination, typically a file name, or "-" for
45 standard output, or (if in Perl) a IO::Handle, Fsdb::IO or
46 Fsdb::BoundedQueue objects.
47 --autorun or --noautorun
49 By default, programs process automatically, but Fsdb::Filter
50 objects in Perl do not run until you invoke the run() method. The
51 "--(no)autorun" option controls that behavior within Perl.
52 --header H
54 Use H as the full Fsdb header, rather than reading a header from
55 then input. This option is particularly useful when using Fsdb
56 under Hadoop, where split files don't have heades.
57 --help
59 Show help.
60 --man
62 Show full manual.
63
65 Fsdb(3)
66
68 dbpipeline
69 dbpipeline(@modules);
70 This shorthand-routine creates a dbpipeline object and then immediately
72 runs it.
73 Thus perl code becomes nearly as terse as shell code:
75 dbpipeline(
77 dbcol(qw(name test1)),
78 dbroweval('_test1 += 5;'),
79 );
80 The following commands currently have shorthand aliases:
82 cgi_to_db(1)
84 combined_log_format_to_db(1)
85 csv_to_db(1)
86 db_to_csv(1)
87 db_to_html_table(1)
88 dbcol(1)
89 dbcolcopylast(1)
90 dbcolcreate(1)
91 dbcoldefine(1)
92 dbcolhisto(1)
93 dbcolmerge(1)
94 dbcolmovingstats(1)
95 dbcolneaten(1)
96 dbcolpercentile(1)
97 dbcolrename(1)
98 dbcolscorrelate(1)
99 dbcolsplittocols(1)
100 dbcolsplittorows(1)
101 dbcolsregression(1)
102 dbcolstats(1)
103 dbcolstatscores(1)
104 dbfilealter(1)
105 dbfilecat(1)
106 dbfilediff(1)
107 dbfilepivot(1)
108 dbfilestripcomments(1)
109 dbfilevalidate(1)
110 dbformmail(1)
111 dbjoin(1)
112 dbmapreduce(1)
113 dbmerge(1)
114 dbmerge2(1)
115 dbmultistats(1)
116 dbrow(1)
117 dbrowaccumulate(1)
118 dbrowcount(1)
119 dbrowdiff(1)
120 dbroweval(1)
121 dbrowuniq(1)
122 dbrvstatdiff(1)
123 dbsort(1)
124 html_table_to_db(1)
125 kitrace_to_db(1)
126 mysql_to_db(1)
127 tabdelim_to_db(1)
128 tcpdump_to_db(1)
129 xml_to_db(1)
130 and
132 dbsubprocess(3)
134 dbpipeline_filter
136 my($result_reader, $fred) = dbpipeline_filter($source, $result_reader_aref, @modules);
137 Set up a pipeline of @MODULES that filters data pushed through it,
139 where the data comes from $SOURCE (any Fsdb::Filter::parse_io_option
140 object, like a Fsdb::IO::Reader object, queue, or filename).
141 Returns a $RESULT_READER Fsdb::IO::Reader object, created with
143 $RESULT_READER_AREF as options. This reader will produce the filtered
144 data, and a $FRED that must be joined to guarantee output has
145 completed.
146 Or if $RESULT_READER_AREF is "[-raw_fh, 1]", it just returns the
148 IO::Handle to the pipe.
149 As an example, this code uses "dbpipeline_filter" to insure the input
151 (from $in which is a filename or Fsdb::IO::Reader) is sorted
152 numerically by column "x":
153 use Fsdb::Filter::dbpipeline qw(dbpipeline_filter dbsort);
155 my($new_in, $new_fred) = dbpipeline_filter($in,
156 [-comment_handler => $self->create_delay_comments_sub],
157 dbsort(qw(--nolog -n x)));
158 while (my $fref = $new_in->read_rowwobj()) {
159 # do something
160 };
161 $new_in->close;
162 $new_fred->join();
163 dbpipeline_sink
165 my($fsdb_writer, $fred) = dbpipeline_sink($writer_arguments_aref, @modules);
166 Set up a pipeline of @MODULES that is a data "sink", where the output
168 is given by a "--output" argument, or goes to standard output (by
169 default). The caller generates input into the pipeline by writing to a
170 newly created $FSDB_WRITER, whose configuration is specified by the
171 mandatory first argument $WRITER_ARGUMENTS_AREF. (These arguments
172 should include the schema.) Returns this writer, and a $FRED that must
173 be joined to guarantee output has completed.
174 If the first argument to modules is "--fred_exit_sub", then the second
176 is taken as a CODE block that runs at fred exit (and the two are not
177 passed to modules).
178 If the first argument to modules is "--fred_description", then the
180 second is taken as a text description of the Fred.
181 dbpipeline_open2
183 my($fsdb_reader_fh, $fsdb_writer, $fred) =
184 dbpipeline_open2($writer_arguments_aref, @modules);
185 Set up a pipeline of @MODULES that is a data sink and source (both!).
187 The caller generates input into the pipeline by writing to a newly
188 created $FSDB_WRITER, whose configuration is specified by the mandatory
189 argument $WRITER_ARGUMENTS_AREF. These arguments should include the
190 schema.) The output of the pipeline comes out to the newly created
191 $FSDB_READER_FH. Returns this read queue and writer, and a $PID that
192 must be joined to guarantee output has completed.
193 (Unfortunately the interface is asymmetric with a read queue but a
195 write "Fsdb::IO" object, because "Fsdb::IO::Reader" blocks on input of
196 the header.)
197 Like IPC::Open2, with all of its pros and cons like potential deadlock.
199 dbpipeline_close2_hash
201 my($href) = dbpipeline_close2_hash($fsdb_read_fh, $fsdb_writer, $pid);
202 Reads and returns one line of output from $FSDB_READER, after closing
204 $FSDB_WRITER and joining the $PID.
205 Useful, for example, to get dbcolstats output cleanly.
207 new
209 $filter = new Fsdb::Filter::dbpipeline(@arguments);
210 set_defaults
212 $filter->set_defaults();
213 Internal: set up defaults.
215 parse_options
217 $filter->parse_options(@ARGV);
218 Internal: parse options
220 setup
222 $filter->_reap();
223 Internal: reap any forked threads.
225 setup
227 $filter->setup();
228 Internal: setup, parse headers.
230 run
232 $filter->run();
233 Internal: run over all IO
235 finish
237 $filter->finish();
238 Internal: we would write a trailer, but we don't because we depend on
240 the last command in the pipeline to do that. We don't actually have a
241 valid output stream.
242
244 Copyright (C) 1991-2018 by John Heidemann <johnh@isi.edu>
245 This program is distributed under terms of the GNU general public
247 license, version 2. See the file COPYING with the distribution for
248 details.
249perl v5.36.0 2023-01-20 Fsdb::Filter::dbpipeline(3)