1Fsdb::Filter::dbpipelinUes(e3r)Contributed Perl DocumentFastdibo:n:Filter::dbpipeline(3)
2
6 dbpipeline - allow db commands to be assembled as pipelines in Perl
7
9 use Fsdb::Filter::dbpipeline qw(:all);
10 dbpipeline(
11 dbrow(qw(name test1)),
12 dbroweval('_test1 += 5;')
13 );
14 Or for more customized versions, see "dbpipeline_filter",
16 "dbpipeline_sink", "dbpipeline_open2", and "dbpipeline_close2_hash".
17
19 This module makes it easy to create pipelines in Perl using separate
20 processes. (In the past we used to use perl threads.)
21 By default (as with all Fsdb modules), input is from STDIN and output
23 to STDOUT. Two helper functions, fromfile and tofile can grab data
24 from files.
25 Dbpipeline differs in several ways from all other Fsdb::Filter modules:
27 it does not have a corresponding Unix command (it is used only from
28 within Perl). It does not log its presence to the output stream (this
29 is arguably a bug, but it doesn't actually do anything).
30
32 Unlike most Fsdb modules, dbpipeline defaults to "--autorun".
33 This module also supports the standard fsdb options:
35 -d Enable debugging output.
37 -i or --input InputSource
39 Read from InputSource, typically a file name, or "-" for standard
40 input, or (if in Perl) a IO::Handle, Fsdb::IO or Fsdb::BoundedQueue
41 objects.
42 -o or --output OutputDestination
44 Write to OutputDestination, typically a file name, or "-" for
45 standard output, or (if in Perl) a IO::Handle, Fsdb::IO or
46 Fsdb::BoundedQueue objects.
47 --autorun or --noautorun
49 By default, programs process automatically, but Fsdb::Filter
50 objects in Perl do not run until you invoke the run() method. The
51 "--(no)autorun" option controls that behavior within Perl.
52 --help
54 Show help.
55 --man
57 Show full manual.
58
60 Fsdb(3)
61
63 dbpipeline
64 dbpipeline(@modules);
65 This shorthand-routine creates a dbpipeline object and then immediately
67 runs it.
68 Thus perl code becomes nearly as terse as shell code:
70 dbpipeline(
72 dbcol(qw(name test1)),
73 dbroweval('_test1 += 5;'),
74 );
75 The following commands currently have shorthand aliases:
77 cgi_to_db(1)
79 combined_log_format_to_db(1)
80 csv_to_db(1)
81 db_to_csv(1)
82 db_to_html_table(1)
83 dbcol(1)
84 dbcolcopylast(1)
85 dbcolcreate(1)
86 dbcoldefine(1)
87 dbcolhisto(1)
88 dbcolmerge(1)
89 dbcolmovingstats(1)
90 dbcolneaten(1)
91 dbcolpercentile(1)
92 dbcolrename(1)
93 dbcolscorrelate(1)
94 dbcolsplittocols(1)
95 dbcolsplittorows(1)
96 dbcolsregression(1)
97 dbcolstats(1)
98 dbcolstatscores(1)
99 dbfilealter(1)
100 dbfilecat(1)
101 dbfilediff(1)
102 dbfilepivot(1)
103 dbfilestripcomments(1)
104 dbfilevalidate(1)
105 dbformmail(1)
106 dbjoin(1)
107 dbmapreduce(1)
108 dbmerge(1)
109 dbmerge2(1)
110 dbmultistats(1)
111 dbrow(1)
112 dbrowaccumulate(1)
113 dbrowcount(1)
114 dbrowdiff(1)
115 dbroweval(1)
116 dbrowuniq(1)
117 dbrvstatdiff(1)
118 dbsort(1)
119 html_table_to_db(1)
120 kitrace_to_db(1)
121 mysql_to_db(1)
122 tabdelim_to_db(1)
123 tcpdump_to_db(1)
124 xml_to_db(1)
125 and
127 dbsubprocess(3)
129 dbpipeline_filter
131 my($result_reader, $fred) = dbpipeline_filter($source, $result_reader_aref, @modules);
132 Set up a pipeline of @MODULES that filters data pushed through it,
134 where the data comes from $SOURCE (any Fsdb::Filter::parse_io_option
135 object, like a Fsdb::IO::Reader object, queue, or filename).
136 Returns a $RESULT_READER Fsdb::IO::Reader object, created with
138 $RESULT_READER_AREF as options. This reader will produce the filtered
139 data, and a $FRED that must be joined to guarantee output has
140 completed.
141 Or if $RESULT_READER_AREF is "[-raw_fh, 1]", it just returns the
143 IO::Handle to the pipe.
144 As an example, this code uses "dbpipeline_filter" to insure the input
146 (from $in which is a filename or Fsdb::IO::Reader) is sorted
147 numerically by column "x":
148 use Fsdb::Filter::dbpipeline qw(dbpipeline_filter dbsort);
150 my($new_in, $new_fred) = dbpipeline_filter($in,
151 [-comment_handler => $self->create_delay_comments_sub],
152 dbsort(qw(--nolog -n x)));
153 while (my $fref = $new_in->read_rowwobj()) {
154 # do something
155 };
156 $new_in->close;
157 $new_fred->join();
158 dbpipeline_sink
160 my($fsdb_writer, $fred) = dbpipeline_sink($writer_arguments_aref, @modules);
161 Set up a pipeline of @MODULES that is a data "sink", where the output
163 is given by a "--output" argument, or goes to standard output (by
164 default). The caller generates input into the pipeline by writing to a
165 newly created $FSDB_WRITER, whose configuration is specified by the
166 mandatory first argument $WRITER_ARGUMENTS_AREF. (These arguments
167 should include the schema.) Returns this writer, and a $FRED that must
168 be joined to guarantee output has completed.
169 If the first argument to modules is "--fred_exit_sub", then the second
171 is taken as a CODE block that runs at fred exit (and the two are not
172 passed to modules).
173 If the first argument to modules is "--fred_description", then the
175 second is taken as a text description of the Fred.
176 dbpipeline_open2
178 my($fsdb_reader_fh, $fsdb_writer, $fred) =
179 dbpipeline_open2($writer_arguments_aref, @modules);
180 Set up a pipeline of @MODULES that is a data sink and source (both!).
182 The caller generates input into the pipeline by writing to a newly
183 created $FSDB_WRITER, whose configuration is specified by the mandatory
184 argument $WRITER_ARGUMENTS_AREF. These arguments should include the
185 schema.) The output of the pipeline comes out to the newly created
186 $FSDB_READER_FH. Returns this read queue and writer, and a $PID that
187 must be joined to guarantee output has completed.
188 (Unfortunately the interface is asymmetric with a read queue but a
190 write "Fsdb::IO" object, because "Fsdb::IO::Reader" blocks on input of
191 the header.)
192 Like IPC::Open2, with all of its pros and cons like potential deadlock.
194 dbpipeline_close2_hash
196 my($href) = dbpipeline_close2_hash($fsdb_read_fh, $fsdb_writer, $pid);
197 Reads and returns one line of output from $FSDB_READER, after closing
199 $FSDB_WRITER and joining the $PID.
200 Useful, for example, to get dbcolstats output cleanly.
202 new
204 $filter = new Fsdb::Filter::dbpipeline(@arguments);
205 set_defaults
207 $filter->set_defaults();
208 Internal: set up defaults.
210 parse_options
212 $filter->parse_options(@ARGV);
213 Internal: parse options
215 setup
217 $filter->_reap();
218 Internal: reap any forked threads.
220 setup
222 $filter->setup();
223 Internal: setup, parse headers.
225 run
227 $filter->run();
228 Internal: run over all IO
230 finish
232 $filter->finish();
233 Internal: we would write a trailer, but we don't because we depend on
235 the last command in the pipeline to do that. We don't actually have a
236 valid output stream.
237
239 Copyright (C) 1991-2018 by John Heidemann <johnh@isi.edu>
240 This program is distributed under terms of the GNU general public
242 license, version 2. See the file COPYING with the distribution for
243 details.
244perl v5.30.0 2019-09-19 Fsdb::Filter::dbpipeline(3)