1Fsdb::Filter::dbpipelinUes(e3r)Contributed Perl DocumentFastdibo:n:Filter::dbpipeline(3)
2
3
4

NAME

6       dbpipeline - allow db commands to be assembled as pipelines in Perl
7

SYNOPSIS

9           use Fsdb::Filter::dbpipeline qw(:all);
10           dbpipeline(
11               dbrow(qw(name test1)),
12               dbroweval('_test1 += 5;')
13           );
14
15       Or for more customized versions, see "dbpipeline_filter",
16       "dbpipeline_sink", "dbpipeline_open2", and "dbpipeline_close2_hash".
17

DESCRIPTION

19       This module makes it easy to create pipelines in Perl using separate
20       processes.  (In the past we used to use perl threads.)
21
22       By default (as with all Fsdb modules), input is from STDIN and output
23       to STDOUT.  Two helper functions, fromfile and tofile can grab data
24       from files.
25
26       Dbpipeline differs in several ways from all other Fsdb::Filter modules:
27       it does not have a corresponding Unix command (it is used only from
28       within Perl).  It does not log its presence to the output stream (this
29       is arguably a bug, but it doesn't actually do anything).
30

OPTIONS

32       Unlike most Fsdb modules, dbpipeline defaults to "--autorun".
33
34       This module also supports the standard fsdb options:
35
36       -d  Enable debugging output.
37
38       -i or --input InputSource
39           Read from InputSource, typically a file name, or "-" for standard
40           input, or (if in Perl) a IO::Handle, Fsdb::IO or Fsdb::BoundedQueue
41           objects.
42
43       -o or --output OutputDestination
44           Write to OutputDestination, typically a file name, or "-" for
45           standard output, or (if in Perl) a IO::Handle, Fsdb::IO or
46           Fsdb::BoundedQueue objects.
47
48       --autorun or --noautorun
49           By default, programs process automatically, but Fsdb::Filter
50           objects in Perl do not run until you invoke the run() method.  The
51           "--(no)autorun" option controls that behavior within Perl.
52
53       --header H
54           Use H as the full Fsdb header, rather than reading a header from
55           then input.  This option is particularly useful when using Fsdb
56           under Hadoop, where split files don't have heades.
57
58       --help
59           Show help.
60
61       --man
62           Show full manual.
63

SEE ALSO

65       Fsdb(3)
66

CLASS FUNCTIONS

68   dbpipeline
69           dbpipeline(@modules);
70
71       This shorthand-routine creates a dbpipeline object and then immediately
72       runs it.
73
74       Thus perl code becomes nearly as terse as shell code:
75
76           dbpipeline(
77               dbcol(qw(name test1)),
78               dbroweval('_test1 += 5;'),
79           );
80
81       The following commands currently have shorthand aliases:
82
83       cgi_to_db(1)
84       combined_log_format_to_db(1)
85       csv_to_db(1)
86       db_to_csv(1)
87       db_to_html_table(1)
88       dbcol(1)
89       dbcolcopylast(1)
90       dbcolcreate(1)
91       dbcoldefine(1)
92       dbcolhisto(1)
93       dbcolmerge(1)
94       dbcolmovingstats(1)
95       dbcolneaten(1)
96       dbcolpercentile(1)
97       dbcolrename(1)
98       dbcolscorrelate(1)
99       dbcolsplittocols(1)
100       dbcolsplittorows(1)
101       dbcolsregression(1)
102       dbcolstats(1)
103       dbcolstatscores(1)
104       dbfilealter(1)
105       dbfilecat(1)
106       dbfilediff(1)
107       dbfilepivot(1)
108       dbfilestripcomments(1)
109       dbfilevalidate(1)
110       dbformmail(1)
111       dbjoin(1)
112       dbmapreduce(1)
113       dbmerge(1)
114       dbmerge2(1)
115       dbmultistats(1)
116       dbrow(1)
117       dbrowaccumulate(1)
118       dbrowcount(1)
119       dbrowdiff(1)
120       dbroweval(1)
121       dbrowuniq(1)
122       dbrvstatdiff(1)
123       dbsort(1)
124       html_table_to_db(1)
125       kitrace_to_db(1)
126       mysql_to_db(1)
127       tabdelim_to_db(1)
128       tcpdump_to_db(1)
129       xml_to_db(1)
130
131       and
132
133       dbsubprocess(3)
134
135   dbpipeline_filter
136           my($result_reader, $fred) = dbpipeline_filter($source, $result_reader_aref, @modules);
137
138       Set up a pipeline of @MODULES that filters data pushed through it,
139       where the data comes from $SOURCE (any Fsdb::Filter::parse_io_option
140       object, like a Fsdb::IO::Reader object, queue, or filename).
141
142       Returns a $RESULT_READER Fsdb::IO::Reader object, created with
143       $RESULT_READER_AREF as options.  This reader will produce the filtered
144       data, and a $FRED that must be joined to guarantee output has
145       completed.
146
147       Or if $RESULT_READER_AREF is "[-raw_fh,  1]", it just returns the
148       IO::Handle to the pipe.
149
150       As an example, this code uses "dbpipeline_filter" to insure the input
151       (from $in which is a filename or Fsdb::IO::Reader) is sorted
152       numerically by column "x":
153
154           use Fsdb::Filter::dbpipeline qw(dbpipeline_filter dbsort);
155           my($new_in, $new_fred) = dbpipeline_filter($in,
156               [-comment_handler => $self->create_delay_comments_sub],
157               dbsort(qw(--nolog -n x)));
158           while (my $fref = $new_in->read_rowwobj()) {
159               # do something
160           };
161           $new_in->close;
162           $new_fred->join();
163
164   dbpipeline_sink
165           my($fsdb_writer, $fred) = dbpipeline_sink($writer_arguments_aref, @modules);
166
167       Set up a pipeline of @MODULES that is a data "sink", where the output
168       is given by a "--output" argument, or goes to standard output (by
169       default).  The caller generates input into the pipeline by writing to a
170       newly created $FSDB_WRITER, whose configuration is specified by the
171       mandatory first argument $WRITER_ARGUMENTS_AREF.  (These arguments
172       should include the schema.)  Returns this writer, and a $FRED that must
173       be joined to guarantee output has completed.
174
175       If the first argument to modules is "--fred_exit_sub", then the second
176       is taken as a CODE block that runs at fred exit (and the two are not
177       passed to modules).
178
179       If the first argument to modules is "--fred_description", then the
180       second is taken as a text description of the Fred.
181
182   dbpipeline_open2
183           my($fsdb_reader_fh, $fsdb_writer, $fred) =
184               dbpipeline_open2($writer_arguments_aref, @modules);
185
186       Set up a pipeline of @MODULES that is a data sink and source (both!).
187       The caller generates input into the pipeline by writing to a newly
188       created $FSDB_WRITER, whose configuration is specified by the mandatory
189       argument $WRITER_ARGUMENTS_AREF.  These arguments should include the
190       schema.)  The output of the pipeline comes out to the newly created
191       $FSDB_READER_FH.  Returns this read queue and writer, and a $PID that
192       must be joined to guarantee output has completed.
193
194       (Unfortunately the interface is asymmetric with a read queue but a
195       write "Fsdb::IO" object, because "Fsdb::IO::Reader" blocks on input of
196       the header.)
197
198       Like IPC::Open2, with all of its pros and cons like potential deadlock.
199
200   dbpipeline_close2_hash
201           my($href) = dbpipeline_close2_hash($fsdb_read_fh, $fsdb_writer, $pid);
202
203       Reads and returns one line of output from $FSDB_READER, after closing
204       $FSDB_WRITER and joining the $PID.
205
206       Useful, for example, to get dbcolstats output cleanly.
207
208   new
209           $filter = new Fsdb::Filter::dbpipeline(@arguments);
210
211   set_defaults
212           $filter->set_defaults();
213
214       Internal: set up defaults.
215
216   parse_options
217           $filter->parse_options(@ARGV);
218
219       Internal: parse options
220
221   setup
222           $filter->_reap();
223
224       Internal: reap any forked threads.
225
226   setup
227           $filter->setup();
228
229       Internal: setup, parse headers.
230
231   run
232           $filter->run();
233
234       Internal: run over all IO
235
236   finish
237           $filter->finish();
238
239       Internal: we would write a trailer, but we don't because we depend on
240       the last command in the pipeline to do that.  We don't actually have a
241       valid output stream.
242
244       Copyright (C) 1991-2018 by John Heidemann <johnh@isi.edu>
245
246       This program is distributed under terms of the GNU general public
247       license, version 2.  See the file COPYING with the distribution for
248       details.
249
250
251
252perl v5.32.0                      2020-11-16       Fsdb::Filter::dbpipeline(3)
Impressum