1Fsdb::Filter::dbpipelinUes(e3r)Contributed Perl DocumentFastdibo:n:Filter::dbpipeline(3)
2
3
4
6 dbpipeline - allow db commands to be assembled as pipelines in Perl
7
9 use Fsdb::Filter::dbpipeline qw(:all);
10 dbpipeline(
11 dbrow(qw(name test1)),
12 dbroweval('_test1 += 5;')
13 );
14
15 Or for more customized versions, see "dbpipeline_filter",
16 "dbpipeline_sink", "dbpipeline_open2", and "dbpipeline_close2_hash".
17
19 This module makes it easy to create pipelines in Perl using separate
20 processes. (In the past we used to use perl threads.)
21
22 By default (as with all Fsdb modules), input is from STDIN and output
23 to STDOUT. Two helper functions, fromfile and tofile can grab data
24 from files.
25
26 Dbpipeline differs in several ways from all other Fsdb::Filter modules:
27 it does not have a corresponding Unix command (it is used only from
28 within Perl). It does not log its presence to the output stream (this
29 is arguably a bug, but it doesn't actually do anything).
30
32 Unlike most Fsdb modules, dbpipeline defaults to "--autorun".
33
34 This module also supports the standard fsdb options:
35
36 -d Enable debugging output.
37
38 -i or --input InputSource
39 Read from InputSource, typically a file name, or "-" for standard
40 input, or (if in Perl) a IO::Handle, Fsdb::IO or Fsdb::BoundedQueue
41 objects.
42
43 -o or --output OutputDestination
44 Write to OutputDestination, typically a file name, or "-" for
45 standard output, or (if in Perl) a IO::Handle, Fsdb::IO or
46 Fsdb::BoundedQueue objects.
47
48 --autorun or --noautorun
49 By default, programs process automatically, but Fsdb::Filter
50 objects in Perl do not run until you invoke the run() method. The
51 "--(no)autorun" option controls that behavior within Perl.
52
53 --header H
54 Use H as the full Fsdb header, rather than reading a header from
55 then input. This option is particularly useful when using Fsdb
56 under Hadoop, where split files don't have heades.
57
58 --help
59 Show help.
60
61 --man
62 Show full manual.
63
65 Fsdb(3)
66
68 dbpipeline
69 dbpipeline(@modules);
70
71 This shorthand-routine creates a dbpipeline object and then immediately
72 runs it.
73
74 Thus perl code becomes nearly as terse as shell code:
75
76 dbpipeline(
77 dbcol(qw(name test1)),
78 dbroweval('_test1 += 5;'),
79 );
80
81 The following commands currently have shorthand aliases:
82
83 cgi_to_db(1)
84 combined_log_format_to_db(1)
85 csv_to_db(1)
86 db_to_csv(1)
87 db_to_html_table(1)
88 dbcol(1)
89 dbcolcopylast(1)
90 dbcolcreate(1)
91 dbcoldefine(1)
92 dbcolhisto(1)
93 dbcolmerge(1)
94 dbcolmovingstats(1)
95 dbcolneaten(1)
96 dbcolpercentile(1)
97 dbcolrename(1)
98 dbcolscorrelate(1)
99 dbcolsplittocols(1)
100 dbcolsplittorows(1)
101 dbcolsregression(1)
102 dbcolstats(1)
103 dbcolstatscores(1)
104 dbfilealter(1)
105 dbfilecat(1)
106 dbfilediff(1)
107 dbfilepivot(1)
108 dbfilestripcomments(1)
109 dbfilevalidate(1)
110 dbformmail(1)
111 dbjoin(1)
112 dbmapreduce(1)
113 dbmerge(1)
114 dbmerge2(1)
115 dbmultistats(1)
116 dbrow(1)
117 dbrowaccumulate(1)
118 dbrowcount(1)
119 dbrowdiff(1)
120 dbroweval(1)
121 dbrowuniq(1)
122 dbrvstatdiff(1)
123 dbsort(1)
124 html_table_to_db(1)
125 kitrace_to_db(1)
126 mysql_to_db(1)
127 tabdelim_to_db(1)
128 tcpdump_to_db(1)
129 xml_to_db(1)
130
131 and
132
133 dbsubprocess(3)
134
135 dbpipeline_filter
136 my($result_reader, $fred) = dbpipeline_filter($source, $result_reader_aref, @modules);
137
138 Set up a pipeline of @MODULES that filters data pushed through it,
139 where the data comes from $SOURCE (any Fsdb::Filter::parse_io_option
140 object, like a Fsdb::IO::Reader object, queue, or filename).
141
142 Returns a $RESULT_READER Fsdb::IO::Reader object, created with
143 $RESULT_READER_AREF as options. This reader will produce the filtered
144 data, and a $FRED that must be joined to guarantee output has
145 completed.
146
147 Or if $RESULT_READER_AREF is "[-raw_fh, 1]", it just returns the
148 IO::Handle to the pipe.
149
150 As an example, this code uses "dbpipeline_filter" to insure the input
151 (from $in which is a filename or Fsdb::IO::Reader) is sorted
152 numerically by column "x":
153
154 use Fsdb::Filter::dbpipeline qw(dbpipeline_filter dbsort);
155 my($new_in, $new_fred) = dbpipeline_filter($in,
156 [-comment_handler => $self->create_delay_comments_sub],
157 dbsort(qw(--nolog -n x)));
158 while (my $fref = $new_in->read_rowwobj()) {
159 # do something
160 };
161 $new_in->close;
162 $new_fred->join();
163
164 dbpipeline_sink
165 my($fsdb_writer, $fred) = dbpipeline_sink($writer_arguments_aref, @modules);
166
167 Set up a pipeline of @MODULES that is a data "sink", where the output
168 is given by a "--output" argument, or goes to standard output (by
169 default). The caller generates input into the pipeline by writing to a
170 newly created $FSDB_WRITER, whose configuration is specified by the
171 mandatory first argument $WRITER_ARGUMENTS_AREF. (These arguments
172 should include the schema.) Returns this writer, and a $FRED that must
173 be joined to guarantee output has completed.
174
175 If the first argument to modules is "--fred_exit_sub", then the second
176 is taken as a CODE block that runs at fred exit (and the two are not
177 passed to modules).
178
179 If the first argument to modules is "--fred_description", then the
180 second is taken as a text description of the Fred.
181
182 dbpipeline_open2
183 my($fsdb_reader_fh, $fsdb_writer, $fred) =
184 dbpipeline_open2($writer_arguments_aref, @modules);
185
186 Set up a pipeline of @MODULES that is a data sink and source (both!).
187 The caller generates input into the pipeline by writing to a newly
188 created $FSDB_WRITER, whose configuration is specified by the mandatory
189 argument $WRITER_ARGUMENTS_AREF. These arguments should include the
190 schema.) The output of the pipeline comes out to the newly created
191 $FSDB_READER_FH. Returns this read queue and writer, and a $PID that
192 must be joined to guarantee output has completed.
193
194 (Unfortunately the interface is asymmetric with a read queue but a
195 write "Fsdb::IO" object, because "Fsdb::IO::Reader" blocks on input of
196 the header.)
197
198 Like IPC::Open2, with all of its pros and cons like potential deadlock.
199
200 dbpipeline_close2_hash
201 my($href) = dbpipeline_close2_hash($fsdb_read_fh, $fsdb_writer, $pid);
202
203 Reads and returns one line of output from $FSDB_READER, after closing
204 $FSDB_WRITER and joining the $PID.
205
206 Useful, for example, to get dbcolstats output cleanly.
207
208 new
209 $filter = new Fsdb::Filter::dbpipeline(@arguments);
210
211 set_defaults
212 $filter->set_defaults();
213
214 Internal: set up defaults.
215
216 parse_options
217 $filter->parse_options(@ARGV);
218
219 Internal: parse options
220
221 setup
222 $filter->_reap();
223
224 Internal: reap any forked threads.
225
226 setup
227 $filter->setup();
228
229 Internal: setup, parse headers.
230
231 run
232 $filter->run();
233
234 Internal: run over all IO
235
236 finish
237 $filter->finish();
238
239 Internal: we would write a trailer, but we don't because we depend on
240 the last command in the pipeline to do that. We don't actually have a
241 valid output stream.
242
244 Copyright (C) 1991-2018 by John Heidemann <johnh@isi.edu>
245
246 This program is distributed under terms of the GNU general public
247 license, version 2. See the file COPYING with the distribution for
248 details.
249
250
251
252perl v5.34.1 2022-04-04 Fsdb::Filter::dbpipeline(3)