1FLOWDUMPER(1)         User Contributed Perl Documentation        FLOWDUMPER(1)
2
3
4

NAME

6       flowdumper - a grep(1)-like utility for raw flow files
7

SYNOPSIS

9          flowdumper [-h] [-v] [-s|S|r|R] [-a|n] [[-I expr] -e expr [-E expr]] [-c] [-B file] [-o output_file] [flow_file [...]]
10
11       but usually just:
12
13          flowdumper [-s] -e expr flow_file [...]
14

DESCRIPTION

16       flowdumper is a grep(1)-like utility for selecting and processing flows
17       from cflowd or flow-tools raw flow files.  The selection criteria are
18       specified by using the "-e" option described below.
19
20       flowdumper's primary features are the ability to:
21
22       ·   Print the content of raw flow files in one of two built-in formats
23           or a format of the users own.  The built-in "long" format is much
24           like that produced by the flowdump command supplied with cflowd.
25           The "short", single-line format is suitable for subsequent post-
26           processing by line-oriented filters like sed(1).
27
28       ·   Act as a filter, reading raw flow input from either file(s) or
29           standard input, and producing filtered raw flow output on standard
30           output.  This is similar to how grep(1) is often used on text
31           files.
32
33       ·   Select flows according to practically any criteria that can be
34           expressed in perl syntax.
35
36       The "flow variables" and other symbols available for use in the "-e"
37       expression are those made available by the Cflow module when used like
38       this:
39
40          use Cflow qw(:flowvars :tcpflags :icmptypes :icmpcodes);
41
42       See the Cflow perl documentation for full details on these values (i.e.
43       "perldoc Cflow".)
44
45       Most perl syntax is allowed in the expressions specified with the "-e",
46       "-I", and "-E" options.  See the perl man pages for full details on
47       operators ("man perlop") and functions ("man perlfunc") available for
48       use in those expressions.
49
50       If run with no arguments, filters standard input to standard output.
51
52       The options and their arguments, roughly in order of usefulness, are:
53
54       "-h"
55           shows the usage information
56
57           mnemonic: 'h'elp
58
59       "-a"
60           print all flows
61
62           implied if "-e" is not specified
63
64           mnemonic: 'a'll
65
66       "-e" expr
67           evaluate this expression once per flow
68
69           mnemonic: 'e'xpression
70
71       "-c"
72           print number of flows matched in input
73
74           mnemonic: 'c'ount
75
76       "-s"
77           print flows in short (one-line) format, ignored with "-n"
78
79           mnemonic: 's'hort
80
81       "-r"
82           print flows in the raw/binary flow file format
83
84           ignored with "-n"
85
86           mnemonic: 'r'aw
87
88       "-R"
89           "repacks" and print flows in the raw/binary flow file format
90
91           requires "-e", ignored with "-n", useful with "-p"
92
93           mnemonic: 'R'epack raw
94
95       "-n"
96           don't print matching flows
97
98           mnemonic: like "perl "-n"" or "sed "-n""
99
100       "-o" output_file
101           send output to the specified file.  A single printf(3) string
102           conversion specifier can be used within the output_file value (such
103           as "/tmp/%s.txt") to make the output file name a function of the
104           input file basename.
105
106           mneomic: 'o'utput file
107
108       "-S"
109           print flows in the "old" short (one-line) format
110
111           ignored with "-n"
112
113           mnemonic: 'S'hort
114
115       "-v"
116           be verbose with messages
117
118           mnemonic: 'v'erbose
119
120       "-V"
121           be very verbose with messages (implies ""-v"")
122
123           mnemonic: 'V'ery verbose
124
125       "-I" expr
126           eval expression initially, before flow processing
127
128           practically useless without "-e"
129
130           mnemonic: 'I'nitial expression
131
132       "-E" expr
133           eval expression after flow processing is complete
134
135           practically useless without "-e"
136
137           mnemonic: 'E'ND expression
138
139       "-B" file
140           Load the specified BGP dump file using Net::ParseRouteTable.
141
142           In your optional expression, you can now refer to these variables:
143
144              $dst_as_path_arrayref
145              $dst_origin_as
146              $dst_peer_as
147              $src_as_path_arrayref
148              $src_origin_as
149              $src_peer_as
150
151           which will cause a lookup.  Their values are undefined if the
152           lookup fails.
153
154           mnemonic: 'B'GP dump file
155
156       "-p" prefix_mappings_file
157           read file containing IPv4 prefix mappings in this format (one per
158           line):
159
160              10.42.69.0/24 -> 10.69.42.0/24
161              ...
162
163           When specifying this option, you can, and should at some point,
164           call the ENCODE subroutine in your expressions to have it encode
165           the IP address flowvars such as $Cflow::exporter, $Cflow::srcaddr,
166           $Cflow::dstaddr, and $Cflow::nexthop.
167
168           mnemonic: 'p'refixes
169

EXAMPLES

171       Print all flows, in a multi-line format, to a pager:
172
173          $ flowdumper -a flows.* |less
174
175       Print all the UDP flows to another file using the raw binary flow
176       format:
177
178          $ flowdumper -re '17 == $protocol' flows.current > udp_flows.current
179
180       Print all TCP flows which have the SYN bit set in the TCP flags:
181
182          $ flowdumper -se '6 == $protocol && ($TH_SYN & $tcp_flags)' flows.*
183
184       Print the first 10 flows to another file using the raw binary flow
185       format:
186
187          $ flowdumper -I '$n = 10' -re '$n-- or exit' flows.*0 > head.cflow
188
189       Print all flows with the start and end time using a two-line format:
190
191          $ flowdumper -se 'print scalar(localtime($startime)), "\n"' flows.*
192
193       Print all flows with the specified source address using a short,
194       single-line format:
195
196          $ flowdumper -se '"10.42.42.42" eq $srcip' flows.*
197
198       Do the same thing in a quicker, but less obvious, way:
199
200          $ flowdumper -I '
201             use Socket;
202             $addr = unpack("N", Socket::inet_aton("10.42.42.42"));
203          ' -se '$addr == $srcaddr'  flows.*
204
205       (This latter method runs quicker because inet_aton(3) is only called
206       once, instead of once per flow.)
207
208       Print all flows with a source address within the specifed
209       network/subnet:
210
211          $ flowdumper \
212          -I 'use Socket;
213              $mask = unpack("N", Socket::inet_aton("10.42.0.0"));
214              $width = 16' \
215          -se '$mask == ((0xffffffff << (32-$width)) & $srcaddr)' flows.*
216
217       Print all flows where either the source or the destination address, but
218       not both, is within the specified set of networks or subnets:
219
220          $ flowdumper \
221          -I 'use Net::Patricia;
222              $pt = Net::Patricia->new;
223              map { $pt->add_string($_, 1) } qw( 10.42.0.0/16
224                                                 10.69.0.0/16 )' \
225          -se '1 == ($pt->match_integer($srcaddr) +
226                     $pt->match_integer($dstaddr))' flows.*
227
228       Count the total number of "talkers" (unique source host addresses) by
229       piping them to sort(1) and wc(1) to count them:
230
231          $ flowdumper \
232          -I 'use Net::Patricia;
233              $pt = Net::Patricia->new;
234              map { $pt->add_string($_, 1) } qw( 10.42.0.0/16
235                                                 10.69.0.0/16 )' \
236          -ne '$pt->match_integer($srcaddr) and print "$srcip\n"' flows.* \
237          |sort -u |wc -l
238
239       Count the total number of "talkers" (unique source host addresses) that
240       are within a the specified networks or subnets:
241
242          $ flowdumper \
243          -I 'use Net::Patricia;
244              $pt = new Net::Patricia;
245              map { $pt->add_string($_, 1) } qw( 10.42.0.0/16
246                                                 10.69.0.0/16 );
247              $talkers = new Net::Patricia' \
248          -ne '$pt->match_integer($srcaddr) &&
249               ($talkers->match_integer($srcaddr) or
250                $talkers->add_string($srcip, 1))' \
251          -E 'printf("%d\n", $talkers->climb( sub { 1 } ))' flows.*
252
253       (For large numbers of flows, this latter method is quicker because it
254       populates a Net::Patricia trie with the unique addresses and counts the
255       resulting nodes rather than having to print them to standard output and
256       then having to sort them to determine how many are unique.)
257
258       Select the TCP flows and "ENCODE" the IP addresses according to the
259       prefix encodings specified in "prefix_encodings.txt":
260
261          $ flowdumper -p prefix_encodings.txt -se '6 == $protocol && ENCODE'
262
263       Produce a new raw flow file with the IP addresses ENCODEd according to
264       the prefix encodings specified in "prefix_encodings.txt":
265
266          $ flowdumper -p prefix_encodings.txt -Re 'ENCODE' flows > flows.enc
267
268       Produce a set of raw flow files that have the $src_as and $dst_as
269       origin AS values filled in based upon a lookup in externally-specified
270       routing table (in the file "router.bgp") and have the IP address info
271       replaces with zeroes (for anonymity):
272
273          $ ssh router "show route protocol bgp terse" > router.bgp # Juniper
274
275          $ flowdumper \
276          -B router.bgp \
277          -e '$src_as = $src_origin_as,
278              $dst_as = $dst_origin_as,
279              (($exporter = 0),
280               ($srcaddr  = 0),
281               ($src_mask = 0),
282               ($dstaddr  = 0),
283               ($dst_mask = 0),
284               ($nexthop  = 0), 1)' \
285          -R \
286          -o /tmp/%s.cflow_enc \
287          flows*
288

NOTES

290       This utility was inspired by Daniel McRobb's flowdump utility which is
291       supplied with cflowd.  flowdumper was originally written as merely a
292       sample of what can be done with the Cflow perl module, but has since
293       been developed into a more complete tool.
294

BUGS

296       When using the "-B" option, routing table entries that contain AS sets
297       at the end of the AS path are quietly discarded.  (It's not so quiet if
298       you also specified "-V".)  It was necessary to discard these, because I
299       did not consider AS sets when designing the API and therefore have no
300       way to communicate more than one origin AS value per for a single
301       source or destination IP address.
302
303       There are perhaps some pathological combinations of options that
304       currently do not produce usage error messages, but should.
305
306       Since the expression syntax is that of perl itself, there are lots of
307       useless expressions that will happily be accepted without complaint.
308       This is particular troublesome when trying to track down typos, for
309       instance, with the flow variable names.
310
311       This script probably has the same bugs as the Cflow module, since it's
312       based upon it.
313

AUTHOR

315       Dave Plonka <plonka@doit.wisc.edu>
316
317       Copyright (C) 1998-2005  Dave Plonka.  This program is free software;
318       you can redistribute it and/or modify it under the terms of the GNU
319       General Public License as published by the Free Software Foundation;
320       either version 2 of the License, or (at your option) any later version.
321

SEE ALSO

323       perl(1), Socket, Net::Netmask, Net::Patricia, Cflow.
324
325
326
327perl v5.30.1                      2020-01-29                     FLOWDUMPER(1)
Impressum