1FLOWDUMPER(1) User Contributed Perl Documentation FLOWDUMPER(1)
2
6 flowdumper - a grep(1)-like utility for raw flow files
7
9 flowdumper [-h] [-v] [-s|S|r|R] [-a|n] [[-I expr] -e expr [-E expr]] [-c] [-B file] [-o output_file] [flow_file [...]]
10 but usually just:
12 flowdumper [-s] -e expr flow_file [...]
14
16 flowdumper is a grep(1)-like utility for selecting and processing flows
17 from cflowd or flow-tools raw flow files. The selection criteria are
18 specified by using the "-e" option described below.
19 flowdumper's primary features are the ability to:
21 • Print the content of raw flow files in one of two built-in formats
23 or a format of the users own. The built-in "long" format is much
24 like that produced by the flowdump command supplied with cflowd.
25 The "short", single-line format is suitable for subsequent post-
26 processing by line-oriented filters like sed(1).
27 • Act as a filter, reading raw flow input from either file(s) or
29 standard input, and producing filtered raw flow output on standard
30 output. This is similar to how grep(1) is often used on text
31 files.
32 • Select flows according to practically any criteria that can be
34 expressed in perl syntax.
35 The "flow variables" and other symbols available for use in the "-e"
37 expression are those made available by the Cflow module when used like
38 this:
39 use Cflow qw(:flowvars :tcpflags :icmptypes :icmpcodes);
41 See the Cflow perl documentation for full details on these values (i.e.
43 "perldoc Cflow".)
44 Most perl syntax is allowed in the expressions specified with the "-e",
46 "-I", and "-E" options. See the perl man pages for full details on
47 operators ("man perlop") and functions ("man perlfunc") available for
48 use in those expressions.
49 If run with no arguments, filters standard input to standard output.
51 The options and their arguments, roughly in order of usefulness, are:
53 "-h"
55 shows the usage information
56 mnemonic: 'h'elp
58 "-a"
60 print all flows
61 implied if "-e" is not specified
63 mnemonic: 'a'll
65 "-e" expr
67 evaluate this expression once per flow
68 mnemonic: 'e'xpression
70 "-c"
72 print number of flows matched in input
73 mnemonic: 'c'ount
75 "-s"
77 print flows in short (one-line) format, ignored with "-n"
78 mnemonic: 's'hort
80 "-r"
82 print flows in the raw/binary flow file format
83 ignored with "-n"
85 mnemonic: 'r'aw
87 "-R"
89 "repacks" and print flows in the raw/binary flow file format
90 requires "-e", ignored with "-n", useful with "-p"
92 mnemonic: 'R'epack raw
94 "-n"
96 don't print matching flows
97 mnemonic: like "perl "-n"" or "sed "-n""
99 "-o" output_file
101 send output to the specified file. A single printf(3) string
102 conversion specifier can be used within the output_file value (such
103 as "/tmp/%s.txt") to make the output file name a function of the
104 input file basename.
105 mneomic: 'o'utput file
107 "-S"
109 print flows in the "old" short (one-line) format
110 ignored with "-n"
112 mnemonic: 'S'hort
114 "-v"
116 be verbose with messages
117 mnemonic: 'v'erbose
119 "-V"
121 be very verbose with messages (implies ""-v"")
122 mnemonic: 'V'ery verbose
124 "-I" expr
126 eval expression initially, before flow processing
127 practically useless without "-e"
129 mnemonic: 'I'nitial expression
131 "-E" expr
133 eval expression after flow processing is complete
134 practically useless without "-e"
136 mnemonic: 'E'ND expression
138 "-B" file
140 Load the specified BGP dump file using Net::ParseRouteTable.
141 In your optional expression, you can now refer to these variables:
143 $dst_as_path_arrayref
145 $dst_origin_as
146 $dst_peer_as
147 $src_as_path_arrayref
148 $src_origin_as
149 $src_peer_as
150 which will cause a lookup. Their values are undefined if the
152 lookup fails.
153 mnemonic: 'B'GP dump file
155 "-p" prefix_mappings_file
157 read file containing IPv4 prefix mappings in this format (one per
158 line):
159 10.42.69.0/24 -> 10.69.42.0/24
161 ...
162 When specifying this option, you can, and should at some point,
164 call the ENCODE subroutine in your expressions to have it encode
165 the IP address flowvars such as $Cflow::exporter, $Cflow::srcaddr,
166 $Cflow::dstaddr, and $Cflow::nexthop.
167 mnemonic: 'p'refixes
169
171 Print all flows, in a multi-line format, to a pager:
172 $ flowdumper -a flows.* |less
174 Print all the UDP flows to another file using the raw binary flow
176 format:
177 $ flowdumper -re '17 == $protocol' flows.current > udp_flows.current
179 Print all TCP flows which have the SYN bit set in the TCP flags:
181 $ flowdumper -se '6 == $protocol && ($TH_SYN & $tcp_flags)' flows.*
183 Print the first 10 flows to another file using the raw binary flow
185 format:
186 $ flowdumper -I '$n = 10' -re '$n-- or exit' flows.*0 > head.cflow
188 Print all flows with the start and end time using a two-line format:
190 $ flowdumper -se 'print scalar(localtime($startime)), "\n"' flows.*
192 Print all flows with the specified source address using a short,
194 single-line format:
195 $ flowdumper -se '"10.42.42.42" eq $srcip' flows.*
197 Do the same thing in a quicker, but less obvious, way:
199 $ flowdumper -I '
201 use Socket;
202 $addr = unpack("N", Socket::inet_aton("10.42.42.42"));
203 ' -se '$addr == $srcaddr' flows.*
204 (This latter method runs quicker because inet_aton(3) is only called
206 once, instead of once per flow.)
207 Print all flows with a source address within the specifed
209 network/subnet:
210 $ flowdumper \
212 -I 'use Socket;
213 $mask = unpack("N", Socket::inet_aton("10.42.0.0"));
214 $width = 16' \
215 -se '$mask == ((0xffffffff << (32-$width)) & $srcaddr)' flows.*
216 Print all flows where either the source or the destination address, but
218 not both, is within the specified set of networks or subnets:
219 $ flowdumper \
221 -I 'use Net::Patricia;
222 $pt = Net::Patricia->new;
223 map { $pt->add_string($_, 1) } qw( 10.42.0.0/16
224 10.69.0.0/16 )' \
225 -se '1 == ($pt->match_integer($srcaddr) +
226 $pt->match_integer($dstaddr))' flows.*
227 Count the total number of "talkers" (unique source host addresses) by
229 piping them to sort(1) and wc(1) to count them:
230 $ flowdumper \
232 -I 'use Net::Patricia;
233 $pt = Net::Patricia->new;
234 map { $pt->add_string($_, 1) } qw( 10.42.0.0/16
235 10.69.0.0/16 )' \
236 -ne '$pt->match_integer($srcaddr) and print "$srcip\n"' flows.* \
237 |sort -u |wc -l
238 Count the total number of "talkers" (unique source host addresses) that
240 are within a the specified networks or subnets:
241 $ flowdumper \
243 -I 'use Net::Patricia;
244 $pt = new Net::Patricia;
245 map { $pt->add_string($_, 1) } qw( 10.42.0.0/16
246 10.69.0.0/16 );
247 $talkers = new Net::Patricia' \
248 -ne '$pt->match_integer($srcaddr) &&
249 ($talkers->match_integer($srcaddr) or
250 $talkers->add_string($srcip, 1))' \
251 -E 'printf("%d\n", $talkers->climb( sub { 1 } ))' flows.*
252 (For large numbers of flows, this latter method is quicker because it
254 populates a Net::Patricia trie with the unique addresses and counts the
255 resulting nodes rather than having to print them to standard output and
256 then having to sort them to determine how many are unique.)
257 Select the TCP flows and "ENCODE" the IP addresses according to the
259 prefix encodings specified in "prefix_encodings.txt":
260 $ flowdumper -p prefix_encodings.txt -se '6 == $protocol && ENCODE'
262 Produce a new raw flow file with the IP addresses ENCODEd according to
264 the prefix encodings specified in "prefix_encodings.txt":
265 $ flowdumper -p prefix_encodings.txt -Re 'ENCODE' flows > flows.enc
267 Produce a set of raw flow files that have the $src_as and $dst_as
269 origin AS values filled in based upon a lookup in externally-specified
270 routing table (in the file "router.bgp") and have the IP address info
271 replaces with zeroes (for anonymity):
272 $ ssh router "show route protocol bgp terse" > router.bgp # Juniper
274 $ flowdumper \
276 -B router.bgp \
277 -e '$src_as = $src_origin_as,
278 $dst_as = $dst_origin_as,
279 (($exporter = 0),
280 ($srcaddr = 0),
281 ($src_mask = 0),
282 ($dstaddr = 0),
283 ($dst_mask = 0),
284 ($nexthop = 0), 1)' \
285 -R \
286 -o /tmp/%s.cflow_enc \
287 flows*
288
290 This utility was inspired by Daniel McRobb's flowdump utility which is
291 supplied with cflowd. flowdumper was originally written as merely a
292 sample of what can be done with the Cflow perl module, but has since
293 been developed into a more complete tool.
294
296 When using the "-B" option, routing table entries that contain AS sets
297 at the end of the AS path are quietly discarded. (It's not so quiet if
298 you also specified "-V".) It was necessary to discard these, because I
299 did not consider AS sets when designing the API and therefore have no
300 way to communicate more than one origin AS value per for a single
301 source or destination IP address.
302 There are perhaps some pathological combinations of options that
304 currently do not produce usage error messages, but should.
305 Since the expression syntax is that of perl itself, there are lots of
307 useless expressions that will happily be accepted without complaint.
308 This is particular troublesome when trying to track down typos, for
309 instance, with the flow variable names.
310 This script probably has the same bugs as the Cflow module, since it's
312 based upon it.
313
315 Dave Plonka <plonka@doit.wisc.edu>
316 Copyright (C) 1998-2005 Dave Plonka. This program is free software;
318 you can redistribute it and/or modify it under the terms of the GNU
319 General Public License as published by the Free Software Foundation;
320 either version 2 of the License, or (at your option) any later version.
321
323 perl(1), Socket, Net::Netmask, Net::Patricia, Cflow.
324perl v5.38.0 2023-07-20 FLOWDUMPER(1)