1tcpflow(1) tcpflow 1.6.1 tcpflow(1)
2
6 tcpflow - TCP flow recorder
7
9 tcpflow [-aBcCDIpsZ] [-b max_bytes] [-d debug_level] [-[eE] scanner]
10 [-f max_fds] [-F[ctTXMkmg]] [-h|--help] [-i iface]
11 [-l file1.pcap file2.pcap...] [-L semlock] [-m min_bytes] [-o outdir]
12 [-r file1.pcap] [-R file0.pcap] [-S name=value] [-T[filename template]]
13 [-U|--relinquish-privileges username] [-v|--verbose] [-V|--version]
14 [-w file] [-x scanner] [-X file.xml] [-z|--chroot directory] [expres‐
15 sion]
16
18 tcpflow is a program that captures data transmitted as part of TCP con‐
19 nections (flows), and stores the data in a way that is convenient for
20 protocol analysis or debugging. Rather than showing packet-by-packet
21 information, tcpflow reconstructs the actual data streams and stores
22 each flow in a separate file for later analysis. tcpflow understands
23 TCP sequence numbers and will correctly reconstruct data streams
24 regardless of retransmissions or out-of-order delivery. tcpflow pro‐
25 vides control over filenames for automatic binning of connections by
26 protocol, IP address or connection number, and has a sophisticated
27 plug-in system for decompressing compressed HTTP connections, undoing
28 MIME encoding, or calling user-provided programs for post-processing.
29 By default tcpflow stores all captured data in files that have names of
31 the form:
32 192.168.101.102.02345-010.011.012.013.45103
34 ...where the contents of the above file would be data transmitted from
36 host 192.168.101.102 port 2345, to host 10.11.12.13 port 45103.
37 If you want to simply process a few hundred thousand packets and see
39 what you have, try this:
40 tcpflow -a -o outdir -Fk -r packets.pcap
42 This will cause tcpflow to perform (-a) all processing, store the out‐
44 put in a directory called outdir, bin the output in directories of 1000
45 connections each, and read its input from the file packets.pcap. More
46 sophisticated processing is possible, of course.
47
49 -a Enable all processing. Same as -e all.
50 -B Force binary output even when printing to console with -C or -c.
52 -b max_bytes
54 Specifies the maximum size of a captured flow. Any bytes beyond
55 max_bytes from the first byte captured will be discarded. The
56 default is to store an unlimited number of bytes per flow. Note:
57 before version 1.4, tcpflow could only store a maximum of 4GiB
58 per flow.
59 -c Console print. Print the contents of packets to stdout as they
61 are received, without storing any captured data to files
62 (implies -s).
63 -C Console print without the packet source and destination details
65 being printed. Print the contents of packets to stdout as they
66 are received, without storing any captured data to files
67 (implies -s).
68 -D Console output should be in hex.
70 -d Debug level. Set the level of debugging messages printed to
72 stderr to debug_level. Higher numbers produce more messages.
73 -d 0 causes completely silent operation. -d 1 , the default,
74 produces minimal status messages. -d 10 produces verbose output
75 equivalent to -v . Numbers higher than 10 can produce a large
76 amount of debugging information useful only to developers.
77 -E name
79 Disable all scanners and then enable scanner name
80 -e name
82 Enable scanner name.
83 -e all Enables all scanners. Same as -a
85 -e http
87 Perform HTTP post-processing ("After" processing). If the output
88 file is
89 208.111.153.175.00080-192.168.001.064.37314,
91 Then the post-processing will create the files:
93 208.111.153.175.00080-192.168.001.064.37314-HTTP
95 208.111.153.175.00080-192.168.001.064.37314-HTTPBODY
96 If the HTTPBODY was compressed with GZIP, you may get a third
98 file as well:
99 208.111.153.175.00080-192.168.001.064.37314-HTTPBODY-GZIP
101 Additional information about these streams, such as their MD5
103 hash value, is also written to the DFXML report file.
104 -e python -S py_path=path -S py_module=module -S py_function=foo
106 Post-process TCP payload by an external python function.
107 The python function must take a single string parameter. The
109 python function can return a string (else the function does must
110 not return). The returned string (if any) is written in the
111 DFXML report file inside the XML tag
112 <scan_python_result>...</scan_python_result>. A sample python
113 script is available within the tcpflow source code in directory
114 python/plugins.
115 Example:
117 tcpflow -r my.cap -e python -S py_path=python/plugins -S py_module=samplePlugin -S py_function=sampleFunction
119 -F[format]
121 Specifies format for output filenames.
122 Format specifiers:
124 c Appends the connection counter to ALL filenames.
126 t Prepends each filename with a Unix timestamp (seconds
128 since epoch).
129 T Prepends each filename with an ISO-8601 timestamp.
131 X Do not output any files (other than the DFXML report
133 file).
134 -FM Include MD5 of each flow in the DFXML report file.
136 -FX Suppresses file output entirely, DFXML report file is still pro‐
138 duced.
139 -Fk bin output in 1K directories
141 -Fm bin output in 1M directories (2 levels)
143 -Fg bin output in 1G directories (3 levels)
145 -fmax_fds
147 Max file descriptors used. Limit the number of file descriptors
148 used by tcpflow to max_fds. Higher numbers use more system
149 resources, but usually perform better. If the underlying oper‐
150 ating system supports the setrlimit() system call, the OS will
151 be asked to enforce the requested limit. The default is for
152 tcpflow to use the maximum number of file descriptors allowed by
153 the OS. The -v option will report how many file descriptors
154 tcpflow is using.
155 -g Output flow information to console in multiple colors. (Blue for
157 client to server flows, red for server to client flows, green
158 for undecided flows.) Note: This option was different from
159 tcpflow 1.3 (-e) and 1.4.4 (-J).
160 -h --help
162 Help. Print usage information and exit.
163 -hh More help. Print more usage information and exit.
165 -i iface
167 Interface name. Capture packets from the network interface
168 named iface. If no interface is specified with -i , a reason‐
169 able default will be used by libpcap automatically.
170 -I Store the reception timestamps (of TCP packets) in a companion
172 file *.findx. Therefore each flow will have two files: (1) the
173 usual file containing payload bytes and (2) the text file con‐
174 taining the corresponding timestamps. This last file *.findx
175 has three columns using the pipe '|' as separator:
176 byte-index|timestamp|length
178 The byte-index column is the postion within the file containing
180 the payload bytes. The timestamp column represents the number
181 of seconds since epoch as a floating point number. The preci‐
182 sion is the microsecond but may also be the nanosecond in a
183 future tcpflow version. The length column is the number of suc‐
184 cessive bytes concerned by timestamp and can include several TCP
185 frames (TCP packets). The extension findx may become from the
186 fact that the timestamps are frame indexed.
187 -L semlock_name
189 Specifies that semlock_name should be used as a Unix semaphore
190 to prevent two different copies of tcpflow running in two dif‐
191 ferent processes but outputting to the same standard output from
192 printing on top of each other. This is an application of Unix
193 named semaphores; bet you have never seen one before.
194 -l Treat the following arguments as filenames with an assumed -r
196 command before each one. This allows you to read a lot of files
197 at once with shell globbing. For example, to process all of the
198 pcap files in the current directory, use this:
199 tcpflow -o out -a -l *.pcap
201 -m min_size
204 Forces a new connection output file when there is a skip in the
205 TCP session of min_size bytes or more.
206 -o outdir
208 Specifies the output directory where the transcript files will
209 be written.
210 -P No purge. Normally tcpflow removes connections from the hash ta‐
212 ble after the connection is closed with a FIN. This conserves
213 memory but takes additional CPU time. Selecting this option
214 causes the std::tr1:unordered_map to grow without bounds, as
215 tcpflow did prior to version 1.1. That makes tcpflow run faster
216 if there are less than 10 million connections, but can lead to
217 out-of-memory errors.
218 -p No promiscuous mode. Normally, tcpflow attempts to put the net‐
220 work interface into promiscuous mode before capturing packets.
221 The -p option tells tcpflow not to put the interface into pro‐
222 miscuous mode. Note that it might already be in promiscuous
223 mode for some other reason.
224 -q Quiet mode --- don't print warnings. Currently the only warning
226 that tcpflow prints is a warning when more than 10,000 files are
227 created that the user should have provided the -Fk, -Fm, or -Fg
228 options. We might have other warnings in the future.
229 --relinquish-privileges=username
231 When tcpflow is run as root, this option changes the user ID and
232 group ID to write files owned by username. The group ID is the
233 first one from the username groups list. This operation is per‐
234 formed just after opening the capture device or just after open‐
235 ing the first input PCAP file. This option does not support
236 multi root-only readable input files as the root privileges are
237 dropped after opening the first file (e.g. -r root-only-
238 access.pcap -R root-only.pcap -l root-only*.pcap). This option
239 has the same behaviour as the tcpdump(1) option having the same
240 name --relinquish-privileges
241 -r Read from file. Read packets from file, which was created using
243 the -w option of tcpdump(1). This option may be repeated any
244 number of times. Standard input is used if file is "-". Note
245 that for this option to be useful, tcpdump's -s option should be
246 used to set the snaplen to the MTU of the interface (e.g., 1500)
247 while capturing packets.
248 -R Read from a file, but only to complete TCP flows. This option is
250 used when tcpflow is used to process a series of files that are
251 captured over time. For each time period n, file file(n).pcap
252 should be processed with -R file(n).pcap, while file(n-1).pcap
253 should be processed with -r file(n-1).pcap.
254 -Sname=value
256 Sets a name parameter to be equal to value for a plug-in. Use
257 -hh to find out all of the settable parameters.
258 -s Strip non-printables. Convert all non-printable characters to
260 the "." character before printing packets to the console or
261 storing them to a file.
262 -T[format]
264 Specifies an arbitrary template for filenames.
265 %A expands to source IP address.
267 %a expands to source IP port.
269 %B expands to destination IP address.
271 %b expands to destination IP port.
273 %T expands to timestamp in ISO8601 format.
275 %t expands to timestamp in Unix time_t format.
277 %V expands to "--" if a VLAN is present.
279 %v expands to the VLAN number if a VLAN is present.
281 %C expands to "c" if the connection count>0.
283 %c expands to the connection count if the connection
285 count>0.
286 %# always expands to the connection count.
288 %N (connection_number ) % 1000
290 %K (connection_number / 1000) % 1000
292 %M (connection_number / 1000000) % 1000
294 %G (connection_number / 1000000000) % 1000
296 %% prints a "%".
298 When the option -T is used, tcpflow ignores options -Fk,
300 -Fm and -Fg.
301 However, the option -T handles '/' within the filename template
302 patern to create sub-directories. For example the following
303 line will create a directory tree out/IP-src/port-src/IP-
304 dst/port-dst.
305 tcpflow -r packets.pcap -o out -T %A/%a/%B/%b/%c%N.flow
307 -V --version
309 Print the version number and exit.
310 -v --verbose
312 Verbose operation. Verbosely describe tcpflow's operation.
313 Equivalent to -d 10.
314 -w filename.pcap
316 Write packets that were not processed to filename.pcap. Typi‐
317 cally this will be UDP packets.
318 -X filename.xml
320 Write a DFXML report to filename.xml. The file contains a record
321 of every tcp connection, how the tcpflow program was compiled,
322 and the computer on which tcpflow was run. By default tcpflow
323 writes the DFXML report in file report.xml.
324 -Z Don't decompress gzip-compressed streams. -K Retain per flow
326 isolated pcap structure.
327 expression
329 selects which packets will be captured. If no expression is
330 given, all packets on the net will be captured. Otherwise, only
331 packets for which expression is `true' will be captured.
332 For the expression syntax, see pcap-filter(7).
334 The expression argument can be passed to tcpflow as either a
336 single Shell argument, or as multiple Shell arguments, whichever
337 is more convenient. Generally, if the expression contains Shell
338 metacharacters, such as backslashes used to escape protocol
339 names, it is easier to pass it as a single, quoted argument
340 rather than to escape the Shell metacharacters. Multiple argu‐
341 ments are concatenated with spaces before being parsed.
342
345 The DFXML report is the XML file written by tcpflow to provide tcpflow
346 build details, command line arguments and information about processed
347 flows.
348 By default the DFXML file is named report.xml. But this filename can
350 be changed using command line option -X.
351 DFXML file respects the DFXML schema defined by project
353 https://github.com/dfxml-working-group/dfxml_schema.
354 Moreover tcpflow adds two extra XML tags, as illustrated by the follow‐
355 ing example:
356 <tcpflow startime='2017-07-22T00:12:21.962782Z' endtime='2017-07-22T00:12:22.097591Z'
358 family='2' mac_daddr='40:3d:78:57:ed:d4' mac_saddr='00:c5:42:d2:cb:f2'
359 src_ipn='141.134.34.12' dst_ipn='192.168.0.40' srcport='80' dstport='38797'
360 packets='4' len='677' caplen='611' />
361 <tcpflow:result scanner="python" path="python/plugins" module="samplePlugin"
363 function="sampleFunction">bla bla bla</tcpflow:result>
364 The first XML tag <tcpflow> provide information about the captured
366 flow. This tag should be renamed <tcpflow:cap> in a future version in
367 order to conform better to DFXML schema.
368 The second XML tag <tcpflow:result> collects processing results. For
370 the moment, only the scanner python uses this feature.
371 The XML attributes of <tcpflow> are:
373 · startime Reception time of first packet
375 · endtime Reception time of last packet
377 · family
379 · mac_daddr Destination MAC address of first packet (printed if
381 any)
382 · mac_saddr Source MAC address of first packet (printed if any)
384 · src_ipn IP source
386 · dst_ipn IP destination
388 · srcport TCP port source
390 · dstport TCP port destination
392 · packets Nummber of packets
394 · out_of_order_count Number of times tcpflow has replaced missing
396 payload by zeros in the flow file, for example when capture does
397 not contain the TCP session begin (printed if any)
398 · violations Number of protocol violations (printed if any)
400 · len Sum of un-truncated length of all packet data (including
402 headers, see https://stackoverflow.com/q/1491660)
403 · caplen Sum of captured bytes of all packet data (including head‐
405 ers, printed if different from len)
406 The XML attributes of <tcpflow:result> are:
408 · scanner Name of the scanner
410 · path Directory of the scanner module (printed if relevant)
412 · module Module name (printed if relevant, used to indicate the
414 python script)
415 · function Function name (printed if relevant, used to indicate
417 the function within the python module)
418
420 To record all packets arriving at or departing from sundown and extract
421 all of the HTTP attachments:
422 tcpflow -e http -o outdir host sundown
423 To record traffic between helios and either hot or ace and bin the
425 results into 1000 files per directory and calculate the MD5 of each
426 flow:
427 tcpflow -X report.xml -e md5 -o outdir -Fk host helios and \( hot or ace \)
428
430 Please send bug reports to simsong@acm.org.
431 tcpflow currently does not understand IP fragments. Flows containing
433 IP fragments will not be recorded correctly.
434
436 Originally by Jeremy Elson <jelson@circlemud.org>. Substantially modi‐
437 fied and maintained by Simson L. Garfinkel <simsong@acm.org>. Network
438 visualization code by Michael Shick <mike@shick.in>
439 The current version of this software is available at
441 http://digitalcorpora.org/downloads/tcpflow/
442 An announcement mailing list for this program is at:
444 http://groups.google.com/group/tcpflow-users
445
447 tcpdump(1), nit(4P), bpf(4), pcap(3), pcap-savefile(5), pcap-filter(7)
448tcpflow 1.6.1 2013-04-13 tcpflow(1)