1tcpflow(1)                       tcpflow 1.6.1                      tcpflow(1)
2
3
4

NAME

6       tcpflow - TCP flow recorder
7

SYNOPSIS

9       tcpflow [-aBcCDIpsZ] [-b max_bytes] [-d debug_level] [-[eE] scanner]
10       [-f max_fds] [-F[ctTXMkmg]] [-h|--help] [-i iface]
11       [-l file1.pcap file2.pcap...]  [-L semlock] [-m min_bytes] [-o outdir]
12       [-r file1.pcap] [-R file0.pcap] [-S name=value] [-T[filename template]]
13       [-U|--relinquish-privileges username] [-v|--verbose] [-V|--version]
14       [-w file] [-x scanner] [-X file.xml] [-z|--chroot directory] [expres‐
15       sion]
16

DESCRIPTION

18       tcpflow is a program that captures data transmitted as part of TCP con‐
19       nections (flows), and stores the data in a way that is convenient for
20       protocol analysis or debugging.  Rather than showing packet-by-packet
21       information, tcpflow reconstructs the actual data streams and stores
22       each flow in a separate file for later analysis.  tcpflow understands
23       TCP sequence numbers and will correctly reconstruct data streams
24       regardless of retransmissions or out-of-order delivery. tcpflow pro‐
25       vides control over filenames for automatic binning of connections by
26       protocol, IP address or connection number, and has a sophisticated
27       plug-in system for decompressing compressed HTTP connections, undoing
28       MIME encoding, or calling user-provided programs for post-processing.
29
30       By default tcpflow stores all captured data in files that have names of
31       the form:
32
33            192.168.101.102.02345-010.011.012.013.45103
34
35       ...where the contents of the above file would be data transmitted from
36       host 192.168.101.102 port 2345, to host 10.11.12.13 port 45103.
37
38       If you want to simply process a few hundred thousand packets and see
39       what you have, try this:
40
41            tcpflow -a -o outdir -Fk -r packets.pcap
42
43       This will cause tcpflow to perform (-a) all processing, store the out‐
44       put in a directory called outdir, bin the output in directories of 1000
45       connections each, and read its input from the file packets.pcap. More
46       sophisticated processing is possible, of course.
47

OPTIONS

49       -a     Enable all processing. Same as -e all.
50
51       -B     Force binary output even when printing to console with -C or -c.
52
53       -b max_bytes
54              Specifies the maximum size of a captured flow.  Any bytes beyond
55              max_bytes from the first byte captured will be discarded.  The
56              default is to store an unlimited number of bytes per flow. Note:
57              before version 1.4, tcpflow could only store a maximum of 4GiB
58              per flow.
59
60       -c     Console print.  Print the contents of packets to stdout as they
61              are received, without storing any captured data to files
62              (implies -s).
63
64       -C     Console print without the packet source and destination details
65              being printed.  Print the contents of packets to stdout as they
66              are received, without storing any captured data to files
67              (implies -s).
68
69       -D     Console output should be in hex.
70
71       -d     Debug level.  Set the level of debugging messages printed to
72              stderr to debug_level.  Higher numbers produce more messages.
73              -d 0 causes completely silent operation.  -d 1 , the default,
74              produces minimal status messages.  -d 10 produces verbose output
75              equivalent to -v .  Numbers higher than 10 can produce a large
76              amount of debugging information useful only to developers.
77
78       -E name
79              Disable all scanners and then enable scanner name
80
81       -e name
82              Enable scanner name.
83
84       -e all Enables all scanners. Same as -a
85
86       -e http
87              Perform HTTP post-processing ("After" processing). If the output
88              file is
89
90                   208.111.153.175.00080-192.168.001.064.37314,
91
92              Then the post-processing will create the files:
93
94                   208.111.153.175.00080-192.168.001.064.37314-HTTP
95                   208.111.153.175.00080-192.168.001.064.37314-HTTPBODY
96
97              If the HTTPBODY was compressed with GZIP, you may get a third
98              file as well:
99
100                   208.111.153.175.00080-192.168.001.064.37314-HTTPBODY-GZIP
101
102              Additional information about these streams, such as their MD5
103              hash value, is also written to the DFXML report file.
104
105       -e python -S py_path=path -S py_module=module -S py_function=foo
106              Post-process TCP payload by an external python function.
107
108              The python function must take a single string parameter.  The
109              python function can return a string (else the function does must
110              not return).  The returned string (if any) is written in the
111              DFXML report file inside the XML tag
112              <scan_python_result>...</scan_python_result>.  A sample python
113              script is available within the tcpflow source code in directory
114              python/plugins.
115
116              Example:
117
118                  tcpflow -r my.cap -e python -S py_path=python/plugins -S py_module=samplePlugin -S py_function=sampleFunction
119
120       -F[format]
121              Specifies format for output filenames.
122
123              Format specifiers:
124
125              c      Appends the connection counter to ALL filenames.
126
127              t      Prepends each filename with a Unix timestamp (seconds
128                     since epoch).
129
130              T      Prepends each filename with an ISO-8601 timestamp.
131
132              X      Do not output any files (other than the DFXML report
133                     file).
134
135       -FM    Include MD5 of each flow in the DFXML report file.
136
137       -FX    Suppresses file output entirely, DFXML report file is still pro‐
138              duced.
139
140       -Fk    bin output in 1K directories
141
142       -Fm    bin output in 1M directories (2 levels)
143
144       -Fg    bin output in 1G directories (3 levels)
145
146       -fmax_fds
147              Max file descriptors used.  Limit the number of file descriptors
148              used by tcpflow to max_fds.  Higher numbers use more system
149              resources, but usually perform better.  If the underlying oper‐
150              ating system supports the setrlimit() system call, the OS will
151              be asked to enforce the requested limit.  The default is for
152              tcpflow to use the maximum number of file descriptors allowed by
153              the OS.  The -v option will report how many file descriptors
154              tcpflow is using.
155
156       -g     Output flow information to console in multiple colors. (Blue for
157              client to server flows, red for server to client flows, green
158              for undecided flows.)  Note: This option was different from
159              tcpflow 1.3 (-e) and 1.4.4 (-J).
160
161       -h --help
162              Help.  Print usage information and exit.
163
164       -hh    More help.  Print more usage information and exit.
165
166       -i iface
167              Interface name.  Capture packets from the network interface
168              named iface.  If no interface is specified with -i , a reason‐
169              able default will be used by libpcap automatically.
170
171       -I     Store the reception timestamps (of TCP packets) in a companion
172              file *.findx.  Therefore each flow will have two files: (1) the
173              usual file containing payload bytes and (2) the text file con‐
174              taining the corresponding timestamps.  This last file *.findx
175              has three columns using the pipe '|' as separator:
176
177                  byte-index|timestamp|length
178
179              The byte-index column is the postion within the file containing
180              the payload bytes.  The timestamp column represents the number
181              of seconds since epoch as a floating point number.  The preci‐
182              sion is the microsecond but may also be the nanosecond in a
183              future tcpflow version.  The length column is the number of suc‐
184              cessive bytes concerned by timestamp and can include several TCP
185              frames (TCP packets).  The extension findx may become from the
186              fact that the timestamps are frame indexed.
187
188       -L semlock_name
189              Specifies that semlock_name should be used as a Unix semaphore
190              to prevent two different copies of tcpflow running in two dif‐
191              ferent processes but outputting to the same standard output from
192              printing on top of each other. This is an application of Unix
193              named semaphores; bet you have never seen one before.
194
195       -l     Treat the following arguments as filenames with an assumed -r
196              command before each one.  This allows you to read a lot of files
197              at once with shell globbing. For example, to process all of the
198              pcap files in the current directory, use this:
199
200                   tcpflow -o out -a -l *.pcap
201
202
203       -m min_size
204              Forces a new connection output file when there is a skip in the
205              TCP session of min_size bytes or more.
206
207       -o outdir
208              Specifies the output directory where the transcript files will
209              be written.
210
211       -P     No purge. Normally tcpflow removes connections from the hash ta‐
212              ble after the connection is closed with a FIN. This conserves
213              memory but takes additional CPU time. Selecting this option
214              causes the std::tr1:unordered_map to grow without bounds, as
215              tcpflow did prior to version 1.1. That makes tcpflow run faster
216              if there are less than 10 million connections, but can lead to
217              out-of-memory errors.
218
219       -p     No promiscuous mode.  Normally, tcpflow attempts to put the net‐
220              work interface into promiscuous mode before capturing packets.
221              The -p option tells tcpflow not to put the interface into pro‐
222              miscuous mode.  Note that it might already be in promiscuous
223              mode for some other reason.
224
225       -q     Quiet mode --- don't print warnings. Currently the only warning
226              that tcpflow prints is a warning when more than 10,000 files are
227              created that the user should have provided the -Fk, -Fm, or -Fg
228              options. We might have other warnings in the future.
229
230       --relinquish-privileges=username
231              When tcpflow is run as root, this option changes the user ID and
232              group ID to write files owned by username.  The group ID is the
233              first one from the username groups list.  This operation is per‐
234              formed just after opening the capture device or just after open‐
235              ing the first input PCAP file.  This option does not support
236              multi root-only readable input files as the root privileges are
237              dropped after opening the first file (e.g.  -r root-only-
238              access.pcap -R root-only.pcap -l root-only*.pcap).  This option
239              has the same behaviour as the tcpdump(1) option having the same
240              name --relinquish-privileges
241
242       -r     Read from file.  Read packets from file, which was created using
243              the -w option of tcpdump(1).  This option may be repeated any
244              number of times. Standard input is used if file is "-".  Note
245              that for this option to be useful, tcpdump's -s option should be
246              used to set the snaplen to the MTU of the interface (e.g., 1500)
247              while capturing packets.
248
249       -R     Read from a file, but only to complete TCP flows. This option is
250              used when tcpflow is used to process a series of files that are
251              captured over time.  For each time period n, file  file(n).pcap
252              should be processed with  -R file(n).pcap, while file(n-1).pcap
253              should be processed with -r file(n-1).pcap.
254
255       -Sname=value
256              Sets a name parameter to be equal to value for a plug-in.  Use
257              -hh to find out all of the settable parameters.
258
259       -s     Strip non-printables.  Convert all non-printable characters to
260              the "." character before printing packets to the console or
261              storing them to a file.
262
263       -T[format]
264              Specifies an arbitrary template for filenames.
265
266              %A     expands to source IP address.
267
268              %a     expands to source IP port.
269
270              %B     expands to destination IP address.
271
272              %b     expands to destination IP port.
273
274              %T     expands to timestamp in ISO8601 format.
275
276              %t     expands to timestamp in Unix time_t format.
277
278              %V     expands to "--" if a VLAN is present.
279
280              %v     expands to the VLAN number if a VLAN is present.
281
282              %C     expands to "c" if the connection count>0.
283
284              %c     expands to the connection count if the connection
285                     count>0.
286
287              %#     always expands to the connection count.
288
289              %N     (connection_number )             % 1000
290
291              %K     (connection_number / 1000)       % 1000
292
293              %M     (connection_number / 1000000)    % 1000
294
295              %G     (connection_number / 1000000000) % 1000
296
297              %%     prints a "%".
298
299              When the option -T is used, tcpflow ignores options -Fk,
300              -Fm and -Fg.
301              However, the option -T handles '/' within the filename template
302              patern to create sub-directories.  For example the following
303              line will create a directory tree out/IP-src/port-src/IP-
304              dst/port-dst.
305
306                  tcpflow -r packets.pcap -o out -T %A/%a/%B/%b/%c%N.flow
307
308       -V --version
309              Print the version number and exit.
310
311       -v --verbose
312              Verbose operation.  Verbosely describe tcpflow's operation.
313              Equivalent to  -d 10.
314
315       -w filename.pcap
316              Write packets that were not processed to filename.pcap. Typi‐
317              cally this will be UDP packets.
318
319       -X filename.xml
320              Write a DFXML report to filename.xml. The file contains a record
321              of every tcp connection, how the tcpflow program was compiled,
322              and the computer on which tcpflow was run.  By default tcpflow
323              writes the DFXML report in file report.xml.
324
325       -Z     Don't decompress gzip-compressed streams.  -K Retain per flow
326              isolated pcap structure.
327
328       expression
329              selects which packets will be captured.  If no expression is
330              given, all packets on the net will be captured.  Otherwise, only
331              packets for which expression is `true' will be captured.
332
333              For the expression syntax, see pcap-filter(7).
334
335              The expression argument can be passed to tcpflow as either a
336              single Shell argument, or as multiple Shell arguments, whichever
337              is more convenient.  Generally, if the expression contains Shell
338              metacharacters, such as backslashes used to escape protocol
339              names, it is easier to pass it as a single, quoted argument
340              rather than to escape the Shell metacharacters.  Multiple argu‐
341              ments are concatenated with spaces before being parsed.
342
343

DFXML report

345       The DFXML report is the XML file written by tcpflow to provide tcpflow
346       build details, command line arguments and information about processed
347       flows.
348
349       By default the DFXML file is named report.xml.  But this filename can
350       be changed using command line option -X.
351
352       DFXML file respects the DFXML schema defined by project
353       https://github.com/dfxml-working-group/dfxml_schema.
354       Moreover tcpflow adds two extra XML tags, as illustrated by the follow‐
355       ing example:
356
357              <tcpflow startime='2017-07-22T00:12:21.962782Z' endtime='2017-07-22T00:12:22.097591Z'
358                       family='2' mac_daddr='40:3d:78:57:ed:d4' mac_saddr='00:c5:42:d2:cb:f2'
359                       src_ipn='141.134.34.12' dst_ipn='192.168.0.40' srcport='80' dstport='38797'
360                       packets='4' len='677' caplen='611' />
361
362              <tcpflow:result scanner="python" path="python/plugins" module="samplePlugin"
363                              function="sampleFunction">bla bla bla</tcpflow:result>
364
365       The first XML tag <tcpflow> provide information about the captured
366       flow.  This tag should be renamed <tcpflow:cap> in a future version in
367       order to conform better to DFXML schema.
368
369       The second XML tag <tcpflow:result> collects processing results.  For
370       the moment, only the scanner python uses this feature.
371
372       The XML attributes of <tcpflow> are:
373
374       ·      startime Reception time of first packet
375
376       ·      endtime Reception time of last packet
377
378       ·      family
379
380       ·      mac_daddr Destination MAC address of first packet (printed if
381              any)
382
383       ·      mac_saddr Source MAC address of first packet (printed if any)
384
385       ·      src_ipn IP source
386
387       ·      dst_ipn IP destination
388
389       ·      srcport TCP port source
390
391       ·      dstport TCP port destination
392
393       ·      packets Nummber of packets
394
395       ·      out_of_order_count Number of times tcpflow has replaced missing
396              payload by zeros in the flow file, for example when capture does
397              not contain the TCP session begin (printed if any)
398
399       ·      violations Number of protocol violations (printed if any)
400
401       ·      len Sum of un-truncated length of all packet data (including
402              headers, see https://stackoverflow.com/q/1491660)
403
404       ·      caplen Sum of captured bytes of all packet data (including head‐
405              ers, printed if different from len)
406
407       The XML attributes of <tcpflow:result> are:
408
409       ·      scanner Name of the scanner
410
411       ·      path Directory of the scanner module (printed if relevant)
412
413       ·      module Module name (printed if relevant, used to indicate the
414              python script)
415
416       ·      function Function name (printed if relevant, used to indicate
417              the function within the python module)
418

EXAMPLES

420       To record all packets arriving at or departing from sundown and extract
421       all of the HTTP attachments:
422              tcpflow -e http -o outdir host sundown
423
424       To record traffic between helios and either hot or ace and bin the
425       results into 1000 files per directory and calculate the MD5 of each
426       flow:
427              tcpflow -X report.xml -e md5 -o outdir -Fk host helios and \( hot or ace \)
428

BUGS

430       Please send bug reports to simsong@acm.org.
431
432       tcpflow currently does not understand IP fragments.  Flows containing
433       IP fragments will not be recorded correctly.
434

AUTHORS

436       Originally by Jeremy Elson <jelson@circlemud.org>.  Substantially modi‐
437       fied and maintained by Simson L. Garfinkel <simsong@acm.org>.  Network
438       visualization code by Michael Shick <mike@shick.in>
439
440       The current version of this software is available at
441              http://digitalcorpora.org/downloads/tcpflow/
442
443       An announcement mailing list for this program is at:
444              http://groups.google.com/group/tcpflow-users
445

SEE ALSO

447       tcpdump(1), nit(4P), bpf(4), pcap(3), pcap-savefile(5), pcap-filter(7)
448
449
450
451tcpflow 1.6.1                     2013-04-13                        tcpflow(1)
Impressum