1FLENT(1) Flent: The FLExible Network Tester FLENT(1)
2
6 flent - Flent: The FLExible Network Tester
7
9 Flent is a wrapper around netperf and similar tools to run predefined
10 tests and aggregate and plot the results. It defines several tests that
11 can be run against one or more hosts, primarily targeted at testing for
12 the presence of bufferbloat under various conditions.
13 The aggregated data is saved in (gzipped) JSON format for later pro‐
15 cessing and/or import into other tools. The JSON format is documented
16 below.
17 Apart from the JSON format, the data can be output as csv values, emacs
19 org mode tables or plots. Each test can specify several different
20 plots, including time-series plots of the values against each other, as
21 well as CDF plots of (e.g.) ping times.
22 Plots can be output to the formats supported by matplotlib by specify‐
24 ing the output filename with -o output.{png,ps,pdf,svg}. If no output
25 file is specified, the plot is displayed using matplotlib’s interactive
26 plot browser, which also allows saving of the output (in .png format).
27 Requirements
29 Flent runs on Python, versions 2.7+ and 3.3+. Plotting requires a func‐
30 tional matplotlib installation (but everything else can run without
31 matplotlib). For the interactive plot viewer, a graphical display (and
32 suitably configured matplotlib) is required.
33 Most tests employ the netperf benchmarking tool to run the tests. Ver‐
35 sion 2.6 or higher is required, and netperf must be compiled with the
36 --enable-demo option passed to ./configure. Some tests use iperf in ad‐
37 dition to, or instead of netperf. Both tools must be available in the
38 PATH.
39 For ICMP ping measurements, the version of ping employed must support
41 output timestamping (the -D parameter to GNU ping). This is not sup‐
42 ported by the BSD and OSX versions of ping. As an alternative to the
43 regular ping command, the fping utility (see http://fping.org) can be
44 employed. In that case fping must be version 3.5 or greater. Flent will
45 attempt to detect the presence of fping in the PATH and check for sup‐
46 port for the -D parameter. If this check is successful, fping will be
47 employed for ping data, otherwise the system ping will be used.
48 The irtt tool is highly recommended for UDP measurements. See
50 https://github.com/peteheist/irtt. Flent will automatically detect if
51 irtt is available in the PATH and use it if it is detected. Note that
52 the server component of irtt needs to be running on the target host(s)
53 as well.
54
56 When run, flent must be supplied either (a) a test name and one or more
57 host names to connect to, or (b) one or more input files containing
58 data from previous runs to post-process.
59 Test names, hostnames and input file names can all be specified as un‐
61 qualified arguments, and flent will do its best to guess which is
62 which. For each argument, if it is an existing file, it is assumed to
63 be an input file, if it is the name of an existing test configuration
64 it’s assumed to be a test name, and if neither of those are true, it is
65 assumed to be a host name. The -i and -H switches can be used to ex‐
66 plicitly specify the interpretation of an argument.
67 Invocation
69 flent [options] <host|test|input file ...>
70 General options
72 -o OUTPUT, --output=OUTPUT
73 File to write processed output to (default standard out).
74 -D DATA_DIR, --data-dir=DATA_DIR
76 Directory to store data files in. Defaults to the current direc‐
77 tory.
78 -i INPUT, --input=INPUT
80 File to read input from (instead of running tests). Input files
81 can also be specified as unqualified arguments without using the
82 -i switch.
83 -f FORMAT, --format=FORMAT
85 Select output format (plot, csv, org_table, stats). Default is
86 no processed output (just writes the JSON data file).
87 -p PLOT, --plot=PLOT
89 Select which plot to output for the given test (implies -f
90 plot). Use the --list-plots option to see available plots.
91 -t TITLE, --title-extra=TITLE
93 Text to add to plot title and data file name.
94 -n NOTE, --note=NOTE
96 Add arbitrary text as a note to be stored in the JSON data file
97 (under the NOTE key in the metadata object).
98 -r RCFILE, --rcfile=RCFILE
100 Load configuration data from RCFILE (default ~/.flentrc). See
101 section below for information on the rc file format.
102 -x, --extended-metadata
104 Collect extended metadata and store it with the data file. May
105 include details of your machine you don’t want to distribute;
106 see the section on the data format below.
107 --remote-metadata=HOSTNAME
109 Collect extended metadata from a remote host. HOSTNAME is passed
110 verbatim to ssh, so can include hosts specified in ~/.ssh/con‐
111 fig. Note that gathering the data can take some time, since it
112 involves executing several remote commands. This option can be
113 specified multiple times and implies --extended-metadata.
114 --gui Run the flent GUI. All other options are used as defaults in the
116 GUI, but can be changed once it is running. The GUI can also be
117 started by running the flent-gui binary. For more information on
118 the GUI, see the gui section.
119 --new-gui-instance
121 Start a new GUI instance. Otherwise, flent will try to connect
122 to an already running GUI instance and have that load any new
123 data files specified as arguments. Implies --gui when passed on
124 the command line, but not when set in the rc file. Note that
125 when multiple GUI instances are running, there is no guarantee
126 as to which instance will get a subsequent open request (if run
127 again without --new-gui-instance).
128 --gui-no-defer
130 Normally, the GUI defers redrawing plots until they are needed
131 to avoid redrawing all open plots every time an option changes.
132 This switch turns off that optimisation in favour of always re‐
133 drawing everything straight away. This is useful when loading a
134 bunch of plots from the command line and then wanting to flip
135 through them without drawing delay.
136 -b BATCH_NAME, --batch-name=BATCH_NAME
138 Run test batch BATCH_NAME (must be specified in a batch file
139 loaded by the --batch-file option). Can be supplied multiple
140 times.
141 -B BATCH_FILE, --batch-file=BATCH_FILE
143 Load batch file BATCH_FILE. Can be specified multiple times, in
144 which case the files will be combined (with identically-named
145 sections being overridden by later files). See appropriate sec‐
146 tion below for an explanation of the batch file format.
147 --batch-override=key=value
149 Override parameter ’key’ in the batch config and set it to
150 ’value’. The key name will be case folded to lower case. Can be
151 specified multiple times.
152 --batch-dry-run
154 Dry batch run. Prints what would be done, but doesn’t actually
155 run any tests.
156 --batch-verbose
158 Be verbose during batch run: Print all commands executed.
159 --batch-no-shuffle
161 Do not randomise the order of test runs within each batch.
162 --batch-repetitions=REPETITIONS
164 Shorthand for --batch-override 'repetitions=REPETITIONS’.
165 --batch-title=TITLE
167 Shorthand for --batch-override 'batch_title=TITLE’.
168 --batch-resume=DIR
170 Try to resume a previously interrupted batch run. The argument
171 is the top-level output directory from the previous run.
172 This will attempt to find a data file in the resume directory
174 and load the BATCH_TIME from the previous run from that and con‐
175 tinue. The assumption is that the output directory and filenames
176 are generated from the batch time, so that they will match with
177 the previous run when the same time is used. Then, tests for
178 which data files already exist will be skipped on this run. If
179 the rest of the batch invocation is different from the one being
180 resumed, results may not be what you want.
181 There's a check to ensure that the generated output path is a
183 subdirectory of the resume directory, and the whole run will be
184 aborted if it isn't.
185 Test configuration options
187 These options affect the behaviour of the test being run and have no
188 effect when parsing input files.
189 -H HOST, --host=HOST
191 Host to connect to for tests. For tests that support it, multi‐
192 ple hosts can be specified by supplying this option multiple
193 times. Hosts can also be specified as unqualified arguments;
194 this parameter guarantees that the argument be interpreted as a
195 host name (rather than being subject to auto-detection between
196 input files, hostnames and test names).
197 --local-bind=IP
199 Local hostname or IP address to bind to (for test tools that
200 support this). Can be specified multiple times for tests that
201 connect to more than one host; if it is, it must be specified as
202 many times as there are hosts.
203 --remote-host=idx=HOSTNAME
205 A remote hostname to connect to when starting a test. The idx is
206 the runner index, which is assigned sequentially to each runner
207 (and so it is not the same as the sequence of hostnames). Look
208 for the 'IDX' key in SERIES_META for a test get the idx used
209 here, but note that the idx assignment depends on the exact ar‐
210 guments to the test.
211 This works by simply prepending 'ssh HOSTNAME' to the runner
213 command, so it relies on the same binaries being in the same
214 places on both machines, and won't work for all runners.
215 This option can be specified multiple times to have multiple
217 runners run on remote hosts.
218 -l LENGTH, --length=LENGTH
220 Base test length (some tests may add some time to this).
221 -s STEP_SIZE, --step-size=STEP_SIZE
223 Measurement data point step size.
224 -d DELAY, --delay=DELAY
226 Number of seconds to delay parts of test (such as bandwidth
227 loaders).
228 -4, --ipv4
230 Use IPv4 for tests (some tests may ignore this).
231 -6, --ipv6
233 Use IPv6 for tests (some tests may ignore this).
234 --socket-timeout=SOCKET_TIMEOUT
236 Socket timeout (in seconds) used for UDP delay measurement, to
237 prevent stalls on packet loss. Only enabled if the installed
238 netperf version is detected to support this (requires SVN ver‐
239 sion of netperf).
240 For the default value, see the output of flent -h. The value of
242 this parameter is an implicit upper bound on how long a
243 round-trip time that can be measured. As such you may need to
244 adjust it if you are experiencing latency above the default
245 value. Set to 0 to disable.
246 --send-size=SEND_SIZE
248 Send size (in bytes) used for TCP tests. Netperf uses the socket
249 buffer size by default, which if too large can cause spikes in
250 the throughput results. Lowering this value will increase CPU
251 usage but also improves the fidelity of the throughput results
252 without having to decrease the socket buffer size.
253 Can be specified multiple times, with each value corresponding
255 to a stream of a test. If only specified once, the same value
256 will be applied to all flows.
257 --test-parameter=key=value
259 Arbitrary test parameter in key=value format. Key will be case
260 folded to lower case. The values are stored with the results
261 metadata, and so can be used for storing arbitrary information
262 relevant for a particular test run.
263 In addition to serving as simple metadata, the test parameters
265 can also affect the behaviour of some test configurations. See
266 the tests section for information on these.
267 This option can be specified multiple times to set multiple test
269 parameters.
270 --swap-up-down
272 Switch upstream and downstream directions for data transfer.
273 This means that ’upload’ will become ’download’ and vice versa.
274 Works by exchanging netperf TCP_MAERTS and TCP_STREAM parame‐
275 ters, so only works for tests that employ these as their data
276 transfer, and only for the TCP streams.
277 --socket-stats
279 Parse socket stats during test. This will capture and parse
280 socket statistics for all TCP upload flows during a test, adding
281 TCP cwnd and RTT values to the test data. Requires the 'ss'
282 utility to be present on the system, and can fail if there are
283 too many simultaneous upload flows; which is why this option is
284 not enabled by default.
285 --marking-name
287 Define a new symbolic name that can be used when specifying flow
288 markings using the 'markings' test parameter. This can be used
289 to make it easier to specify custom diffserv markings on flows
290 by using symbolic names for each marking value instead of the
291 hex codes. Values specified here will be used in addition to the
292 common values (listed below), and cannot override the built-in
293 names. Names will be case-folded when matching.
294 The list of symbolic markings natively supported, along with
296 their hex expansions, are:
297 AF11: 0x28 CS0: 0x00
299 AF12: 0x30 CS1: 0x20
300 AF13: 0x38 CS2: 0x40
301 AF21: 0x48 CS3: 0x60
302 AF22: 0x50 CS4: 0x80
303 AF23: 0x58 CS5: 0xa0
304 AF31: 0x68 CS6: 0xc0
305 AF32: 0x70 CS7: 0xe0
306 AF33: 0x78 EF: 0xb8
307 AF41: 0x88
308 AF42: 0x90
309 AF43: 0x98
310 Note that the hexadecimal values denote the value of the full
312 ToS byte (including the two ECN bits), so they need to be
313 right-shifted by two bits to get the corresponding diffserv code
314 points.
315 Plot configuration options
317 These options are used to configure the appearance of plot output and
318 only make sense combined with -f plot.
319 --label-x=LABEL
321 --label-y=LABEL
323 Override the figure axis labels. Can be specified twice, corre‐
324 sponding to figures with multiple axes.
325 -I, --invert-latency-y
327 Invert latency data series axis (typically the Y-axis), making
328 plots show ’better' values upwards.
329 -z, --zero-y
331 Always start Y axis of plot at zero, instead of autoscaling the
332 axis. Autoscaling is still enabled for the upper bound. This
333 also disables log scale if enabled.
334 --log-scale={log2,log10}
336 Use the specified logarithmic scale on plots.
337 --norm-factor=FACTOR
339 Data normalisation factor. Divide all data points by this value.
340 Can be specified multiple times, in which case each value corre‐
341 sponds to a data series.
342 --bounds-x=BOUNDS
344 --bounds-y=BOUNDS
346 Specify bounds of the plot axes. If specifying one number, that
347 will become the upper bound. Specify two numbers separated by a
348 comma to specify both upper and lower bounds. To specify just
349 the lower bound, add a comma afterwards. Can be specified twice,
350 corresponding to figures with multiple axes.
351 -S, --scale-mode
353 Treat file names (except for the first one) passed as unquali‐
354 fied arguments as if passed as --scale-data (default as if
355 passed as --input).
356 --concatenate
358 Concatenate multiple result sets into one data series. This
359 means that each data file will have its time axis shifted by the
360 preceding series duration and appended to the first data set
361 specified. Only works for data sets from the same test, obvi‐
362 ously.
363 --absolute-time
365 Plot data points with absolute UNIX time on the x-axis. This re‐
366 quires the absolute starting time for the test run to be stored
367 in the data file, and so it won’t work with data files that pre‐
368 dates this feature.
369 --subplot-combine
371 When plotting multiple data series, plot each one on a separate
372 subplot instead of combining them into one plot. This mode is
373 not supported for all plot types, and only works when
374 --scale-mode is disabled.
375 --skip-missing-series
377 Skip missing series entirely from bar plots, instead of leaving
378 an empty space for it.
379 --no-print-n
381 Do not print the number of data points on combined plots. When
382 using plot types that combines results from several test runs,
383 the number of data series in each combined data point is nor‐
384 mally added after the series name, (n=X) for X data series. This
385 option turns that off.
386 --no-annotation
388 Exclude annotation with hostnames, time and test length from
389 plots.
390 --figure-note=NOTE, --fig-note=NOTE
392 Add a note (arbitrary text) to the bottom-left of the figure.
393 --no-title
395 Exclude title from plots.
396 --override-title=TITLE
398 Override plot title with this string. Completely discards the
399 configured title (from the test configuration), as well as the
400 title stored in the data set, and replaces it with the value
401 supplied here. This is useful to override the plot title at the
402 time of plotting, for instance to add a title to an aggregate
403 plot from several data series. When this parameter is specified,
404 --no-title has no effect.
405 --no-labels
407 Hides tick labels from box and bar plots.
408 --no-markers
410 Don’t use line markers to differentiate data series on plots.
411 --no-legend
413 Exclude legend from plots.
414 --horizontal-legend
416 Place a horizontal legend below the plot instead of a vertical
417 one next to it. Doesn't always work well if there are too many
418 items in the legend.
419 --legend-title=LEGEND_TITLE
421 Override legend title on plot.
422 --legend-placement=LEGEND_PLACEMENT
424 Control legend placement. Enabling this option will place the
425 legend inside the plot at the specified location. Can be one of
426 'best', 'upper right', 'upper left', 'lower left', 'lower
427 right', 'right', 'center left', 'center right', 'lower center',
428 'upper center' or 'center'.
429 --legend-columns=LEGEND_COLUMNS
431 Set the number of columns in the legend.
433 --reverse-legend
435 Reverse the order of items in the legend. This can be useful to
436 make the legend order match the data series in some cases.
437 --filter-legend
439 Filter legend labels by removing the longest common substring
440 from all entries. This is not particularly smart, so use with
441 care.
442 --replace-legend=src=dest
444 Replace 'src' with 'dst' in legends. Can be specified multiple
445 times.
446 --filter-regexp=REGEXP
448 Filter the plot legend by the supplied regular expression. Note
449 that for combining several plot results, the regular expression
450 is also applied before the grouping logic, meaning that a too
451 wide filter can mess up the grouping.
452 --override-label=LABEL
454 Override dataset label. Can be specified multiple times when
455 multiple datasets are being plotted, in which case the order of
456 labels corresponds to the order of datasets.
457 Like --override-title, this is applied at the time of plotting.
459 --filter-series=SERIES
461 Filter out specified series from plot. Can be specified multiple
462 times.
463 --split-group=LABEL
465 Split data sets into groups when creating box plots. Specify
466 this option multiple times to define the new groups; the value
467 of each option is the group name.
468 Say you're plotting nine datasets which are really testing two
470 variables with three values each. In this case, it can be useful
471 to have the box plot of the results be split into three parts
472 (corresponding to the values of one variable) with each three
473 boxes in each of them (corresponding to the values of the second
474 variable). This option makes this possible; simply specify it
475 three times with the labels to be used for the three groups.
476 A constraint on this option is that the number of datasets being
478 plotted must be divisible by the number of groups.
479 --colours=COLOURS
481 Comma-separated list of colours to be used for the plot colour
482 cycle. Can be specified in any format understood by matplotlib
483 (including HTML hex values prefixed with a #).
484 Yes, this option uses British spelling. No, American spelling is
486 not supported.
487 --override-colour-mode=MODE
489 Override colour_mode attribute. This changes the way colours are
490 assigned to bar plots. The default is 'groups' which assigns a
491 separate colour to each group of data series. The alternative is
492 'series' which assigns a separate colour to each series, repeat‐
493 ing them for each data group.
494 --override-group-by=GROUP
496 Override the group_by setting for combination plots. This is
497 useful to, for instance, switch to splitting up combined data
498 sets by batch run instead of by file name.
499 --combine-save-dir=DIRNAME
501 When doing a combination plot save the intermediate data to
502 DIRNAME. This can then be used for subsequent plotting to avoid
503 having to load all the source data files again on each plot.
504 --figure-width=FIG_WIDTH
506 Figure width in inches. Used when saving plots to file and for
507 default size of the interactive plot window.
508 --figure-height=FIG_HEIGHT
510 Figure height in inches. Used when saving plots to file and for
511 default size of the interactive plot window.
512 --figure-dpi=FIG_DPI
514 Figure DPI. Used when saving plots to raster format files.
515 --fallback-layout
517 Use the fallback layout engine (tight_layout built in to mat‐
518 plotlib). Use this if text is cut off on saved figures. The
519 downside to the fallback engine is that the size of the figure
520 (as specified by --figure-width and --figure-height) is no
521 longer kept constant.)
522 --no-matplotlibrc
524 Don’t load included matplotlibrc values. Use this if autodetec‐
525 tion of custom matplotlibrc fails and flent is inadvertently
526 overriding rc values.
527 --no-hover-highlight
529 Don't highlight data series on hover in interactive plot views.
530 Use this if redrawing is too slow, or the highlighting is unde‐
531 sired for other reasons.
532 --scale-data=SCALE_DATA
534 Additional data files to consider when scaling the plot axes
535 (for plotting several plots with identical axes). Note, this
536 displays only the first data set, but with axis scaling taking
537 into account the additional data sets. Can be supplied multiple
538 times; see also --scale-mode.
539 Test tool-related options
541 --control-host=HOST
542 Hostname for the test control connection (for test tools that
543 support this). Default: First hostname of test target.
544 When running tests that uses D-ITG as a test tool (such as the
546 voip-* tests), this switch controls where flent will look for
547 the D-ITG control server (see section below on running tests
548 with D-ITG). For Netperf-based tests, this option is passed to
549 Netperf to control where to point the control connection. This
550 is useful to, for instance, to run the control server communica‐
551 tion over a separate control network so as to not interfere with
552 test traffic.
553 There is also a per-flow setting for this for tests that connect
555 to multiple hosts; see the control_hosts test parameter in
556 tests. If both are set, the per-flow setting takes precedence
557 for those tests that use it.
558 --control-local-bind=IP
560 Local hostname or IP to bind control connection to (for test
561 tools that support it; currently netperf). If not supplied, the
562 value for --local-bind will be used. Note that if this value is
563 passed but --local-bind is not, netperf will use the value spec‐
564 ified here to bind the data connections to as well.
565 --netperf-control-port=PORT
567 Port for Netperf control server. Default: 12865.
568 --ditg-control-port=PORT
570 Port for D-ITG control server. Default: 8000.
571 --ditg-control-secret=SECRET
573 Secret for D-ITG control server authentication. Default: ’’.
574 --http-getter-urllist=FILENAME
576 When running HTTP tests, the http-getter tool is used to fetch
577 URLs (see https://github.com/tohojo/http-getter). This option
578 specifies the filename containing the list of HTTP URLs to get.
579 Can also be a URL, which will then be downloaded as part of each
580 test iteration. If not specified, this is set to
581 http://<hostname>/filelist.txt where <hostname> is the first
582 test hostname.
583 --http-getter-dns-servers=DNS_SERVERS
585 DNS servers to use for http-getter lookups. Format is
586 host[:port][,host[:port]]... This option will only work if
587 libcurl supports it (needs to be built with the ares resolver).
588 Default is none (use the system resolver).
589 --http-getter-timeout=MILLISECONDS
591 Timeout for HTTP connections. Default is to use the test length.
592 --http-getter-workers=NUMBER
594 Number of workers to use for getting HTTP urls. Default is 4.
595 Misc and debugging options:
597 -L LOG_FILE, --log-file=LOG_FILE
598 Write debug log (test program output) to log file.
599 --list-tests
601 List available tests and exit.
602 --list-plots
604 List available plots for selected test and exit.
605 -V, --version
607 Show Flent version information and exit.
608 -v, --verbose
610 Enable verbose logging to console.
611 -q, --quiet
613 Disable normal logging to console (and only log warnings and er‐
614 rors).
615 --debug-error
617 Print full exception backtraces to console.
618 -h, --help
620 Show usage help message and exit.
621 Signals
623 Flent will abort what it is currently doing on receiving a SIGINT --
624 this includes killing all runners, cleaning up temporary files and
625 shutting down as gracefully as possible. Runners are killed with
626 SIGTERM in this mode, and their output is discarded. If a batch run is
627 in progress, the current test will be interrupted in this way, and the
628 rest of the batch run is aborted. Previously completed tests and their
629 results are not aborted. Post-commands marked as ’essential’ will be
630 run after the test is interrupted. Additionally, flent converts SIGTERM
631 into SIGINT internally and reacts accordingly.
632 Upon receiving a SIGUSR1, flent will try to gracefully abort the test
634 it is currently running, and parse the output of the runners to the ex‐
635 tent that any such output exists. That is, each runner will be killed
636 by a SIGINT, which will cause a graceful shutdown for at least ping and
637 netperf (although netperf running in TCP_MAERTS mode will bug out when
638 interrupted like this, so end-of-tests statistics will be missing).
639 Flent will only react once to a SIGUSR1, sending exactly one SIGINT to
640 the active runners, then wait for them to exit. This may take several
641 seconds in the case of netperf. If the runners for some reason fail to
642 exit, flent will be stuck and will need to be killed with SIGINT. If
643 running in batch mode, SIGUSR1 will only affect the currently running
644 test; subsequent tests will still be run.
645
647 Test are supplied as Python files and can specify commands to run etc.
648 For a full list of the tests supported by flent, see the --list-tests
649 option.
650 The Realtime Response Under Load (RRUL) test
652 This test exists in a couple of variants and is a partial implementa‐
653 tion of the RRUL specification as written by Dave Taht (see
654 https://github.com/dtaht/deBloat/blob/master/spec/rrule.doc?raw=true).
655 It works by running RTT measurement using ICMP ping and UDP roundtrip
656 time measurement, while loading up the link with eight TCP streams
657 (four downloads, four uploads). This quite reliably saturates the mea‐
658 sured link (wherever the bottleneck might be), and thus exposes
659 bufferbloat when it is present.
660 Simple TCP flow tests
662 These tests combine a TCP flow (either in one direction, or both) with
663 an ICMP ping measurement. It’s a simpler test than RRUL, but in some
664 cases the single TCP flow can be sufficient to saturate the link.
665 UDP flood test
667 This test runs iperf configured to emit 100Mbps of UDP packets targeted
668 at the test host, while measuring RTT using ICMP ping. It is useful for
669 observing latency in the face of a completely unresponsive packet
670 stream.
671 Test parameters
673 Some test parameters (set with --test-parameter) affect the way tests
674 behave. These are:
675 upload_streams
677 download_streams
679 bidir_streams
681 These set the number of streams for the tests that are config‐
682 urable. The tcp_nup, tcp_ndown and rrul_be_nflows tests under‐
683 stand upload_streams and download_streams, while the rrul_var
684 test understands bidir_streams. If any of these parameter is set
685 to the special value num_cpus the number of streams will be set
686 to the number of CPUs on the system (if this information is
687 available).
688 tcp_cong_control
690 Set the congestion control used for TCP flows, for platforms
691 that supports setting it. This can be specified as a simple
692 string to set the same value for upstream and downstream, or two
693 comma-separated values to set it separately for the upstream and
694 downstream directions. On Linux, any value in the sysctl
695 net.ipv4.tcp_allowed_congestion_control can be used.
696 If a congestion control is specified that is not available on
698 the system running the test, setting it will simply fail. In ad‐
699 dition, some tests override the congestion control for one or
700 more flows. The actual congestion control used is stored in the
701 CONG_CONTROL per-test metadata field.
702 udp_bandwidth
704 udp_bandwidths
706 udp_pktsize
708 udp_pktsizes
710 This sets the bandwidth and packet size of each UDP stream in
711 the udp_* tests. The option is passed to iperf so can be in any
712 syntax the iperf understands (e.g. 20M for 20 Mbps).
713 When running multiple UDP streams use the plural versions of the
715 options (udp_bandwidths and udp_pktsizes) to specify individual
716 per-stream values (comma-separated per stream), or the singular
717 versions to specify the same value for all streams.
718 burst_length
720 burst_ports
722 burst_psize
724 burst_tos
726 These set the length, number of ports to use, packet size and
727 TOS value for the packet bursts generated in the burst* tests.
728 cpu_stats_hosts
730 netstat_hosts
732 qdisc_stats_hosts
734 wifi_stats_hosts
736 These set hostnames to gather statistics from from during the
737 test. The hostnames are passed to SSH, so can be anything under‐
738 stood by SSH (including using username@host syntax, or using
739 hosts defined in ~/.ssh/config). This will attempt to run re‐
740 mote commands on these hosts to gather the required statistics,
741 so passwordless login has to be enabled for. Multiple hostnames
742 can be specified, separated by commas.
743 CPU stats and netstat output is global to the machine being con‐
745 nected to. The qdisc and WiFi stats need extra parameters to
746 work. These are qdisc_stats_interfaces, wifi_stats_interfaces
747 and wifi_stats_stations. The two former specify which interfaces
748 to gather statistics from. These are paired with the hostnames,
749 and so must contain the same number of elements (also comma-sep‐
750 arated) as the _hosts variables. To specify multiple interfaces
751 on the same host, duplicate the hostname. The wifi_stats_sta‐
752 tions parameter specifies MAC addresses of stations to gather
753 statistics for. This list is the same for all hosts, but only
754 stations present in debugfs on each host are actually captured.
755 The qdisc stats gather statistics output from tc -s, while the
757 WiFi stats gather statistics from debugfs. These are gathered by
758 looping in a shell script; however, for better performance, the
759 tc_iterate and wifistats_iterate programmes available in the
760 misc/ directory of the source code tarball can be installed. On
761 low-powered systems this can be critical to get correct statis‐
762 tics. The helper programmes are packaged for LEDE/OpenWrt in the
763 flent-tools package.
764 ping_hosts
766 ping_local_binds
768 ping_labels
770 These are used to define one or more extra host names that will
771 receive a ping flow while a test is run. The ping_hosts variable
772 simply specifies hostnames to ping (several can be specified by
773 separating them with commas). The ping_local_binds variable
774 sets local IP address(es) to bind to for the extra ping flows.
775 If specified, it must contain the same number of local addresses
776 as the number of ping hosts. The same local address can be spec‐
777 ified multiple times, however. The ping_labels variable is used
778 to label each of the ping flows; if not specified, Flent will
779 create a default label based on the target hostname for each
780 flow.
781 voip_host
783 voip_local_bind
785 voip_control_host
787 voip_marking
789 Similar to the ping variants above, these parameters specify a
790 hostname that will receive a VoIP test. However, unlike the ping
791 parameters, only one hostname can be specified for VoIP tests,
792 and that end-host needs to have either D-ITG (and the control
793 server) or the IRTT server running. The marking setting controls
794 which DiffServ marking is applied to the VoIP flow and defaults
795 to no marking being set.
796 control_hosts
798 Hostnames to use for the control connections for the rtt_fair*
799 tests. Comma-separated. If specified, it must contain as many
800 hostnames as the number of target hostnames specified for the
801 test.
802 markings
804 labels Flow markings to use for each of the flows in the rtt_fair*
806 tests. Comma-separated values of markings understood by Netperf
807 (such as "CS0"). Only supports setting the same marking on both
808 the upstream and downstream packets of each flow (so no
809 "CS0,CS0" setting as can be used for Netperf). If not set, de‐
810 faults to CS0 (best effort). If set, each value corresponds to a
811 flow, and any extra flows will be set to CS0. By default each
812 flow will be labelled according to its marking; to override
813 these labels, use the labels parameter.
814 stream_delays
816 Specify a per-stream delay (in seconds) for the different
817 streams started up by a test. Use commas to separate values for
818 the different streams. This can be used to create tests with
819 staggered start times, for example to test TCP flow startup con‐
820 vergence times. What exactly constitutes a stream depends on the
821 test. For example, the rtt_fair* tests considers each hostname a
822 stream, whether or not there is one or two flows going to that
823 host.
824
826 Flent comes equipped with a GUI to browse and plot previously captured
827 datasets. The GUI requires PyQt4; if this is installed, it can be
828 launched with the --gui parameter, or by launching the flent-gui bi‐
829 nary. Additionally, if Flent is launched without parameters and with‐
830 out a controlling terminal, the GUI will be launched automatically.
831 The GUI can be used for interactively plotting previously captured
833 datasets, and makes it easy to compare results from several test runs.
834 It presents a tabbed interface to graphs of data files, allows dynamic
835 configuration of plots, and includes a metadata browser. For each
836 loaded data file, additional data files can be loaded and added to the
837 plot, similar to what happens when specifying multiple input files for
838 plotting on the command line. A checkbox controls whether the added
839 data files are added as separate entries to the plot, or whether they
840 are used for scaling the output (mirroring the --scale-mode) command
841 line switch.
842 The GUI also incorporates matplotlib’s interactive browsing toolbar,
844 enabling panning and zooming of the plot area, dynamic configuration of
845 plot and axis parameters and labels and saving the plots to file. The
846 exact dynamic features supported depends on the installed version of
847 matplotlib.
848
850 The RC file
851 Some of the command line options can be specified in an rc file. By de‐
852 fault, flent looks for this in ~/.flentrc, but an alternative location
853 can be specified with the --rcfile command line option.
854 The rc file allows options to be specified globally, an optionally
856 overridden for specific tests. For an explanation of the options, refer
857 to the annotated example rc file, by default installed to
858 /usr/share/doc/flent/flentrc.example.
859 Batch Files
861 Flent supports reading batch files to automate running several tests
862 and do setup/teardown of test environment etc. This greatly aids repro‐
863 ducibility of tests.
864 The batch file is parsed as an ini file, and can have three types of
866 sections: batches, commands and args. Each section also has a name;
867 type and name are separated with two colons. 'Batches' are named tests
868 that can be selected from the command line, 'commands' are system com‐
869 mands to be run before or after each test run, and 'args' are used in
870 the looping mechanism (which allows repeating tests multiple times with
871 different parameters).
872 Variables in sections control the operation of Flent and can be modi‐
874 fied in several ways: Sections of the same type can inherit from each
875 other and the variables in an 'arg' section will be interpolated into
876 the batch definition on each iteration of a loop. In addition, variable
877 contents can be substituted into other variables by using the ${var‐
878 name} syntax. These three operations are resolved in this order (inher‐
879 itance, arg interpolation and variable substitution).
880 An annotated example batchfile is distributed with the source code, and
882 is by default installed to /usr/share/doc/flent/batchfile.example.
883
885 The aggregated test data is saved in a file called
886 <test_name>-<date>.<title>.flent.gz (the title part is omitted if no
887 title is specified by the -t parameter). This file contains the data
888 points generated during the test, as well as some metadata.
889 The top-level object keys
891 version
892 The file format version as an integer.
893 x_values
895 An array of the x values for the test data (typically the time
896 values for timeseries data).
897 results
899 A JSON object containing the result data series. The keys are
900 the data series names; the value for each key is an array of y
901 values for that data series. The data array has the same length
902 as the x_values array, but there may be missing data points
903 (signified by null values).
904 metadata
906 An object containing various data points about the test run. The
907 metadata values are read in as configuration parameters when the
908 data set is loaded in for further processing. Not all tests use
909 all the parameters, but they are saved anyway.
910 raw_values
912 An array of objects for each data series. Each element of the
913 array contains the raw values as parsed from the test tool cor‐
914 responding to that data series.
915 Metadata keys
917 NAME The test name.
918 TITLE Any extra title specified by the --title-extra parameter when
920 the test was run.
921 HOSTS List of the server hostnames connected to during the test.
923 LOCAL_HOST
925 The hostname of the machine that ran the test.
926 LENGTH Test length in seconds, as specified by the --length parameter.
928 TOTAL_LENGTH
930 Actual data series length, after the test has added time to the
931 LENGTH.
932 STEP_SIZE
934 Time step size granularity.
935 TIME ISO timestamp of the time the test was initiated.
937 NOTE Arbitrary text as entered with the --note switch when the test
939 was run.
940 FLENT_VERSION
942 Version of Flent that generated the data file.
943 IP_VERSION
945 IP version used to run test (as specified by command line param‐
946 eters, or auto-detected from getaddrinfo() if unspecified).
947 KERNEL_NAME
949 The kernel name as reported by uname -s.
950 KERNEL_RELEASE
952 The kernel release as reported by uname -r.
953 MODULE_VERSIONS
955 The sha1sum of certain interesting Linux kernel modules, if
956 available. Can be used to match test data to specific code ver‐
957 sions, if the kernel build is instrumented to, e.g., set the
958 build ID to a git revision.
959 SYSCTLS
961 The values of several networking-related sysctls on the host (if
962 available; Linux only).
963 EGRESS_INFO
965 Interface name, qdisc, offload, driver and BQL configuration of
966 the interface used to reach the test target. This requires that
967 the ip binary is present on Linux, but can be extracted from
968 route on BSD. Qdisc information requires the tc binary to be
969 present, and offload information requires ethtool.
970 If the --remote-metadata is used, the extended metadata info is
972 gathered for each of the hostnames specified. This is gathered
973 under the REMOTE_METADATA key in the metadata object, keyed by
974 the hostname values passed to --remote-metadata. Additionally,
975 the REMOTE_METADATA object will contain an object called
976 INGRESS_INFO which is a duplicate of EGRESS_INFO, but with the
977 destination IP exchanged for the source address of the host run‐
978 ning flent. The assumption here is that --remote-metadata is
979 used to capture metadata of a router known to be in the test
980 path, in which case INGRESS_INFO will contain information about
981 the reverse path from the router (which is ingress from the
982 point of view of the host running flent). If the host being
983 queried for remote metadata is off the path, the contents of
984 INGRESS_INFO will probably be the same as that of EGRESS_INFO .
985 Extended metadata
987 If the --extended-metadata switch is turned on, the following addi‐
988 tional values are collected and stored (to the extent they are avail‐
989 able from the platform):
990 IP_ADDRS
992 IP addresses assigned to the machine running flent.
993 GATEWAYS
995 IP addresses of all detected default gateways on the system, and
996 the interfaces they are reachable through. Only available if the
997 netstat binary is present on the system.
998 EGRESS_INFO
1000 In the EGRESS_INFO key, the IP address of the next-hop router
1001 and the interface MAC address are added if extended metadata is
1002 enabled.
1003
1005 The following output formats are currently supported by Flent:
1006 Plot output (-f plot)
1008 Output test data as one of a series of graphical plots of timeseries
1009 data or summarised as a CDF plot. Each test supplies a number of dif‐
1010 ferent plots; the list of plots for a given test is output by the
1011 --list-plots switch (which must be supplied along with a test name).
1012 The plots are drawn by matplotlib, and can be displayed on the screen
1014 interactively (requires a graphical display), or output to a file in
1015 svg, pdf, ps and png formats. Using the -o switch turns out file output
1016 (the file format is inferred from the file name), while not supplying
1017 the switch turns on the interactive plot viewer.
1018 Tabulated output (-f csv and -f org_table)
1020 These formats output the numeric data in a tabulated format to be con‐
1021 sumed by other applications. The csv output format is a comma-separated
1022 output that can be imported into e.g. spreadsheets, while org_table
1023 outputs a tabulated output in the table format supported by Emacs org
1024 mode. The data is output in text format to standard output, or written
1025 to a file if invoked with the -o parameter.
1026 Statistics output (-f stats)
1028 This output format outputs various statistics about the test data, such
1029 as total bandwidth consumed, and various statistical measures
1030 (min/max/mean/median/std dev/variance) for each data source specified
1031 in the relevant test (this can include some data sources not includes
1032 on plots). The data is output in text format to standard output, or
1033 written to a file if invoked with the -o parameter.
1034 Metadata output (-f metadata)
1036 This output format outputs the test metadata as pretty-printed JSON
1037 (also suitable for human consumption). It is output as a list of ob‐
1038 jects, where each object corresponds to the metadata of one test.
1039 Mostly useful for inspecting metadata of stored data files.
1040
1042 Running Tests With The D-ITG Tool
1043 This version of flent has experimental support for running and parsing
1044 the output of the D-ITG test tool (see
1045 http://traffic.comics.unina.it/software/ITG/). Flent supports parsing
1046 the one-way delay as measured by D-ITG. However, in order to do so, the
1047 data needs to be collected at the receiver end, statistics extracted,
1048 and the result passed back to flent on the sending side.
1049 To perform this function, flent supports a control server which will
1051 listen to XML-RPC requests, spawn an appropriate ITGRecv instance and,
1052 after the test is done, parse its output and make it available for
1053 flent to retrieve. This control server is available as a Python file
1054 that by default is installed in /usr/share/doc/flent/misc. It currently
1055 requires a patched version of D-ITG v2.8.1. The patch is also included
1056 in the same directory.
1057 Note that the D-ITG server is finicky and not designed with security in
1059 mind. For this reason, the control server includes HMAC authentication
1060 to only allow authenticated clients to run a test against the server;
1061 however there is currently no support for enforcement of this in e.g.
1062 firewall rules. Please bear this in mind when running a publicly reach‐
1063 able ITGRecv instance (with or without the control server). Another se‐
1064 curity issue with the control server is that the Python XML-RPC library
1065 by default is vulnerable to XML entity expansion attacks. For this
1066 reason, it is highly recommended to install the defusedxml library
1067 (available at https://pypi.python.org/pypi/defusedxml/) on the host
1068 running the control server. The server will try to find the library on
1069 startup and refuse to run if it is not available, unless explicitly
1070 told otherwise.
1071 Due to the hassle of using D-ITG, it is recommended to install irtt in‐
1073 stead and use that for VoIP tests.
1074 Bugs
1076 Under some conditions (such as severe bufferbloat), the UDP RTT mea‐
1077 surements done by netperf can experience packet loss to the extent that
1078 the test aborts completely, which can cause missing data points for
1079 some measurement series. The --socket-timeout feature can alleviate
1080 this, but requires a recent SVN version of netperf to work. Flent tries
1081 to detect if netperf supports this option and enables it for the UDP
1082 measurements if it does. Using irtt for UDP measurements is a way to
1083 alleviate this; Flent will automatically detect the availability of
1084 irtt and use it if available.
1085 Probably many other bugs. Please report any found to
1087 https://github.com/tohojo/flent/issues and include the output of flent
1088 --version in the report. A debug log (as obtained with flent
1089 --log-file) is also often useful.
1090
1092 Toke Høiland-Jørgensen
1093
1095 2012-2021, Toke Høiland-Jørgensen and contributors. Source code is
1096 GPLv3. Documentation is CC-BY-SA. Neither license affects any output
1097 produced by Flent
10982.0.1 June 24, 2021 FLENT(1)