1FLENT(1) Flent: The FLExible Network Tester FLENT(1)
2
6 flent - Flent: The FLExible Network Tester
7
9 Flent is a wrapper around netperf and similar tools to run predefined
10 tests and aggregate and plot the results. It defines several tests that
11 can be run against one or more hosts, primarily targeted at testing for
12 the presence of bufferbloat under various conditions.
13 The aggregated data is saved in (gzipped) JSON format for later pro‐
15 cessing and/or import into other tools. The JSON format is documented
16 below.
17 Apart from the JSON format, the data can be output as csv values, emacs
19 org mode tables or plots. Each test can specify several different
20 plots, including time-series plots of the values against each other, as
21 well as CDF plots of (e.g.) ping times.
22 Plots can be output to the formats supported by matplotlib by specify‐
24 ing the output filename with -o output.{png,ps,pdf,svg}. If no output
25 file is specified, the plot is displayed using matplotlib’s interactive
26 plot browser, which also allows saving of the output (in .png format).
27 Requirements
29 Flent runs on Python, versions 2.7+ and 3.3+. Plotting requires a func‐
30 tional matplotlib installation (but everything else can run without
31 matplotlib). For the interactive plot viewer, a graphical display (and
32 suitably configured matplotlib) is required.
33 Most tests employ the netperf benchmarking tool to run the tests. Ver‐
35 sion 2.6 or higher is required, and netperf must be compiled with the
36 --enable-demo option passed to ./configure. Some tests use iperf in
37 addition to, or instead of netperf. Both tools must be available in the
38 PATH.
39 For ICMP ping measurements, the version of ping employed must support
41 output timestamping (the -D parameter to GNU ping). This is not sup‐
42 ported by the BSD and OSX versions of ping. As an alternative to the
43 regular ping command, the fping utility (see http://fping.org) can be
44 employed. In that case fping must be version 3.5 or greater. Flent will
45 attempt to detect the presence of fping in the PATH and check for sup‐
46 port for the -D parameter. If this check is successful, fping will be
47 employed for ping data, otherwise the system ping will be used.
48 The irtt tool is highly recommended for UDP measurements. See
50 https://github.com/peteheist/irtt. Flent will automatically detect if
51 irtt is available in the PATH and use it if it is detected. Note that
52 the server component of irtt needs to be running on the target host(s)
53 as well.
54
56 When run, flent must be supplied either (a) a test name and one or more
57 host names to connect to, or (b) one or more input files containing
58 data from previous runs to post-process.
59 Test names, hostnames and input file names can all be specified as
61 unqualified arguments, and flent will do its best to guess which is
62 which. For each argument, if it is an existing file, it is assumed to
63 be an input file, if it is the name of an existing test configuration
64 it’s assumed to be a test name, and if neither of those are true, it is
65 assumed to be a host name. The -i and -H switches can be used to
66 explicitly specify the interpretation of an argument.
67 Invocation
69 flent [options] <host|test|input file ...>
70 General options
72 -o OUTPUT, --output=OUTPUT
73 File to write processed output to (default standard out).
74 -D DATA_DIR, --data-dir=DATA_DIR
76 Directory to store data files in. Defaults to the current direc‐
77 tory.
78 -i INPUT, --input=INPUT
80 File to read input from (instead of running tests). Input files
81 can also be specified as unqualified arguments without using the
82 -i switch.
83 -f FORMAT, --format=FORMAT
85 Select output format (plot, csv, org_table, stats). Default is
86 no processed output (just writes the JSON data file).
87 -p PLOT, --plot=PLOT
89 Select which plot to output for the given test (implies -f
90 plot). Use the --list-plots option to see available plots.
91 -t TITLE, --title-extra=TITLE
93 Text to add to plot title and data file name.
94 -n NOTE, --note=NOTE
96 Add arbitrary text as a note to be stored in the JSON data file
97 (under the NOTE key in the metadata object).
98 -r RCFILE, --rcfile=RCFILE
100 Load configuration data from RCFILE (default ~/.flentrc). See
101 section below for information on the rc file format.
102 -x, --extended-metadata
104 Collect extended metadata and store it with the data file. May
105 include details of your machine you don’t want to distribute;
106 see the section on the data format below.
107 --remote-metadata=HOSTNAME
109 Collect extended metadata from a remote host. HOSTNAME is passed
110 verbatim to ssh, so can include hosts specified in ~/.ssh/con‐
111 fig. Note that gathering the data can take some time, since it
112 involves executing several remote commands. This option can be
113 specified multiple times and implies --extended-metadata.
114 --gui Run the flent GUI. All other options are used as defaults in the
116 GUI, but can be changed once it is running. The GUI can also be
117 started by running the flent-gui binary. For more information on
118 the GUI, see the gui section.
119 --new-gui-instance
121 Start a new GUI instance. Otherwise, flent will try to connect
122 to an already running GUI instance and have that load any new
123 data files specified as arguments. Implies --gui when passed on
124 the command line, but not when set in the rc file. Note that
125 when multiple GUI instances are running, there is no guarantee
126 as to which instance will get a subsequent open request (if run
127 again without --new-gui-instance).
128 --gui-no-defer
130 Normally, the GUI defers redrawing plots until they are needed
131 to avoid redrawing all open plots every time an option changes.
132 This switch turns off that optimisation in favour of always
133 redrawing everything straight away. This is useful when loading
134 a bunch of plots from the command line and then wanting to flip
135 through them without drawing delay.
136 -b BATCH_NAME, --batch-name=BATCH_NAME
138 Run test batch BATCH_NAME (must be specified in a batch file
139 loaded by the --batch-file option). Can be supplied multiple
140 times.
141 -B BATCH_FILE, --batch-file=BATCH_FILE
143 Load batch file BATCH_FILE. Can be specified multiple times, in
144 which case the files will be combined (with identically-named
145 sections being overridden by later files). See appropriate sec‐
146 tion below for an explanation of the batch file format.
147 --batch-override=key=value
149 Override parameter ’key’ in the batch config and set it to
150 ’value’. The key name will be case folded to lower case. Can be
151 specified multiple times.
152 --batch-dry-run
154 Dry batch run. Prints what would be done, but doesn’t actually
155 run any tests.
156 --batch-verbose
158 Be verbose during batch run: Print all commands executed.
159 --batch-no-shuffle
161 Do not randomise the order of test runs within each batch.
162 --batch-repetitions=REPETITIONS
164 Shorthand for --batch-override 'repetitions=REPETITIONS’.
165 --batch-title=TITLE
167 Shorthand for --batch-override 'batch_title=TITLE’.
168 --batch-resume=DIR
170 Try to resume a previously interrupted batch run. The argument
171 is the top-level output directory from the previous run.
172 This will attempt to find a data file in the resume directory
174 and load the BATCH_TIME from the previous run from that and con‐
175 tinue. The assumption is that the output directory and filenames
176 are generated from the batch time, so that they will match with
177 the previous run when the same time is used. Then, tests for
178 which data files already exist will be skipped on this run. If
179 the rest of the batch invocation is different from the one being
180 resumed, results may not be what you want.
181 There's a check to ensure that the generated output path is a
183 subdirectory of the resume directory, and the whole run will be
184 aborted if it isn't.
185 Test configuration options
187 These options affect the behaviour of the test being run and have no
188 effect when parsing input files.
189 -H HOST, --host=HOST
191 Host to connect to for tests. For tests that support it, multi‐
192 ple hosts can be specified by supplying this option multiple
193 times. Hosts can also be specified as unqualified arguments;
194 this parameter guarantees that the argument be interpreted as a
195 host name (rather than being subject to auto-detection between
196 input files, hostnames and test names).
197 --local-bind=IP
199 Local hostname or IP address to bind to (for test tools that
200 support this). Can be specified multiple times for tests that
201 connect to more than one host; if it is, it must be specified as
202 many times as there are hosts.
203 --remote-host=idx=HOSTNAME
205 A remote hostname to connect to when starting a test. The idx is
206 the runner index, which is assigned sequentially to each runner
207 (and so it is not the same as the sequence of hostnames). Look
208 for the 'IDX' key in SERIES_META for a test get the idx used
209 here, but note that the idx assignment depends on the exact
210 arguments to the test.
211 This works by simply prepending 'ssh HOSTNAME' to the runner
213 command, so it relies on the same binaries being in the same
214 places on both machines, and won't work for all runners.
215 This option can be specified multiple times to have multiple
217 runners run on remote hosts.
218 -l LENGTH, --length=LENGTH
220 Base test length (some tests may add some time to this).
221 -s STEP_SIZE, --step-size=STEP_SIZE
223 Measurement data point step size.
224 -d DELAY, --delay=DELAY
226 Number of seconds to delay parts of test (such as bandwidth
227 loaders).
228 -4, --ipv4
230 Use IPv4 for tests (some tests may ignore this).
231 -6, --ipv6
233 Use IPv6 for tests (some tests may ignore this).
234 --socket-timeout=SOCKET_TIMEOUT
236 Socket timeout (in seconds) used for UDP delay measurement, to
237 prevent stalls on packet loss. Only enabled if the installed
238 netperf version is detected to support this (requires SVN ver‐
239 sion of netperf).
240 For the default value, see the output of flent -h. The value of
242 this parameter is an implicit upper bound on how long a
243 round-trip time that can be measured. As such you may need to
244 adjust it if you are experiencing latency above the default
245 value. Set to 0 to disable.
246 --test-parameter=key=value
248 Arbitrary test parameter in key=value format. Key will be case
249 folded to lower case. The values are stored with the results
250 metadata, and so can be used for storing arbitrary information
251 relevant for a particular test run.
252 In addition to serving as simple metadata, the test parameters
254 can also affect the behaviour of some test configurations. See
255 the tests section for information on these.
256 This option can be specified multiple times to set multiple test
258 parameters.
259 --swap-up-down
261 Switch upstream and downstream directions for data transfer.
262 This means that ’upload’ will become ’download’ and vice versa.
263 Works by exchanging netperf TCP_MAERTS and TCP_STREAM parame‐
264 ters, so only works for tests that employ these as their data
265 transfer, and only for the TCP streams.
266 --socket-stats
268 Parse socket stats during test. This will capture and parse
269 socket statistics for all TCP upload flows during a test, adding
270 TCP cwnd and RTT values to the test data. Requires the 'ss'
271 utility to be present on the system, and can fail if there are
272 too many simultaneous upload flows; which is why this option is
273 not enabled by default.
274 Plot configuration options
276 These options are used to configure the appearance of plot output and
277 only make sense combined with -f plot.
278 --label-x=LABEL
280 --label-y=LABEL
282 Override the figure axis labels. Can be specified twice, corre‐
283 sponding to figures with multiple axes.
284 -I, --invert-latency-y
286 Invert latency data series axis (typically the Y-axis), making
287 plots show ’better' values upwards.
288 -z, --zero-y
290 Always start Y axis of plot at zero, instead of autoscaling the
291 axis. Autoscaling is still enabled for the upper bound. This
292 also disables log scale if enabled.
293 --log-scale={log2,log10}
295 Use the specified logarithmic scale on plots.
296 --norm-factor=FACTOR
298 Data normalisation factor. Divide all data points by this value.
299 Can be specified multiple times, in which case each value corre‐
300 sponds to a data series.
301 --bounds-x=BOUNDS
303 --bounds-y=BOUNDS
305 Specify bounds of the plot axes. If specifying one number, that
306 will become the upper bound. Specify two numbers separated by a
307 comma to specify both upper and lower bounds. To specify just
308 the lower bound, add a comma afterwards. Can be specified twice,
309 corresponding to figures with multiple axes.
310 -S, --scale-mode
312 Treat file names (except for the first one) passed as unquali‐
313 fied arguments as if passed as --scale-data (default as if
314 passed as --input).
315 --concatenate
317 Concatenate multiple result sets into one data series. This
318 means that each data file will have its time axis shifted by the
319 preceding series duration and appended to the first data set
320 specified. Only works for data sets from the same test, obvi‐
321 ously.
322 --absolute-time
324 Plot data points with absolute UNIX time on the x-axis. This
325 requires the absolute starting time for the test run to be
326 stored in the data file, and so it won’t work with data files
327 that predates this feature.
328 --subplot-combine
330 When plotting multiple data series, plot each one on a separate
331 subplot instead of combining them into one plot. This mode is
332 not supported for all plot types, and only works when
333 --scale-mode is disabled.
334 --skip-missing-series
336 Skip missing series entirely from bar plots, instead of leaving
337 an empty space for it.
338 --no-print-n
340 Do not print the number of data points on combined plots. When
341 using plot types that combines results from several test runs,
342 the number of data series in each combined data point is nor‐
343 mally added after the series name, (n=X) for X data series. This
344 option turns that off.
345 --no-annotation
347 Exclude annotation with hostnames, time and test length from
348 plots.
349 --figure-note=NOTE, --fig-note=NOTE
351 Add a note (arbitrary text) to the bottom-left of the figure.
352 --no-title
354 Exclude title from plots.
355 --override-title=TITLE
357 Override plot title with this string. Completely discards the
358 configured title (from the test configuration), as well as the
359 title stored in the data set, and replaces it with the value
360 supplied here. This is useful to override the plot title at the
361 time of plotting, for instance to add a title to an aggregate
362 plot from several data series. When this parameter is specified,
363 --no-title has no effect.
364 --no-labels
366 Hides tick labels from box and bar plots.
367 --no-markers
369 Don’t use line markers to differentiate data series on plots.
370 --no-legend
372 Exclude legend from plots.
373 --horizontal-legend
375 Place a horizontal legend below the plot instead of a vertical
376 one next to it. Doesn't always work well if there are too many
377 items in the legend.
378 --legend-title=LEGEND_TITLE
380 Override legend title on plot.
381 --legend-placement=LEGEND_PLACEMENT
383 Control legend placement. Enabling this option will place the
384 legend inside the plot at the specified location. Can be one of
385 'best', 'upper right', 'upper left', 'lower left', 'lower
386 right', 'right', 'center left', 'center right', 'lower center',
387 'upper center' or 'center'.
388 --legend-columns=LEGEND_COLUMNS
390 Set the number of columns in the legend.
392 --reverse-legend
394 Reverse the order of items in the legend. This can be useful to
395 make the legend order match the data series in some cases.
396 --filter-legend
398 Filter legend labels by removing the longest common substring
399 from all entries. This is not particularly smart, so use with
400 care.
401 --replace-legend=src=dest
403 Replace 'src' with 'dst' in legends. Can be specified multiple
404 times.
405 --filter-regexp=REGEXP
407 Filter the plot legend by the supplied regular expression. Note
408 that for combining several plot results, the regular expression
409 is also applied before the grouping logic, meaning that a too
410 wide filter can mess up the grouping.
411 --override-label=LABEL
413 Override dataset label. Can be specified multiple times when
414 multiple datasets are being plotted, in which case the order of
415 labels corresponds to the order of datasets.
416 Like --override-title, this is applied at the time of plotting.
418 --filter-series=SERIES
420 Filter out specified series from plot. Can be specified multiple
421 times.
422 --split-group=LABEL
424 Split data sets into groups when creating box plots. Specify
425 this option multiple times to define the new groups; the value
426 of each option is the group name.
427 Say you're plotting nine datasets which are really testing two
429 variables with three values each. In this case, it can be useful
430 to have the box plot of the results be split into three parts
431 (corresponding to the values of one variable) with each three
432 boxes in each of them (corresponding to the values of the second
433 variable). This option makes this possible; simply specify it
434 three times with the labels to be used for the three groups.
435 A constraint on this option is that the number of datasets being
437 plotted must be divisible by the number of groups.
438 --colours=COLOURS
440 Comma-separated list of colours to be used for the plot colour
441 cycle. Can be specified in any format understood by matplotlib
442 (including HTML hex values prefixed with a #).
443 Yes, this option uses British spelling. No, American spelling is
445 not supported.
446 --override-colour-mode=MODE
448 Override colour_mode attribute. This changes the way colours are
449 assigned to bar plots. The default is 'groups' which assigns a
450 separate colour to each group of data series. The alternative is
451 'series' which assigns a separate colour to each series, repeat‐
452 ing them for each data group.
453 --override-group-by=GROUP
455 Override the group_by setting for combination plots. This is
456 useful to, for instance, switch to splitting up combined data
457 sets by batch run instead of by file name.
458 --combine-save-dir=DIRNAME
460 When doing a combination plot save the intermediate data to
461 DIRNAME. This can then be used for subsequent plotting to avoid
462 having to load all the source data files again on each plot.
463 --figure-width=FIG_WIDTH
465 Figure width in inches. Used when saving plots to file and for
466 default size of the interactive plot window.
467 --figure-height=FIG_HEIGHT
469 Figure height in inches. Used when saving plots to file and for
470 default size of the interactive plot window.
471 --figure-dpi=FIG_DPI
473 Figure DPI. Used when saving plots to raster format files.
474 --fallback-layout
476 Use the fallback layout engine (tight_layout built in to mat‐
477 plotlib). Use this if text is cut off on saved figures. The
478 downside to the fallback engine is that the size of the figure
479 (as specified by --figure-width and --figure-height) is no
480 longer kept constant.)
481 --no-matplotlibrc
483 Don’t load included matplotlibrc values. Use this if autodetec‐
484 tion of custom matplotlibrc fails and flent is inadvertently
485 overriding rc values.
486 --no-hover-highlight
488 Don't highlight data series on hover in interactive plot views.
489 Use this if redrawing is too slow, or the highlighting is unde‐
490 sired for other reasons.
491 --scale-data=SCALE_DATA
493 Additional data files to consider when scaling the plot axes
494 (for plotting several plots with identical axes). Note, this
495 displays only the first data set, but with axis scaling taking
496 into account the additional data sets. Can be supplied multiple
497 times; see also --scale-mode.
498 Test tool-related options
500 --control-host=HOST
501 Hostname for the test control connection (for test tools that
502 support this). Default: First hostname of test target.
503 When running tests that uses D-ITG as a test tool (such as the
505 voip-* tests), this switch controls where flent will look for
506 the D-ITG control server (see section below on running tests
507 with D-ITG). For Netperf-based tests, this option is passed to
508 Netperf to control where to point the control connection. This
509 is useful to, for instance, to run the control server communica‐
510 tion over a separate control network so as to not interfere with
511 test traffic.
512 There is also a per-flow setting for this for tests that connect
514 to multiple hosts; see the control_hosts test parameter in
515 tests. If both are set, the per-flow setting takes precedence
516 for those tests that use it.
517 --control-local-bind=IP
519 Local hostname or IP to bind control connection to (for test
520 tools that support it; currently netperf). If not supplied, the
521 value for --local-bind will be used. Note that if this value is
522 passed but --local-bind is not, netperf will use the value spec‐
523 ified here to bind the data connections to as well.
524 --netperf-control-port=PORT
526 Port for Netperf control server. Default: 12865.
527 --ditg-control-port=PORT
529 Port for D-ITG control server. Default: 8000.
530 --ditg-control-secret=SECRET
532 Secret for D-ITG control server authentication. Default: ’’.
533 --http-getter-urllist=FILENAME
535 When running HTTP tests, the http-getter tool is used to fetch
536 URLs (see https://github.com/tohojo/http-getter). This option
537 specifies the filename containing the list of HTTP URLs to get.
538 Can also be a URL, which will then be downloaded as part of each
539 test iteration. If not specified, this is set to
540 http://<hostname>/filelist.txt where <hostname> is the first
541 test hostname.
542 --http-getter-dns-servers=DNS_SERVERS
544 DNS servers to use for http-getter lookups. Format is
545 host[:port][,host[:port]]... This option will only work if
546 libcurl supports it (needs to be built with the ares resolver).
547 Default is none (use the system resolver).
548 --http-getter-timeout=MILLISECONDS
550 Timeout for HTTP connections. Default is to use the test length.
551 --http-getter-workers=NUMBER
553 Number of workers to use for getting HTTP urls. Default is 4.
554 Misc and debugging options:
556 -L LOG_FILE, --log-file=LOG_FILE
557 Write debug log (test program output) to log file.
558 --list-tests
560 List available tests and exit.
561 --list-plots
563 List available plots for selected test and exit.
564 -V, --version
566 Show Flent version information and exit.
567 -v, --verbose
569 Enable verbose logging to console.
570 -q, --quiet
572 Disable normal logging to console (and only log warnings and
573 errors).
574 --debug-error
576 Print full exception backtraces to console.
577 -h, --help
579 Show usage help message and exit.
580 Signals
582 Flent will abort what it is currently doing on receiving a SIGINT --
583 this includes killing all runners, cleaning up temporary files and
584 shutting down as gracefully as possible. Runners are killed with
585 SIGTERM in this mode, and their output is discarded. If a batch run is
586 in progress, the current test will be interrupted in this way, and the
587 rest of the batch run is aborted. Previously completed tests and their
588 results are not aborted. Post-commands marked as ’essential’ will be
589 run after the test is interrupted. Additionally, flent converts SIGTERM
590 into SIGINT internally and reacts accordingly.
591 Upon receiving a SIGUSR1, flent will try to gracefully abort the test
593 it is currently running, and parse the output of the runners to the
594 extent that any such output exists. That is, each runner will be killed
595 by a SIGINT, which will cause a graceful shutdown for at least ping and
596 netperf (although netperf running in TCP_MAERTS mode will bug out when
597 interrupted like this, so end-of-tests statistics will be missing).
598 Flent will only react once to a SIGUSR1, sending exactly one SIGINT to
599 the active runners, then wait for them to exit. This may take several
600 seconds in the case of netperf. If the runners for some reason fail to
601 exit, flent will be stuck and will need to be killed with SIGINT. If
602 running in batch mode, SIGUSR1 will only affect the currently running
603 test; subsequent tests will still be run.
604
606 Test are supplied as Python files and can specify commands to run etc.
607 For a full list of the tests supported by flent, see the --list-tests
608 option.
609 The Realtime Response Under Load (RRUL) test
611 This test exists in a couple of variants and is a partial implementa‐
612 tion of the RRUL specification as written by Dave Taht (see
613 https://github.com/dtaht/deBloat/blob/master/spec/rrule.doc?raw=true).
614 It works by running RTT measurement using ICMP ping and UDP roundtrip
615 time measurement, while loading up the link with eight TCP streams
616 (four downloads, four uploads). This quite reliably saturates the mea‐
617 sured link (wherever the bottleneck might be), and thus exposes
618 bufferbloat when it is present.
619 Simple TCP flow tests
621 These tests combine a TCP flow (either in one direction, or both) with
622 an ICMP ping measurement. It’s a simpler test than RRUL, but in some
623 cases the single TCP flow can be sufficient to saturate the link.
624 UDP flood test
626 This test runs iperf configured to emit 100Mbps of UDP packets targeted
627 at the test host, while measuring RTT using ICMP ping. It is useful for
628 observing latency in the face of a completely unresponsive packet
629 stream.
630 Test parameters
632 Some test parameters (set with --test-parameter) affect the way tests
633 behave. These are:
634 upload_streams
636 download_streams
638 These set the number of upload or download streams for the
639 tcp_nup, tcp_ndown and rrul_be_nflows tests. If set to the spe‐
640 cial value num_cpus the number of streams will be set to the
641 number of CPUs on the system (if this information is available).
642 tcp_cong_control
644 Set the congestion control used for TCP flows, for platforms
645 that supports setting it. This can be specified as a simple
646 string to set the same value for upstream and downstream, or two
647 comma-separated values to set it separately for the upstream and
648 downstream directions. On Linux, any value in the sysctl
649 net.ipv4.tcp_allowed_congestion_control can be used.
650 If a congestion control is specified that is not available on
652 the system running the test, setting it will simply fail. In
653 addition, some tests override the congestion control for one or
654 more flows. The actual congestion control used is stored in the
655 CONG_CONTROL per-test metadata field.
656 udp_bandwidth
658 This sets the bandwidth of each UDP stream in the udp_* tests.
659 The option is passed to iperf so can be in any syntax the iperf
660 understands (e.g. 20M for 20 Mbps).
661 burst_length
663 burst_ports
665 burst_psize
667 burst_tos
669 These set the length, number of ports to use, packet size and
670 TOS value for the packet bursts generated in the burst* tests.
671 cpu_stats_hosts
673 netstat_hosts
675 qdisc_stats_hosts
677 wifi_stats_hosts
679 These set hostnames to gather statistics from from during the
680 test. The hostnames are passed to SSH, so can be anything under‐
681 stood by SSH (including using username@host syntax, or using
682 hosts defined in ~/.ssh/config). This will attempt to run
683 remote commands on these hosts to gather the required statis‐
684 tics, so passwordless login has to be enabled for. Multiple
685 hostnames can be specified, separated by commas.
686 CPU stats and netstat output is global to the machine being con‐
688 nected to. The qdisc and WiFi stats need extra parameters to
689 work. These are qdisc_stats_interfaces, wifi_stats_interfaces
690 and wifi_stats_stations. The two former specify which interfaces
691 to gather statistics from. These are paired with the hostnames,
692 and so must contain the same number of elements (also comma-sep‐
693 arated) as the _hosts variables. To specify multiple interfaces
694 on the same host, duplicate the hostname. The wifi_stats_sta‐
695 tions parameter specifies MAC addresses of stations to gather
696 statistics for. This list is the same for all hosts, but only
697 stations present in debugfs on each host are actually captured.
698 The qdisc stats gather statistics output from tc -s, while the
700 WiFi stats gather statistics from debugfs. These are gathered by
701 looping in a shell script; however, for better performance, the
702 tc_iterate and wifistats_iterate programmes available in the
703 misc/ directory of the source code tarball can be installed. On
704 low-powered systems this can be critical to get correct statis‐
705 tics. The helper programmes are packaged for LEDE/OpenWrt in the
706 flent-tools package.
707 ping_hosts
709 ping_local_binds
711 These are used to define one or more extra host names that will
712 receive a ping flow while a test is run. The ping_hosts variable
713 simply specifies hostnames to ping (several can be specified by
714 separating them with commas). The ping_local_binds variable
715 sets local IP address(es) to bind to for the extra ping flows.
716 If specified, it must contain the same number of local addresses
717 as the number of ping hosts. The same local address can be spec‐
718 ified multiple times, however.
719 voip_host
721 voip_local_bind
723 voip_control_host
725 voip_marking
727 Similar to the ping variants above, these parameters specify a
728 hostname that will receive a VoIP test. However, unlike the ping
729 parameters, only one hostname can be specified for VoIP tests,
730 and that end-host needs to have either D-ITG (and the control
731 server) or the IRTT server running. The marking setting controls
732 which DiffServ marking is applied to the VoIP flow and defaults
733 to no marking being set.
734 control_hosts
736 Hostnames to use for the control connections for the rtt_fair*
737 tests. Comma-separated. If specified, it must contain as many
738 hostnames as the number of target hostnames specified for the
739 test.
740 markings
742 Flow markings to use for each of the flows in the rtt_fair*
743 tests. Comma-separated values of markings understood by Netperf
744 (such as "CS0"). Only supports setting the same marking on both
745 the upstream and downstream packets of each flow (so no
746 "CS0,CS0" setting as can be used for Netperf). If not set,
747 defaults to CS0 (best effort). If set, each value corresponds to
748 a flow, and any extra flows will be set to CS0.
749
751 Flent comes equipped with a GUI to browse and plot previously captured
752 datasets. The GUI requires PyQt4; if this is installed, it can be
753 launched with the --gui parameter, or by launching the flent-gui
754 binary. Additionally, if Flent is launched without parameters and
755 without a controlling terminal, the GUI will be launched automatically.
756 The GUI can be used for interactively plotting previously captured
758 datasets, and makes it easy to compare results from several test runs.
759 It presents a tabbed interface to graphs of data files, allows dynamic
760 configuration of plots, and includes a metadata browser. For each
761 loaded data file, additional data files can be loaded and added to the
762 plot, similar to what happens when specifying multiple input files for
763 plotting on the command line. A checkbox controls whether the added
764 data files are added as separate entries to the plot, or whether they
765 are used for scaling the output (mirroring the --scale-mode) command
766 line switch.
767 The GUI also incorporates matplotlib’s interactive browsing toolbar,
769 enabling panning and zooming of the plot area, dynamic configuration of
770 plot and axis parameters and labels and saving the plots to file. The
771 exact dynamic features supported depends on the installed version of
772 matplotlib.
773
775 The RC file
776 Some of the command line options can be specified in an rc file. By
777 default, flent looks for this in ~/.flentrc, but an alternative loca‐
778 tion can be specified with the --rcfile command line option.
779 The rc file allows options to be specified globally, an optionally
781 overridden for specific tests. For an explanation of the options, refer
782 to the annotated example rc file, by default installed to
783 /usr/share/doc/flent/flentrc.example.
784 Batch Files
786 Flent supports reading batch files to automate running several tests
787 and do setup/teardown of test environment etc. This greatly aids repro‐
788 ducibility of tests.
789 The batch file is parsed as an ini file, and can have three types of
791 sections: batches, commands and args. Each section also has a name;
792 type and name are separated with two colons. 'Batches' are named tests
793 that can be selected from the command line, 'commands' are system com‐
794 mands to be run before or after each test run, and 'args' are used in
795 the looping mechanism (which allows repeating tests multiple times with
796 different parameters).
797 Variables in sections control the operation of Flent and can be modi‐
799 fied in several ways: Sections of the same type can inherit from each
800 other and the variables in an 'arg' section will be interpolated into
801 the batch definition on each iteration of a loop. In addition, variable
802 contents can be substituted into other variables by using the ${var‐
803 name} syntax. These three operations are resolved in this order (inher‐
804 itance, arg interpolation and variable substitution).
805 An annotated example batchfile is distributed with the source code, and
807 is by default installed to /usr/share/doc/flent/batchfile.example.
808
810 The aggregated test data is saved in a file called
811 <test_name>-<date>.<title>.flent.gz (the title part is omitted if no
812 title is specified by the -t parameter). This file contains the data
813 points generated during the test, as well as some metadata.
814 The top-level object keys
816 version
817 The file format version as an integer.
818 x_values
820 An array of the x values for the test data (typically the time
821 values for timeseries data).
822 results
824 A JSON object containing the result data series. The keys are
825 the data series names; the value for each key is an array of y
826 values for that data series. The data array has the same length
827 as the x_values array, but there may be missing data points
828 (signified by null values).
829 metadata
831 An object containing various data points about the test run. The
832 metadata values are read in as configuration parameters when the
833 data set is loaded in for further processing. Not all tests use
834 all the parameters, but they are saved anyway.
835 raw_values
837 An array of objects for each data series. Each element of the
838 array contains the raw values as parsed from the test tool cor‐
839 responding to that data series.
840 Metadata keys
842 NAME The test name.
843 TITLE Any extra title specified by the --title-extra parameter when
845 the test was run.
846 HOSTS List of the server hostnames connected to during the test.
848 LOCAL_HOST
850 The hostname of the machine that ran the test.
851 LENGTH Test length in seconds, as specified by the --length parameter.
853 TOTAL_LENGTH
855 Actual data series length, after the test has added time to the
856 LENGTH.
857 STEP_SIZE
859 Time step size granularity.
860 TIME ISO timestamp of the time the test was initiated.
862 NOTE Arbitrary text as entered with the --note switch when the test
864 was run.
865 FLENT_VERSION
867 Version of Flent that generated the data file.
868 IP_VERSION
870 IP version used to run test (as specified by command line param‐
871 eters, or auto-detected from getaddrinfo() if unspecified).
872 KERNEL_NAME
874 The kernel name as reported by uname -s.
875 KERNEL_RELEASE
877 The kernel release as reported by uname -r.
878 MODULE_VERSIONS
880 The sha1sum of certain interesting Linux kernel modules, if
881 available. Can be used to match test data to specific code ver‐
882 sions, if the kernel build is instrumented to, e.g., set the
883 build ID to a git revision.
884 SYSCTLS
886 The values of several networking-related sysctls on the host (if
887 available; Linux only).
888 EGRESS_INFO
890 Interface name, qdisc, offload, driver and BQL configuration of
891 the interface used to reach the test target. This requires that
892 the ip binary is present on Linux, but can be extracted from
893 route on BSD. Qdisc information requires the tc binary to be
894 present, and offload information requires ethtool.
895 If the --remote-metadata is used, the extended metadata info is
897 gathered for each of the hostnames specified. This is gathered
898 under the REMOTE_METADATA key in the metadata object, keyed by
899 the hostname values passed to --remote-metadata. Additionally,
900 the REMOTE_METADATA object will contain an object called
901 INGRESS_INFO which is a duplicate of EGRESS_INFO, but with the
902 destination IP exchanged for the source address of the host run‐
903 ning flent. The assumption here is that --remote-metadata is
904 used to capture metadata of a router known to be in the test
905 path, in which case INGRESS_INFO will contain information about
906 the reverse path from the router (which is ingress from the
907 point of view of the host running flent). If the host being
908 queried for remote metadata is off the path, the contents of
909 INGRESS_INFO will probably be the same as that of EGRESS_INFO .
910 Extended metadata
912 If the --extended-metadata switch is turned on, the following addi‐
913 tional values are collected and stored (to the extent they are avail‐
914 able from the platform):
915 IP_ADDRS
917 IP addresses assigned to the machine running flent.
918 GATEWAYS
920 IP addresses of all detected default gateways on the system, and
921 the interfaces they are reachable through. Only available if the
922 netstat binary is present on the system.
923 EGRESS_INFO
925 In the EGRESS_INFO key, the IP address of the next-hop router
926 and the interface MAC address are added if extended metadata is
927 enabled.
928
930 The following output formats are currently supported by Flent:
931 Plot output (-f plot)
933 Output test data as one of a series of graphical plots of timeseries
934 data or summarised as a CDF plot. Each test supplies a number of dif‐
935 ferent plots; the list of plots for a given test is output by the
936 --list-plots switch (which must be supplied along with a test name).
937 The plots are drawn by matplotlib, and can be displayed on the screen
939 interactively (requires a graphical display), or output to a file in
940 svg, pdf, ps and png formats. Using the -o switch turns out file output
941 (the file format is inferred from the file name), while not supplying
942 the switch turns on the interactive plot viewer.
943 Tabulated output (-f csv and -f org_table)
945 These formats output the numeric data in a tabulated format to be con‐
946 sumed by other applications. The csv output format is a comma-separated
947 output that can be imported into e.g. spreadsheets, while org_table
948 outputs a tabulated output in the table format supported by Emacs org
949 mode. The data is output in text format to standard output, or written
950 to a file if invoked with the -o parameter.
951 Statistics output (-f stats)
953 This output format outputs various statistics about the test data, such
954 as total bandwidth consumed, and various statistical measures
955 (min/max/mean/median/std dev/variance) for each data source specified
956 in the relevant test (this can include some data sources not includes
957 on plots). The data is output in text format to standard output, or
958 written to a file if invoked with the -o parameter.
959 Metadata output (-f metadata)
961 This output format outputs the test metadata as pretty-printed JSON
962 (also suitable for human consumption). It is output as a list of
963 objects, where each object corresponds to the metadata of one test.
964 Mostly useful for inspecting metadata of stored data files.
965
967 Running Tests With The D-ITG Tool
968 This version of flent has experimental support for running and parsing
969 the output of the D-ITG test tool (see
970 http://traffic.comics.unina.it/software/ITG/). Flent supports parsing
971 the one-way delay as measured by D-ITG. However, in order to do so, the
972 data needs to be collected at the receiver end, statistics extracted,
973 and the result passed back to flent on the sending side.
974 To perform this function, flent supports a control server which will
976 listen to XML-RPC requests, spawn an appropriate ITGRecv instance and,
977 after the test is done, parse its output and make it available for
978 flent to retrieve. This control server is available as a Python file
979 that by default is installed in /usr/share/doc/flent/misc. It currently
980 requires a patched version of D-ITG v2.8.1. The patch is also included
981 in the same directory.
982 Note that the D-ITG server is finicky and not designed with security in
984 mind. For this reason, the control server includes HMAC authentication
985 to only allow authenticated clients to run a test against the server;
986 however there is currently no support for enforcement of this in e.g.
987 firewall rules. Please bear this in mind when running a publicly reach‐
988 able ITGRecv instance (with or without the control server). Another
989 security issue with the control server is that the Python XML-RPC
990 library by default is vulnerable to XML entity expansion attacks. For
991 this reason, it is highly recommended to install the defusedxml library
992 (available at https://pypi.python.org/pypi/defusedxml/) on the host
993 running the control server. The server will try to find the library on
994 startup and refuse to run if it is not available, unless explicitly
995 told otherwise.
996 Due to the hassle of using D-ITG, it is recommended to install irtt
998 instead and use that for VoIP tests.
999 Bugs
1001 Under some conditions (such as severe bufferbloat), the UDP RTT mea‐
1002 surements done by netperf can experience packet loss to the extent that
1003 the test aborts completely, which can cause missing data points for
1004 some measurement series. The --socket-timeout feature can alleviate
1005 this, but requires a recent SVN version of netperf to work. Flent tries
1006 to detect if netperf supports this option and enables it for the UDP
1007 measurements if it does. Using irtt for UDP measurements is a way to
1008 alleviate this; Flent will automatically detect the availability of
1009 irtt and use it if available.
1010 Probably many other bugs. Please report any found to
1012 https://github.com/tohojo/flent/issues and include the output of flent
1013 --version in the report. A debug log (as obtained with flent
1014 --log-file) is also often useful.
1015
1017 Toke Høiland-Jørgensen
1018
1020 2012-2017, Toke Høiland-Jørgensen and contributors. Source code is
1021 GPLv3. Documentation is CC-BY-SA
10221.3.2 November 26, 2019 FLENT(1)