1FLENT(1) Flent: The FLExible Network Tester FLENT(1)
2
6 flent - Flent: The FLExible Network Tester
7
9 Flent is a wrapper around netperf and similar tools to run predefined
10 tests and aggregate and plot the results. It defines several tests that
11 can be run against one or more hosts, primarily targeted at testing for
12 the presence of bufferbloat under various conditions.
13 The aggregated data is saved in (gzipped) JSON format for later pro‐
15 cessing and/or import into other tools. The JSON format is documented
16 below.
17 Apart from the JSON format, the data can be output as csv values, emacs
19 org mode tables or plots. Each test can specify several different
20 plots, including time-series plots of the values against each other, as
21 well as CDF plots of (e.g.) ping times.
22 Plots can be output to the formats supported by matplotlib by specify‐
24 ing the output filename with -o output.{png,ps,pdf,svg}. If no output
25 file is specified, the plot is displayed using matplotlib’s interactive
26 plot browser, which also allows saving of the output (in .png format).
27 Requirements
29 Flent runs on Python, versions 2.7+ and 3.3+. Plotting requires a func‐
30 tional matplotlib installation (but everything else can run without
31 matplotlib). For the interactive plot viewer, a graphical display (and
32 suitably configured matplotlib) is required.
33 Most tests employ the netperf benchmarking tool to run the tests. Ver‐
35 sion 2.6 or higher is required, and netperf must be compiled with the
36 --enable-demo option passed to ./configure. Some tests use iperf in ad‐
37 dition to, or instead of netperf. Both tools must be available in the
38 PATH.
39 For ICMP ping measurements, the version of ping employed must support
41 output timestamping (the -D parameter to GNU ping). This is not sup‐
42 ported by the BSD and OSX versions of ping. As an alternative to the
43 regular ping command, the fping utility (see http://fping.org) can be
44 employed. In that case fping must be version 3.5 or greater. Flent will
45 attempt to detect the presence of fping in the PATH and check for sup‐
46 port for the -D parameter. If this check is successful, fping will be
47 employed for ping data, otherwise the system ping will be used.
48 The irtt tool is highly recommended for UDP measurements. See
50 https://github.com/peteheist/irtt. Flent will automatically detect if
51 irtt is available in the PATH and use it if it is detected. Note that
52 the server component of irtt needs to be running on the target host(s)
53 as well.
54
56 When run, flent must be supplied either (a) a test name and one or more
57 host names to connect to, or (b) one or more input files containing
58 data from previous runs to post-process.
59 Test names, hostnames and input file names can all be specified as un‐
61 qualified arguments, and flent will do its best to guess which is
62 which. For each argument, if it is an existing file, it is assumed to
63 be an input file, if it is the name of an existing test configuration
64 it’s assumed to be a test name, and if neither of those are true, it is
65 assumed to be a host name. The -i and -H switches can be used to ex‐
66 plicitly specify the interpretation of an argument.
67 Invocation
69 flent [options] <host|test|input file ...>
70 General options
72 -o OUTPUT, --output=OUTPUT
73 File to write processed output to (default standard out).
74 -D DATA_DIR, --data-dir=DATA_DIR
76 Directory to store data files in. Defaults to the current direc‐
77 tory.
78 -i INPUT, --input=INPUT
80 File to read input from (instead of running tests). Input files
81 can also be specified as unqualified arguments without using the
82 -i switch.
83 -f FORMAT, --format=FORMAT
85 Select output format (plot, csv, org_table, stats). Default is
86 no processed output (just writes the JSON data file).
87 -p PLOT, --plot=PLOT
89 Select which plot to output for the given test (implies -f
90 plot). Use the --list-plots option to see available plots.
91 -t TITLE, --title-extra=TITLE
93 Text to add to plot title and data file name.
94 -n NOTE, --note=NOTE
96 Add arbitrary text as a note to be stored in the JSON data file
97 (under the NOTE key in the metadata object).
98 -r RCFILE, --rcfile=RCFILE
100 Load configuration data from RCFILE (default ~/.flentrc). See
101 section below for information on the rc file format.
102 -x, --extended-metadata
104 Collect extended metadata and store it with the data file. May
105 include details of your machine you don’t want to distribute;
106 see the section on the data format below.
107 --remote-metadata=HOSTNAME
109 Collect extended metadata from a remote host. HOSTNAME is passed
110 verbatim to ssh, so can include hosts specified in ~/.ssh/con‐
111 fig. Note that gathering the data can take some time, since it
112 involves executing several remote commands. This option can be
113 specified multiple times and implies --extended-metadata.
114 --gui Run the flent GUI. All other options are used as defaults in the
116 GUI, but can be changed once it is running. The GUI can also be
117 started by running the flent-gui binary. For more information on
118 the GUI, see the The Flent GUI section.
119 --new-gui-instance
121 Start a new GUI instance. Otherwise, flent will try to connect
122 to an already running GUI instance and have that load any new
123 data files specified as arguments. Implies --gui when passed on
124 the command line, but not when set in the rc file. Note that
125 when multiple GUI instances are running, there is no guarantee
126 as to which instance will get a subsequent open request (if run
127 again without --new-gui-instance).
128 --gui-no-defer
130 Normally, the GUI defers redrawing plots until they are needed
131 to avoid redrawing all open plots every time an option changes.
132 This switch turns off that optimisation in favour of always re‐
133 drawing everything straight away. This is useful when loading a
134 bunch of plots from the command line and then wanting to flip
135 through them without drawing delay.
136 -b BATCH_NAME, --batch-name=BATCH_NAME
138 Run test batch BATCH_NAME (must be specified in a batch file
139 loaded by the --batch-file option). Can be supplied multiple
140 times.
141 -B BATCH_FILE, --batch-file=BATCH_FILE
143 Load batch file BATCH_FILE. Can be specified multiple times, in
144 which case the files will be combined (with identically-named
145 sections being overridden by later files). See appropriate sec‐
146 tion below for an explanation of the batch file format.
147 --batch-override=key=value
149 Override parameter ’key’ in the batch config and set it to
150 ’value’. The key name will be case folded to lower case. Can be
151 specified multiple times.
152 --batch-dry-run
154 Dry batch run. Prints what would be done, but doesn’t actually
155 run any tests.
156 --batch-verbose
158 Be verbose during batch run: Print all commands executed.
159 --batch-no-shuffle
161 Do not randomise the order of test runs within each batch.
162 --batch-repetitions=REPETITIONS
164 Shorthand for --batch-override 'repetitions=REPETITIONS’.
165 --batch-title=TITLE
167 Shorthand for --batch-override 'batch_title=TITLE’.
168 --batch-resume=DIR
170 Try to resume a previously interrupted batch run. The argument
171 is the top-level output directory from the previous run.
172 This will attempt to find a data file in the resume directory
174 and load the BATCH_TIME from the previous run from that and con‐
175 tinue. The assumption is that the output directory and filenames
176 are generated from the batch time, so that they will match with
177 the previous run when the same time is used. Then, tests for
178 which data files already exist will be skipped on this run. If
179 the rest of the batch invocation is different from the one being
180 resumed, results may not be what you want.
181 There's a check to ensure that the generated output path is a
183 subdirectory of the resume directory, and the whole run will be
184 aborted if it isn't.
185 Test configuration options
187 These options affect the behaviour of the test being run and have no
188 effect when parsing input files.
189 -H HOST, --host=HOST
191 Host to connect to for tests. For tests that support it, multi‐
192 ple hosts can be specified by supplying this option multiple
193 times. Hosts can also be specified as unqualified arguments;
194 this parameter guarantees that the argument be interpreted as a
195 host name (rather than being subject to auto-detection between
196 input files, hostnames and test names).
197 --local-bind=IP
199 Local hostname or IP address to bind to (for test tools that
200 support this). Can be specified multiple times for tests that
201 connect to more than one host; if it is, it must be specified as
202 many times as there are hosts.
203 --remote-host=idx=HOSTNAME
205 A remote hostname to connect to when starting a test. The idx is
206 the runner index, which is assigned sequentially to each runner
207 (and so it is not the same as the sequence of hostnames). Look
208 for the 'IDX' key in SERIES_META for a test get the idx used
209 here, but note that the idx assignment depends on the exact ar‐
210 guments to the test.
211 This works by simply prepending 'ssh HOSTNAME' to the runner
213 command, so it relies on the same binaries being in the same
214 places on both machines, and won't work for all runners.
215 This option can be specified multiple times to have multiple
217 runners run on remote hosts.
218 -l LENGTH, --length=LENGTH
220 Base test length (some tests may add some time to this).
221 -s STEP_SIZE, --step-size=STEP_SIZE
223 Measurement data point step size.
224 -d DELAY, --delay=DELAY
226 Number of seconds to delay parts of test (such as bandwidth
227 loaders).
228 -4, --ipv4
230 Use IPv4 for tests (some tests may ignore this).
231 -6, --ipv6
233 Use IPv6 for tests (some tests may ignore this).
234 --socket-timeout=SOCKET_TIMEOUT
236 Socket timeout (in seconds) used for UDP delay measurement, to
237 prevent stalls on packet loss. Only enabled if the installed
238 netperf version is detected to support this (requires SVN ver‐
239 sion of netperf).
240 For the default value, see the output of flent -h. The value of
242 this parameter is an implicit upper bound on how long a
243 round-trip time that can be measured. As such you may need to
244 adjust it if you are experiencing latency above the default
245 value. Set to 0 to disable.
246 --send-size=SEND_SIZE
248 Send size (in bytes) used for TCP tests. Netperf uses the socket
249 buffer size by default, which if too large can cause spikes in
250 the throughput results. Lowering this value will increase CPU
251 usage but also improves the fidelity of the throughput results
252 without having to decrease the socket buffer size.
253 Can be specified multiple times, with each value corresponding
255 to a stream of a test. If only specified once, the same value
256 will be applied to all flows.
257 --test-parameter=key=value
259 Arbitrary test parameter in key=value format. Key will be case
260 folded to lower case. The values are stored with the results
261 metadata, and so can be used for storing arbitrary information
262 relevant for a particular test run.
263 In addition to serving as simple metadata, the test parameters
265 can also affect the behaviour of some test configurations. See
266 the Supplied Tests section for information on these.
267 This option can be specified multiple times to set multiple test
269 parameters.
270 --swap-up-down
272 Switch upstream and downstream directions for data transfer.
273 This means that ’upload’ will become ’download’ and vice versa.
274 Works by exchanging netperf TCP_MAERTS and TCP_STREAM parame‐
275 ters, so only works for tests that employ these as their data
276 transfer, and only for the TCP streams.
277 --socket-stats
279 Parse socket stats during test. This will capture and parse
280 socket statistics for all TCP upload flows during a test, adding
281 TCP cwnd and RTT values to the test data. Requires the 'ss'
282 utility to be present on the system, and can fail if there are
283 too many simultaneous upload flows; which is why this option is
284 not enabled by default.
285 --marking-name
287 Define a new symbolic name that can be used when specifying flow
288 markings using the 'markings' test parameter. This can be used
289 to make it easier to specify custom diffserv markings on flows
290 by using symbolic names for each marking value instead of the
291 hex codes. Values specified here will be used in addition to the
292 common values (listed below), and cannot override the built-in
293 names. Names will be case-folded when matching.
294 The list of symbolic markings natively supported, along with
296 their hex expansions, are:
297 AF11: 0x28 CS0: 0x00
299 AF12: 0x30 CS1: 0x20
300 AF13: 0x38 CS2: 0x40
301 AF21: 0x48 CS3: 0x60
302 AF22: 0x50 CS4: 0x80
303 AF23: 0x58 CS5: 0xa0
304 AF31: 0x68 CS6: 0xc0
305 AF32: 0x70 CS7: 0xe0
306 AF33: 0x78 EF: 0xb8
307 AF41: 0x88
308 AF42: 0x90
309 AF43: 0x98
310 Note that the hexadecimal values denote the value of the full
312 ToS byte (including the two ECN bits), so they need to be
313 right-shifted by two bits to get the corresponding diffserv code
314 points.
315 Plot configuration options
317 These options are used to configure the appearance of plot output and
318 only make sense combined with -f plot.
319 --label-x=LABEL
321 --label-y=LABEL
323 Override the figure axis labels. Can be specified twice, corre‐
324 sponding to figures with multiple axes.
325 -I, --invert-latency-y
327 Invert latency data series axis (typically the Y-axis), making
328 plots show ’better' values upwards.
329 -z, --zero-y
331 Always start Y axis of plot at zero, instead of autoscaling the
332 axis. Autoscaling is still enabled for the upper bound. This
333 also disables log scale if enabled.
334 --log-scale={log2,log10}
336 Use the specified logarithmic scale on plots.
337 --norm-factor=FACTOR
339 Data normalisation factor. Divide all data points by this value.
340 Can be specified multiple times, in which case each value corre‐
341 sponds to a data series.
342 --bounds-x=BOUNDS
344 --bounds-y=BOUNDS
346 Specify bounds of the plot axes. If specifying one number, that
347 will become the upper bound. Specify two numbers separated by a
348 comma to specify both upper and lower bounds. To specify just
349 the lower bound, add a comma afterwards. Can be specified twice,
350 corresponding to figures with multiple axes.
351 -S, --scale-mode
353 Treat file names (except for the first one) passed as unquali‐
354 fied arguments as if passed as --scale-data (default as if
355 passed as --input).
356 --concatenate
358 Concatenate multiple result sets into one data series. This
359 means that each data file will have its time axis shifted by the
360 preceding series duration and appended to the first data set
361 specified. Only works for data sets from the same test, obvi‐
362 ously.
363 --absolute-time
365 Plot data points with absolute UNIX time on the x-axis. This re‐
366 quires the absolute starting time for the test run to be stored
367 in the data file, and so it won’t work with data files that pre‐
368 dates this feature.
369 --subplot-combine
371 When plotting multiple data series, plot each one on a separate
372 subplot instead of combining them into one plot. This mode is
373 not supported for all plot types, and only works when
374 --scale-mode is disabled.
375 --skip-missing-series
377 Skip missing series entirely from bar plots, instead of leaving
378 an empty space for it.
379 --no-print-n
381 Do not print the number of data points on combined plots. When
382 using plot types that combines results from several test runs,
383 the number of data series in each combined data point is nor‐
384 mally added after the series name, (n=X) for X data series. This
385 option turns that off.
386 --no-annotation
388 Exclude annotation with hostnames, time and test length from
389 plots.
390 --figure-note=NOTE, --fig-note=NOTE
392 Add a note (arbitrary text) to the bottom-left of the figure.
393 --no-title
395 Exclude title from plots.
396 --override-title=TITLE
398 Override plot title with this string. Completely discards the
399 configured title (from the test configuration), as well as the
400 title stored in the data set, and replaces it with the value
401 supplied here. This is useful to override the plot title at the
402 time of plotting, for instance to add a title to an aggregate
403 plot from several data series. When this parameter is specified,
404 --no-title has no effect.
405 --no-labels
407 Hides tick labels from box and bar plots.
408 --no-markers
410 Don’t use line markers to differentiate data series on plots.
411 --no-legend
413 Exclude legend from plots.
414 --horizontal-legend
416 Place a horizontal legend below the plot instead of a vertical
417 one next to it. Doesn't always work well if there are too many
418 items in the legend.
419 --legend-title=LEGEND_TITLE
421 Override legend title on plot.
422 --legend-placement=LEGEND_PLACEMENT
424 Control legend placement. Enabling this option will place the
425 legend inside the plot at the specified location. Can be one of
426 'best', 'upper right', 'upper left', 'lower left', 'lower
427 right', 'right', 'center left', 'center right', 'lower center',
428 'upper center' or 'center'.
429 --legend-columns=LEGEND_COLUMNS
431 Set the number of columns in the legend.
433 --reverse-legend
435 Reverse the order of items in the legend. This can be useful to
436 make the legend order match the data series in some cases.
437 --filter-legend
439 Filter legend labels by removing the longest common substring
440 from all entries. This is not particularly smart, so use with
441 care.
442 --replace-legend=src=dest
444 Replace 'src' with 'dst' in legends. Can be specified multiple
445 times.
446 --filter-regexp=REGEXP
448 Filter the plot legend by the supplied regular expression (re‐
449 moving any text from the legend that matches the expression).
450 Can be specified multiple times, in which case the filters will
451 be applied in the order specified. Note that for combining sev‐
452 eral plot results, the regular expression is also applied before
453 the grouping logic, meaning that a too wide filter can mess up
454 the grouping.
455 --override-label=LABEL
457 Override dataset label. Can be specified multiple times when
458 multiple datasets are being plotted, in which case the order of
459 labels corresponds to the order of datasets.
460 Like --override-title, this is applied at the time of plotting.
462 --filter-series=SERIES
464 Filter out specified series from plot. Can be specified multiple
465 times.
466 --split-group=LABEL
468 Split data sets into groups when creating box plots. Specify
469 this option multiple times to define the new groups; the value
470 of each option is the group name.
471 Say you're plotting nine datasets which are really testing two
473 variables with three values each. In this case, it can be useful
474 to have the box plot of the results be split into three parts
475 (corresponding to the values of one variable) with each three
476 boxes in each of them (corresponding to the values of the second
477 variable). This option makes this possible; simply specify it
478 three times with the labels to be used for the three groups.
479 A constraint on this option is that the number of datasets being
481 plotted must be divisible by the number of groups.
482 --colours=COLOURS
484 Comma-separated list of colours to be used for the plot colour
485 cycle. Can be specified in any format understood by matplotlib
486 (including HTML hex values prefixed with a #).
487 Yes, this option uses British spelling. No, American spelling is
489 not supported.
490 --override-colour-mode=MODE
492 Override colour_mode attribute. This changes the way colours are
493 assigned to bar plots. The default is 'groups' which assigns a
494 separate colour to each group of data series. The alternative is
495 'series' which assigns a separate colour to each series, repeat‐
496 ing them for each data group.
497 --override-group-by=GROUP
499 Override the group_by setting for combination plots. This is
500 useful to, for instance, switch to splitting up combined data
501 sets by batch run instead of by file name.
502 --combine-save-dir=DIRNAME
504 When doing a combination plot save the intermediate data to
505 DIRNAME. This can then be used for subsequent plotting to avoid
506 having to load all the source data files again on each plot.
507 --figure-width=FIG_WIDTH
509 Figure width in inches. Used when saving plots to file and for
510 default size of the interactive plot window.
511 --figure-height=FIG_HEIGHT
513 Figure height in inches. Used when saving plots to file and for
514 default size of the interactive plot window.
515 --figure-dpi=FIG_DPI
517 Figure DPI. Used when saving plots to raster format files.
518 --fallback-layout
520 Use the fallback layout engine (tight_layout built in to mat‐
521 plotlib). Use this if text is cut off on saved figures. The
522 downside to the fallback engine is that the size of the figure
523 (as specified by --figure-width and --figure-height) is no
524 longer kept constant.)
525 --no-matplotlibrc
527 Don’t load included matplotlibrc values. Use this if autodetec‐
528 tion of custom matplotlibrc fails and flent is inadvertently
529 overriding rc values.
530 --no-hover-highlight
532 Don't highlight data series on hover in interactive plot views.
533 Use this if redrawing is too slow, or the highlighting is unde‐
534 sired for other reasons.
535 --scale-data=SCALE_DATA
537 Additional data files to consider when scaling the plot axes
538 (for plotting several plots with identical axes). Note, this
539 displays only the first data set, but with axis scaling taking
540 into account the additional data sets. Can be supplied multiple
541 times; see also --scale-mode.
542 Test tool-related options
544 --control-host=HOST
545 Hostname for the test control connection (for test tools that
546 support this). Default: First hostname of test target.
547 When running tests that uses D-ITG as a test tool (such as the
549 voip-* tests), this switch controls where flent will look for
550 the D-ITG control server (see section below on running tests
551 with D-ITG). For Netperf-based tests, this option is passed to
552 Netperf to control where to point the control connection. This
553 is useful to, for instance, to run the control server communica‐
554 tion over a separate control network so as to not interfere with
555 test traffic.
556 There is also a per-flow setting for this for tests that connect
558 to multiple hosts; see the control_hosts test parameter in
559 Supplied Tests. If both are set, the per-flow setting takes
560 precedence for those tests that use it.
561 --control-local-bind=IP
563 Local hostname or IP to bind control connection to (for test
564 tools that support it; currently netperf). If not supplied, the
565 value for --local-bind will be used. Note that if this value is
566 passed but --local-bind is not, netperf will use the value spec‐
567 ified here to bind the data connections to as well.
568 --netperf-control-port=PORT
570 Port for Netperf control server. Default: 12865.
571 --ditg-control-port=PORT
573 Port for D-ITG control server. Default: 8000.
574 --ditg-control-secret=SECRET
576 Secret for D-ITG control server authentication. Default: ’’.
577 --http-getter-urllist=FILENAME
579 When running HTTP tests, the http-getter tool is used to fetch
580 URLs (see https://github.com/tohojo/http-getter). This option
581 specifies the filename containing the list of HTTP URLs to get.
582 Can also be a URL, which will then be downloaded as part of each
583 test iteration. If not specified, this is set to
584 http://<hostname>/filelist.txt where <hostname> is the first
585 test hostname.
586 --http-getter-dns-servers=DNS_SERVERS
588 DNS servers to use for http-getter lookups. Format is
589 host[:port][,host[:port]]... This option will only work if
590 libcurl supports it (needs to be built with the ares resolver).
591 Default is none (use the system resolver).
592 --http-getter-timeout=MILLISECONDS
594 Timeout for HTTP connections. Default is to use the test length.
595 --http-getter-workers=NUMBER
597 Number of workers to use for getting HTTP urls. Default is 4.
598 --irtt-sampling-interval=MILLISECONDS
600 Override the sampling interval passed to irtt, in milliseconds.
601 Can be used to run irtt with a higher sampling frequency than
602 the rest of the test. If set, this will override the sampling
603 interval for all instances of irtt used in the test.
604 Misc and debugging options:
606 -L LOG_FILE, --log-file=LOG_FILE
607 Write debug log (test program output) to log file.
608 --list-tests
610 List available tests and exit.
611 --list-plots
613 List available plots for selected test and exit.
614 -V, --version
616 Show Flent version information and exit.
617 -v, --verbose
619 Enable verbose logging to console.
620 -q, --quiet
622 Disable normal logging to console (and only log warnings and er‐
623 rors).
624 --debug-error
626 Print full exception backtraces to console.
627 -h, --help
629 Show usage help message and exit.
630 Signals
632 Flent will abort what it is currently doing on receiving a SIGINT --
633 this includes killing all runners, cleaning up temporary files and
634 shutting down as gracefully as possible. Runners are killed with
635 SIGTERM in this mode, and their output is discarded. If a batch run is
636 in progress, the current test will be interrupted in this way, and the
637 rest of the batch run is aborted. Previously completed tests and their
638 results are not aborted. Post-commands marked as ’essential’ will be
639 run after the test is interrupted. Additionally, flent converts SIGTERM
640 into SIGINT internally and reacts accordingly.
641 Upon receiving a SIGUSR1, flent will try to gracefully abort the test
643 it is currently running, and parse the output of the runners to the ex‐
644 tent that any such output exists. That is, each runner will be killed
645 by a SIGINT, which will cause a graceful shutdown for at least ping and
646 netperf (although netperf running in TCP_MAERTS mode will bug out when
647 interrupted like this, so end-of-tests statistics will be missing).
648 Flent will only react once to a SIGUSR1, sending exactly one SIGINT to
649 the active runners, then wait for them to exit. This may take several
650 seconds in the case of netperf. If the runners for some reason fail to
651 exit, flent will be stuck and will need to be killed with SIGINT. If
652 running in batch mode, SIGUSR1 will only affect the currently running
653 test; subsequent tests will still be run.
654
656 Test are supplied as Python files and can specify commands to run etc.
657 For a full list of the tests supported by flent, see the --list-tests
658 option.
659 The Realtime Response Under Load (RRUL) test
661 This test exists in a couple of variants and is a partial implementa‐
662 tion of the RRUL specification as written by Dave Taht (see
663 https://www.bufferbloat.net/projects/bloat/wiki/RRUL_Spec/). It works
664 by running RTT measurement using ICMP ping and UDP roundtrip time mea‐
665 surement, while loading up the link with eight TCP streams (four down‐
666 loads, four uploads). This quite reliably saturates the measured link
667 (wherever the bottleneck might be), and thus exposes bufferbloat when
668 it is present.
669 Simple TCP flow tests
671 These tests combine a TCP flow (either in one direction, or both) with
672 an ICMP ping measurement. It’s a simpler test than RRUL, but in some
673 cases the single TCP flow can be sufficient to saturate the link.
674 UDP flood test
676 This test runs iperf configured to emit 100Mbps of UDP packets targeted
677 at the test host, while measuring RTT using ICMP ping. It is useful for
678 observing latency in the face of a completely unresponsive packet
679 stream.
680 Test parameters
682 Some test parameters (set with --test-parameter) affect the way tests
683 behave. These are:
684 upload_streams
686 download_streams
688 bidir_streams
690 These set the number of streams for the tests that are config‐
691 urable. The tcp_nup, tcp_ndown and rrul_be_nflows tests under‐
692 stand upload_streams and download_streams, while the rrul_var
693 test understands bidir_streams. If any of these parameter is set
694 to the special value num_cpus the number of streams will be set
695 to the number of CPUs on the system (if this information is
696 available).
697 tcp_cong_control
699 Set the congestion control used for TCP flows, for platforms
700 that supports setting it. This can be specified as a simple
701 string to set the same value for upstream and downstream, or two
702 comma-separated values to set it separately for the upstream and
703 downstream directions. On Linux, any value in the sysctl
704 net.ipv4.tcp_allowed_congestion_control can be used.
705 If a congestion control is specified that is not available on
707 the system running the test, setting it will simply fail. In ad‐
708 dition, some tests override the congestion control for one or
709 more flows. The actual congestion control used is stored in the
710 CONG_CONTROL per-test metadata field.
711 udp_bandwidth
713 udp_bandwidths
715 udp_pktsize
717 udp_pktsizes
719 This sets the bandwidth and packet size of each UDP stream in
720 the udp_* tests. The option is passed to iperf so can be in any
721 syntax the iperf understands (e.g. 20M for 20 Mbps).
722 When running multiple UDP streams use the plural versions of the
724 options (udp_bandwidths and udp_pktsizes) to specify individual
725 per-stream values (comma-separated per stream), or the singular
726 versions to specify the same value for all streams.
727 burst_length
729 burst_ports
731 burst_psize
733 burst_tos
735 These set the length, number of ports to use, packet size and
736 TOS value for the packet bursts generated in the burst* tests.
737 cpu_stats_hosts
739 netstat_hosts
741 qdisc_stats_hosts
743 wifi_stats_hosts
745 These set hostnames to gather statistics from from during the
746 test. The hostnames are passed to SSH, so can be anything under‐
747 stood by SSH (including using username@host syntax, or using
748 hosts defined in ~/.ssh/config). This will attempt to run re‐
749 mote commands on these hosts to gather the required statistics,
750 so passwordless login has to be enabled for. Multiple hostnames
751 can be specified, separated by commas.
752 CPU stats and netstat output is global to the machine being con‐
754 nected to. The qdisc and WiFi stats need extra parameters to
755 work. These are qdisc_stats_interfaces, wifi_stats_interfaces
756 and wifi_stats_stations. The two former specify which interfaces
757 to gather statistics from. These are paired with the hostnames,
758 and so must contain the same number of elements (also comma-sep‐
759 arated) as the _hosts variables. To specify multiple interfaces
760 on the same host, duplicate the hostname. The wifi_stats_sta‐
761 tions parameter specifies MAC addresses of stations to gather
762 statistics for. This list is the same for all hosts, but only
763 stations present in debugfs on each host are actually captured.
764 The qdisc stats gather statistics output from tc -s, while the
766 WiFi stats gather statistics from debugfs. These are gathered by
767 looping in a shell script; however, for better performance, the
768 tc_iterate and wifistats_iterate programmes available in the
769 misc/ directory of the source code tarball can be installed. On
770 low-powered systems this can be critical to get correct statis‐
771 tics. The helper programmes are packaged for LEDE/OpenWrt in the
772 flent-tools package.
773 ping_hosts
775 ping_local_binds
777 ping_labels
779 These are used to define one or more extra host names that will
780 receive a ping flow while a test is run. The ping_hosts variable
781 simply specifies hostnames to ping (several can be specified by
782 separating them with commas). The ping_local_binds variable
783 sets local IP address(es) to bind to for the extra ping flows.
784 If specified, it must contain the same number of local addresses
785 as the number of ping hosts. The same local address can be spec‐
786 ified multiple times, however. The ping_labels variable is used
787 to label each of the ping flows; if not specified, Flent will
788 create a default label based on the target hostname for each
789 flow.
790 voip_host
792 voip_local_bind
794 voip_control_host
796 voip_marking
798 Similar to the ping variants above, these parameters specify a
799 hostname that will receive a VoIP test. However, unlike the ping
800 parameters, only one hostname can be specified for VoIP tests,
801 and that end-host needs to have either D-ITG (and the control
802 server) or the IRTT server running. The marking setting controls
803 which DiffServ marking is applied to the VoIP flow and defaults
804 to no marking being set.
805 control_hosts
807 Hostnames to use for the control connections for the rtt_fair*
808 tests. Comma-separated. If specified, it must contain as many
809 hostnames as the number of target hostnames specified for the
810 test.
811 markings
813 labels Flow markings to use for each of the flows in the rtt_fair*
815 tests. Comma-separated values of markings understood by Netperf
816 (such as "CS0"). Only supports setting the same marking on both
817 the upstream and downstream packets of each flow (so no
818 "CS0,CS0" setting as can be used for Netperf). If not set, de‐
819 faults to CS0 (best effort). If set, each value corresponds to a
820 flow, and any extra flows will be set to CS0. By default each
821 flow will be labelled according to its marking; to override
822 these labels, use the labels parameter.
823 stream_delays
825 Specify a per-stream delay (in seconds) for the different
826 streams started up by a test. Use commas to separate values for
827 the different streams. This can be used to create tests with
828 staggered start times, for example to test TCP flow startup con‐
829 vergence times. What exactly constitutes a stream depends on the
830 test. For example, the rtt_fair* tests considers each hostname a
831 stream, whether or not there is one or two flows going to that
832 host.
833
835 Flent comes equipped with a GUI to browse and plot previously captured
836 datasets. The GUI requires PyQt4; if this is installed, it can be
837 launched with the --gui parameter, or by launching the flent-gui bi‐
838 nary. Additionally, if Flent is launched without parameters and with‐
839 out a controlling terminal, the GUI will be launched automatically.
840 The GUI can be used for interactively plotting previously captured
842 datasets, and makes it easy to compare results from several test runs.
843 It presents a tabbed interface to graphs of data files, allows dynamic
844 configuration of plots, and includes a metadata browser. For each
845 loaded data file, additional data files can be loaded and added to the
846 plot, similar to what happens when specifying multiple input files for
847 plotting on the command line. A checkbox controls whether the added
848 data files are added as separate entries to the plot, or whether they
849 are used for scaling the output (mirroring the --scale-mode) command
850 line switch.
851 The GUI also incorporates matplotlib’s interactive browsing toolbar,
853 enabling panning and zooming of the plot area, dynamic configuration of
854 plot and axis parameters and labels and saving the plots to file. The
855 exact dynamic features supported depends on the installed version of
856 matplotlib.
857
859 The RC file
860 Some of the command line options can be specified in an rc file. By de‐
861 fault, flent looks for this in ~/.flentrc, but an alternative location
862 can be specified with the --rcfile command line option.
863 The rc file allows options to be specified globally, an optionally
865 overridden for specific tests. For an explanation of the options, refer
866 to the annotated example rc file, by default installed to
867 /usr/share/doc/flent/flentrc.example.
868 Batch Files
870 Flent supports reading batch files to automate running several tests
871 and do setup/teardown of test environment etc. This greatly aids repro‐
872 ducibility of tests.
873 The batch file is parsed as an ini file, and can have three types of
875 sections: batches, commands and args. Each section also has a name;
876 type and name are separated with two colons. 'Batches' are named tests
877 that can be selected from the command line, 'commands' are system com‐
878 mands to be run before or after each test run, and 'args' are used in
879 the looping mechanism (which allows repeating tests multiple times with
880 different parameters).
881 Variables in sections control the operation of Flent and can be modi‐
883 fied in several ways: Sections of the same type can inherit from each
884 other and the variables in an 'arg' section will be interpolated into
885 the batch definition on each iteration of a loop. In addition, variable
886 contents can be substituted into other variables by using the ${var‐
887 name} syntax. These three operations are resolved in this order (inher‐
888 itance, arg interpolation and variable substitution).
889 An annotated example batchfile is distributed with the source code, and
891 is by default installed to /usr/share/doc/flent/batchfile.example.
892
894 The aggregated test data is saved in a file called
895 <test_name>-<date>.<title>.flent.gz (the title part is omitted if no
896 title is specified by the -t parameter). This file contains the data
897 points generated during the test, as well as some metadata.
898 The top-level object keys
900 version
901 The file format version as an integer.
902 x_values
904 An array of the x values for the test data (typically the time
905 values for timeseries data).
906 results
908 A JSON object containing the result data series. The keys are
909 the data series names; the value for each key is an array of y
910 values for that data series. The data array has the same length
911 as the x_values array, but there may be missing data points
912 (signified by null values).
913 metadata
915 An object containing various data points about the test run. The
916 metadata values are read in as configuration parameters when the
917 data set is loaded in for further processing. Not all tests use
918 all the parameters, but they are saved anyway.
919 raw_values
921 An array of objects for each data series. Each element of the
922 array contains the raw values as parsed from the test tool cor‐
923 responding to that data series.
924 Metadata keys
926 NAME The test name.
927 TITLE Any extra title specified by the --title-extra parameter when
929 the test was run.
930 HOSTS List of the server hostnames connected to during the test.
932 LOCAL_HOST
934 The hostname of the machine that ran the test.
935 LENGTH Test length in seconds, as specified by the --length parameter.
937 TOTAL_LENGTH
939 Actual data series length, after the test has added time to the
940 LENGTH.
941 STEP_SIZE
943 Time step size granularity.
944 TIME ISO timestamp of the time the test was initiated.
946 NOTE Arbitrary text as entered with the --note switch when the test
948 was run.
949 FLENT_VERSION
951 Version of Flent that generated the data file.
952 IP_VERSION
954 IP version used to run test (as specified by command line param‐
955 eters, or auto-detected from getaddrinfo() if unspecified).
956 KERNEL_NAME
958 The kernel name as reported by uname -s.
959 KERNEL_RELEASE
961 The kernel release as reported by uname -r.
962 MODULE_VERSIONS
964 The sha1sum of certain interesting Linux kernel modules, if
965 available. Can be used to match test data to specific code ver‐
966 sions, if the kernel build is instrumented to, e.g., set the
967 build ID to a git revision.
968 SYSCTLS
970 The values of several networking-related sysctls on the host (if
971 available; Linux only).
972 EGRESS_INFO
974 Interface name, qdisc, offload, driver and BQL configuration of
975 the interface used to reach the test target. This requires that
976 the ip binary is present on Linux, but can be extracted from
977 route on BSD. Qdisc information requires the tc binary to be
978 present, and offload information requires ethtool.
979 If the --remote-metadata is used, the extended metadata info is
981 gathered for each of the hostnames specified. This is gathered
982 under the REMOTE_METADATA key in the metadata object, keyed by
983 the hostname values passed to --remote-metadata. Additionally,
984 the REMOTE_METADATA object will contain an object called
985 INGRESS_INFO which is a duplicate of EGRESS_INFO, but with the
986 destination IP exchanged for the source address of the host run‐
987 ning flent. The assumption here is that --remote-metadata is
988 used to capture metadata of a router known to be in the test
989 path, in which case INGRESS_INFO will contain information about
990 the reverse path from the router (which is ingress from the
991 point of view of the host running flent). If the host being
992 queried for remote metadata is off the path, the contents of
993 INGRESS_INFO will probably be the same as that of EGRESS_INFO .
994 Extended metadata
996 If the --extended-metadata switch is turned on, the following addi‐
997 tional values are collected and stored (to the extent they are avail‐
998 able from the platform):
999 IP_ADDRS
1001 IP addresses assigned to the machine running flent.
1002 GATEWAYS
1004 IP addresses of all detected default gateways on the system, and
1005 the interfaces they are reachable through. Only available if the
1006 netstat binary is present on the system.
1007 EGRESS_INFO
1009 In the EGRESS_INFO key, the IP address of the next-hop router
1010 and the interface MAC address are added if extended metadata is
1011 enabled.
1012
1014 The following output formats are currently supported by Flent:
1015 Plot output (-f plot)
1017 Output test data as one of a series of graphical plots of timeseries
1018 data or summarised as a CDF plot. Each test supplies a number of dif‐
1019 ferent plots; the list of plots for a given test is output by the
1020 --list-plots switch (which must be supplied along with a test name).
1021 The plots are drawn by matplotlib, and can be displayed on the screen
1023 interactively (requires a graphical display), or output to a file in
1024 svg, pdf, ps and png formats. Using the -o switch turns out file output
1025 (the file format is inferred from the file name), while not supplying
1026 the switch turns on the interactive plot viewer.
1027 Tabulated output (-f csv and -f org_table)
1029 These formats output the numeric data in a tabulated format to be con‐
1030 sumed by other applications. The csv output format is a comma-separated
1031 output that can be imported into e.g. spreadsheets, while org_table
1032 outputs a tabulated output in the table format supported by Emacs org
1033 mode. The data is output in text format to standard output, or written
1034 to a file if invoked with the -o parameter.
1035 Statistics output (-f stats)
1037 This output format outputs various statistics about the test data, such
1038 as total bandwidth consumed, and various statistical measures
1039 (min/max/mean/median/std dev/variance) for each data source specified
1040 in the relevant test (this can include some data sources not includes
1041 on plots). The data is output in text format to standard output, or
1042 written to a file if invoked with the -o parameter.
1043 Metadata output (-f metadata)
1045 This output format outputs the test metadata as pretty-printed JSON
1046 (also suitable for human consumption). It is output as a list of ob‐
1047 jects, where each object corresponds to the metadata of one test.
1048 Mostly useful for inspecting metadata of stored data files.
1049
1051 Running Tests With The D-ITG Tool
1052 This version of flent has experimental support for running and parsing
1053 the output of the D-ITG test tool (see
1054 http://traffic.comics.unina.it/software/ITG/). Flent supports parsing
1055 the one-way delay as measured by D-ITG. However, in order to do so, the
1056 data needs to be collected at the receiver end, statistics extracted,
1057 and the result passed back to flent on the sending side.
1058 To perform this function, flent supports a control server which will
1060 listen to XML-RPC requests, spawn an appropriate ITGRecv instance and,
1061 after the test is done, parse its output and make it available for
1062 flent to retrieve. This control server is available as a Python file
1063 that by default is installed in /usr/share/doc/flent/misc. It currently
1064 requires a patched version of D-ITG v2.8.1. The patch is also included
1065 in the same directory.
1066 Note that the D-ITG server is finicky and not designed with security in
1068 mind. For this reason, the control server includes HMAC authentication
1069 to only allow authenticated clients to run a test against the server;
1070 however there is currently no support for enforcement of this in e.g.
1071 firewall rules. Please bear this in mind when running a publicly reach‐
1072 able ITGRecv instance (with or without the control server). Another se‐
1073 curity issue with the control server is that the Python XML-RPC library
1074 by default is vulnerable to XML entity expansion attacks. For this
1075 reason, it is highly recommended to install the defusedxml library
1076 (available at https://pypi.python.org/pypi/defusedxml/) on the host
1077 running the control server. The server will try to find the library on
1078 startup and refuse to run if it is not available, unless explicitly
1079 told otherwise.
1080 Due to the hassle of using D-ITG, it is recommended to install irtt in‐
1082 stead and use that for VoIP tests.
1083 Bugs
1085 Under some conditions (such as severe bufferbloat), the UDP RTT mea‐
1086 surements done by netperf can experience packet loss to the extent that
1087 the test aborts completely, which can cause missing data points for
1088 some measurement series. The --socket-timeout feature can alleviate
1089 this, but requires a recent SVN version of netperf to work. Flent tries
1090 to detect if netperf supports this option and enables it for the UDP
1091 measurements if it does. Using irtt for UDP measurements is a way to
1092 alleviate this; Flent will automatically detect the availability of
1093 irtt and use it if available.
1094 Probably many other bugs. Please report any found to
1096 https://github.com/tohojo/flent/issues and include the output of flent
1097 --version in the report. A debug log (as obtained with flent
1098 --log-file) is also often useful.
1099
1101 Toke Høiland-Jørgensen
1102
1104 2012-2022, Toke Høiland-Jørgensen and contributors. Source code is
1105 GPLv3. Documentation is CC-BY-SA. Neither license affects any output
1106 produced by Flent
11072.1.1 November 02, 2022 FLENT(1)