1xdp-bench(8) A simple XDP benchmarking tool xdp-bench(8)
2
6 XDP-bench - a simple XDP benchmarking tool
7
9 XDP-bench is a benchmarking utility for exercising the different opera‐
10 tion modes of XDP. It is intended to be a simple program demonstrating
11 the various operating modes; these include dropping packets, hairpin
12 forwarding (using the XDP_TX return code), and redirection using the
13 various in-kernel packet redirection facilities.
14 The drop and TX modes support various options to control whether packet
17 data is touched (read or written) before being dropped or transmitted.
18 The redirection modes support using the simple ifindex-based bpf_redi‐
19 rect helper, the bpf_redirect_map helper using a cpumap as its target,
20 bpf_redirect_map using a devmap as its target, and the devmap's broad‐
21 cast mode which allows redirecting to multiple devices.
22 There is more information on the meaning of the output in both default
25 (terse) and extended output mode, in the Output Format Description sec‐
26 tion below.
27 Running xdp-bench
30 The syntax for running xdp-bench is:
31 Usage: xdp-bench COMMAND [options]
33 COMMAND can be one of:
35 drop - Drop all packets on an interface
36 pass - Pass all packets to the network stack
37 tx - Transmit packets back out on an interface (hairpin forwarding)
38 redirect - XDP redirect using the bpf_redirect() helper
39 redirect-cpu - XDP CPU redirect using BPF_MAP_TYPE_CPUMAP
40 redirect-map - XDP redirect using BPF_MAP_TYPE_DEVMAP
41 redirect-multi - XDP multi-redirect using BPF_MAP_TYPE_DEVMAP and the BPF_F_BROADCAST flag
42 Each command, and its options are explained below. Or use xdp-bench
45 COMMAND --help to see the options for each command.
46
49 In this mode, xdp-bench installs an XDP program on an interface that
50 simply drops all packets. There are options to control what to do with
51 the packet before dropping it (touch the packet data or not), as well
52 as which statistics to gather. This is a basic benchmark for the base‐
53 line (best-case) performance of XDP on an interface.
54 The syntax for the drop command is:
57 xdp-bench drop [options] <ifname>
60 Where <ifname> is the name of the interface the XDP program should be
63 installed on.
64 The supported options are:
67 -p, --packet-operation <ACTION>
70 Specify which operation should be taken on the packet before dropping
71 it. The following actions are available:
72 no-touch - Drop the packet without touching the packet data
74 touch - Read a field in the packet header before dropping
75 parse-ip - Parse the IP header field before dropping
76 swap-macs - Swap the source and destination MAC addresses before dropping
77 Whether to touch the packet before dropping it can have a significant
80 performance impact as this requires bringing packet data into the CPU
81 cache (and flushing it back out if writing).
82 The default for this option is no-touch.
85 -l, --load-mode <MODE>
88 Specify which mechanism xdp-bench should use to load the packet data
89 when parsing the IP header (used with -p parse-ip). The following modes
90 are available:
91 dpa - Use traditional Direct Packet Access from the XDP program
93 load-bytes - Use the xdp_load_bytes() helper function to load the data
94 This can be used to benchmark the various packet access modes supported
97 by the kernel.
98 The default for this option is dpa.
101 -r, --rxq-stats
104 If set, the XDP program will also gather statistics on which receive
105 queue index each packet was received on. This is displayed in the ex‐
106 tended output mode along with per-CPU data (which, depending on the
107 hardware configuration may or may not be equivalent).
108 -i, --interval <SECONDS>
111 Set the polling interval for collecting all statistics and displaying
112 them to the output. The unit of interval is in seconds.
113 -e, --extended
116 Start xdp-bench in "extended" output mode. If not set, xdp-bench will
117 start in "terse" mode. The output mode can be switched by hitting
118 C-$ while the program is running. See also the Output Format Descrip‐
119 tion section below.
120 -m, --mode
123 Selects the XDP program mode (native or skb). Note that native XDP mode
124 is the default, and loading the redirect program in skb manner is nei‐
125 ther performant, nor recommended. However, this option is useful if the
126 interface driver lacks native XDP support, or when simply testing the
127 tool.
128 -v, --verbose
131 Enable verbose logging. Supply twice to enable verbose logging from the
132 underlying libxdp and libbpf libraries.
133 --version
136 Show the application version and exit.
137 -h, --help
140 Display a summary of the available options
141
144 In this mode, xdp-bench installs an XDP program on an interface that
145 passes all packets to the network stack after processing them (return‐
146 ing XDP_PASS). There are options to control what to do with the packet
147 before passing it (touch the packet data or not), as well as which sta‐
148 tistics to gather. This is a basic benchmark for the overhead of in‐
149 stalling an XDP program on an interface while still running the regular
150 network stack.
151 The syntax for the pass command is:
154 xdp-bench pass [options] <ifname>
157 Where <ifname> is the name of the interface the XDP program should be
160 installed on.
161 The supported options are:
164 -p, --packet-operation <ACTION>
167 Specify which operation should be taken on the packet before passing
168 it. The following actions are available:
169 no-touch - Pass the packet without touching the packet data
171 touch - Read a field in the packet header before passing
172 parse-ip - Parse the IP header field before passing
173 swap-macs - Swap the source and destination MAC addresses before passing
174 The default for this option is no-touch.
177 -l, --load-mode <MODE>
180 Specify which mechanism xdp-bench should use to load the packet data
181 when parsing the IP header (used with -p parse-ip). The following modes
182 are available:
183 dpa - Use traditional Direct Packet Access from the XDP program
185 load-bytes - Use the xdp_load_bytes() helper function to load the data
186 This can be used to benchmark the various packet access modes supported
189 by the kernel.
190 The default for this option is dpa.
193 -r, --rxq-stats
196 If set, the XDP program will also gather statistics on which receive
197 queue index each packet was received on. This is displayed in the ex‐
198 tended output mode along with per-CPU data (which, depending on the
199 hardware configuration may or may not be equivalent).
200 -i, --interval <SECONDS>
203 Set the polling interval for collecting all statistics and displaying
204 them to the output. The unit of interval is in seconds.
205 -e, --extended
208 Start xdp-bench in "extended" output mode. If not set, xdp-bench will
209 start in "terse" mode. The output mode can be switched by hitting
210 C-$ while the program is running. See also the Output Format Descrip‐
211 tion section below.
212 -m, --mode
215 Selects the XDP program mode (native or skb). Note that native XDP mode
216 is the default, and loading the redirect program in skb manner is nei‐
217 ther performant, nor recommended. However, this option is useful if the
218 interface driver lacks native XDP support, or when simply testing the
219 tool.
220 -v, --verbose
223 Enable verbose logging. Supply twice to enable verbose logging from the
224 underlying libxdp and libbpf libraries.
225 --version
228 Show the application version and exit.
229 -h, --help
232 Display a summary of the available options
233
236 In this mode, xdp-bench installs an XDP program on an interface that
237 performs so-called "hairpin forwarding", which means each packet is
238 transmitted back out the same interface (using the XDP_TX return
239 code).. There are options to control what to do with the packet before
240 transmitting it (touch the packet data or not), as well as which sta‐
241 tistics to gather.
242 The syntax for the tx command is:
245 xdp-bench tx [options] <ifname>
248 Where <ifname> is the name of the interface the XDP program should be
251 installed on.
252 The supported options are:
255 -p, --packet-operation <ACTION>
258 Specify which operation should be taken on the packet before transmit‐
259 ting it. The following actions are available:
260 no-touch - Transmit the packet without touching the packet data
262 touch - Read a field in the packet header before transmitting
263 parse-ip - Parse the IP header field before transmitting
264 swap-macs - Swap the source and destination MAC addresses before transmitting
265 To allow the packet to be successfully transmitted back to the sender,
268 the MAC addresses have to be swapped, so that the source MAC matches
269 the network device. However, there is a performance overhead in doing
270 swapping, so this option allows this function to be turned off.
271 The default for this option is swap-macs.
274 -l, --load-mode <MODE>
277 Specify which mechanism xdp-bench should use to load the packet data
278 when parsing the IP header (used with -p parse-ip). The following modes
279 are available:
280 dpa - Use traditional Direct Packet Access from the XDP program
282 load-bytes - Use the xdp_load_bytes() helper function to load the data
283 This can be used to benchmark the various packet access modes supported
286 by the kernel.
287 The default for this option is dpa.
290 -r, --rxq-stats
293 If set, the XDP program will also gather statistics on which receive
294 queue index each packet was received on. This is displayed in the ex‐
295 tended output mode along with per-CPU data (which, depending on the
296 hardware configuration may or may not be equivalent).
297 -i, --interval <SECONDS>
300 Set the polling interval for collecting all statistics and displaying
301 them to the output. The unit of interval is in seconds.
302 -e, --extended
305 Start xdp-bench in "extended" output mode. If not set, xdp-bench will
306 start in "terse" mode. The output mode can be switched by hitting
307 C-$ while the program is running. See also the Output Format Descrip‐
308 tion section below.
309 -m, --mode
312 Selects the XDP program mode (native or skb). Note that native XDP mode
313 is the default, and loading the redirect program in skb manner is nei‐
314 ther performant, nor recommended. However, this option is useful if the
315 interface driver lacks native XDP support, or when simply testing the
316 tool.
317 -v, --verbose
320 Enable verbose logging. Supply twice to enable verbose logging from the
321 underlying libxdp and libbpf libraries.
322 --version
325 Show the application version and exit.
326 -h, --help
329 Display a summary of the available options
330
333 In this mode, xdp-bench sets up packet redirection between the two in‐
334 terfaces supplied on the command line using the bpf_redirect BPF helper
335 triggered on packet reception on the ingress interface.
336 The syntax for the redirect command is:
339 xdp-bench redirect [options] <ifname_in> <ifname_out>
342 Where <ifname_in> is the name of the input interface from where packets
345 will be redirect to the output interface <ifname_out>.
346 The supported options are:
349 -i, --interval <SECONDS>
352 Set the polling interval for collecting all statistics and displaying
353 them to the output. The unit of interval is in seconds.
354 -s, --stats
357 Enable statistics for successful redirection. This option comes with a
358 per packet tracing overhead, for recording all successful redirections.
359 -e, --extended
362 Start xdp-bench in "extended" output mode. If not set, xdp-bench will
363 start in "terse" mode. The output mode can be switched by hitting
364 C-$ while the program is running. See also the Output Format Descrip‐
365 tion section below.
366 -m, --mode
369 Selects the XDP program mode (native or skb). Note that native XDP mode
370 is the default, and loading the redirect program in skb manner is nei‐
371 ther performant, nor recommended. However, this option is useful if the
372 interface driver lacks native XDP support, or when simply testing the
373 tool.
374 -v, --verbose
377 Enable verbose logging. Supply twice to enable verbose logging from the
378 underlying libxdp and libbpf libraries.
379 --version
382 Show the application version and exit.
383 -h, --help
386 Display a summary of the available options
387
390 In this mode, xdp-bench sets up packet redirection using the bpf_redi‐
391 rect_map BPF helper triggered on packet reception on the ingress inter‐
392 face, using a cpumap as its target. Hence, this tool can be used to re‐
393 direct packets on an interface from one CPU to another. In addition to
394 this, the tool then supports redirecting the packet to another output
395 device when it is processed on the target CPU.
396 The syntax for the redirect-cpu command is:
399 xdp-bench redirect-cpu [options] <ifname> -c 0 ... -c N
402 Where <ifname> is the name of the input interface from where packets
405 will be redirect to the target CPU list specified using -c.
406 The supported options are:
409 -c, --cpu <CPU>
412 Specify a possible target CPU index. This option must be passed at
413 least once, and can be passed multiple times to specify a list of CPUs.
414 Which CPU is chosen for a given packet depends on the value of the
415 --program-mode option, described below.
416 -p, --program-mode <MODE>
419 Specify a program that embeds a predefined policy deciding how packets
420 are redirected to different CPUs. The following options are available:
421 no-touch - Redirect without touching packet data
423 touch - Read packet data before redirecting
424 round-robin - Cycle between target CPUs in a round-robin fashion (for each packet)
425 l4-proto - Choose the target CPU based on the layer-4 protocol of packet
426 l4-filter - Like l4-proto, but drop UDP packets with destination port 9 (used by pktgen)
427 l4-hash - Use source and destination IP hashing to pick target CPU
428 The no-touch and touch modes always redirect packets to the same CPU
431 (the first value supplied to --cpu). The round-robin and l4-hash modes
432 distribute packets between all the CPUs supplied as --cpu arguments,
433 while l4-proto and l4-filter send TCP and unrecognised packets to CPU
434 index 0, UDP packets to CPU index 1 and ICMP packets to CPU index 2
435 (where the index refers to the order the actual CPUs are given on the
436 command line).
437 The default for this option is l4-hash.
440 -r --remote-action <ACTION>
443 If this option is set, a separate program is installed into the cpumap,
444 which will be invoked on the remote CPU after the packet is processed
445 there. The action can be either drop or pass which will drop the packet
446 or pass it to the regular networking stack, respectively. Or it can be
447 redirect, which will cause the packet to be redirected to another in‐
448 terface and transmitted out that interface on the remote CPU. If this
449 option is set to redirect the target device must be specified using
450 --redirect-device.
451 The default for this option is disabled.
454 -r, --redirect-device <IFNAME>
457 Specify the device to redirect the packet to when it is received on the
458 target CPU. Note that this option can only be specified with --re‐
459 mote-action redirect.
460 -q, --qsize <PACKETS>
463 Set the queue size for the per-CPU cpumap ring buffer used for redi‐
464 recting packets from multiple CPUs to one CPU. The default value is
465 2048 packets.
466 -x, --stress-mode
469 Stress the cpumap implementation by deallocating and reallocating the
470 cpumap ring buffer on each polling interval.
471 -i, --interval <SECONDS>
474 Set the polling interval for collecting all statistics and displaying
475 them to the output. The unit of interval is in seconds.
476 -s, --stats
479 Enable statistics for successful redirection. This option comes with a
480 per packet tracing overhead, for recording all successful redirections.
481 -e, --extended
484 Start xdp-bench in "extended" output mode. If not set, xdp-bench will
485 start in "terse" mode. The output mode can be switched by hitting
486 C-$ while the program is running. See also the Output Format Descrip‐
487 tion section below.
488 -m, --mode
491 Selects the XDP program mode (native or skb). Note that native XDP mode
492 is the default, and loading the redirect program in skb manner is nei‐
493 ther performant, nor recommended. However, this option is useful if the
494 interface driver lacks native XDP support, or when simply testing the
495 tool.
496 -v, --verbose
499 Enable verbose logging. Supply twice to enable verbose logging from the
500 underlying libxdp and libbpf libraries.
501 --version
504 Show the application version and exit.
505 -h, --help
508 Display a summary of the available options
509
512 In this mode, xdp-bench sets up packet redirection between two inter‐
513 faces supplied on the command line using the bpf_redirect_map() BPF
514 helper triggered on packet reception on the ingress interface, using a
515 devmap as its target.
516 The syntax for the redirect-map command is:
519 xdp-bench redirect-map [options] <ifname_in> <ifname_out>
522 Where <ifname_in> is the name of the input interface from where packets
525 will be redirect to the output interface <ifname_out>.
526 The supported options are:
529 -X, --load-egress
532 Load a program in the devmap entry used for redirection, so that it is
533 invoked after the packet is redirected to the target device, before it
534 is transmitted out of the output interface. The remote program will up‐
535 date the packet data so its source MAC address matches the one of the
536 destination interface.
537 -i, --interval <SECONDS>
540 Set the polling interval for collecting all statistics and displaying
541 them to the output. The unit of interval is in seconds.
542 -s, --stats
545 Enable statistics for successful redirection. This option comes with a
546 per packet tracing overhead, for recording all successful redirections.
547 -e, --extended
550 Start xdp-bench in "extended" output mode. If not set, xdp-bench will
551 start in "terse" mode. The output mode can be switched by hitting
552 C-$ while the program is running. See also the Output Format Descrip‐
553 tion section below.
554 -m, --mode
557 Selects the XDP program mode (native or skb). Note that native XDP mode
558 is the default, and loading the redirect program in skb manner is nei‐
559 ther performant, nor recommended. However, this option is useful if the
560 interface driver lacks native XDP support, or when simply testing the
561 tool.
562 -v, --verbose
565 Enable verbose logging. Supply twice to enable verbose logging from the
566 underlying libxdp and libbpf libraries.
567 --version
570 Show the application version and exit.
571 -h, --help
574 Display a summary of the available options
575
578 In this mode, xdp-bench sets up one-to-many packet redirection between
579 interfaces supplied on the command line, using the bpf_redirect_map BPF
580 helper triggered on packet reception on the ingress interface, using a
581 devmap as its target. The packet is broadcast to all output interfaces
582 specified on the command line, using devmap's packet broadcast feature.
583 The syntax for the redirect-multi command is:
586 xdp-bench redirect-multi [options] <ifname_in> <ifname_out1> ... <if‐
589 name_outN>
590 Where <ifname_in> is the name of the input interface from where packets
593 will be redirect to one or many output interface(s).
594 The supported options are:
597 -X, --load-egress
600 Load a program in the devmap entry used for redirection, so that it is
601 invoked after the packet is redirected to the target device, before it
602 is transmitted out of the output interface. The remote program will up‐
603 date the packet data so its source MAC address matches the one of the
604 destination interface.
605 -i, --interval <SECONDS>
608 Set the polling interval for collecting all statistics and displaying
609 them to the output. The unit of interval is in seconds.
610 -s, --stats
613 Enable statistics for successful redirection. This option comes with a
614 per packet tracing overhead, for recording all successful redirections.
615 -e, --extended
618 Start xdp-bench in "extended" output mode. If not set, xdp-bench will
619 start in "terse" mode. The output mode can be switched by hitting
620 C-$ while the program is running. See also the Output Format Descrip‐
621 tion section below.
622 -m, --mode
625 Selects the XDP program mode (native or skb). Note that native XDP mode
626 is the default, and loading the redirect program in skb manner is nei‐
627 ther performant, nor recommended. However, this option is useful if the
628 interface driver lacks native XDP support, or when simply testing the
629 tool.
630 -v, --verbose
633 Enable verbose logging. Supply twice to enable verbose logging from the
634 underlying libxdp and libbpf libraries.
635 --version
638 Show the application version and exit.
639 -h, --help
642 Display a summary of the available options
643
647 By default, redirect success statistics are disabled, use --stats to
648 enable. The terse output mode is default, extended output mode can be
649 activated using the --extended command line option.
650 SIGQUIT (Ctrl + \) can be used to switch the mode dynamically at run‐
653 time.
654 Terse mode displays at most the following fields:
657 rx/s Number of packets received per second
658 redir/s Number of packets successfully redirected per second
659 err,drop/s Aggregated count of errors per second (including dropped packets when not using the drop command)
660 xmit/s Number of packets transmitted on the output device per second
661 Extended output mode displays at most the following fields:
664 FIELD DESCRIPTION
665 receive Displays the number of packets received and errors encountered
666 Whenever an error or packet drop occurs, details of per CPU error
668 and drop statistics will be expanded inline in terse mode.
669 pkt/s - Packets received per second
670 drop/s - Packets dropped per second
671 error/s - Errors encountered per second
672 redirect - Displays the number of packets successfully redirected
673 Errors encountered are expanded under redirect_err field
674 Note that passing -s to enable it has a per packet overhead
675 redir/s - Packets redirected successfully per second
676 redirect_err Displays the number of packets that failed redirection
679 The errno is expanded under this field with per CPU count
681 The recognized errors are:
682 EINVAL: Invalid redirection
683 ENETDOWN: Device being redirected to is down
684 EMSGSIZE: Packet length too large for device
685 EOPNOTSUPP: Operation not supported
686 ENOSPC: No space in ptr_ring of cpumap kthread
687 error/s - Packets that failed redirection per second
689 enqueue to cpu N Displays the number of packets enqueued to bulk queue of CPU N
692 Expands to cpu:FROM->N to display enqueue stats for each CPU enqueuing to CPU N
693 Received packets can be associated with the CPU redirect program is enqueuing
694 packets to.
695 pkt/s - Packets enqueued per second from other CPU to CPU N
696 drop/s - Packets dropped when trying to enqueue to CPU N
697 bulk-avg - Average number of packets processed for each event
698 kthread Displays the number of packets processed in CPUMAP kthread for each CPU
701 Packets consumed from ptr_ring in kthread, and its xdp_stats (after calling
702 CPUMAP bpf prog) are expanded below this. xdp_stats are expanded as a total and
703 then per-CPU to associate it to each CPU's pinned CPUMAP kthread.
704 pkt/s - Packets consumed per second from ptr_ring
705 drop/s - Packets dropped per second in kthread
706 sched - Number of times kthread called schedule()
707 xdp_stats (also expands to per-CPU counts)
709 pass/s - XDP_PASS count for CPUMAP program execution
710 drop/s - XDP_DROP count for CPUMAP program execution
711 redir/s - XDP_REDIRECT count for CPUMAP program execution
712 xdp_exception Displays xdp_exception tracepoint events
715 This can occur due to internal driver errors, unrecognized
717 XDP actions and due to explicit user trigger by use of XDP_ABORTED
718 Each action is expanded below this field with its count
719 hit/s - Number of times the tracepoint was hit per second
720 devmap_xmit Displays devmap_xmit tracepoint events
723 This tracepoint is invoked for successful transmissions on output
725 device but these statistics are not available for generic XDP mode,
726 hence they will be omitted from the output when using SKB mode
727 xmit/s - Number of packets that were transmitted per second
728 drop/s - Number of packets that failed transmissions per second
729 drv_err/s - Number of internal driver errors per second
730 bulk-avg - Average number of packets processed for each event
731
734 Please report any bugs on Github: https://github.com/xdp-project/xdp-
735 tools/issues
736
739 Earlier xdp-redirect tools were written by Jesper Dangaard Brouer and
740 John Fastabend. They were then rewritten to support more features by
741 Kumar Kartikeya Dwivedi, who also ported them to xdp-tools together
742 with Toke Høiland-Jørgensen. This man page was written by Kumar Kar‐
743 tikeya Dwivedi and Toke Høiland-Jørgensen.
744V1.4.1 OCTOBER 20, 2023 xdp-bench(8)