1fi_tcp(7) Libfabric v1.18.1 fi_tcp(7)
2
6 fi_tcp - Provider that runs over TCP/IP
7
9 The tcp provider is usable on all operation systems supported by lib‐
10 fabric. It runs over TCP (SOCK_STREAM) sockets, and includes the abil‐
11 ity to leverage operation specific features, such as support for zero-
12 copy and io_uring. The provider implements a custom protocol over
13 TCP/IP needed to support the libfabric communication APIs.
14
16 The following features are supported
17 Endpoint types
19 FI_EP_MSG FI_EP_RDM
20 Endpoint capabilities
22 FI_MSG, FI_RMA, FI_TAGGED, FI_RMA_PMEM, FI_RMA_EVENT, FI_MUL‐
23 TI_RECV, FI_DIRECTED_RECV
24 Shared Rx Context
26 The tcp provider supports shared receive context
27
29 The tcp provider may be configured using several environment variables.
30 A subset of supported variables is defined below. For a full list, use
31 the fi_info utility application. For example, `fi_info -g tcp' will
32 show all environment variables defined for the tcp provider.
33 FI_TCP_IFACE
35 A specific network interface can be requested with this variable
36 FI_TCP_PORT_LOW_RANGE/FI_TCP_PORT_HIGH_RANGE
38 These variables are used to set the range of ports to be used by
39 the tcp provider for its passive endpoint creation. This is
40 useful where only a range of ports are allowed by firewall for
41 tcp connections.
42 FI_TCP_TX_SIZE
44 Transmit context size. This is the number of transmit requests
45 that an application may post to the provider before receiving
46 -FI_EAGAIN. Default: 256 for msg endpoints, 64k for rdm.
47 FI_TCP_RX_SIZE
49 Receive context size. This is the number of receive buffers
50 that the application may post to the provider. Default: 256 for
51 msg endpoints, 64k for rdm.
52 FI_TCP_MAX_INJECT
54 Maximum size of inject messages and the maximum size of an unex‐
55 pected message that may be buffered at the receiver. Default
56 128 bytes.
57 FI_TCP_STAGING_SBUF_SIZE
59 Size of buffer used to coalesce iovec’s or send requests before
60 posting to the kernel. The staging buffer is used when the
61 socket is busy and cannot accept new data. In that case, the
62 data can be queued in the staging buffer until the socket re‐
63 sumes sending. This optimizes transfering a series of back-to-
64 back small messages to the same target. Default: 9000 bytes.
65 Set to 0 to disable.
66 FI_TCP_PREFETCH_RBUF_SIZE
68 Size of the buffer used to prefetch received data from the ker‐
69 nel. When starting to receive a new message, the provider will
70 request that the kernel fill the prefetch buffer and process re‐
71 ceived data from there. This reduces the number of kernel calls
72 needed to receive a series of small messages. Default: 9000
73 bytes. Set to 0 to disable.
74 FI_TCP_ZEROCOPY_SIZE
76 Lower threshold where zero copy transfers will be used, if sup‐
77 ported by the platform, set to -1 to disable. Default: dis‐
78 abled.
79 FI_TCP_TRACE_MSG
81 If enabled, will log transport message information on all sent
82 and received messages. Must be paired with FI_LOG_LEVEL=trace
83 to print the message details.
84 FI_TCP_IO_URING
86 Uses io_uring for socket operations if available, rather than
87 going through the standard socket APIs (i.e. connect, accept,
88 send, recv). Default: disabled.
89
91 The tcp provider supports both msg and rdm endpoints directly. Support
92 for rdm endpoints is available starting at libfabric version v1.18.0,
93 and comes from the merge back of the net provider found in libfabric
94 versions v1.16 and v1.17. For compatibility with older libfabric ver‐
95 sions, the tcp provider may be paired with the ofi_rxm provider as an
96 alternative solution for rdm endpoint support. It is recommended that
97 applications that do not need wire compatibility with older versions of
98 libfabric use the rdm endpoint support directly from the tcp provider.
99 This will provide the best performance.
100
102 fabric(7), fi_provider(7), fi_getinfo(3)
103
105 OpenFabrics.
106Libfabric Programmer’s Manual 2023-03-11 fi_tcp(7)