1fi_psm2(7) Libfabric v1.10.0 fi_psm2(7)
2
6 fi_psm2 - The PSM2 Fabric Provider
7
9 The psm2 provider runs over the PSM 2.x interface that is supported by
10 the Intel Omni-Path Fabric. PSM 2.x has all the PSM 1.x features plus
11 a set of new functions with enhanced capabilities. Since PSM 1.x and
12 PSM 2.x are not ABI compatible the psm2 provider only works with PSM
13 2.x and doesn't support Intel TrueScale Fabric.
14
16 The psm2 provider doesn't support all the features defined in the lib‐
17 fabric API. Here are some of the limitations:
18 Endpoint types
20 Only support non-connection based types FI_DGRAM and FI_RDM
21 Endpoint capabilities
23 Endpoints can support any combination of data transfer capabili‐
24 ties FI_TAGGED, FI_MSG, FI_ATOMICS, and FI_RMA. These capabili‐
25 ties can be further refined by FI_SEND, FI_RECV, FI_READ,
26 FI_WRITE, FI_REMOTE_READ, and FI_REMOTE_WRITE to limit the di‐
27 rection of operations.
28 FI_MULTI_RECV is supported for non-tagged message queue only.
30 Scalable endpoints are supported if the underlying PSM2 library sup‐
32 ports multiple endpoints. This condition must be satisfied both when
33 the provider is built and when the provider is used. See the Scalable
34 endpoints section for more information.
35 Other supported capabilities include FI_TRIGGER, FI_REMOTE_CQ_DATA,
37 FI_RMA_EVENT, FI_SOURCE, and FI_SOURCE_ERR. Furthermore,
38 FI_NAMED_RX_CTX is supported when scalable endpoints are enabled.
39 Modes FI_CONTEXT is required for the FI_TAGGED and FI_MSG capabili‐
41 ties. That means, any request belonging to these two categories
42 that generates a completion must pass as the operation context a
43 valid pointer to type struct fi_context, and the space refer‐
44 enced by the pointer must remain untouched until the request has
45 completed. If none of FI_TAGGED and FI_MSG is asked for, the
46 FI_CONTEXT mode is not required.
47 Progress
49 The psm2 provider requires manual progress. The application is
50 expected to call fi_cq_read or fi_cntr_read function from time
51 to time when no other libfabric function is called to ensure
52 progress is made in a timely manner. The provider does support
53 auto progress mode. However, the performance can be signifi‐
54 cantly impacted if the application purely depends on the
55 provider to make auto progress.
56 Scalable endpoints
58 Scalable endpoints support depends on the multi-EP feature of
59 the PSM2 library. If the PSM2 library supports this feature,
60 the availability is further controlled by an environment vari‐
61 able PSM2_MULTI_EP. The psm2 provider automatically sets this
62 variable to 1 if it is not set. The feature can be disabled ex‐
63 plicitly by setting PSM2_MULTI_EP to 0.
64 When creating a scalable endpoint, the exact number of contexts re‐
66 quested should be set in the "fi_info" structure passed to the fi_scal‐
67 able_ep function. This number should be set in "fi_info->ep_at‐
68 tr->tx_ctx_cnt" or "fi_info->ep_attr->rx_ctx_cnt" or both, whichever
69 greater is used. The psm2 provider allocates all requested contexts
70 upfront when the scalable endpoint is created. The same context is
71 used for both Tx and Rx.
72 For optimal performance, it is advised to avoid having multiple threads
74 accessing the same context, either directly by posting
75 send/recv/read/write request, or indirectly by polling associated com‐
76 pletion queues or counters.
77 Using the scalable endpoint as a whole in communication functions is
79 not supported. Instead, individual tx context or rx context of the
80 scalable endpoint should be used. Similarly, using the address of the
81 scalable endpoint as the source address or destination address doesn't
82 collectively address all the tx/rx contexts. It addresses only the
83 first tx/rx context, instead.
84 Shared Tx contexts
86 In order to achieve the purpose of saving PSM context by using
87 shared Tx context, the endpoints bound to the shared Tx contexts
88 need to be Tx only. The reason is that Rx capability always re‐
89 quires a PSM context, which can also be automatically used for
90 Tx. As the result, allocating a shared Tx context for Rx capa‐
91 ble endpoints actually consumes one extra context instead of
92 saving some.
93 Unsupported features
95 These features are unsupported: connection management, passive
96 endpoint, and shared receive context.
97
99 The psm2 provider checks for the following environment variables:
100 FI_PSM2_UUID
102 PSM requires that each job has a unique ID (UUID). All the pro‐
103 cesses in the same job need to use the same UUID in order to be
104 able to talk to each other. The PSM reference manual advises to
105 keep UUID unique to each job. In practice, it generally works
106 fine to reuse UUID as long as (1) no two jobs with the same UUID
107 are running at the same time; and (2) previous jobs with the
108 same UUID have exited normally. If running into "resource busy"
109 or "connection failure" issues with unknown reason, it is advis‐
110 able to manually set the UUID to a value different from the de‐
111 fault.
112 The default UUID is 00FF00FF-0000-0000-0000-00FF0F0F00FF.
114 FI_PSM2_NAME_SERVER
116 The psm2 provider has a simple built-in name server that can be
117 used to resolve an IP address or host name into a transport ad‐
118 dress needed by the fi_av_insert call. The main purpose of this
119 name server is to allow simple client-server type applications
120 (such as those in fabtests) to be written purely with libfabric,
121 without using any out-of-band communication mechanism. For such
122 applications, the server would run first to allow endpoints be
123 created and registered with the name server, and then the client
124 would call fi_getinfo with the node parameter set to the IP ad‐
125 dress or host name of the server. The resulting fi_info struc‐
126 ture would have the transport address of the endpoint created by
127 the server in the dest_addr field. Optionally the service pa‐
128 rameter can be used in addition to node. Notice that the ser‐
129 vice number is interpreted by the provider and is not a TCP/IP
130 port number.
131 The name server is on by default. It can be turned off by setting the
133 variable to 0. This may save a small amount of resource since a sepa‐
134 rate thread is created when the name server is on.
135 The provider detects OpenMPI and MPICH runs and changes the default
137 setting to off.
138 FI_PSM2_TAGGED_RMA
140 The RMA functions are implemented on top of the PSM Active Mes‐
141 sage functions. The Active Message functions have limit on the
142 size of data can be transferred in a single message. Large
143 transfers can be divided into small chunks and be pipe-lined.
144 However, the bandwidth is sub-optimal by doing this way.
145 The psm2 provider use PSM tag-matching message queue functions to
147 achieve higher bandwidth for large size RMA. It takes advantage of the
148 extra tag bits available in PSM2 to separate the RMA traffic from the
149 regular tagged message queue.
150 The option is on by default. To turn it off set the variable to 0.
152 FI_PSM2_DELAY
154 Time (seconds) to sleep before closing PSM endpoints. This is a
155 workaround for a bug in some versions of PSM library.
156 The default setting is 0.
158 FI_PSM2_TIMEOUT
160 Timeout (seconds) for gracefully closing PSM endpoints. A
161 forced closing will be issued if timeout expires.
162 The default setting is 5.
164 FI_PSM2_CONN_TIMEOUT
166 Timeout (seconds) for establishing connection between two PSM
167 endpoints.
168 The default setting is 5.
170 FI_PSM2_PROG_INTERVAL
172 When auto progress is enabled (asked via the hints to fi_get‐
173 info), a progress thread is created to make progress calls from
174 time to time. This option set the interval (microseconds) be‐
175 tween progress calls.
176 The default setting is 1 if affinity is set, or 1000 if not. See
178 FI_PSM2_PROG_AFFINITY.
179 FI_PSM2_PROG_AFFINITY
181 When set, specify the set of CPU cores to set the progress
182 thread affinity to. The format is
183 <start>[:<end>[:<stride>]][,<start>[:<end>[:<stride>]]]*, where
184 each triplet <start>:<end>:<stride> defines a block of core_ids.
185 Both <start> and <end> can be either the core_id (when >=0) or
186 core_id - num_cores (when <0).
187 By default affinity is not set.
189 FI_PSM2_INJECT_SIZE
191 Maximum message size allowed for fi_inject and fi_tinject calls.
192 This is an experimental feature to allow some applications to
193 override default inject size limitation. When the inject size
194 is larger than the default value, some inject calls might block.
195 The default setting is 64.
197 FI_PSM2_LOCK_LEVEL
199 When set, dictate the level of locking being used by the
200 provider. Level 2 means all locks are enabled. Level 1 dis‐
201 ables some locks and is suitable for runs that limit the access
202 to each PSM2 context to a single thread. Level 0 disables all
203 locks and thus is only suitable for single threaded runs.
204 To use level 0 or level 1, wait object and auto progress mode cannot be
206 used because they introduce internal threads that may break the condi‐
207 tions needed for these levels.
208 The default setting is 2.
210 FI_PSM2_LAZY_CONN
212 There are two strategies on when to establish connections be‐
213 tween the PSM2 endpoints that OFI endpoints are built on top of.
214 In eager connection mode, connections are established when ad‐
215 dresses are inserted into the address vector. In lazy connec‐
216 tion mode, connections are established when addresses are used
217 the first time in communication. Eager connection mode has
218 slightly lower critical path overhead but lazy connection mode
219 scales better.
220 This option controls how the two connection modes are used. When set
222 to 1, lazy connection mode is always used. When set to 0, eager con‐
223 nection mode is used when required conditions are all met and lazy con‐
224 nection mode is used otherwise. The conditions for eager connection
225 mode are: (1) multiple endpoint (and scalable endpoint) support is dis‐
226 abled by explicitly setting PSM2_MULTI_EP=0; and (2) the address vector
227 type is FI_AV_MAP.
228 The default setting is 0.
230 FI_PSM2_DISCONNECT
232 The provider has a mechanism to automatically send disconnection
233 notifications to all connected peers before the local endpoint
234 is closed. As the response, the peers call psm2_ep_disconnect
235 to clean up the connection state at their side. This allows the
236 same PSM2 epid be used by different dynamically started process‐
237 es (clients) to communicate with the same peer (server). This
238 mechanism, however, introduce extra overhead to the finalization
239 phase. For applications that never reuse epids within the same
240 session such overhead is unnecessary.
241 This option controls whether the automatic disconnection notification
243 mechanism should be enabled. For client-server application mentioned
244 above, the client side should set this option to 1, but the server
245 should set it to 0.
246 The default setting is 0.
248 FI_PSM2_TAG_LAYOUT
250 Select how the 96-bit PSM2 tag bits are organized. Currently
251 three choices are available: tag60 means 32-4-60 partitioning
252 for CQ data, internal protocol flags, and application tag.
253 tag64 means 4-28-64 partitioning for internal protocol flags, CQ
254 data, and application tag. auto means to choose either tag60 or
255 tag64 based on the hints passed to fi_getinfo -- tag60 is used
256 if remote CQ data support is requested explicitly, either by
257 passing non-zero value via hints->domain_attr->cq_data_size or
258 by including FI_REMOTE_CQ_DATA in hints->caps, otherwise tag64
259 is used. If tag64 is the result of automatic selection, fi_get‐
260 info also returns a second instance of the provider with tag60
261 layout.
262 The default setting is auto.
264 Notice that if the provider is compiled with macro PSMX2_TAG_LAYOUT de‐
266 fined to 1 (means tag60) or 2 (means tag64), the choice is fixed at
267 compile time and this runtime option will be disabled.
268
270 fabric(7), fi_provider(7), fi_psm(7),
271
273 OpenFabrics.
274Libfabric Programmer's Manual 2019-04-09 fi_psm2(7)