1fi_shm(7) Libfabric v1.14.0 fi_shm(7)
2
6 fi_shm - The SHM Fabric Provider
7
9 The SHM provider is a complete provider that can be used on Linux sys‐
10 tems supporting shared memory and process_vm_readv/process_vm_writev
11 calls. The provider is intended to provide high-performance communica‐
12 tion between processes on the same system.
13
15 This release contains an initial implementation of the SHM provider
16 that offers the following support:
17 Endpoint types
19 The provider supports only endpoint type FI_EP_RDM.
20 Endpoint capabilities
22 Endpoints cna support any combinations of the following data
23 transfer capabilities: FI_MSG, FI_TAGGED, FI_RMA, amd FI_ATOM‐
24 ICS. These capabilities can be further defined by FI_SEND,
25 FI_RECV, FI_READ, FI_WRITE, FI_REMOTE_READ, and FI_REMOTE_WRITE
26 to limit the direction of operations.
27 Modes The provider does not require the use of any mode bits.
29 Progress
31 The SHM provider supports FI_PROGRESS_MANUAL. Receive side data
32 buffers are not modified outside of completion processing rou‐
33 tines. The provider processes messages using three different
34 methods, based on the size of the message. For messages smaller
35 than 4096 bytes, tx completions are generated immediately after
36 the send. For larger messages, tx completions are not generated
37 until the receiving side has processed the message.
38 Address Format
40 The SHM provider uses the address format FI_ADDR_STR, which fol‐
41 lows the general format pattern “[prefix]://[addr]”. The appli‐
42 cation can provide addresses through the node or hints parame‐
43 ter. As long as the address is in a valid FI_ADDR_STR format
44 (contains “://”), the address will be used as is. If the appli‐
45 cation input is incorrectly formatted or no input was provided,
46 the SHM provider will resolve it according to the following SHM
47 provider standards:
48 (flags & FI_SOURCE) ? src_addr : dest_addr = - if (node && service) :
50 “fi_ns://node:service” - if (service) : “fi_ns://service” - if (node &&
51 !service) : “fi_shm://node” - if (!node && !service) : “fi_shm://PID”
52 !(flags & FI_SOURCE) - src_addr = “fi_shm://PID”
54 In other words, if the application provides a source and/or destination
56 address in an acceptable FI_ADDR_STR format (contains “://”), the call
57 to util_getinfo will successfully fill in src_addr and dest_addr with
58 the provided input. If the input is not in an ADDR_STR format, the
59 shared memory provider will then create a proper FI_ADDR_STR address
60 with either the “fi_ns://” (node/service format) or “fi_shm://” (shm
61 format) prefixes signaling whether the addr is a “unique” address and
62 does or does not need an extra endpoint name identifier appended in or‐
63 der to make it unique. For the shared memory provider, we assume that
64 the service (with or without a node) is enough to make it unique, but a
65 node alone is not sufficient. If only a node is provided, the
66 “fi_shm://” prefix is used to signify that it is not a unique address.
67 If no node or service are provided (and in the case of setting the src
68 address without FI_SOURCE and no hints), the process ID will be used as
69 a default address. On endpoint creation, if the src_addr has the
70 “fi_shm://” prefix, the provider will append “:[uid]:[ep_idx]” as a
71 unique endpoint name (essentially, in place of a service). In the case
72 of the “fi_ns://” prefix (or any other prefix if one was provided by
73 the application), no supplemental information is required to make it
74 unique and it will remain with only the application-defined address.
75 Note that the actual endpoint name will not include the FI_ADDR_STR
76 "*://" prefix since it cannot be included in any shared memory region
77 names. The provider will strip off the prefix before setting the end‐
78 point name. As a result, the addresses “fi_prefix1://my_node:my_ser‐
79 vice” and “fi_prefix2://my_node:my_service” would result in endpoints
80 and regions of the same name. The application can also override the
81 endpoint name after creating an endpoint using setname() without any
82 address format restrictions.
83 Msg flags The provider currently only supports the FI_REMOTE_CQ_DATA
85 msg flag.
86 MR registration mode The provider implements FI_MR_VIRT_ADDR memory
88 mode.
89 Atomic operations The provider supports all combinations of datatype
91 and operations as long as the message is less than 4096 bytes (or 2048
92 for compare operations).
93
95 The SHM provider has hard-coded maximums for supported queue sizes and
96 data transfers. These values are reflected in the related fabric at‐
97 tribute structures
98 EPs must be bound to both RX and TX CQs.
100 No support for counters.
102
104 The shm provider checks for the following environment variables:
105 FI_SHM_SAR_THRESHOLD
107 Maximum message size to use segmentation protocol before switch‐
108 ing to mmap (only valid when CMA is not available). Default:
109 SIZE_MAX (18446744073709551615)
110 FI_SHM_TX_SIZE
112 Maximum number of outstanding tx operations. Default 1024
113 FI_SHM_RX_SIZE
115 Maximum number of outstanding rx operations. Default 1024
116 FI_SHM_DISABLE_CMA
118 Manually disables CMA. Default false
119
121 fabric(7), fi_provider(7), fi_getinfo(3)
122
124 OpenFabrics.
125Libfabric Programmer’s Manual 2021-04-20 fi_shm(7)