1fi_av_set(3) Libfabric v1.17.0 fi_av_set(3)
2
6 fi_av_set - Address vector set operations
7 fi_av_set / fi_close
9 Open or close an address vector set
10 fi_av_set_union
12 Perform a set union operation on two AV sets
13 fi_av_set_intersect
15 Perform a set intersect operation on two AV sets
16 fi_av_set_diff
18 Perform a set difference operation on two AV sets
19 fi_av_set_insert
21 Add an address to an AV set
22 fi_av_set_remove
24 Remove an address from an AV set
25 fi_av_set_addr
27 Obtain a collective address for current addresses in an AV set
28
30 #include <rdma/fi_collective.h>
31 int fi_av_set(struct fid_av *av, struct fi_av_set_attr *attr,
33 struct fid_av_set **set, void * context);
34 int fi_av_set_union(struct fid_av_set *dst, const struct fid_av_set *src);
36 int fi_av_set_intersect(struct fid_av_set *dst, const struct fid_av_set *src);
38 int fi_av_set_diff(struct fid_av_set *dst, const struct fid_av_set *src);
40 int fi_av_set_insert(struct fid_av_set *set, fi_addr_t addr);
42 int fi_av_set_remove(struct fid_av_set *set, fi_addr_t addr);
44 int fi_av_set_addr(struct fid_av_set *set, fi_addr_t *coll_addr);
46 int fi_close(struct fid *av_set);
48
50 av Address vector
51 set Address vector set
53 dst Address vector set updated by set operation
55 src Address vector set providing input to a set operation
57 attr Address vector set attributes
59 context
61 User specified context associated with the address vector set
62 flags Additional flags to apply to the operation.
64 addr Destination address to insert to remove from AV set.
66 coll_addr
68 Address identifying collective group.
69
71 An address vector set (AV set) represents an ordered subset of address‐
72 es of an address vector. AV sets are used to identify the participants
73 in a collective operation. Endpoints use the fi_join_collective() op‐
74 eration to associate itself with an AV set. The join collective opera‐
75 tion provides an fi_addr that is used when communicating with a collec‐
76 tive group.
77 The creation and manipulation of an AV set is a local operation. No
79 fabric traffic is exchanged between peers. As a result, each peer is
80 responsible for creating matching AV sets as part of their collective
81 membership definition. See fi_collective(3) for a discussion of mem‐
82 bership models.
83 fi_av_set
85 The fi_av_set call creates a new AV set. The initial properties of the
86 AV set are specified through the struct fi_av_set_attr parameter. This
87 structure is defined below, and allows including a subset of addresses
88 in the AV set as part of AV set creation. Addresses may be added or
89 removed from an AV set using the AV set interfaces defined below.
90 fi_av_set_attr
92 struct fi_av_set_attr {
93 size_t count;
94 fi_addr_t start_addr;
95 fi_addr_t end_addr;
96 uint64_t stride;
97 size_t comm_key_size;
98 uint8_t *comm_key;
99 uint64_t flags;
100 };
101 count Indicates the expected the number of members that will be a part
103 of the AV set. The provider uses this to optimize resource al‐
104 locations. If count is 0, the provider will select a size based
105 on available system configuration data or underlying limita‐
106 tions.
107 start_addr / end_addr
109 The starting and ending addresses, inclusive, to include as part
110 of the AV set. The use of start and end address require that
111 the associated AV have been created as type FI_AV_TABLE. Valid
112 addresses in the AV which fall within the specified range and
113 which meet other requirements (such as stride) will be added as
114 initial members to the AV set. The start_addr and end_addr must
115 be set to FI_ADDR_NOTAVAIL if creating an empty AV set, a commu‐
116 nication key is being provided, or the AV is of type FI_AV_MAP.
117 The number of addresses between start_addr and end_addr must be less
119 than or equal to the specified count value.
120 stride The number of entries between successive addresses included in
122 the AV set. The AV set will include all addresses from
123 start_addr + stride x i, for increasing, non-negative, integer
124 values of i, up to end_addr. A stride of 1 indicates that all
125 addresses between start_addr and end_addr should be added to the
126 AV set. Stride should be set to 0 unless the start_addr and
127 end_addr fields are valid.
128 comm_key_size
130 The length of the communication key in bytes. This field should
131 be 0 if a communication key is not available.
132 comm_key
134 If supported by the fabric, this represents a key associated
135 with the AV set. The communication key is used by applications
136 that directly manage collective membership through a fabric man‐
137 agement agent or resource manager. The key is used to convey
138 that results of the membership setup to the underlying provider.
139 The use and format of a communication key is fabric provider
140 specific.
141 flags Flags may be used to configure the AV set, including restricting
143 which collective operations the AV set needs to support. See
144 the flags section for a list of flags that may be specified when
145 creating the AV set.
146 fi_av_set_union
148 The AV set union call adds all addresses in the source AV set that are
149 not in the destination AV set to the destination AV set. Where order‐
150 ing matters, the newly inserted addresses are placed at the end of the
151 AV set.
152 fi_av_set_intersect
154 The AV set intersect call remove all addresses from the destination AV
155 set that are not also members of the source AV set. The order of the
156 addresses in the destination AV set is unchanged.
157 fi_av_set_diff
159 The AV set difference call removes all address from the destination AV
160 set that are also members of the source AV set. The order of the ad‐
161 dresses in the destination AV set is unchanged.
162 fi_av_set_insert
164 The AV set insert call appends the specified address to the end of the
165 AV set.
166 fi_av_set_remove
168 The AV set remove call removes the specified address from the given AV
169 set. The order of the remaining addresses in the AV set is unchanged.
170 fi_av_set_addr
172 Returns an address that may be used to communicate with all current
173 members of an AV set. This is a local operation only that does not in‐
174 volve network communication. The returned address may be used as input
175 into fi_join_collective. Note that attempting to use the address re‐
176 turned from fi_av_set_addr (e.g. passing it to fi_join_collective)
177 while simultaneously modifying the addresses stored in an AV set re‐
178 sults in undefined behavior.
179 fi_close
181 Closes an AV set and releases all resources associated with it. Any
182 operations active at the time an AV set is closed will be aborted, with
183 the result of the collective undefined.
184
186 The following flags may be specified as part of AV set creation.
187 FI_UNIVERSE
189 When set, then the AV set will be created containing all ad‐
190 dresses stored in the corresponding AV.
191 FI_BARRIER_SET
193 If set, the AV set will be configured to support barrier opera‐
194 tions.
195 FI_BROADCAST_SET
197 If set, the AV set will be configured to support broadcast oper‐
198 ations.
199 FI_ALLTOALL_SET
201 If set, the AV set will be configured to support all to all op‐
202 erations.
203 FI_ALLREDUCE_SET
205 If set, the AV set will be configured to support all reduce op‐
206 erations.
207 FI_ALLGATHER_SET
209 If set, the AV set will be configured to support all gather op‐
210 erations.
211 FI_REDUCE_SCATTER_SET
213 If set, the AV set will be configured to support reduce scatter
214 operations.
215 FI_REDUCE_SET
217 If set, the AV set will be configured to support reduce opera‐
218 tions.
219 FI_SCATTER_SET
221 If set, the AV set will be configured to support scatter opera‐
222 tions.
223 FI_GATHER_SET
225 If set, the AV set will be configured to support gather opera‐
226 tions.
227
229 Developers who are familiar with MPI will find that AV sets are similar
230 to MPI groups, and may act as a direct mapping in some, but not all,
231 situations.
232 By default an AV set will be created to support all collective opera‐
234 tions supported by the underlying provider (see fi_query_collective).
235 Users may reduce resource requirements by specifying only those collec‐
236 tion operations needed by the AV set through the use of creation flags:
237 FI_BARRIER_SET, FI_BROADCAST_SET, etc. If no such flags are specified,
238 the AV set will be configured to support any that are supported. It is
239 an error for a user to request an unsupported collective.
240
242 Returns 0 on success. On error, a negative value corresponding to fab‐
243 ric errno is returned. Fabric errno values are defined in rdma/fi_er‐
244 rno.h.
245
247 fi_av(3), fi_collective(3)
248
250 OpenFabrics.
251Libfabric Programmer’s Manual 2022-12-11 fi_av_set(3)