1ibacm(1) ibacm ibacm(1)
2
6 ibacm - address and route resolution services for InfiniBand.
7
9 ibacm [-D] [-P] [-A addr_file] [-O option_file]
10
12 The IB ACM implements and provides a framework for name, address, and
13 route (path) resolution services over InfiniBand. It is intended to
14 address connection setup scalability issues running MPI applications on
15 large clusters. The IB ACM provides information needed to establish a
16 connection, but does not implement the CM protocol.
17 A primary user of the ibacm service is the librdmacm library. This
19 enables applications to make use of the ibacm service without code
20 changes or needing to be aware that the service is in use. librdmacm
21 versions 1.0.12 - 1.0.15 can invoke IB ACM services when built using
22 the --with-ib_acm option. Version 1.0.16 and newer of librdmacm will
23 automatically use the IB ACM if it is installed. The IB ACM services
24 tie in under the rdma_resolve_addr, rdma_resolve_route, and rdma_getad‐
25 drinfo routines. For maximum benefit, the rdma_getaddrinfo routine
26 should be used, however existing applications should still see signifi‐
27 cant connection scaling benefits using the calls available in librdmacm
28 1.0.11 and previous releases.
29 The IB ACM is focused on being scalable, efficient, and extensible. It
31 implements a plugin architecture that allows a vendor to supply its
32 proprietary provider in addition to the default provider. The current
33 default provider implementation ibacmp limits network traffic, SA
34 interactions, and centralized services. Ibacmp supports multiple reso‐
35 lution protocols in order to handle different fabric topologies.
36 The IB ACM package is comprised of three components: the ibacm core
38 service, the default provider ibacmp shared library, and a test/config‐
39 uration utility - ib_acme. All three are userspace components and are
40 available for Linux. Additional details are given below.
41
43 -D run in daemon mode (default)
44 -P run as standard process
46 -A addr_file
48 address configuration file
49 -O option_file
51 option configuration file
52 --systemd
54 Enable systemd integration. This includes optional socket acti‐
55 vation of the daemon's listening socket.
56
58 1. Prerequisites: libibverbs and libibumad must be installed. The IB
59 stack should be running with IPoIB configured. These steps assume that
60 the user has administrative privileges.
61 2. Install the IB ACM package. This installs ibacm, ibacmp, ib_acme,
63 and init.d scripts.
64 3. Run 'ibacm' as administrator to start the ibacm daemon.
66 4. Optionally, run 'ib_acme -d <dest_ip> -v' to verify that the ibacm
68 service is running.
69 5. Install librdmacm, using the build option --with-ib_acm if needed.
71 This build option is not needed with librdmacm 1.0.17 or newer. The
72 librdmacm will automatically use the ibacm service. On failures, the
73 librdmacm will fall back to normal resolution.
74 6. You can use ib_acme -P to gather performance statistics from the
76 local ibacm daemon to see if the service is working correctly. Simi‐
77 larly, the command ib_acme -e could be used to enumerate all endpoints
78 created by the local ibacm service.
79
81 ib_acme:
82 The ib_acme program serves a dual role. It acts as a utility to test
84 ibacm operation and help verify if the ibacm service and selected pro‐
85 tocol is usable for a given cluster configuration. Additionally, it
86 automatically generates ibacm configuration files to assist with or
87 eliminate manual setup.
88 ibacm configuration files:
90 The ibacm service relies on two configuration files.
92 The ibacm_addr.cfg file contains name and address mappings for each IB
94 <device, port, pkey> endpoint. Although the names in the
95 ibacm_addr.cfg file can be anything, ib_acme maps the host name to the
96 IB endpoints. IP addresses, on the other hand, are assigned dynami‐
97 cally. If the address file cannot be found, the ibacm service will
98 attempt to create one using default values.
99 The ibacm_opts.cfg file provides a set of configurable options for the
101 ibacm core service and default provider, such as timeout, number of
102 retries, logging level, etc. ib_acme generates the ibacm_opts.cfg file
103 using static information. If an option file cannot be found, ibacm
104 will use default values.
105 ibacm:
107 The ibacm service is responsible for resolving names and addresses to
109 InfiniBand path information and caching such data. It should execute
110 with administrative privileges.
111 The ibacm implements a client interface over TCP sockets, which is
113 abstracted by the librdmacm library. One or more providers can be
114 loaded by the core service, depending on the configuration. In the
115 default provider ibacmp, one or more back-end protocols are used to
116 satisfy user requests. Although ibacmp supports standard SA path
117 record queries on the back-end, it also supports a resolution protocol
118 based on multicast traffic. The latter is not usable on all fabric
119 topologies, specifically ones that may not have reversible paths or
120 fabrics using torus routing. Users should use the ib_acme utility to
121 verify that multicast protocol is usable before running other applica‐
122 tions.
123 Conceptually, the default provider ibacmp implements an ARP like proto‐
125 col and either uses IB multicast records to construct path record data
126 or queries the SA directly, depending on the selected route protocol.
127 By default, the ibacmp provider uses and caches SA path record queries.
128 Specifically, all IB endpoints join a number of multicast groups. Mul‐
130 ticast groups differ based on rates, mtu, sl, etc., and are priori‐
131 tized. All participating endpoints must be able to communicate on the
132 lowest priority multicast group. The ibacmp assigns one or more
133 names/addresses to each IB endpoint using the ibacm_addr.cfg file.
134 Clients provide source and destination names or addresses as input to
135 the service, and receive as output path record data.
136 The service maps a client's source name/address to a local IB endpoint.
138 If the destination name/address is not cached locally in the default
139 provider, it sends a multicast request out on the lowest priority mul‐
140 ticast group on the local endpoint. The request carries a list of mul‐
141 ticast groups that the sender can use. The recipient of the request
142 selects the highest priority multicast group that it can use as well
143 and returns that information directly to the sender. The request data
144 is cached by all endpoints that receive the multicast request message.
145 The source endpoint also caches the response and uses the multicast
146 group that was selected to construct or obtain path record data, which
147 is returned to the client.
148 The current implementation of the provider ibacmp has several addi‐
150 tional restrictions:
151 - The ibacmp is limited in its handling of dynamic changes. ibacm must
153 be stopped and restarted if a cluster is reconfigured.
154 - Support for IPv6 has not been verified.
156 - The number of multicast groups that an endpoint can support is lim‐
158 ited to 2.
159 The ibacmp contains several internal caches. These include caches for
161 GID and LID destination addresses. These caches can be optionally pre‐
162 loaded. ibacm supports the OpenSM dump_pr plugin "full" PathRecord for‐
163 mat which is used to preload these caches. The file format is speci‐
164 fied in the ibacm_opts.cfg file via the route_preload setting which
165 should be set to full_opensm_v1 for this file format. Default format
166 is none which does not preload these caches. See dump_pr.notes.txt in
167 dump_pr for more information on the full_opensm_v1 file format and how
168 to configure OpenSM to generate this file.
169 Additionally, the name, IPv4, and IPv6 caches can be be preloaded by
171 using the addr_preload option. The default is none which does not
172 preload these caches. To preload these caches, set this option to
173 acm_hosts and configure the addr_data_file appropriately.
174
176 ibacm(7), ib_acme(1), rdma_cm(7)
177ibacm 2014-06-16 ibacm(1)