1ibacm(1) ibacm ibacm(1)
2
6 ibacm - address and route resolution services for InfiniBand.
7
9 ibacm [-D] [-P] [-A addr_file] [-O option_file]
10
12 The IB ACM implements and provides a framework for name, address, and
13 route (path) resolution services over InfiniBand. It is intended to
14 address connection setup scalability issues running MPI applications on
15 large clusters. The IB ACM provides information needed to establish a
16 connection, but does not implement the CM protocol.
17 A primary user of the ibacm service is the librdmacm library. This
19 enables applications to make use of the ibacm service without code
20 changes or needing to be aware that the service is in use. librdmacm
21 versions 1.0.12 - 1.0.15 can invoke IB ACM services when built using
22 the --with-ib_acm option. Version 1.0.16 and newer of librdmacm will
23 automatically use the IB ACM if it is installed. The IB ACM services
24 tie in under the rdma_resolve_addr, rdma_resolve_route, and rdma_getad‐
25 drinfo routines. For maximum benefit, the rdma_getaddrinfo routine
26 should be used, however existing applications should still see signifi‐
27 cant connection scaling benefits using the calls available in librdmacm
28 1.0.11 and previous releases.
29 The IB ACM is focused on being scalable and efficient. The current
31 implementation limits network traffic, SA interactions, and centralized
32 services. ACM supports multiple resolution protocols in order to han‐
33 dle different fabric topologies.
34 The IB ACM package is comprised of two components: the ibacm service
36 and a test/configuration utility - ib_acme. Both are userspace compo‐
37 nents and are available for Linux and Windows. Additional details are
38 given below.
39
41 -D run in daemon mode (default)
42 -P run as standard process
44 -A addr_file
46 address configuration file
47 -O option_file
49 option configuration file
50
52 1. Prerequisites: libibverbs and libibumad must be installed. The IB
53 stack should be running with IPoIB configured. These steps assume that
54 the user has administrative privileges.
55 2. Install the IB ACM package. This installs ibacm, ib_acme, and
57 init.d scripts.
58 3. Run 'ibacm' as administrator to start the ibacm daemon.
60 4. Optionally, run 'ib_acme -d <dest_ip> -v' to verify that the ibacm
62 service is running.
63 5. Install librdmacm, using the build option --with-ib_acm if needed.
65 This build option is not needed with librdmacm 1.0.17 or newer. The
66 librdmacm will automatically use the ibacm service. On failures, the
67 librdmacm will fall back to normal resolution.
68 6. You can use ib_acme -P to gather performance statistics from the
70 local ibacm daemon to see if the service is working correctly.
71
73 ib_acme:
74 The ib_acme program serves a dual role. It acts as a utility to test
76 ibacm operation and help verify if the ibacm service and selected pro‐
77 tocol is usable for a given cluster configuration. Additionally, it
78 automatically generates ibacm configuration files to assist with or
79 eliminate manual setup.
80 ibacm configuration files:
82 The ibacm service relies on two configuration files.
84 The ibacm_addr.cfg file contains name and address mappings for each IB
86 <device, port, pkey> endpoint. Although the names in the
87 ibacm_addr.cfg file can be anything, ib_acme maps the host name and IP
88 addresses to the IB endpoints. If the address file cannot be found,
89 the ibacm service will attempt to create one using default values.
90 The ibacm_opts.cfg file provides a set of configurable options for the
92 ibacm service, such as timeout, number of retries, logging level, etc.
93 ib_acme generates the ibacm_opts.cfg file using static information. If
94 an option file cannot be found, ibacm will use default values.
95 ibacm:
97 The ibacm service is responsible for resolving names and addresses to
99 InfiniBand path information and caching such data. It should execute
100 with administrative privileges.
101 The ibacm implements a client interface over TCP sockets, which is
103 abstracted by the librdmacm library. One or more back-end protocols
104 are used by the ibacm service to satisfy user requests. Although the
105 ibacm supports standard SA path record queries on the back-end, it also
106 supports a resolution protocol based on multicast traffic. The latter
107 is not usable on all fabric topologies, specifically ones that may not
108 have reversible paths or fabrics using torus routing. Users should use
109 the ib_acme utility to verify that multicast protocol is usable before
110 running other applications.
111 Conceptually, the ibacm service implements an ARP like protocol and
113 either uses IB multicast records to construct path record data or
114 queries the SA directly, depending on the selected route protocol. By
115 default, the ibacm services uses and caches SA path record queries.
116 Specifically, all IB endpoints join a number of multicast groups. Mul‐
118 ticast groups differ based on rates, mtu, sl, etc., and are priori‐
119 tized. All participating endpoints must be able to communicate on the
120 lowest priority multicast group. The ibacm assigns one or more
121 names/addresses to each IB endpoint using the ibacm_addr.cfg file.
122 Clients provide source and destination names or addresses as input to
123 the service, and receive as output path record data.
124 The service maps a client's source name/address to a local IB endpoint.
126 If a client does not provide a source address, then the ibacm service
127 will select one based on the destination and local routing tables. If
128 the destination name/address is not cached locally, it sends a multi‐
129 cast request out on the lowest priority multicast group on the local
130 endpoint. The request carries a list of multicast groups that the
131 sender can use. The recipient of the request selects the highest pri‐
132 ority multicast group that it can use as well and returns that informa‐
133 tion directly to the sender. The request data is cached by all end‐
134 points that receive the multicast request message. The source endpoint
135 also caches the response and uses the multicast group that was selected
136 to construct or obtain path record data, which is returned to the
137 client.
138 The current implementation of the IB ACM has several additional
140 restrictions:
141 - The ibacm is limited in its handling of dynamic changes. ibacm must
143 be stopped and restarted if a cluster is reconfigured.
144 - Cached data does not timed out and is only updated if a new resolu‐
146 tion request is received from a different QPN than a cached request.
147 - Support for IPv6 has not been verified.
149 - The number of addresses that can be assigned to a single endpoint is
151 limited to 4.
152 - The number of multicast groups that an endpoint can support is lim‐
154 ited to 2.
155 The ibacm contains several internal caches. These include caches for
157 GID and LID destination addresses. These caches can be optionally pre‐
158 loaded. ibacm supports the OpenSM dump_pr plugin "full" PathRecord for‐
159 mat which is used to preload these caches. The file format is speci‐
160 fied in the ibacm_opts.cfg file via the route_preload setting which
161 should be set to full_opensm_v1 for this file format. Default format
162 is none which does not preload these caches. See dump_pr.notes.txt in
163 dump_pr for more information on the full_opensm_v1 file format and how
164 to configure OpenSM to generate this file.
165 Additionally, the name, IPv4, and IPv6 caches can be be preloaded by
167 using the addr_preload option. The default is none which does not
168 preload these caches. To preload these caches, set this option to
169 acm_hosts and configure the addr_data_file appropriately.
170
172 ibacm(7), ib_acme(1), rdma_cm(7)
173ibacm 2013-07-18 ibacm(1)