1ibacm(1)                             ibacm                            ibacm(1)
2
3
4

NAME

6       ibacm - address and route resolution services for InfiniBand.
7

SYNOPSIS

9       ibacm [-D] [-P] [-A addr_file] [-O option_file]
10

DESCRIPTION

12       The  IB  ACM implements and provides a framework for name, address, and
13       route (path) resolution services over InfiniBand.  It  is  intended  to
14       address connection setup scalability issues running MPI applications on
15       large clusters.  The IB ACM provides information needed to establish  a
16       connection, but does not implement the CM protocol.
17
18       A  primary  user  of  the ibacm service is the librdmacm library.  This
19       enables applications to make use of  the  ibacm  service  without  code
20       changes  or  needing to be aware that the service is in use.  librdmacm
21       versions 1.0.12 - 1.0.15 can invoke IB ACM services  when  built  using
22       the  --with-ib_acm  option.  Version 1.0.16 and newer of librdmacm will
23       automatically use the IB ACM if it is installed.  The IB  ACM  services
24       tie in under the rdma_resolve_addr, rdma_resolve_route, and rdma_getad‐
25       drinfo routines.  For maximum  benefit,  the  rdma_getaddrinfo  routine
26       should be used, however existing applications should still see signifi‐
27       cant connection scaling benefits using the calls available in librdmacm
28       1.0.11 and previous releases.
29
30       The IB ACM is focused on being scalable, efficient, and extensible.  It
31       implements a plugin architecture that allows a  vendor  to  supply  its
32       proprietary  provider in addition to the default provider.  The current
33       default provider  implementation  ibacmp  limits  network  traffic,  SA
34       interactions, and centralized services.  Ibacmp supports multiple reso‐
35       lution protocols in order to handle different fabric topologies.
36
37       The IB ACM package is comprised of three  components:  the  ibacm  core
38       service, the default provider ibacmp shared library, and a test/config‐
39       uration utility - ib_acme.  All three are userspace components and  are
40       available for Linux.  Additional details are given below.
41

OPTIONS

43       -D     run in daemon mode (default)
44
45       -P     run as standard process
46
47       -A addr_file
48              address configuration file
49
50       -O option_file
51              option configuration file
52
53       --systemd
54              Enable  systemd integration. This includes optional socket acti‐
55              vation of the daemon's listening socket.
56

QUICK START GUIDE

58       1. Prerequisites: libibverbs and libibumad must be installed.   The  IB
59       stack should be running with IPoIB configured.  These steps assume that
60       the user has administrative privileges.
61
62       2. Install the IB ACM package.  This installs ibacm,  ibacmp,  ib_acme,
63       and init.d scripts.
64
65       3. Run 'ibacm' as administrator to start the ibacm daemon.
66
67       4.  Optionally,  run 'ib_acme -d <dest_ip> -v' to verify that the ibacm
68       service is running.
69
70       5. Install librdmacm, using the build option --with-ib_acm  if  needed.
71       This  build  option  is not needed with librdmacm 1.0.17 or newer.  The
72       librdmacm will automatically use the ibacm service.  On  failures,  the
73       librdmacm will fall back to normal resolution.
74
75       6.  You  can  use  ib_acme -P to gather performance statistics from the
76       local ibacm daemon to see if the service is working  correctly.   Simi‐
77       larly,  the command ib_acme -e could be used to enumerate all endpoints
78       created by the local ibacm service.
79

NOTES

81       ib_acme:
82
83       The ib_acme program serves a dual role.  It acts as a utility  to  test
84       ibacm  operation and help verify if the ibacm service and selected pro‐
85       tocol is usable for a given cluster configuration.    Additionally,  it
86       automatically  generates  ibacm  configuration  files to assist with or
87       eliminate manual setup.
88
89       ibacm configuration files:
90
91       The ibacm service relies on two configuration files.
92
93       The ibacm_addr.cfg file contains name and address mappings for each  IB
94       <device,   port,   pkey>   endpoint.    Although   the   names  in  the
95       ibacm_addr.cfg file can be anything, ib_acme maps the host name to  the
96       IB  endpoints.   IP  addresses, on the other hand, are assigned dynami‐
97       cally.  If the address file cannot be found,  the  ibacm  service  will
98       attempt to create one using default values.
99
100       The  ibacm_opts.cfg file provides a set of configurable options for the
101       ibacm core service and default provider, such  as  timeout,  number  of
102       retries, logging level, etc.  ib_acme generates the ibacm_opts.cfg file
103       using static information.  If an option file  cannot  be  found,  ibacm
104       will use default values.
105
106       ibacm:
107
108       The  ibacm  service is responsible for resolving names and addresses to
109       InfiniBand path information and caching such data.  It  should  execute
110       with administrative privileges.
111
112       The  ibacm  implements  a  client  interface over TCP sockets, which is
113       abstracted by the librdmacm library.  One  or  more  providers  can  be
114       loaded  by  the  core  service, depending on the configuration.  In the
115       default provider ibacmp, one or more back-end  protocols  are  used  to
116       satisfy  user  requests.   Although  ibacmp  supports  standard SA path
117       record queries on the back-end, it also supports a resolution  protocol
118       based  on  multicast  traffic.   The latter is not usable on all fabric
119       topologies, specifically ones that may not  have  reversible  paths  or
120       fabrics  using  torus routing.  Users should use the ib_acme utility to
121       verify that multicast protocol is usable before running other  applica‐
122       tions.
123
124       Conceptually, the default provider ibacmp implements an ARP like proto‐
125       col and either uses IB multicast records to construct path record  data
126       or  queries  the SA directly, depending on the selected route protocol.
127       By default, the ibacmp provider uses and caches SA path record queries.
128
129       Specifically, all IB endpoints join a number of multicast groups.  Mul‐
130       ticast  groups  differ  based  on rates, mtu, sl, etc., and are priori‐
131       tized.  All participating endpoints must be able to communicate on  the
132       lowest  priority  multicast  group.   The  ibacmp  assigns  one or more
133       names/addresses to each IB  endpoint  using  the  ibacm_addr.cfg  file.
134       Clients  provide  source and destination names or addresses as input to
135       the service, and receive as output path record data.
136
137       The service maps a client's source name/address to a local IB endpoint.
138       If  the  destination  name/address is not cached locally in the default
139       provider, it sends a multicast request out on the lowest priority  mul‐
140       ticast group on the local endpoint.  The request carries a list of mul‐
141       ticast groups that the sender can use.  The recipient  of  the  request
142       selects  the  highest  priority multicast group that it can use as well
143       and returns that information directly to the sender.  The request  data
144       is  cached by all endpoints that receive the multicast request message.
145       The source endpoint also caches the response  and  uses  the  multicast
146       group  that was selected to construct or obtain path record data, which
147       is returned to the client.
148
149       The current implementation of the provider  ibacmp  has  several  addi‐
150       tional restrictions:
151
152       - The ibacmp is limited in its handling of dynamic changes.  ibacm must
153       be stopped and restarted if a cluster is reconfigured.
154
155       - Support for IPv6 has not been verified.
156
157       - The number of multicast groups that an endpoint can support  is  lim‐
158       ited to 2.
159
160       The  ibacmp contains several internal caches.  These include caches for
161       GID and LID destination addresses.  These caches can be optionally pre‐
162       loaded. ibacm supports the OpenSM dump_pr plugin "full" PathRecord for‐
163       mat which is used to preload these caches.  The file format  is  speci‐
164       fied  in  the  ibacm_opts.cfg  file via the route_preload setting which
165       should be set to full_opensm_v1 for this file format.   Default  format
166       is  none which does not preload these caches.  See dump_pr.notes.txt in
167       dump_pr for more information on the full_opensm_v1 file format and  how
168       to configure OpenSM to generate this file.
169
170       Additionally,  the  name,  IPv4, and IPv6 caches can be be preloaded by
171       using the addr_preload option.  The default  is  none  which  does  not
172       preload  these  caches.  To  preload  these  caches, set this option to
173       acm_hosts and configure the addr_data_file appropriately.
174

SEE ALSO

176       ibacm(7), ib_acme(1), rdma_cm(7)
177
178
179
180ibacm                             2014-06-16                          ibacm(1)
Impressum