1NM-CLOUD-SETUP(8) Automatic Network Configuratio NM-CLOUD-SETUP(8)
2
6 nm-cloud-setup - Overview of Automatic Network Configuration in Cloud
7
9 When running a virtual machine in a public cloud environment, it is
10 desirable to automatically configure the network of that VM. In simple
11 setups, the VM only has one network interface and the public cloud
12 supports automatic configuration via DHCP, DHCP6 or IPv6 autoconf.
13 However, the virtual machine might have multiple network interfaces, or
14 multiple IP addresses and IP subnets on one interface which cannot be
15 configured via DHCP. Also, the administrator may reconfigure the
16 network while the machine is running. NetworkManager's nm-cloud-setup
17 is a tool that automatically picks up such configuration in cloud
18 environments and updates the network configuration of the host.
19 Multiple cloud providers are supported. See the section called
21 “SUPPORTED CLOUD PROVIDERS”.
22
24 The goal of nm-cloud-setup is to be configuration-less and work
25 automatically. All you need is to opt-in to the desired cloud providers
26 (see the section called “ENVIRONMENT VARIABLES”) and run
27 /usr/libexec/nm-cloud-setup.
28 Usually this is done by enabling the nm-cloud-setup.service systemd
30 service and let it run periodically. For that there is both a
31 nm-cloud-setup.timer systemd timer and a NetworkManager dispatcher
32 script.
33
35 nm-cloud-setup configures the network by fetching the configuration
36 from the well-known meta data server of the cloud provider. That means,
37 it already needs the network configured to the point where it can reach
38 the meta data server. Commonly that means, that a simple connection
39 profile is activated that possibly uses DHCP to get the primary IP
40 address. NetworkManager will create such a profile for ethernet devices
41 automatically if it is not configured otherwise via "no-auto-default"
42 setting in NetworkManager.conf. One possible alternative may be to
43 create such an initial profile with nmcli device connect "$DEVICE" or
44 nmcli connection add type ethernet ....
45 By setting the user-data org.freedesktop.nm-cloud-setup.skip=yes on the
47 profile, nm-cloud-setup will skip the device.
48 nm-cloud-setup modifies the run time configuration akin to nmcli device
50 modify. With this approach, the configuration is not persisted and only
51 preserved until the device disconnects.
52 /usr/libexec/nm-cloud-setup
54 The binary /usr/libexec/nm-cloud-setup does most of the work. It
55 supports no command line arguments but can be configured via
56 environment variables. See the section called “ENVIRONMENT VARIABLES”
57 for the supported environment variables.
58 By default, all cloud providers are disabled unless you opt-in by
60 enabling one or several providers. If cloud providers are enabled, the
61 program tries to fetch the host's configuration from a meta data server
62 of the cloud via HTTP. If configuration could be not fetched, no cloud
63 provider are detected and the program quits. If host configuration is
64 obtained, the corresponding cloud provider is successfully detected.
65 Then the network of the host will be configured.
66 It is intended to re-run nm-cloud-setup every time when the
68 configuration (maybe) changes. The tool is idempotent, so it should be
69 OK to also run it more often than necessary. You could run
70 /usr/libexec/nm-cloud-setup directly. However it may be preferable to
71 restart the nm-cloud-setup systemd service instead or use the timer or
72 dispatcher script to run it periodically (see below).
73 nm-cloud-setup.service systemd unit
75 Usually /usr/libexec/nm-cloud-setup is not run directly, but only by
76 systemctl restart nm-cloud-setup.service. This ensures that the tool
77 only runs once at any time. It also allows to integrate with the
78 nm-cloud-setup systemd timer, and to enable/disable the service via
79 systemd.
80 As you need to set environment variable to configure nm-cloud-setup
82 binary, you can do so via systemd override files. Try systemctl edit
83 nm-cloud-setup.service.
84 nm-cloud-setup.timer systemd timer
86 /usr/libexec/nm-cloud-setup is intended to run whenever an update is
87 necessary. For example, during boot when when changing the network
88 configuration of the virtual machine via the cloud provider.
89 One way to do this, is by enabling the nm-cloud-setup.timer systemd
91 timer with systemctl enable --now nm-cloud-setup.timer.
92 /usr/lib/NetworkManager/dispatcher.d/90-nm-cloud-setup.sh
94 There is also a NetworkManager dispatcher script that will run for
95 example when an interface is activated by NetworkManager. Together with
96 the nm-cloud-setup.timer systemd timer this script is to automatically
97 pick up changes to the network.
98 The dispatcher script will do nothing, unless the systemd service is
100 enabled. To use the dispatcher script you should therefor run systemctl
101 enable nm-cloud-setup.service once.
102
104 The following environment variables are used to configure
105 /usr/libexec/nm-cloud-setup. You may want to configure them with a
106 drop-in for the systemd service. For example by calling systemctl edit
107 nm-cloud-setup.service and configuring [Service] Environment=, as
108 described in systemd.exec(5) manual.
109 • NM_CLOUD_SETUP_LOG: control the logging verbosity. Set it to one of
111 TRACE, DEBUG, INFO, WARN, ERR or OFF. The program will print
112 message on stdout and the default level is WARN.
113 • NM_CLOUD_SETUP_AZURE: boolean, whether Microsoft Azure support is
115 enabled. Defaults to no.
116 • NM_CLOUD_SETUP_EC2: boolean, whether Amazon EC2 (AWS) support is
118 enabled. Defaults to no.
119 • NM_CLOUD_SETUP_GCP: boolean, whether Google GCP support is enabled.
121 Defaults to no.
122 • NM_CLOUD_SETUP_ALIYUN: boolean, whether Alibaba Cloud (Aliyun)
124 support is enabled. Defaults to no.
125
127 As detailed before, nm-cloud-setup needs to be explicitly enabled. As
128 it runs as a systemd service and timer, that basically means to enable
129 and configure those. This can be done by dropping the correct files and
130 symlinks to disk.
131 The following example enables nm-cloud-setup for Amazon EC2 cloud:
133 dnf install -y NetworkManager-cloud-setup
135 mkdir -p /etc/systemd/system/nm-cloud-setup.service.d
137 cat > /etc/systemd/system/nm-cloud-setup.service.d/10-enable-ec2.conf << EOF
138 [Service]
139 Environment=NM_CLOUD_SETUP_EC2=yes
140 EOF
141 # systemctl enable nm-cloud-setup.service
143 mkdir -p /etc/systemd/system/NetworkManager.service.wants/
144 ln -s /usr/lib/systemd/system/nm-cloud-setup.service /etc/systemd/system/NetworkManager.service.wants/nm-cloud-setup.service
145 # systemctl enable nm-cloud-setup.timer
147 mkdir -p /etc/systemd/system/timers.target.wants/
148 ln -s /etc/systemd/system/timers.target.wants/nm-cloud-setup.timer /usr/lib/systemd/system/nm-cloud-setup.timer
149 # systemctl daemon-reload
151
155 Amazon EC2 (AWS)
156 For AWS, the tools tries to fetch configuration from
157 http://169.254.169.254/. Currently, it only configures IPv4 and does
158 nothing about IPv6. It will do the following.
159 • First fetch http://169.254.169.254/latest/meta-data/ to determine
161 whether the expected API is present. This determines whether EC2
162 environment is detected and whether to proceed to configure the
163 host using EC2 meta data.
164 • Fetch
166 http://169.254.169.254/2018-09-24/meta-data/network/interfaces/macs/
167 to get the list of available interface. Interfaces are identified
168 by their MAC address.
169 • Then for each interface fetch
171 http://169.254.169.254/2018-09-24/meta-data/network/interfaces/macs/$MAC/subnet-ipv4-cidr-block
172 and
173 http://169.254.169.254/2018-09-24/meta-data/network/interfaces/macs/$MAC/local-ipv4s.
174 Thereby we get a list of local IPv4 addresses and one CIDR subnet
175 block.
176 • Then nm-cloud-setup iterates over all interfaces for which it could
178 fetch IP configuration. If no ethernet device for the respective
179 MAC address is found, it is skipped. Also, if the device is
180 currently not activated in NetworkManager or if the currently
181 activated profile has a user-data
182 org.freedesktop.nm-cloud-setup.skip=yes, it is skipped.
183 If only one interface and one address is configured, then the tool
185 does nothing and leaves the automatic configuration that was
186 obtained via DHCP.
187 Otherwise, the tool will change the runtime configuration of the
189 device.
190 • Add static IPv4 addresses for all the configured addresses from
192 local-ipv4s with prefix length according to
193 subnet-ipv4-cidr-block. For example, we might have here 2 IP
194 addresses like "172.16.5.3/24,172.16.5.4/24".
195 • Choose a route table 30400 + the index of the interface and add
197 a default route 0.0.0.0/0. The gateway is the first IP address
198 in the CIDR subnet block. For example, we might get a route
199 "0.0.0.0/0 172.16.5.1 10 table=30400".
200 Also choose a route table 30200 + the interface index. This
202 contains a direct routes to the subnets of this interface.
203 • Finally, add a policy routing rule for each address. For
205 example "priority 30200 from 172.16.5.3/32 table 30200,
206 priority 30200 from 172.16.5.4/32 table 30200". and "priority
207 30400 from 172.16.5.3/32 table 30400, priority 30400 from
208 172.16.5.4/32 table 30400" The 30200+ rules select the table to
209 reach the subnet directly, while the 30400+ rules use the
210 default route. Also add a rule "priority 30350 table main
211 suppress_prefixlength 0". This has a priority between the two
212 previous rules and causes a lookup of routes in the main table
213 while ignoring the default route. The purpose of this is so
214 that other specific routes in the main table are honored over
215 the default route in table 30400+.
216 With above example, this roughly corresponds for interface eth0 to
218 nmcli device modify "eth0" ipv4.addresses
219 "172.16.5.3/24,172.16.5.4/24" ipv4.routes "172.16.5.0/24 0.0.0.0 10
220 table=30200, 0.0.0.0/0 172.16.5.1 10 table=30400"
221 ipv4.routing-rules "priority 30200 from 172.16.5.3/32 table 30200,
222 priority 30200 from 172.16.5.4/32 table 30200, priority 20350 table
223 main suppress_prefixlength 0, priority 30400 from 172.16.5.3/32
224 table 30400, priority 30400 from 172.16.5.4/32 table 30400". Note
225 that this replaces the previous addresses, routes and rules with
226 the new information. But also note that this only changes the run
227 time configuration of the device. The connection profile on disk is
228 not affected.
229 Google Cloud Platform (GCP)
231 For GCP, the meta data is fetched from URIs starting with
232 http://metadata.google.internal/computeMetadata/v1/ with a HTTP header
233 "Metadata-Flavor: Google". Currently, the tool only configures IPv4 and
234 does nothing about IPv6. It will do the following.
235 • First fetch
237 http://metadata.google.internal/computeMetadata/v1/instance/id to
238 detect whether the tool runs on Google Cloud Platform. Only if the
239 platform is detected, it will continue fetching the configuration.
240 • Fetch
242 http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/
243 to get the list of available interface indexes. These indexes can
244 be used for further lookups.
245 • Then, for each interface fetch
247 http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/$IFACE_INDEX/mac
248 to get the corresponding MAC address of the found interfaces. The
249 MAC address is used to identify the device later on.
250 • Then, for each interface with a MAC address fetch
252 http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/$IFACE_INDEX/forwarded-ips/
253 and then all the found IP addresses at
254 http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/$IFACE_INDEX/forwarded-ips/$FIPS_INDEX.
255 • At this point, we have a list of all interfaces (by MAC address)
257 and their configured IPv4 addresses.
258 For each device, we lookup the currently applied connection in
260 NetworkManager. That implies, that the device is currently
261 activated in NetworkManager. If no such device was in
262 NetworkManager, or if the profile has user-data
263 org.freedesktop.nm-cloud-setup.skip=yes, we skip the device. Now
264 for each found IP address we add a static route "$FIPS_ADDR/32
265 0.0.0.0 100 type=local" and reapply the change.
266 The effect is not unlike calling nmcli device modify "$DEVICE"
268 ipv4.routes "$FIPS_ADDR/32 0.0.0.0 100 type=local [,...]" for all
269 relevant devices and all found addresses.
270 Microsoft Azure
272 For Azure, the meta data is fetched from URIs starting with
273 http://169.254.169.254/metadata/instance with a URL parameter
274 "?format=text&api-version=2017-04-02" and a HTTP header
275 "Metadata:true". Currently, the tool only configures IPv4 and does
276 nothing about IPv6. It will do the following.
277 • First fetch
279 http://169.254.169.254/metadata/instance?format=text&api-version=2017-04-02
280 to detect whether the tool runs on Azure Cloud. Only if the
281 platform is detected, it will continue fetching the configuration.
282 • Fetch
284 http://169.254.169.254/metadata/instance/network/interface/?format=text&api-version=2017-04-02
285 to get the list of available interface indexes. These indexes can
286 be used for further lookups.
287 • Then, for each interface fetch
289 http://169.254.169.254/metadata/instance/network/interface/$IFACE_INDEX/macAddress?format=text&api-version=2017-04-02
290 to get the corresponding MAC address of the found interfaces. The
291 MAC address is used to identify the device later on.
292 • Then, for each interface with a MAC address fetch
294 http://169.254.169.254/metadata/instance/network/interface/$IFACE_INDEX/ipv4/ipAddress/?format=text&api-version=2017-04-02
295 to get the list of (indexes of) IP addresses on that interface.
296 • Then, for each IP address index fetch the address at
298 http://169.254.169.254/metadata/instance/network/interface/$IFACE_INDEX/ipv4/ipAddress/$ADDR_INDEX/privateIpAddress?format=text&api-version=2017-04-02.
299 Also fetch the size of the subnet and prefix for the interface from
300 http://169.254.169.254/metadata/instance/network/interface/$IFACE_INDEX/ipv4/subnet/0/address/?format=text&api-version=2017-04-02.
301 and
302 http://169.254.169.254/metadata/instance/network/interface/$IFACE_INDEX/ipv4/subnet/0/prefix/?format=text&api-version=2017-04-02.
303 • At this point, we have a list of all interfaces (by MAC address)
305 and their configured IPv4 addresses.
306 Then the tool configures the system like doing for AWS environment.
308 That is, using source based policy routing with the tables/rules
309 30200/30400.
310 Alibaba Cloud (Aliyun)
312 For Aliyun, the tools tries to fetch configuration from
313 http://100.100.100.200/. Currently, it only configures IPv4 and does
314 nothing about IPv6. It will do the following.
315 • First fetch http://100.100.100.200/2016-01-01/meta-data/ to
317 determine whether the expected API is present. This determines
318 whether Aliyun environment is detected and whether to proceed to
319 configure the host using Aliyun meta data.
320 • Fetch
322 http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/
323 to get the list of available interface. Interfaces are identified
324 by their MAC address.
325 • Then for each interface fetch
327 http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/$MAC/vpc-cidr-block,
328 http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/$MAC/private-ipv4s,
329 http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/$MAC/netmask
330 and
331 http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/$MAC/gateway.
332 Thereby we get a list of private IPv4 addresses, one CIDR subnet
333 block and private IPv4 addresses prefix.
334 • Then nm-cloud-setup iterates over all interfaces for which it could
336 fetch IP configuration. If no ethernet device for the respective
337 MAC address is found, it is skipped. Also, if the device is
338 currently not activated in NetworkManager or if the currently
339 activated profile has a user-data
340 org.freedesktop.nm-cloud-setup.skip=yes, it is skipped. Also, there
341 is only one interface and one IP address, the tool does nothing.
342 Then the tool configures the system like doing for AWS environment.
344 That is, using source based policy routing with the tables/rules
345 30200/30400. One difference to AWS is that the gateway is also
346 fetched via metadata instead of using the first IP address in the
347 subnet.
348
350 NetworkManager(8) nmcli(1)
351NetworkManager 1.32.12 NM-CLOUD-SETUP(8)