1ovn-architecture(7) OVN Manual ovn-architecture(7)
2
6 ovn-architecture - Open Virtual Network architecture
7
9 OVN, the Open Virtual Network, is a system to support logical network
10 abstraction in virtual machine and container environments. OVN comple‐
11 ments the existing capabilities of OVS to add native support for logi‐
12 cal network abstractions, such as logical L2 and L3 overlays and secu‐
13 rity groups. Services such as DHCP are also desirable features. Just
14 like OVS, OVN’s design goal is to have a production-quality implementa‐
15 tion that can operate at significant scale.
16 A physical network comprises physical wires, switches, and routers. A
18 virtual network extends a physical network into a hypervisor or con‐
19 tainer platform, bridging VMs or containers into the physical network.
20 An OVN logical network is a network implemented in software that is in‐
21 sulated from physical (and thus virtual) networks by tunnels or other
22 encapsulations. This allows IP and other address spaces used in logical
23 networks to overlap with those used on physical networks without caus‐
24 ing conflicts. Logical network topologies can be arranged without re‐
25 gard for the topologies of the physical networks on which they run.
26 Thus, VMs that are part of a logical network can migrate from one phys‐
27 ical machine to another without network disruption. See Logical Net‐
28 works, below, for more information.
29 The encapsulation layer prevents VMs and containers connected to a log‐
31 ical network from communicating with nodes on physical networks. For
32 clustering VMs and containers, this can be acceptable or even desir‐
33 able, but in many cases VMs and containers do need connectivity to
34 physical networks. OVN provides multiple forms of gateways for this
35 purpose. See Gateways, below, for more information.
36 An OVN deployment consists of several components:
38 • A Cloud Management System (CMS), which is OVN’s ultimate
40 client (via its users and administrators). OVN integra‐
41 tion requires installing a CMS-specific plugin and re‐
42 lated software (see below). OVN initially targets Open‐
43 Stack as CMS.
44 We generally speak of ``the’’ CMS, but one can imagine
46 scenarios in which multiple CMSes manage different parts
47 of an OVN deployment.
48 • An OVN Database physical or virtual node (or, eventually,
50 cluster) installed in a central location.
51 • One or more (usually many) hypervisors. Hypervisors must
53 run Open vSwitch and implement the interface described in
54 Documentation/topics/integration.rst in the Open vSwitch
55 source tree. Any hypervisor platform supported by Open
56 vSwitch is acceptable.
57 • Zero or more gateways. A gateway extends a tunnel-based
59 logical network into a physical network by bidirection‐
60 ally forwarding packets between tunnels and a physical
61 Ethernet port. This allows non-virtualized machines to
62 participate in logical networks. A gateway may be a phys‐
63 ical host, a virtual machine, or an ASIC-based hardware
64 switch that supports the vtep(5) schema.
65 Hypervisors and gateways are together called transport
67 node or chassis.
68 The diagram below shows how the major components of OVN and related
70 software interact. Starting at the top of the diagram, we have:
71 • The Cloud Management System, as defined above.
73 • The OVN/CMS Plugin is the component of the CMS that in‐
75 terfaces to OVN. In OpenStack, this is a Neutron plugin.
76 The plugin’s main purpose is to translate the CMS’s no‐
77 tion of logical network configuration, stored in the
78 CMS’s configuration database in a CMS-specific format,
79 into an intermediate representation understood by OVN.
80 This component is necessarily CMS-specific, so a new
82 plugin needs to be developed for each CMS that is inte‐
83 grated with OVN. All of the components below this one in
84 the diagram are CMS-independent.
85 • The OVN Northbound Database receives the intermediate
87 representation of logical network configuration passed
88 down by the OVN/CMS Plugin. The database schema is meant
89 to be ``impedance matched’’ with the concepts used in a
90 CMS, so that it directly supports notions of logical
91 switches, routers, ACLs, and so on. See ovn-nb(5) for de‐
92 tails.
93 The OVN Northbound Database has only two clients: the
95 OVN/CMS Plugin above it and ovn-northd below it.
96 • ovn-northd(8) connects to the OVN Northbound Database
98 above it and the OVN Southbound Database below it. It
99 translates the logical network configuration in terms of
100 conventional network concepts, taken from the OVN North‐
101 bound Database, into logical datapath flows in the OVN
102 Southbound Database below it.
103 • The OVN Southbound Database is the center of the system.
105 Its clients are ovn-northd(8) above it and ovn-con‐
106 troller(8) on every transport node below it.
107 The OVN Southbound Database contains three kinds of data:
109 Physical Network (PN) tables that specify how to reach
110 hypervisor and other nodes, Logical Network (LN) tables
111 that describe the logical network in terms of ``logical
112 datapath flows,’’ and Binding tables that link logical
113 network components’ locations to the physical network.
114 The hypervisors populate the PN and Port_Binding tables,
115 whereas ovn-northd(8) populates the LN tables.
116 OVN Southbound Database performance must scale with the
118 number of transport nodes. This will likely require some
119 work on ovsdb-server(1) as we encounter bottlenecks.
120 Clustering for availability may be needed.
121 The remaining components are replicated onto each hypervisor:
123 • ovn-controller(8) is OVN’s agent on each hypervisor and
125 software gateway. Northbound, it connects to the OVN
126 Southbound Database to learn about OVN configuration and
127 status and to populate the PN table and the Chassis col‐
128 umn in Binding table with the hypervisor’s status. South‐
129 bound, it connects to ovs-vswitchd(8) as an OpenFlow con‐
130 troller, for control over network traffic, and to the lo‐
131 cal ovsdb-server(1) to allow it to monitor and control
132 Open vSwitch configuration.
133 • ovs-vswitchd(8) and ovsdb-server(1) are conventional com‐
135 ponents of Open vSwitch.
136 CMS
138 |
139 |
140 +-----------|-----------+
141 | | |
142 | OVN/CMS Plugin |
143 | | |
144 | | |
145 | OVN Northbound DB |
146 | | |
147 | | |
148 | ovn-northd |
149 | | |
150 +-----------|-----------+
151 |
152 |
153 +-------------------+
154 | OVN Southbound DB |
155 +-------------------+
156 |
157 |
158 +------------------+------------------+
159 | | |
160 HV 1 | | HV n |
161 +---------------|---------------+ . +---------------|---------------+
162 | | | . | | |
163 | ovn-controller | . | ovn-controller |
164 | | | | . | | | |
165 | | | | | | | |
166 | ovs-vswitchd ovsdb-server | | ovs-vswitchd ovsdb-server |
167 | | | |
168 +-------------------------------+ +-------------------------------+
169 Information Flow in OVN
172 Configuration data in OVN flows from north to south. The CMS, through
173 its OVN/CMS plugin, passes the logical network configuration to
174 ovn-northd via the northbound database. In turn, ovn-northd compiles
175 the configuration into a lower-level form and passes it to all of the
176 chassis via the southbound database.
177 Status information in OVN flows from south to north. OVN currently pro‐
179 vides only a few forms of status information. First, ovn-northd popu‐
180 lates the up column in the northbound Logical_Switch_Port table: if a
181 logical port’s chassis column in the southbound Port_Binding table is
182 nonempty, it sets up to true, otherwise to false. This allows the CMS
183 to detect when a VM’s networking has come up.
184 Second, OVN provides feedback to the CMS on the realization of its con‐
186 figuration, that is, whether the configuration provided by the CMS has
187 taken effect. This feature requires the CMS to participate in a se‐
188 quence number protocol, which works the following way:
189 1. When the CMS updates the configuration in the northbound
191 database, as part of the same transaction, it increments the
192 value of the nb_cfg column in the NB_Global table. (This is
193 only necessary if the CMS wants to know when the configura‐
194 tion has been realized.)
195 2. When ovn-northd updates the southbound database based on a
197 given snapshot of the northbound database, it copies nb_cfg
198 from northbound NB_Global into the southbound database
199 SB_Global table, as part of the same transaction. (Thus, an
200 observer monitoring both databases can determine when the
201 southbound database is caught up with the northbound.)
202 3. After ovn-northd receives confirmation from the southbound
204 database server that its changes have committed, it updates
205 sb_cfg in the northbound NB_Global table to the nb_cfg ver‐
206 sion that was pushed down. (Thus, the CMS or another ob‐
207 server can determine when the southbound database is caught
208 up without a connection to the southbound database.)
209 4. The ovn-controller process on each chassis receives the up‐
211 dated southbound database, with the updated nb_cfg. This
212 process in turn updates the physical flows installed in the
213 chassis’s Open vSwitch instances. When it receives confirma‐
214 tion from Open vSwitch that the physical flows have been up‐
215 dated, it updates nb_cfg in its own Chassis record in the
216 southbound database.
217 5. ovn-northd monitors the nb_cfg column in all of the Chassis
219 records in the southbound database. It keeps track of the
220 minimum value among all the records and copies it into the
221 hv_cfg column in the northbound NB_Global table. (Thus, the
222 CMS or another observer can determine when all of the hyper‐
223 visors have caught up to the northbound configuration.)
224 Chassis Setup
226 Each chassis in an OVN deployment must be configured with an Open
227 vSwitch bridge dedicated for OVN’s use, called the integration bridge.
228 System startup scripts may create this bridge prior to starting
229 ovn-controller if desired. If this bridge does not exist when ovn-con‐
230 troller starts, it will be created automatically with the default con‐
231 figuration suggested below. The ports on the integration bridge in‐
232 clude:
233 • On any chassis, tunnel ports that OVN uses to maintain
235 logical network connectivity. ovn-controller adds, up‐
236 dates, and removes these tunnel ports.
237 • On a hypervisor, any VIFs that are to be attached to log‐
239 ical networks. For instances connected through software
240 emulated ports such as TUN/TAP or VETH pairs, the hyper‐
241 visor itself will normally create ports and plug them
242 into the integration bridge. For instances connected
243 through representor ports, typically used with hardware
244 offload, the ovn-controller may on CMS direction consult
245 a VIF plug provider for representor port lookup and plug
246 them into the integration bridge (please refer to Docu‐
247 mentation/top‐
248 ics/vif-plug-providers/vif-plug-providers.rst
249 for more information). In both cases the conventions de‐
250 scribed in Documentation/topics/integration.rst in the
251 Open vSwitch source tree is followed to ensure mapping
252 between OVN logical port and VIF. (This is pre-existing
253 integration work that has already been done on hypervi‐
254 sors that support OVS.)
255 • On a gateway, the physical port used for logical network
257 connectivity. System startup scripts add this port to the
258 bridge prior to starting ovn-controller. This can be a
259 patch port to another bridge, instead of a physical port,
260 in more sophisticated setups.
261 Other ports should not be attached to the integration bridge. In par‐
263 ticular, physical ports attached to the underlay network (as opposed to
264 gateway ports, which are physical ports attached to logical networks)
265 must not be attached to the integration bridge. Underlay physical ports
266 should instead be attached to a separate Open vSwitch bridge (they need
267 not be attached to any bridge at all, in fact).
268 The integration bridge should be configured as described below. The ef‐
270 fect of each of these settings is documented in
271 ovs-vswitchd.conf.db(5):
272 fail-mode=secure
274 Avoids switching packets between isolated logical net‐
275 works before ovn-controller starts up. See Controller
276 Failure Settings in ovs-vsctl(8) for more information.
277 other-config:disable-in-band=true
279 Suppresses in-band control flows for the integration
280 bridge. It would be unusual for such flows to show up
281 anyway, because OVN uses a local controller (over a Unix
282 domain socket) instead of a remote controller. It’s pos‐
283 sible, however, for some other bridge in the same system
284 to have an in-band remote controller, and in that case
285 this suppresses the flows that in-band control would or‐
286 dinarily set up. Refer to the documentation for more in‐
287 formation.
288 The customary name for the integration bridge is br-int, but another
290 name may be used.
291 Logical Networks
293 Logical network concepts in OVN include logical switches and logical
294 routers, the logical version of Ethernet switches and IP routers, re‐
295 spectively. Like their physical cousins, logical switches and routers
296 can be connected into sophisticated topologies. Logical switches and
297 routers are ordinarily purely logical entities, that is, they are not
298 associated or bound to any physical location, and they are implemented
299 in a distributed manner at each hypervisor that participates in OVN.
300 Logical switch ports (LSPs) are points of connectivity into and out of
302 logical switches. There are many kinds of logical switch ports. The
303 most ordinary kind represent VIFs, that is, attachment points for VMs
304 or containers. A VIF logical port is associated with the physical loca‐
305 tion of its VM, which might change as the VM migrates. (A VIF logical
306 port can be associated with a VM that is powered down or suspended.
307 Such a logical port has no location and no connectivity.)
308 Logical router ports (LRPs) are points of connectivity into and out of
310 logical routers. A LRP connects a logical router either to a logical
311 switch or to another logical router. Logical routers only connect to
312 VMs, containers, and other network nodes indirectly, through logical
313 switches.
314 Logical switches and logical routers have distinct kinds of logical
316 ports, so properly speaking one should usually talk about logical
317 switch ports or logical router ports. However, an unqualified ``logical
318 port’’ usually refers to a logical switch port.
319 When a VM sends a packet to a VIF logical switch port, the Open vSwitch
321 flow tables simulate the packet’s journey through that logical switch
322 and any other logical routers and logical switches that it might en‐
323 counter. This happens without transmitting the packet across any physi‐
324 cal medium: the flow tables implement all of the switching and routing
325 decisions and behavior. If the flow tables ultimately decide to output
326 the packet at a logical port attached to another hypervisor (or another
327 kind of transport node), then that is the time at which the packet is
328 encapsulated for physical network transmission and sent.
329 Logical Switch Port Types
331 OVN supports a number of kinds of logical switch ports. VIF ports that
333 connect to VMs or containers, described above, are the most ordinary
334 kind of LSP. In the OVN northbound database, VIF ports have an empty
335 string for their type. This section describes some of the additional
336 port types.
337 A router logical switch port connects a logical switch to a logical
339 router, designating a particular LRP as its peer.
340 A localnet logical switch port bridges a logical switch to a physical
342 VLAN. A logical switch may have one or more localnet ports. Such a log‐
343 ical switch is used in two scenarios:
344 • With one or more router logical switch ports, to attach
346 L3 gateway routers and distributed gateways to a physical
347 network.
348 • With one or more VIF logical switch ports, to attach VMs
350 or containers directly to a physical network. In this
351 case, the logical switch is not really logical, since it
352 is bridged to the physical network rather than insulated
353 from it, and therefore cannot have independent but over‐
354 lapping IP address namespaces, etc. A deployment might
355 nevertheless choose such a configuration to take advan‐
356 tage of the OVN control plane and features such as port
357 security and ACLs.
358 When a logical switch contains multiple localnet ports, the following
360 is assumed.
361 • Each chassis has a bridge mapping for one of the localnet
363 physical networks only.
364 • To facilitate interconnectivity between VIF ports of the
366 switch that are located on different chassis with differ‐
367 ent physical network connectivity, the fabric implements
368 L3 routing between these adjacent physical network seg‐
369 ments.
370 Note: nothing said above implies that a chassis cannot be plugged to
372 multiple physical networks as long as they belong to different
373 switches.
374 A localport logical switch port is a special kind of VIF logical switch
376 port. These ports are present in every chassis, not bound to any par‐
377 ticular one. Traffic to such a port will never be forwarded through a
378 tunnel, and traffic from such a port is expected to be destined only to
379 the same chassis, typically in response to a request it received. Open‐
380 Stack Neutron uses a localport port to serve metadata to VMs. A meta‐
381 data proxy process is attached to this port on every host and all VMs
382 within the same network will reach it at the same IP/MAC address with‐
383 out any traffic being sent over a tunnel. For further details, see the
384 OpenStack documentation for networking-ovn.
385 LSP types vtep and l2gateway are used for gateways. See Gateways, be‐
387 low, for more information.
388 Implementation Details
390 These concepts are details of how OVN is implemented internally. They
392 might still be of interest to users and administrators.
393 Logical datapaths are an implementation detail of logical networks in
395 the OVN southbound database. ovn-northd translates each logical switch
396 or router in the northbound database into a logical datapath in the
397 southbound database Datapath_Binding table.
398 For the most part, ovn-northd also translates each logical switch port
400 in the OVN northbound database into a record in the southbound database
401 Port_Binding table. The latter table corresponds roughly to the north‐
402 bound Logical_Switch_Port table. It has multiple types of logical port
403 bindings, of which many types correspond directly to northbound LSP
404 types. LSP types handled this way include VIF (empty string), localnet,
405 localport, vtep, and l2gateway.
406 The Port_Binding table has some types of port binding that do not cor‐
408 respond directly to logical switch port types. The common is patch port
409 bindings, known as logical patch ports. These port bindings always oc‐
410 cur in pairs, and a packet that enters on either side comes out on the
411 other. ovn-northd connects logical switches and logical routers to‐
412 gether using logical patch ports.
413 Port bindings with types vtep, l2gateway, l3gateway, and chassisredi‐
415 rect are used for gateways. These are explained in Gateways, below.
416 Gateways
418 Gateways provide limited connectivity between logical networks and
419 physical ones. They can also provide connectivity between different OVN
420 deployments. This section will focus on the former, and the latter will
421 be described in details in section OVN Deployments Interconnection.
422 OVN support multiple kinds of gateways.
424 VTEP Gateways
426 A ``VTEP gateway’’ connects an OVN logical network to a physical (or
428 virtual) switch that implements the OVSDB VTEP schema that accompanies
429 Open vSwitch. (The ``VTEP gateway’’ term is a misnomer, since a VTEP is
430 just a VXLAN Tunnel Endpoint, but it is a well established name.) See
431 Life Cycle of a VTEP gateway, below, for more information.
432 The main intended use case for VTEP gateways is to attach physical
434 servers to an OVN logical network using a physical top-of-rack switch
435 that supports the OVSDB VTEP schema.
436 L2 Gateways
438 A L2 gateway simply attaches a designated physical L2 segment available
440 on some chassis to a logical network. The physical network effectively
441 becomes part of the logical network.
442 To set up a L2 gateway, the CMS adds an l2gateway LSP to an appropriate
444 logical switch, setting LSP options to name the chassis on which it
445 should be bound. ovn-northd copies this configuration into a southbound
446 Port_Binding record. On the designated chassis, ovn-controller forwards
447 packets appropriately to and from the physical segment.
448 L2 gateway ports have features in common with localnet ports. However,
450 with a localnet port, the physical network becomes the transport be‐
451 tween hypervisors. With an L2 gateway, packets are still transported
452 between hypervisors over tunnels and the l2gateway port is only used
453 for the packets that are on the physical network. The application for
454 L2 gateways is similar to that for VTEP gateways, e.g. to add non-vir‐
455 tualized machines to a logical network, but L2 gateways do not require
456 special support from top-of-rack hardware switches.
457 L3 Gateway Routers
459 As described above under Logical Networks, ordinary OVN logical routers
461 are distributed: they are not implemented in a single place but rather
462 in every hypervisor chassis. This is a problem for stateful services
463 such as SNAT and DNAT, which need to be implemented in a centralized
464 manner.
465 To allow for this kind of functionality, OVN supports L3 gateway
467 routers, which are OVN logical routers that are implemented in a desig‐
468 nated chassis. Gateway routers are typically used between distributed
469 logical routers and physical networks. The distributed logical router
470 and the logical switches behind it, to which VMs and containers attach,
471 effectively reside on each hypervisor. The distributed router and the
472 gateway router are connected by another logical switch, sometimes re‐
473 ferred to as a ``join’’ logical switch. (OVN logical routers may be
474 connected to one another directly, without an intervening switch, but
475 the OVN implementation only supports gateway logical routers that are
476 connected to logical switches. Using a join logical switch also reduces
477 the number of IP addresses needed on the distributed router.) On the
478 other side, the gateway router connects to another logical switch that
479 has a localnet port connecting to the physical network.
480 The following diagram shows a typical situation. One or more logical
482 switches LS1, ..., LSn connect to distributed logical router LR1, which
483 in turn connects through LSjoin to gateway logical router GLR, which
484 also connects to logical switch LSlocal, which includes a localnet port
485 to attach to the physical network.
486 LSlocal
488 |
489 GLR
490 |
491 LSjoin
492 |
493 LR1
494 |
495 +----+----+
496 | | |
497 LS1 ... LSn
498 To configure an L3 gateway router, the CMS sets options:chassis in the
501 router’s northbound Logical_Router to the chassis’s name. In response,
502 ovn-northd uses a special l3gateway port binding (instead of a patch
503 binding) in the southbound database to connect the logical router to
504 its neighbors. In turn, ovn-controller tunnels packets to this port
505 binding to the designated L3 gateway chassis, instead of processing
506 them locally.
507 DNAT and SNAT rules may be associated with a gateway router, which pro‐
509 vides a central location that can handle one-to-many SNAT (aka IP mas‐
510 querading). Distributed gateway ports, described below, also support
511 NAT.
512 Distributed Gateway Ports
514 A distributed gateway port is a logical router port that is specially
516 configured to designate one distinguished chassis, called the gateway
517 chassis, for centralized processing. A distributed gateway port should
518 connect to a logical switch that has an LSP that connects externally,
519 that is, either a localnet LSP or a connection to another OVN deploy‐
520 ment (see OVN Deployments Interconnection). Packets that traverse the
521 distributed gateway port are processed without involving the gateway
522 chassis when they can be, but when needed they do take an extra hop
523 through it.
524 The following diagram illustrates the use of a distributed gateway
526 port. A number of logical switches LS1, ..., LSn connect to distributed
527 logical router LR1, which in turn connects through the distributed
528 gateway port to logical switch LSlocal that includes a localnet port to
529 attach to the physical network.
530 LSlocal
532 |
533 LR1
534 |
535 +----+----+
536 | | |
537 LS1 ... LSn
538 ovn-northd creates two southbound Port_Binding records to represent a
541 distributed gateway port, instead of the usual one. One of these is a
542 patch port binding named for the LRP, which is used for as much traffic
543 as it can. The other one is a port binding with type chassisredirect,
544 named cr-port. The chassisredirect port binding has one specialized
545 job: when a packet is output to it, the flow table causes it to be tun‐
546 neled to the gateway chassis, at which point it is automatically output
547 to the patch port binding. Thus, the flow table can output to this port
548 binding in cases where a particular task has to happen on the gateway
549 chassis. The chassisredirect port binding is not otherwise used (for
550 example, it never receives packets).
551 The CMS may configure distributed gateway ports three different ways.
553 See Distributed Gateway Ports in the documentation for Logi‐
554 cal_Router_Port in ovn-nb(5) for details.
555 Distributed gateway ports support high availability. When more than one
557 chassis is specified, OVN only uses one at a time as the gateway chas‐
558 sis. OVN uses BFD to monitor gateway connectivity, preferring the high‐
559 est-priority gateway that is online.
560 A logical router can have multiple distributed gateway ports, each con‐
562 necting different external networks. Load balancing is not yet sup‐
563 ported for logical routers with more than one distributed gateway port
564 configured.
565 Physical VLAN MTU Issues
567 Consider the preceding diagram again:
569 LSlocal
571 |
572 LR1
573 |
574 +----+----+
575 | | |
576 LS1 ... LSn
577 Suppose that each logical switch LS1, ..., LSn is bridged to a physical
580 VLAN-tagged network attached to a localnet port on LSlocal, over a dis‐
581 tributed gateway port on LR1. If a packet originating on LSi is des‐
582 tined to the external network, OVN sends it to the gateway chassis over
583 a tunnel. There, the packet traverses LR1’s logical router pipeline,
584 possibly undergoes NAT, and eventually ends up at LSlocal’s localnet
585 port. If all of the physical links in the network have the same MTU,
586 then the packet’s transit across a tunnel causes an MTU problem: tunnel
587 overhead prevents a packet that uses the full physical MTU from cross‐
588 ing the tunnel to the gateway chassis (without fragmentation).
589 OVN offers two solutions to this problem, the reside-on-redirect-chas‐
591 sis and redirect-type options. Both solutions require each logical
592 switch LS1, ..., LSn to include a localnet logical switch port LN1,
593 ..., LNn respectively, that is present on each chassis. Both cause
594 packets to be sent over the localnet ports instead of tunnels. They
595 differ in which packets-some or all-are sent this way. The most promi‐
596 nent tradeoff between these options is that reside-on-redirect-chassis
597 is easier to configure and that redirect-type performs better for east-
598 west traffic.
599 The first solution is the reside-on-redirect-chassis option for logical
601 router ports. Setting this option on a LRP from (e.g.) LS1 to LR1 dis‐
602 ables forwarding from LS1 to LR1 except on the gateway chassis. On
603 chassis other than the gateway chassis, this single change means that
604 packets that would otherwise have been forwarded to LR1 are instead
605 forwarded to LN1. The instance of LN1 on the gateway chassis then re‐
606 ceives the packet and forwards it to LR1. The packet traverses the LR1
607 logical router pipeline, possibly undergoes NAT, and eventually ends up
608 at LSlocal’s localnet port. The packet never traverses a tunnel, avoid‐
609 ing the MTU issue.
610 This option has the further consequence of centralizing ``distributed’’
612 logical router LR1, since no packets are forwarded from LS1 to LR1 on
613 any chassis other than the gateway chassis. Therefore, east-west traf‐
614 fic passes through the gateway chassis, not just north-south. (The
615 naive ``fix’’ of allowing east-west traffic to flow directly between
616 chassis over LN1 does not work because routing sets the Ethernet source
617 address to LR1’s source address. Seeing this single Ethernet source ad‐
618 dress originate from all of the chassis will confuse the physical
619 switch.)
620 Do not set the reside-on-redirect-chassis option on a distributed gate‐
622 way port. In the diagram above, it would be set on the LRPs connecting
623 LS1, ..., LSn to LR1.
624 The second solution is the redirect-type option for distributed gateway
626 ports. Setting this option to bridged causes packets that are redi‐
627 rected to the gateway chassis to go over the localnet ports instead of
628 being tunneled. This option does not change how OVN treats packets not
629 redirected to the gateway chassis.
630 The redirect-type option requires the administrator or the CMS to con‐
632 figure each participating chassis with a unique Ethernet address for
633 the logical router by setting ovn-chassis-mac-mappings in the Open
634 vSwitch database, for use by ovn-controller. This makes it more diffi‐
635 cult to configure than reside-on-redirect-chassis.
636 Set the redirect-type option on a distributed gateway port.
638 Using Distributed Gateway Ports For Scalability
640 Although the primary goal of distributed gateway ports is to provide
642 connectivity to external networks, there is a special use case for
643 scalability.
644 In some deployments, such as the ones using ovn-kubernetes, logical
646 switches are bound to individual chassises, and are connected by a dis‐
647 tributed logical router. In such deployments, the chassis level logical
648 switches are centralized on the chassis instead of distributed, which
649 means the ovn-controller on each chassis doesn’t need to process flows
650 and ports of logical switches on other chassises. However, without any
651 specific hint, ovn-controller would still process all the logical
652 switches as if they are fully distributed. In this case, distributed
653 gateway port can be very useful. The chassis level logical switches can
654 be connected to the distributed router using distributed gateway ports,
655 by setting the gateway chassis (or HA chassis groups with only a single
656 chassis in it) to the chassis that each logical switch is bound to.
657 ovn-controller would then skip processing the logical switches on all
658 the other chassises, largely improving the scalability, especially when
659 there are a big number of chassises.
660 Life Cycle of a VIF
662 Tables and their schemas presented in isolation are difficult to under‐
663 stand. Here’s an example.
664 A VIF on a hypervisor is a virtual network interface attached either to
666 a VM or a container running directly on that hypervisor (This is dif‐
667 ferent from the interface of a container running inside a VM).
668 The steps in this example refer often to details of the OVN and OVN
670 Northbound database schemas. Please see ovn-sb(5) and ovn-nb(5), re‐
671 spectively, for the full story on these databases.
672 1. A VIF’s life cycle begins when a CMS administrator creates a
674 new VIF using the CMS user interface or API and adds it to a
675 switch (one implemented by OVN as a logical switch). The CMS
676 updates its own configuration. This includes associating
677 unique, persistent identifier vif-id and Ethernet address
678 mac with the VIF.
679 2. The CMS plugin updates the OVN Northbound database to in‐
681 clude the new VIF, by adding a row to the Logi‐
682 cal_Switch_Port table. In the new row, name is vif-id, mac
683 is mac, switch points to the OVN logical switch’s Logi‐
684 cal_Switch record, and other columns are initialized appro‐
685 priately.
686 3. ovn-northd receives the OVN Northbound database update. In
688 turn, it makes the corresponding updates to the OVN South‐
689 bound database, by adding rows to the OVN Southbound data‐
690 base Logical_Flow table to reflect the new port, e.g. add a
691 flow to recognize that packets destined to the new port’s
692 MAC address should be delivered to it, and update the flow
693 that delivers broadcast and multicast packets to include the
694 new port. It also creates a record in the Binding table and
695 populates all its columns except the column that identifies
696 the chassis.
697 4. On every hypervisor, ovn-controller receives the Logi‐
699 cal_Flow table updates that ovn-northd made in the previous
700 step. As long as the VM that owns the VIF is powered off,
701 ovn-controller cannot do much; it cannot, for example, ar‐
702 range to send packets to or receive packets from the VIF,
703 because the VIF does not actually exist anywhere.
704 5. Eventually, a user powers on the VM that owns the VIF. On
706 the hypervisor where the VM is powered on, the integration
707 between the hypervisor and Open vSwitch (described in Docu‐
708 mentation/topics/integration.rst in the Open vSwitch source
709 tree) adds the VIF to the OVN integration bridge and stores
710 vif-id in external_ids:iface-id to indicate that the inter‐
711 face is an instantiation of the new VIF. (None of this code
712 is new in OVN; this is pre-existing integration work that
713 has already been done on hypervisors that support OVS.)
714 6. On the hypervisor where the VM is powered on, ovn-controller
716 notices external_ids:iface-id in the new Interface. In re‐
717 sponse, in the OVN Southbound DB, it updates the Binding ta‐
718 ble’s chassis column for the row that links the logical port
719 from external_ids: iface-id to the hypervisor. Afterward,
720 ovn-controller updates the local hypervisor’s OpenFlow ta‐
721 bles so that packets to and from the VIF are properly han‐
722 dled.
723 7. Some CMS systems, including OpenStack, fully start a VM only
725 when its networking is ready. To support this, ovn-northd
726 notices the chassis column updated for the row in Binding
727 table and pushes this upward by updating the up column in
728 the OVN Northbound database’s Logical_Switch_Port table to
729 indicate that the VIF is now up. The CMS, if it uses this
730 feature, can then react by allowing the VM’s execution to
731 proceed.
732 8. On every hypervisor but the one where the VIF resides,
734 ovn-controller notices the completely populated row in the
735 Binding table. This provides ovn-controller the physical lo‐
736 cation of the logical port, so each instance updates the
737 OpenFlow tables of its switch (based on logical datapath
738 flows in the OVN DB Logical_Flow table) so that packets to
739 and from the VIF can be properly handled via tunnels.
740 9. Eventually, a user powers off the VM that owns the VIF. On
742 the hypervisor where the VM was powered off, the VIF is
743 deleted from the OVN integration bridge.
744 10. On the hypervisor where the VM was powered off, ovn-con‐
746 troller notices that the VIF was deleted. In response, it
747 removes the Chassis column content in the Binding table for
748 the logical port.
749 11. On every hypervisor, ovn-controller notices the empty Chas‐
751 sis column in the Binding table’s row for the logical port.
752 This means that ovn-controller no longer knows the physical
753 location of the logical port, so each instance updates its
754 OpenFlow table to reflect that.
755 12. Eventually, when the VIF (or its entire VM) is no longer
757 needed by anyone, an administrator deletes the VIF using the
758 CMS user interface or API. The CMS updates its own configu‐
759 ration.
760 13. The CMS plugin removes the VIF from the OVN Northbound data‐
762 base, by deleting its row in the Logical_Switch_Port table.
763 14. ovn-northd receives the OVN Northbound update and in turn
765 updates the OVN Southbound database accordingly, by removing
766 or updating the rows from the OVN Southbound database Logi‐
767 cal_Flow table and Binding table that were related to the
768 now-destroyed VIF.
769 15. On every hypervisor, ovn-controller receives the Logi‐
771 cal_Flow table updates that ovn-northd made in the previous
772 step. ovn-controller updates OpenFlow tables to reflect the
773 update, although there may not be much to do, since the VIF
774 had already become unreachable when it was removed from the
775 Binding table in a previous step.
776 Life Cycle of a Container Interface Inside a VM
778 OVN provides virtual network abstractions by converting information
779 written in OVN_NB database to OpenFlow flows in each hypervisor. Secure
780 virtual networking for multi-tenants can only be provided if OVN con‐
781 troller is the only entity that can modify flows in Open vSwitch. When
782 the Open vSwitch integration bridge resides in the hypervisor, it is a
783 fair assumption to make that tenant workloads running inside VMs cannot
784 make any changes to Open vSwitch flows.
785 If the infrastructure provider trusts the applications inside the con‐
787 tainers not to break out and modify the Open vSwitch flows, then con‐
788 tainers can be run in hypervisors. This is also the case when contain‐
789 ers are run inside the VMs and Open vSwitch integration bridge with
790 flows added by OVN controller resides in the same VM. For both the
791 above cases, the workflow is the same as explained with an example in
792 the previous section ("Life Cycle of a VIF").
793 This section talks about the life cycle of a container interface (CIF)
795 when containers are created in the VMs and the Open vSwitch integration
796 bridge resides inside the hypervisor. In this case, even if a container
797 application breaks out, other tenants are not affected because the con‐
798 tainers running inside the VMs cannot modify the flows in the Open
799 vSwitch integration bridge.
800 When multiple containers are created inside a VM, there are multiple
802 CIFs associated with them. The network traffic associated with these
803 CIFs need to reach the Open vSwitch integration bridge running in the
804 hypervisor for OVN to support virtual network abstractions. OVN should
805 also be able to distinguish network traffic coming from different CIFs.
806 There are two ways to distinguish network traffic of CIFs.
807 One way is to provide one VIF for every CIF (1:1 model). This means
809 that there could be a lot of network devices in the hypervisor. This
810 would slow down OVS because of all the additional CPU cycles needed for
811 the management of all the VIFs. It would also mean that the entity cre‐
812 ating the containers in a VM should also be able to create the corre‐
813 sponding VIFs in the hypervisor.
814 The second way is to provide a single VIF for all the CIFs (1:many
816 model). OVN could then distinguish network traffic coming from differ‐
817 ent CIFs via a tag written in every packet. OVN uses this mechanism and
818 uses VLAN as the tagging mechanism.
819 1. A CIF’s life cycle begins when a container is spawned inside
821 a VM by the either the same CMS that created the VM or a
822 tenant that owns that VM or even a container Orchestration
823 System that is different than the CMS that initially created
824 the VM. Whoever the entity is, it will need to know the vif-
825 id that is associated with the network interface of the VM
826 through which the container interface’s network traffic is
827 expected to go through. The entity that creates the con‐
828 tainer interface will also need to choose an unused VLAN in‐
829 side that VM.
830 2. The container spawning entity (either directly or through
832 the CMS that manages the underlying infrastructure) updates
833 the OVN Northbound database to include the new CIF, by
834 adding a row to the Logical_Switch_Port table. In the new
835 row, name is any unique identifier, parent_name is the vif-
836 id of the VM through which the CIF’s network traffic is ex‐
837 pected to go through and the tag is the VLAN tag that iden‐
838 tifies the network traffic of that CIF.
839 3. ovn-northd receives the OVN Northbound database update. In
841 turn, it makes the corresponding updates to the OVN South‐
842 bound database, by adding rows to the OVN Southbound data‐
843 base’s Logical_Flow table to reflect the new port and also
844 by creating a new row in the Binding table and populating
845 all its columns except the column that identifies the chas‐
846 sis.
847 4. On every hypervisor, ovn-controller subscribes to the
849 changes in the Binding table. When a new row is created by
850 ovn-northd that includes a value in parent_port column of
851 Binding table, the ovn-controller in the hypervisor whose
852 OVN integration bridge has that same value in vif-id in ex‐
853 ternal_ids:iface-id updates the local hypervisor’s OpenFlow
854 tables so that packets to and from the VIF with the particu‐
855 lar VLAN tag are properly handled. Afterward it updates the
856 chassis column of the Binding to reflect the physical loca‐
857 tion.
858 5. One can only start the application inside the container af‐
860 ter the underlying network is ready. To support this,
861 ovn-northd notices the updated chassis column in Binding ta‐
862 ble and updates the up column in the OVN Northbound data‐
863 base’s Logical_Switch_Port table to indicate that the CIF is
864 now up. The entity responsible to start the container appli‐
865 cation queries this value and starts the application.
866 6. Eventually the entity that created and started the con‐
868 tainer, stops it. The entity, through the CMS (or directly)
869 deletes its row in the Logical_Switch_Port table.
870 7. ovn-northd receives the OVN Northbound update and in turn
872 updates the OVN Southbound database accordingly, by removing
873 or updating the rows from the OVN Southbound database Logi‐
874 cal_Flow table that were related to the now-destroyed CIF.
875 It also deletes the row in the Binding table for that CIF.
876 8. On every hypervisor, ovn-controller receives the Logi‐
878 cal_Flow table updates that ovn-northd made in the previous
879 step. ovn-controller updates OpenFlow tables to reflect the
880 update.
881 Architectural Physical Life Cycle of a Packet
883 This section describes how a packet travels from one virtual machine or
884 container to another through OVN. This description focuses on the phys‐
885 ical treatment of a packet; for a description of the logical life cycle
886 of a packet, please refer to the Logical_Flow table in ovn-sb(5).
887 This section mentions several data and metadata fields, for clarity
889 summarized here:
890 tunnel key
892 When OVN encapsulates a packet in Geneve or another tun‐
893 nel, it attaches extra data to it to allow the receiving
894 OVN instance to process it correctly. This takes differ‐
895 ent forms depending on the particular encapsulation, but
896 in each case we refer to it here as the ``tunnel key.’’
897 See Tunnel Encapsulations, below, for details.
898 logical datapath field
900 A field that denotes the logical datapath through which a
901 packet is being processed. OVN uses the field that Open‐
902 Flow 1.1+ simply (and confusingly) calls ``metadata’’ to
903 store the logical datapath. (This field is passed across
904 tunnels as part of the tunnel key.)
905 logical input port field
907 A field that denotes the logical port from which the
908 packet entered the logical datapath. OVN stores this in
909 Open vSwitch extension register number 14.
910 Geneve and STT tunnels pass this field as part of the
912 tunnel key. Ramp switch VXLAN tunnels do not explicitly
913 carry a logical input port, but since they are used to
914 communicate with gateways that from OVN’s perspective
915 consist of only a single logical port, so that OVN can
916 set the logical input port field to this one on ingress
917 to the OVN logical pipeline. As for regular VXLAN tun‐
918 nels, they don’t carry input port field at all. This puts
919 additional limitations on cluster capabilities that are
920 described in Tunnel Encapsulations section.
921 logical output port field
923 A field that denotes the logical port from which the
924 packet will leave the logical datapath. This is initial‐
925 ized to 0 at the beginning of the logical ingress pipe‐
926 line. OVN stores this in Open vSwitch extension register
927 number 15.
928 Geneve, STT and regular VXLAN tunnels pass this field as
930 part of the tunnel key. Ramp switch VXLAN tunnels do not
931 transmit the logical output port field, and since they do
932 not carry a logical output port field in the tunnel key,
933 when a packet is received from ramp switch VXLAN tunnel
934 by an OVN hypervisor, the packet is resubmitted to table
935 8 to determine the output port(s); when the packet
936 reaches table 39, these packets are resubmitted to table
937 40 for local delivery by checking a MLF_RCV_FROM_RAMP
938 flag, which is set when the packet arrives from a ramp
939 tunnel.
940 conntrack zone field for logical ports
942 A field that denotes the connection tracking zone for
943 logical ports. The value only has local significance and
944 is not meaningful between chassis. This is initialized to
945 0 at the beginning of the logical ingress pipeline. OVN
946 stores this in Open vSwitch extension register number 13.
947 conntrack zone fields for routers
949 Fields that denote the connection tracking zones for
950 routers. These values only have local significance and
951 are not meaningful between chassis. OVN stores the zone
952 information for north to south traffic (for DNATting or
953 ECMP symmetric replies) in Open vSwitch extension regis‐
954 ter number 11 and zone information for south to north
955 traffic (for SNATing) in Open vSwitch extension register
956 number 12.
957 logical flow flags
959 The logical flags are intended to handle keeping context
960 between tables in order to decide which rules in subse‐
961 quent tables are matched. These values only have local
962 significance and are not meaningful between chassis. OVN
963 stores the logical flags in Open vSwitch extension regis‐
964 ter number 10.
965 VLAN ID
967 The VLAN ID is used as an interface between OVN and con‐
968 tainers nested inside a VM (see Life Cycle of a container
969 interface inside a VM, above, for more information).
970 Initially, a VM or container on the ingress hypervisor sends a packet
972 on a port attached to the OVN integration bridge. Then:
973 1. OpenFlow table 0 performs physical-to-logical translation.
975 It matches the packet’s ingress port. Its actions annotate
976 the packet with logical metadata, by setting the logical
977 datapath field to identify the logical datapath that the
978 packet is traversing and the logical input port field to
979 identify the ingress port. Then it resubmits to table 8 to
980 enter the logical ingress pipeline.
981 Packets that originate from a container nested within a VM
983 are treated in a slightly different way. The originating
984 container can be distinguished based on the VIF-specific
985 VLAN ID, so the physical-to-logical translation flows addi‐
986 tionally match on VLAN ID and the actions strip the VLAN
987 header. Following this step, OVN treats packets from con‐
988 tainers just like any other packets.
989 Table 0 also processes packets that arrive from other chas‐
991 sis. It distinguishes them from other packets by ingress
992 port, which is a tunnel. As with packets just entering the
993 OVN pipeline, the actions annotate these packets with logi‐
994 cal datapath metadata. For tunnel types that support it,
995 they are also annotated with logical ingress port metadata.
996 In addition, the actions set the logical output port field,
997 which is available because in OVN tunneling occurs after the
998 logical output port is known. These pieces of information
999 are obtained from the tunnel encapsulation metadata (see
1000 Tunnel Encapsulations for encoding details). Then the ac‐
1001 tions resubmit to table 38 to enter the logical egress pipe‐
1002 line.
1003 2. OpenFlow tables 8 through 31 execute the logical ingress
1005 pipeline from the Logical_Flow table in the OVN Southbound
1006 database. These tables are expressed entirely in terms of
1007 logical concepts like logical ports and logical datapaths. A
1008 big part of ovn-controller’s job is to translate them into
1009 equivalent OpenFlow (in particular it translates the table
1010 numbers: Logical_Flow tables 0 through 23 become OpenFlow
1011 tables 8 through 31).
1012 Each logical flow maps to one or more OpenFlow flows. An ac‐
1014 tual packet ordinarily matches only one of these, although
1015 in some cases it can match more than one of these flows
1016 (which is not a problem because all of them have the same
1017 actions). ovn-controller uses the first 32 bits of the logi‐
1018 cal flow’s UUID as the cookie for its OpenFlow flow or
1019 flows. (This is not necessarily unique, since the first 32
1020 bits of a logical flow’s UUID is not necessarily unique.)
1021 Some logical flows can map to the Open vSwitch ``conjunctive
1023 match’’ extension (see ovs-fields(7)). Flows with a conjunc‐
1024 tion action use an OpenFlow cookie of 0, because they can
1025 correspond to multiple logical flows. The OpenFlow flow for
1026 a conjunctive match includes a match on conj_id.
1027 Some logical flows may not be represented in the OpenFlow
1029 tables on a given hypervisor, if they could not be used on
1030 that hypervisor. For example, if no VIF in a logical switch
1031 resides on a given hypervisor, and the logical switch is not
1032 otherwise reachable on that hypervisor (e.g. over a series
1033 of hops through logical switches and routers starting from a
1034 VIF on the hypervisor), then the logical flow may not be
1035 represented there.
1036 Most OVN actions have fairly obvious implementations in
1038 OpenFlow (with OVS extensions), e.g. next; is implemented as
1039 resubmit, field = constant; as set_field. A few are worth
1040 describing in more detail:
1041 output:
1043 Implemented by resubmitting the packet to table 37.
1044 If the pipeline executes more than one output action,
1045 then each one is separately resubmitted to table 37.
1046 This can be used to send multiple copies of the
1047 packet to multiple ports. (If the packet was not mod‐
1048 ified between the output actions, and some of the
1049 copies are destined to the same hypervisor, then us‐
1050 ing a logical multicast output port would save band‐
1051 width between hypervisors.)
1052 get_arp(P, A);
1054 get_nd(P, A);
1055 Implemented by storing arguments into OpenFlow fields,
1056 then resubmitting to table 66, which ovn-controller
1057 populates with flows generated from the MAC_Binding ta‐
1058 ble in the OVN Southbound database. If there is a match
1059 in table 66, then its actions store the bound MAC in
1060 the Ethernet destination address field.
1061 (The OpenFlow actions save and restore the OpenFlow
1063 fields used for the arguments, so that the OVN actions
1064 do not have to be aware of this temporary use.)
1065 put_arp(P, A, E);
1067 put_nd(P, A, E);
1068 Implemented by storing the arguments into OpenFlow
1069 fields, then outputting a packet to ovn-controller,
1070 which updates the MAC_Binding table.
1071 (The OpenFlow actions save and restore the OpenFlow
1073 fields used for the arguments, so that the OVN actions
1074 do not have to be aware of this temporary use.)
1075 R = lookup_arp(P, A, M);
1077 R = lookup_nd(P, A, M);
1078 Implemented by storing arguments into OpenFlow fields,
1079 then resubmitting to table 67, which ovn-controller
1080 populates with flows generated from the MAC_Binding ta‐
1081 ble in the OVN Southbound database. If there is a match
1082 in table 67, then its actions set the logical flow flag
1083 MLF_LOOKUP_MAC.
1084 (The OpenFlow actions save and restore the OpenFlow
1086 fields used for the arguments, so that the OVN actions
1087 do not have to be aware of this temporary use.)
1088 3. OpenFlow tables 37 through 41 implement the output action in
1090 the logical ingress pipeline. Specifically, table 37 serves
1091 as an entry point to egress pipeline. Table 37 detects IP
1092 packets that are too big for a corresponding interface. Ta‐
1093 ble 38 produces ICMPv4 Fragmentation Needed (or ICMPv6 Too
1094 Big) errors and deliver them back to the offending port. ta‐
1095 ble 39 handles packets to remote hypervisors, table 40 han‐
1096 dles packets to the local hypervisor, and table 41 checks
1097 whether packets whose logical ingress and egress port are
1098 the same should be discarded.
1099 Logical patch ports are a special case. Logical patch ports
1101 do not have a physical location and effectively reside on
1102 every hypervisor. Thus, flow table 40, for output to ports
1103 on the local hypervisor, naturally implements output to uni‐
1104 cast logical patch ports too. However, applying the same
1105 logic to a logical patch port that is part of a logical mul‐
1106 ticast group yields packet duplication, because each hyper‐
1107 visor that contains a logical port in the multicast group
1108 will also output the packet to the logical patch port. Thus,
1109 multicast groups implement output to logical patch ports in
1110 table 39.
1111 Each flow in table 39 matches on a logical output port for
1113 unicast or multicast logical ports that include a logical
1114 port on a remote hypervisor. Each flow’s actions implement
1115 sending a packet to the port it matches. For unicast logical
1116 output ports on remote hypervisors, the actions set the tun‐
1117 nel key to the correct value, then send the packet on the
1118 tunnel port to the correct hypervisor. (When the remote hy‐
1119 pervisor receives the packet, table 0 there will recognize
1120 it as a tunneled packet and pass it along to table 40.) For
1121 multicast logical output ports, the actions send one copy of
1122 the packet to each remote hypervisor, in the same way as for
1123 unicast destinations. If a multicast group includes a logi‐
1124 cal port or ports on the local hypervisor, then its actions
1125 also resubmit to table 40. Table 39 also includes:
1126 • A higher-priority rule to match packets received from
1128 ramp switch tunnels, based on flag MLF_RCV_FROM_RAMP,
1129 and resubmit these packets to table 40 for local de‐
1130 livery. Packets received from ramp switch tunnels
1131 reach here because of a lack of logical output port
1132 field in the tunnel key and thus these packets needed
1133 to be submitted to table 8 to determine the output
1134 port.
1135 • A higher-priority rule to match packets received from
1137 ports of type localport, based on the logical input
1138 port, and resubmit these packets to table 40 for lo‐
1139 cal delivery. Ports of type localport exist on every
1140 hypervisor and by definition their traffic should
1141 never go out through a tunnel.
1142 • A higher-priority rule to match packets that have the
1144 MLF_LOCAL_ONLY logical flow flag set, and whose des‐
1145 tination is a multicast address. This flag indicates
1146 that the packet should not be delivered to remote hy‐
1147 pervisors, even if the multicast destination includes
1148 ports on remote hypervisors. This flag is used when
1149 ovn-controller is the originator of the multicast
1150 packet. Since each ovn-controller instance is origi‐
1151 nating these packets, the packets only need to be de‐
1152 livered to local ports.
1153 • A fallback flow that resubmits to table 40 if there
1155 is no other match.
1156 Flows in table 40 resemble those in table 39 but for logical
1158 ports that reside locally rather than remotely. For unicast
1159 logical output ports on the local hypervisor, the actions
1160 just resubmit to table 41. For multicast output ports that
1161 include one or more logical ports on the local hypervisor,
1162 for each such logical port P, the actions change the logical
1163 output port to P, then resubmit to table 41.
1164 A special case is that when a localnet port exists on the
1166 datapath, remote port is connected by switching to the lo‐
1167 calnet port. In this case, instead of adding a flow in table
1168 39 to reach the remote port, a flow is added in table 40 to
1169 switch the logical outport to the localnet port, and resub‐
1170 mit to table 40 as if it were unicasted to a logical port on
1171 the local hypervisor.
1172 Table 41 matches and drops packets for which the logical in‐
1174 put and output ports are the same and the MLF_ALLOW_LOOPBACK
1175 flag is not set. It also drops MLF_LOCAL_ONLY packets di‐
1176 rected to a localnet port. It resubmits other packets to ta‐
1177 ble 42.
1178 4. OpenFlow tables 42 through 62 execute the logical egress
1180 pipeline from the Logical_Flow table in the OVN Southbound
1181 database. The egress pipeline can perform a final stage of
1182 validation before packet delivery. Eventually, it may exe‐
1183 cute an output action, which ovn-controller implements by
1184 resubmitting to table 64. A packet for which the pipeline
1185 never executes output is effectively dropped (although it
1186 may have been transmitted through a tunnel across a physical
1187 network).
1188 The egress pipeline cannot change the logical output port or
1190 cause further tunneling.
1191 5. Table 64 bypasses OpenFlow loopback when MLF_ALLOW_LOOPBACK
1193 is set. Logical loopback was handled in table 41, but Open‐
1194 Flow by default also prevents loopback to the OpenFlow
1195 ingress port. Thus, when MLF_ALLOW_LOOPBACK is set, OpenFlow
1196 table 64 saves the OpenFlow ingress port, sets it to zero,
1197 resubmits to table 65 for logical-to-physical transforma‐
1198 tion, and then restores the OpenFlow ingress port, effec‐
1199 tively disabling OpenFlow loopback prevents. When MLF_AL‐
1200 LOW_LOOPBACK is unset, table 64 flow simply resubmits to ta‐
1201 ble 65.
1202 6. OpenFlow table 65 performs logical-to-physical translation,
1204 the opposite of table 0. It matches the packet’s logical
1205 egress port. Its actions output the packet to the port at‐
1206 tached to the OVN integration bridge that represents that
1207 logical port. If the logical egress port is a container
1208 nested with a VM, then before sending the packet the actions
1209 push on a VLAN header with an appropriate VLAN ID.
1210 Logical Routers and Logical Patch Ports
1212 Typically logical routers and logical patch ports do not have a physi‐
1213 cal location and effectively reside on every hypervisor. This is the
1214 case for logical patch ports between logical routers and logical
1215 switches behind those logical routers, to which VMs (and VIFs) attach.
1216 Consider a packet sent from one virtual machine or container to another
1218 VM or container that resides on a different subnet. The packet will
1219 traverse tables 0 to 65 as described in the previous section Architec‐
1220 tural Physical Life Cycle of a Packet, using the logical datapath rep‐
1221 resenting the logical switch that the sender is attached to. At table
1222 39, the packet will use the fallback flow that resubmits locally to ta‐
1223 ble 40 on the same hypervisor. In this case, all of the processing from
1224 table 0 to table 65 occurs on the hypervisor where the sender resides.
1225 When the packet reaches table 65, the logical egress port is a logical
1227 patch port. ovn-controller implements output to the logical patch is
1228 packet by cloning and resubmitting directly to the first OpenFlow flow
1229 table in the ingress pipeline, setting the logical ingress port to the
1230 peer logical patch port, and using the peer logical patch port’s logi‐
1231 cal datapath (that represents the logical router).
1232 The packet re-enters the ingress pipeline in order to traverse tables 8
1234 to 65 again, this time using the logical datapath representing the log‐
1235 ical router. The processing continues as described in the previous sec‐
1236 tion Architectural Physical Life Cycle of a Packet. When the packet
1237 reaches table 65, the logical egress port will once again be a logical
1238 patch port. In the same manner as described above, this logical patch
1239 port will cause the packet to be resubmitted to OpenFlow tables 8 to
1240 65, this time using the logical datapath representing the logical
1241 switch that the destination VM or container is attached to.
1242 The packet traverses tables 8 to 65 a third and final time. If the des‐
1244 tination VM or container resides on a remote hypervisor, then table 39
1245 will send the packet on a tunnel port from the sender’s hypervisor to
1246 the remote hypervisor. Finally table 65 will output the packet directly
1247 to the destination VM or container.
1248 The following sections describe two exceptions, where logical routers
1250 and/or logical patch ports are associated with a physical location.
1251 Gateway Routers
1253 A gateway router is a logical router that is bound to a physical loca‐
1255 tion. This includes all of the logical patch ports of the logical
1256 router, as well as all of the peer logical patch ports on logical
1257 switches. In the OVN Southbound database, the Port_Binding entries for
1258 these logical patch ports use the type l3gateway rather than patch, in
1259 order to distinguish that these logical patch ports are bound to a
1260 chassis.
1261 When a hypervisor processes a packet on a logical datapath representing
1263 a logical switch, and the logical egress port is a l3gateway port rep‐
1264 resenting connectivity to a gateway router, the packet will match a
1265 flow in table 39 that sends the packet on a tunnel port to the chassis
1266 where the gateway router resides. This processing in table 39 is done
1267 in the same manner as for VIFs.
1268 Distributed Gateway Ports
1270 This section provides additional details on distributed gateway ports,
1272 outlined earlier.
1273 The primary design goal of distributed gateway ports is to allow as
1275 much traffic as possible to be handled locally on the hypervisor where
1276 a VM or container resides. Whenever possible, packets from the VM or
1277 container to the outside world should be processed completely on that
1278 VM’s or container’s hypervisor, eventually traversing a localnet port
1279 instance or a tunnel to the physical network or a different OVN deploy‐
1280 ment. Whenever possible, packets from the outside world to a VM or con‐
1281 tainer should be directed through the physical network directly to the
1282 VM’s or container’s hypervisor.
1283 In order to allow for the distributed processing of packets described
1285 in the paragraph above, distributed gateway ports need to be logical
1286 patch ports that effectively reside on every hypervisor, rather than
1287 l3gateway ports that are bound to a particular chassis. However, the
1288 flows associated with distributed gateway ports often need to be asso‐
1289 ciated with physical locations, for the following reasons:
1290 • The physical network that the localnet port is attached
1292 to typically uses L2 learning. Any Ethernet address used
1293 over the distributed gateway port must be restricted to a
1294 single physical location so that upstream L2 learning is
1295 not confused. Traffic sent out the distributed gateway
1296 port towards the localnet port with a specific Ethernet
1297 address must be sent out one specific instance of the
1298 distributed gateway port on one specific chassis. Traffic
1299 received from the localnet port (or from a VIF on the
1300 same logical switch as the localnet port) with a specific
1301 Ethernet address must be directed to the logical switch’s
1302 patch port instance on that specific chassis.
1303 Due to the implications of L2 learning, the Ethernet ad‐
1305 dress and IP address of the distributed gateway port need
1306 to be restricted to a single physical location. For this
1307 reason, the user must specify one chassis associated with
1308 the distributed gateway port. Note that traffic travers‐
1309 ing the distributed gateway port using other Ethernet ad‐
1310 dresses and IP addresses (e.g. one-to-one NAT) is not re‐
1311 stricted to this chassis.
1312 Replies to ARP and ND requests must be restricted to a
1314 single physical location, where the Ethernet address in
1315 the reply resides. This includes ARP and ND replies for
1316 the IP address of the distributed gateway port, which are
1317 restricted to the chassis that the user associated with
1318 the distributed gateway port.
1319 • In order to support one-to-many SNAT (aka IP masquerad‐
1321 ing), where multiple logical IP addresses spread across
1322 multiple chassis are mapped to a single external IP ad‐
1323 dress, it will be necessary to handle some of the logical
1324 router processing on a specific chassis in a centralized
1325 manner. Since the SNAT external IP address is typically
1326 the distributed gateway port IP address, and for simplic‐
1327 ity, the same chassis associated with the distributed
1328 gateway port is used.
1329 The details of flow restrictions to specific chassis are described in
1331 the ovn-northd documentation.
1332 While most of the physical location dependent aspects of distributed
1334 gateway ports can be handled by restricting some flows to specific
1335 chassis, one additional mechanism is required. When a packet leaves the
1336 ingress pipeline and the logical egress port is the distributed gateway
1337 port, one of two different sets of actions is required at table 39:
1338 • If the packet can be handled locally on the sender’s hy‐
1340 pervisor (e.g. one-to-one NAT traffic), then the packet
1341 should just be resubmitted locally to table 40, in the
1342 normal manner for distributed logical patch ports.
1343 • However, if the packet needs to be handled on the chassis
1345 associated with the distributed gateway port (e.g. one-
1346 to-many SNAT traffic or non-NAT traffic), then table 39
1347 must send the packet on a tunnel port to that chassis.
1348 In order to trigger the second set of actions, the chassisredirect type
1350 of southbound Port_Binding has been added. Setting the logical egress
1351 port to the type chassisredirect logical port is simply a way to indi‐
1352 cate that although the packet is destined for the distributed gateway
1353 port, it needs to be redirected to a different chassis. At table 39,
1354 packets with this logical egress port are sent to a specific chassis,
1355 in the same way that table 39 directs packets whose logical egress port
1356 is a VIF or a type l3gateway port to different chassis. Once the packet
1357 arrives at that chassis, table 40 resets the logical egress port to the
1358 value representing the distributed gateway port. For each distributed
1359 gateway port, there is one type chassisredirect port, in addition to
1360 the distributed logical patch port representing the distributed gateway
1361 port.
1362 High Availability for Distributed Gateway Ports
1364 OVN allows you to specify a prioritized list of chassis for a distrib‐
1366 uted gateway port. This is done by associating multiple Gateway_Chassis
1367 rows with a Logical_Router_Port in the OVN_Northbound database.
1368 When multiple chassis have been specified for a gateway, all chassis
1370 that may send packets to that gateway will enable BFD on tunnels to all
1371 configured gateway chassis. The current master chassis for the gateway
1372 is the highest priority gateway chassis that is currently viewed as ac‐
1373 tive based on BFD status.
1374 For more information on L3 gateway high availability, please refer to
1376 http://docs.ovn.org/en/latest/topics/high-availability.html.
1377 Restrictions of Distributed Gateway Ports
1379 Distributed gateway ports are used to connect to an external network,
1381 which can be a physical network modeled by a logical switch with a lo‐
1382 calnet port, and can also be a logical switch that interconnects dif‐
1383 ferent OVN deployments (see OVN Deployments Interconnection). Usually
1384 there can be many logical routers connected to the same external logi‐
1385 cal switch, as shown in below diagram.
1386 +--LS-EXT-+
1388 | | |
1389 | | |
1390 LR1 ... LRn
1391 In this diagram, there are n logical routers connected to a logical
1394 switch LS-EXT, each with a distributed gateway port, so that traffic
1395 sent to external world is redirected to the gateway chassis that is as‐
1396 signed to the distributed gateway port of respective logical router.
1397 In the logical topology, nothing can prevent an user to add a route be‐
1399 tween the logical routers via the connected distributed gateway ports
1400 on LS-EXT. However, the route works only if the LS-EXT is a physical
1401 network (modeled by a logical switch with a localnet port). In that
1402 case the packet will be delivered between the gateway chassises through
1403 the localnet port via physical network. If the LS-EXT is a regular log‐
1404 ical switch (backed by tunneling only, as in the use case of OVN inter‐
1405 connection), then the packet will be dropped on the source gateway
1406 chassis. The limitation is due the fact that distributed gateway ports
1407 are tied to physical location, and without physical network connection,
1408 we will end up with either dropping the packet or transferring it over
1409 the tunnels which could cause bigger problems such as broadcast packets
1410 being redirect repeatedly by different gateway chassises.
1411 With the limitation in mind, if a user do want the direct connectivity
1413 between the logical routers, it is better to create an internal logical
1414 switch connected to the logical routers with regular logical router
1415 ports, which are completely distributed and the packets don’t have to
1416 leave a chassis unless necessary, which is more optimal than routing
1417 via the distributed gateway ports.
1418 ARP request and ND NS packet processing
1420 Due to the fact that ARP requests and ND NA packets are usually broad‐
1422 cast packets, for performance reasons, OVN deals with requests that
1423 target OVN owned IP addresses (i.e., IP addresses configured on the
1424 router ports, VIPs, NAT IPs) in a specific way and only forwards them
1425 to the logical router that owns the target IP address. This behavior is
1426 different than that of traditional switches and implies that other
1427 routers/hosts connected to the logical switch will not learn the MAC/IP
1428 binding from the request packet.
1429 All other ARP and ND packets are flooded in the L2 broadcast domain and
1431 to all attached logical patch ports.
1432 VIFs on the logical switch connected by a distributed gateway port
1434 Typically the logical switch connected by a distributed gateway port is
1436 for external connectivity, usually to a physical network through a lo‐
1437 calnet port on the logical switch, or to a remote OVN deployment
1438 through OVN Interconnection. In these cases there is no VIF ports re‐
1439 quired on the logical switch.
1440 While not very common, it is still possible to create VIF ports on the
1442 logical switch connected by a distributed gateway port, but there is a
1443 limitation that the logical ports need to reside on the gateway chassis
1444 where the distributed gateway port resides to get connectivity to other
1445 logical switches through the distributed gateway port. There is no lim‐
1446 itation for the VIFs to connect within the logical switch, or beyond
1447 the logical switch through other regular distributed logical router
1448 ports.
1449 A special case is when using distributed gateway ports for scalability
1451 purpose, as mentioned earlier in this document. The logical switches
1452 connected by distributed gateway ports are not for connectivity but
1453 just for regular VIFs. However, the above limitation usually does not
1454 matter because in this use case all the VIFs on the logical switch are
1455 located on the same chassis with the distributed gateway port that con‐
1456 nects the logical switch.
1457 Multiple localnet logical switches connected to a Logical Router
1459 It is possible to have multiple logical switches each with a localnet
1460 port (representing physical networks) connected to a logical router, in
1461 which one localnet logical switch may provide the external connectivity
1462 via a distributed gateway port and rest of the localnet logical
1463 switches use VLAN tagging in the physical network. It is expected that
1464 ovn-bridge-mappings is configured appropriately on the chassis for all
1465 these localnet networks.
1466 East West routing
1468 East-West routing between these localnet VLAN tagged logical switches
1470 work almost the same way as normal logical switches. When the VM sends
1471 such a packet, then:
1472 1. It first enters the ingress pipeline, and then egress pipe‐
1474 line of the source localnet logical switch datapath. It then
1475 enters the ingress pipeline of the logical router datapath
1476 via the logical router port in the source chassis.
1477 2. Routing decision is taken.
1479 3. From the router datapath, packet enters the ingress pipeline
1481 and then egress pipeline of the destination localnet logical
1482 switch datapath and goes out of the integration bridge to
1483 the provider bridge ( belonging to the destination logical
1484 switch) via the localnet port. While sending the packet to
1485 provider bridge, we also replace router port MAC as source
1486 MAC with a chassis unique MAC.
1487 This chassis unique MAC is configured as global ovs config
1489 on each chassis (eg. via "ovs-vsctl set open . external-ids:
1490 ovn-chassis-mac-mappings="phys:aa:bb:cc:dd:ee:$i$i""). For
1491 more details, see ovn-controller(8).
1492 If the above is not configured, then source MAC would be the
1494 router port MAC. This could create problem if we have more
1495 than one chassis. This is because, since the router port is
1496 distributed, the same (MAC,VLAN) tuple will seen by physical
1497 network from other chassis as well, which could cause these
1498 issues:
1499 • Continuous MAC moves in top-of-rack switch (ToR).
1501 • ToR dropping the traffic, which is causing continuous
1503 MAC moves.
1504 • ToR blocking the ports from which MAC moves are hap‐
1506 pening.
1507 4. The destination chassis receives the packet via the localnet
1509 port and sends it to the integration bridge. Before entering
1510 the integration bridge the source mac of the packet will be
1511 replaced with router port mac again. The packet enters the
1512 ingress pipeline and then egress pipeline of the destination
1513 localnet logical switch and finally gets delivered to the
1514 destination VM port.
1515 External traffic
1517 The following happens when a VM sends an external traffic (which re‐
1519 quires NATting) and the chassis hosting the VM doesn’t have a distrib‐
1520 uted gateway port.
1521 1. The packet first enters the ingress pipeline, and then
1523 egress pipeline of the source localnet logical switch data‐
1524 path. It then enters the ingress pipeline of the logical
1525 router datapath via the logical router port in the source
1526 chassis.
1527 2. Routing decision is taken. Since the gateway router or the
1529 distributed gateway port doesn’t reside in the source chas‐
1530 sis, the traffic is redirected to the gateway chassis via
1531 the tunnel port.
1532 3. The gateway chassis receives the packet via the tunnel port
1534 and the packet enters the egress pipeline of the logical
1535 router datapath. NAT rules are applied here. The packet then
1536 enters the ingress pipeline and then egress pipeline of the
1537 localnet logical switch datapath which provides external
1538 connectivity and finally goes out via the localnet port of
1539 the logical switch which provides external connectivity.
1540 Although this works, the VM traffic is tunnelled when sent from the
1542 compute chassis to the gateway chassis. In order for it to work prop‐
1543 erly, the MTU of the localnet logical switches must be lowered to ac‐
1544 count for the tunnel encapsulation.
1545 Centralized routing for localnet VLAN tagged logical switches connected to
1547 a Logical Router
1548 To overcome the tunnel encapsulation problem described in the previous
1549 section, OVN supports the option of enabling centralized routing for
1550 localnet VLAN tagged logical switches. CMS can configure the option op‐
1551 tions:reside-on-redirect-chassis to true for each Logical_Router_Port
1552 which connects to the localnet VLAN tagged logical switches. This
1553 causes the gateway chassis (hosting the distributed gateway port) to
1554 handle all the routing for these networks, making it centralized. It
1555 will reply to the ARP requests for the logical router port IPs.
1556 If the logical router doesn’t have a distributed gateway port connect‐
1558 ing to the localnet logical switch which provides external connectiv‐
1559 ity, or if it has more than one distributed gateway ports, then this
1560 option is ignored by OVN.
1561 The following happens when a VM sends an east-west traffic which needs
1563 to be routed:
1564 1. The packet first enters the ingress pipeline, and then
1566 egress pipeline of the source localnet logical switch data‐
1567 path and is sent out via a localnet port of the source lo‐
1568 calnet logical switch (instead of sending it to router pipe‐
1569 line).
1570 2. The gateway chassis receives the packet via a localnet port
1572 of the source localnet logical switch and sends it to the
1573 integration bridge. The packet then enters the ingress pipe‐
1574 line, and then egress pipeline of the source localnet logi‐
1575 cal switch datapath and enters the ingress pipeline of the
1576 logical router datapath.
1577 3. Routing decision is taken.
1579 4. From the router datapath, packet enters the ingress pipeline
1581 and then egress pipeline of the destination localnet logical
1582 switch datapath. It then goes out of the integration bridge
1583 to the provider bridge ( belonging to the destination logi‐
1584 cal switch) via a localnet port.
1585 5. The destination chassis receives the packet via a localnet
1587 port and sends it to the integration bridge. The packet en‐
1588 ters the ingress pipeline and then egress pipeline of the
1589 destination localnet logical switch and finally delivered to
1590 the destination VM port.
1591 The following happens when a VM sends an external traffic which re‐
1593 quires NATting:
1594 1. The packet first enters the ingress pipeline, and then
1596 egress pipeline of the source localnet logical switch data‐
1597 path and is sent out via a localnet port of the source lo‐
1598 calnet logical switch (instead of sending it to router pipe‐
1599 line).
1600 2. The gateway chassis receives the packet via a localnet port
1602 of the source localnet logical switch and sends it to the
1603 integration bridge. The packet then enters the ingress pipe‐
1604 line, and then egress pipeline of the source localnet logi‐
1605 cal switch datapath and enters the ingress pipeline of the
1606 logical router datapath.
1607 3. Routing decision is taken and NAT rules are applied.
1609 4. From the router datapath, packet enters the ingress pipeline
1611 and then egress pipeline of the localnet logical switch
1612 datapath which provides external connectivity. It then goes
1613 out of the integration bridge to the provider bridge (be‐
1614 longing to the logical switch which provides external con‐
1615 nectivity) via a localnet port.
1616 The following happens for the reverse external traffic.
1618 1. The gateway chassis receives the packet from a localnet port
1620 of the logical switch which provides external connectivity.
1621 The packet then enters the ingress pipeline and then egress
1622 pipeline of the localnet logical switch (which provides ex‐
1623 ternal connectivity). The packet then enters the ingress
1624 pipeline of the logical router datapath.
1625 2. The ingress pipeline of the logical router datapath applies
1627 the unNATting rules. The packet then enters the ingress
1628 pipeline and then egress pipeline of the source localnet
1629 logical switch. Since the source VM doesn’t reside in the
1630 gateway chassis, the packet is sent out via a localnet port
1631 of the source logical switch.
1632 3. The source chassis receives the packet via a localnet port
1634 and sends it to the integration bridge. The packet enters
1635 the ingress pipeline and then egress pipeline of the source
1636 localnet logical switch and finally gets delivered to the
1637 source VM port.
1638 As an alternative to reside-on-redirect-chassis, OVN supports VLAN-
1640 based redirection. Whereas reside-on-redirect-chassis centralizes all
1641 router functionality, VLAN-based redirection only changes how OVN redi‐
1642 rects packets to the gateway chassis. By setting options:redirect-type
1643 to bridged on a distributed gateway port, OVN redirects packets to the
1644 gateway chassis using the localnet port of the router’s peer logical
1645 switch, instead of a tunnel.
1646 If the logical router doesn’t have a distributed gateway port connect‐
1648 ing to the localnet logical switch which provides external connectiv‐
1649 ity, or if it has more than one distributed gateway ports, then this
1650 option is ignored by OVN.
1651 Following happens for bridged redirection:
1653 1. On compute chassis, packet passes though logical router’s
1655 ingress pipeline.
1656 2. If logical outport is gateway chassis attached router port
1658 then packet is "redirected" to gateway chassis using peer
1659 logical switch’s localnet port.
1660 3. This redirected packet has destination mac as router port
1662 mac (the one to which gateway chassis is attached). Its VLAN
1663 id is that of localnet port (peer logical switch of the log‐
1664 ical router port).
1665 4. On the gateway chassis packet will enter the logical router
1667 pipeline again and this time it will passthrough egress
1668 pipeline as well.
1669 5. Reverse traffic packet flows stays the same.
1671 Some guidelines and expections with bridged redirection:
1673 1. Since router port mac is destination mac, hence it has to be
1675 ensured that physical network learns it on ONLY from the
1676 gateway chassis. Which means that ovn-chassis-mac-mappings
1677 should be configure on all the compute nodes, so that physi‐
1678 cal network never learn router port mac from compute nodes.
1679 2. Since packet enters logical router ingress pipeline twice
1681 (once on compute chassis and again on gateway chassis),
1682 hence ttl will be decremented twice.
1683 3. Default redirection type continues to be overlay. User can
1685 switch the redirect-type between bridged and overlay by
1686 changing the value of options:redirect-type
1687 Life Cycle of a VTEP gateway
1689 A gateway is a chassis that forwards traffic between the OVN-managed
1690 part of a logical network and a physical VLAN, extending a tunnel-based
1691 logical network into a physical network.
1692 The steps below refer often to details of the OVN and VTEP database
1694 schemas. Please see ovn-sb(5), ovn-nb(5) and vtep(5), respectively, for
1695 the full story on these databases.
1696 1. A VTEP gateway’s life cycle begins with the administrator
1698 registering the VTEP gateway as a Physical_Switch table en‐
1699 try in the VTEP database. The ovn-controller-vtep connected
1700 to this VTEP database, will recognize the new VTEP gateway
1701 and create a new Chassis table entry for it in the
1702 OVN_Southbound database.
1703 2. The administrator can then create a new Logical_Switch table
1705 entry, and bind a particular vlan on a VTEP gateway’s port
1706 to any VTEP logical switch. Once a VTEP logical switch is
1707 bound to a VTEP gateway, the ovn-controller-vtep will detect
1708 it and add its name to the vtep_logical_switches column of
1709 the Chassis table in the OVN_Southbound database. Note, the
1710 tunnel_key column of VTEP logical switch is not filled at
1711 creation. The ovn-controller-vtep will set the column when
1712 the corresponding vtep logical switch is bound to an OVN
1713 logical network.
1714 3. Now, the administrator can use the CMS to add a VTEP logical
1716 switch to the OVN logical network. To do that, the CMS must
1717 first create a new Logical_Switch_Port table entry in the
1718 OVN_Northbound database. Then, the type column of this entry
1719 must be set to "vtep". Next, the vtep-logical-switch and
1720 vtep-physical-switch keys in the options column must also be
1721 specified, since multiple VTEP gateways can attach to the
1722 same VTEP logical switch. Next, the addresses column of this
1723 logical port must be set to "unknown", it will add a prior‐
1724 ity 0 entry in "ls_in_l2_lkup" stage of logical switch
1725 ingress pipeline. So, traffic with unrecorded mac by OVN
1726 would go through the Logical_Switch_Port to physical net‐
1727 work.
1728 4. The newly created logical port in the OVN_Northbound data‐
1730 base and its configuration will be passed down to the
1731 OVN_Southbound database as a new Port_Binding table entry.
1732 The ovn-controller-vtep will recognize the change and bind
1733 the logical port to the corresponding VTEP gateway chassis.
1734 Configuration of binding the same VTEP logical switch to a
1735 different OVN logical networks is not allowed and a warning
1736 will be generated in the log.
1737 5. Beside binding to the VTEP gateway chassis, the ovn-con‐
1739 troller-vtep will update the tunnel_key column of the VTEP
1740 logical switch to the corresponding Datapath_Binding table
1741 entry’s tunnel_key for the bound OVN logical network.
1742 6. Next, the ovn-controller-vtep will keep reacting to the con‐
1744 figuration change in the Port_Binding in the OVN_Northbound
1745 database, and updating the Ucast_Macs_Remote table in the
1746 VTEP database. This allows the VTEP gateway to understand
1747 where to forward the unicast traffic coming from the ex‐
1748 tended external network.
1749 7. Eventually, the VTEP gateway’s life cycle ends when the ad‐
1751 ministrator unregisters the VTEP gateway from the VTEP data‐
1752 base. The ovn-controller-vtep will recognize the event and
1753 remove all related configurations (Chassis table entry and
1754 port bindings) in the OVN_Southbound database.
1755 8. When the ovn-controller-vtep is terminated, all related con‐
1757 figurations in the OVN_Southbound database and the VTEP
1758 database will be cleaned, including Chassis table entries
1759 for all registered VTEP gateways and their port bindings,
1760 and all Ucast_Macs_Remote table entries and the Logi‐
1761 cal_Switch tunnel keys.
1762 OVN Deployments Interconnection
1764 It is not uncommon for an operator to deploy multiple OVN clusters, for
1765 two main reasons. Firstly, an operator may prefer to deploy one OVN
1766 cluster for each availability zone, e.g. in different physical regions,
1767 to avoid single point of failure. Secondly, there is always an upper
1768 limit for a single OVN control plane to scale.
1769 Although the control planes of the different availability zone (AZ)s
1771 are independent from each other, the workloads from different AZs may
1772 need to communicate across the zones. The OVN interconnection feature
1773 provides a native way to interconnect different AZs by L3 routing
1774 through transit overlay networks between logical routers of different
1775 AZs.
1776 A global OVN Interconnection Northbound database is introduced for the
1778 operator (probably through CMS systems) to configure transit logical
1779 switches that connect logical routers from different AZs. A transit
1780 switch is similar to a regular logical switch, but it is used for in‐
1781 terconnection purpose only. Typically, each transit switch can be used
1782 to connect all logical routers that belong to same tenant across all
1783 AZs.
1784 A dedicated daemon process ovn-ic, OVN interconnection controller, in
1786 each AZ will consume this data and populate corresponding logical
1787 switches to their own northbound databases for each AZ, so that logical
1788 routers can be connected to the transit switch by creating patch port
1789 pairs in their northbound databases. Any router ports connected to the
1790 transit switches are considered interconnection ports, which will be
1791 exchanged between AZs.
1792 Physically, when workloads from different AZs communicate, packets need
1794 to go through multiple hops: source chassis, source gateway, destina‐
1795 tion gateway and destination chassis. All these hops are connected
1796 through tunnels so that the packets never leave overlay networks. A
1797 distributed gateway port is required to connect the logical router to a
1798 transit switch, with a gateway chassis specified, so that the traffic
1799 can be forwarded through the gateway chassis.
1800 A global OVN Interconnection Southbound database is introduced for ex‐
1802 changing control plane information between the AZs. The data in this
1803 database is populated and consumed by the ovn-ic, of each AZ. The main
1804 information in this database includes:
1805 • Datapath bindings for transit switches, which mainly con‐
1807 tains the tunnel keys generated for each transit switch.
1808 Separate key ranges are reserved for transit switches so
1809 that they will never conflict with any tunnel keys lo‐
1810 cally assigned for datapaths within each AZ.
1811 • Availability zones, which are registered by ovn-ic from
1813 each AZ.
1814 • Gateways. Each AZ specifies chassises that are supposed
1816 to work as interconnection gateways, and the ovn-ic will
1817 populate this information to the interconnection south‐
1818 bound DB. The ovn-ic from all the other AZs will learn
1819 the gateways and populate to their own southbound DB as a
1820 chassis.
1821 • Port bindings for logical switch ports created on the
1823 transit switch. Each AZ maintains their logical router to
1824 transit switch connections independently, but ovn-ic au‐
1825 tomatically populates local port bindings on transit
1826 switches to the global interconnection southbound DB, and
1827 learns remote port bindings from other AZs back to its
1828 own northbound and southbound DBs, so that logical flows
1829 can be produced and then translated to OVS flows locally,
1830 which finally enables data plane communication.
1831 • Routes that are advertised between different AZs. If en‐
1833 abled, routes are automatically exchanged by ovn-ic. Both
1834 static routes and directly connected subnets are adver‐
1835 tised. Options in options column of the NB_Global table
1836 of OVN_NB database control the behavior of route adver‐
1837 tisement, such as enable/disable the advertising/learning
1838 routes, whether default routes are advertised/learned,
1839 and blacklisted CIDRs. See ovn-nb(5) for more details.
1840 The tunnel keys for transit switch datapaths and related port bindings
1842 must be agreed across all AZs. This is ensured by generating and stor‐
1843 ing the keys in the global interconnection southbound database. Any
1844 ovn-ic from any AZ can allocate the key, but race conditions are solved
1845 by enforcing unique index for the column in the database.
1846 Once each AZ’s NB and SB databases are populated with interconnection
1848 switches and ports, and agreed upon the tunnel keys, data plane commu‐
1849 nication between the AZs are established.
1850 When VXLAN tunneling is enabled in an OVN cluster, due to the limited
1852 range available for VNIs, Interconnection feature is not supported.
1853 A day in the life of a packet crossing AZs
1855 1. An IP packet is sent out from a VIF on a hypervisor (HV1) of
1857 AZ1, with destination IP belonging to a VIF in AZ2.
1858 2. In HV1’s OVS flow tables, the packet goes through logical
1860 switch and logical router pipelines, and in a logical router
1861 pipeline, the routing stage finds out the next hop for the
1862 destination IP, which belongs to a remote logical router
1863 port in AZ2, and the output port, which is a chassis-redi‐
1864 rect port located on an interconnection gateway (GW1 in
1865 AZ1), so HV1 sends the packet to GW1 through tunnel.
1866 3. On GW1, it continues with the logical router pipe line and
1868 switches to the transit switch’s pipeline through the peer
1869 port of the chassis redirect port. In the transit switch’s
1870 pipeline it outputs to the remote logical port which is lo‐
1871 cated on a gateway (GW2) in AZ2, so the GW1 sends the packet
1872 to GW2 in tunnel.
1873 4. On GW2, it continues with the transit switch pipeline and
1875 switches to the logical router pipeline through the peer
1876 port, which is a chassis redirect port that is located on
1877 GW2. The logical router pipeline then forwards the packet to
1878 relevant logical pipelines according to the destination IP
1879 address, and figures out the MAC and location of the desti‐
1880 nation VIF port - a hypervisor (HV2). The GW2 then sends the
1881 packet to HV2 in tunnel.
1882 5. On HV2, the packet is delivered to the final destination VIF
1884 port by the logical switch egress pipeline, just the same
1885 way as for intra-AZ communications.
1886 Native OVN services for external logical ports
1888 To support OVN native services (like DHCP/IPv6 RA/DNS lookup) to the
1889 cloud resources which are external, OVN supports external logical
1890 ports.
1891 Below are some of the use cases where external ports can be used.
1893 • VMs connected to SR-IOV nics - Traffic from these VMs by
1895 passes the kernel stack and local ovn-controller do not
1896 bind these ports and cannot serve the native services.
1897 • When CMS supports provisioning baremetal servers.
1899 OVN will provide the native services if CMS has done the below configu‐
1901 ration in the OVN Northbound Database.
1902 • A row is created in Logical_Switch_Port, configuring the
1904 addresses column and setting the type to external.
1905 • ha_chassis_group column is configured.
1907 • The HA chassis which belongs to the HA chassis group has
1909 the ovn-bridge-mappings configured and has proper L2 con‐
1910 nectivity so that it can receive the DHCP and other re‐
1911 lated request packets from these external resources.
1912 • The Logical_Switch of this port has a localnet port.
1914 • Native OVN services are enabled by configuring the DHCP
1916 and other options like the way it is done for the normal
1917 logical ports.
1918 It is recommended to use the same HA chassis group for all the external
1920 ports of a logical switch. Otherwise, the physical switch might see MAC
1921 flap issue when different chassis provide the native services. For ex‐
1922 ample when supporting native DHCPv4 service, DHCPv4 server mac (config‐
1923 ured in options:server_mac column in table DHCP_Options) originating
1924 from different ports can cause MAC flap issue. The MAC of the logical
1925 router IP(s) can also flap if the same HA chassis group is not set for
1926 all the external ports of a logical switch.
1927
1929 Role-Based Access Controls for the Southbound DB
1930 In order to provide additional security against the possibility of an
1931 OVN chassis becoming compromised in such a way as to allow rogue soft‐
1932 ware to make arbitrary modifications to the southbound database state
1933 and thus disrupt the OVN network, role-based access controls (see
1934 ovsdb-server(1) for additional details) are provided for the southbound
1935 database.
1936 The implementation of role-based access controls (RBAC) requires the
1938 addition of two tables to an OVSDB schema: the RBAC_Role table, which
1939 is indexed by role name and maps the the names of the various tables
1940 that may be modifiable for a given role to individual rows in a permis‐
1941 sions table containing detailed permission information for that role,
1942 and the permission table itself which consists of rows containing the
1943 following information:
1944 Table Name
1946 The name of the associated table. This column exists pri‐
1947 marily as an aid for humans reading the contents of this
1948 table.
1949 Auth Criteria
1951 A set of strings containing the names of columns (or col‐
1952 umn:key pairs for columns containing string:string maps).
1953 The contents of at least one of the columns or column:key
1954 values in a row to be modified, inserted, or deleted must
1955 be equal to the ID of the client attempting to act on the
1956 row in order for the authorization check to pass. If the
1957 authorization criteria is empty, authorization checking
1958 is disabled and all clients for the role will be treated
1959 as authorized.
1960 Insert/Delete
1962 Row insertion/deletion permission; boolean value indicat‐
1963 ing whether insertion and deletion of rows is allowed for
1964 the associated table. If true, insertion and deletion of
1965 rows is allowed for authorized clients.
1966 Updatable Columns
1968 A set of strings containing the names of columns or col‐
1969 umn:key pairs that may be updated or mutated by autho‐
1970 rized clients. Modifications to columns within a row are
1971 only permitted when the authorization check for the
1972 client passes and all columns to be modified are included
1973 in this set of modifiable columns.
1974 RBAC configuration for the OVN southbound database is maintained by
1976 ovn-northd. With RBAC enabled, modifications are only permitted for the
1977 Chassis, Encap, Port_Binding, and MAC_Binding tables, and are re‐
1978 stricted as follows:
1979 Chassis
1981 Authorization: client ID must match the chassis name.
1982 Insert/Delete: authorized row insertion and deletion are
1984 permitted.
1985 Update: The columns nb_cfg, external_ids, encaps, and
1987 vtep_logical_switches may be modified when authorized.
1988 Encap Authorization: client ID must match the chassis name.
1990 Insert/Delete: row insertion and row deletion are permit‐
1992 ted.
1993 Update: The columns type, options, and ip can be modi‐
1995 fied.
1996 Port_Binding
1998 Authorization: disabled (all clients are considered au‐
1999 thorized. A future enhancement may add columns (or keys
2000 to external_ids) in order to control which chassis are
2001 allowed to bind each port.
2002 Insert/Delete: row insertion/deletion are not permitted
2004 (ovn-northd maintains rows in this table.
2005 Update: Only modifications to the chassis column are per‐
2007 mitted.
2008 MAC_Binding
2010 Authorization: disabled (all clients are considered to be
2011 authorized).
2012 Insert/Delete: row insertion/deletion are permitted.
2014 Update: The columns logical_port, ip, mac, and datapath
2016 may be modified by ovn-controller.
2017 IGMP_Group
2019 Authorization: disabled (all clients are considered to be
2020 authorized).
2021 Insert/Delete: row insertion/deletion are permitted.
2023 Update: The columns address, chassis, datapath, and ports
2025 may be modified by ovn-controller.
2026 Enabling RBAC for ovn-controller connections to the southbound database
2028 requires the following steps:
2029 1. Creating SSL certificates for each chassis with the certifi‐
2031 cate CN field set to the chassis name (e.g. for a chassis
2032 with external-ids:system-id=chassis-1, via the command
2033 "ovs-pki -u req+sign chassis-1 switch").
2034 2. Configuring each ovn-controller to use SSL when connecting
2036 to the southbound database (e.g. via "ovs-vsctl set open .
2037 external-ids:ovn-remote=ssl:x.x.x.x:6642").
2038 3. Configuring a southbound database SSL remote with "ovn-con‐
2040 troller" role (e.g. via "ovn-sbctl set-connection
2041 role=ovn-controller pssl:6642").
2042 Encrypt Tunnel Traffic with IPsec
2044 OVN tunnel traffic goes through physical routers and switches. These
2045 physical devices could be untrusted (devices in public network) or
2046 might be compromised. Enabling encryption to the tunnel traffic can
2047 prevent the traffic data from being monitored and manipulated.
2048 The tunnel traffic is encrypted with IPsec. The CMS sets the ipsec col‐
2050 umn in the northbound NB_Global table to enable or disable IPsec encry‐
2051 tion. If ipsec is true, all OVN tunnels will be encrypted. If ipsec is
2052 false, no OVN tunnels will be encrypted.
2053 When CMS updates the ipsec column in the northbound NB_Global table,
2055 ovn-northd copies the value to the ipsec column in the southbound
2056 SB_Global table. ovn-controller in each chassis monitors the southbound
2057 database and sets the options of the OVS tunnel interface accordingly.
2058 OVS tunnel interface options are monitored by the ovs-monitor-ipsec
2059 daemon which configures IKE daemon to set up IPsec connections.
2060 Chassis authenticates each other by using certificate. The authentica‐
2062 tion succeeds if the other end in tunnel presents a certificate signed
2063 by a trusted CA and the common name (CN) matches the expected chassis
2064 name. The SSL certificates used in role-based access controls (RBAC)
2065 can be used in IPsec. Or use ovs-pki to create different certificates.
2066 The certificate is required to be x.509 version 3, and with CN field
2067 and subjectAltName field being set to the chassis name.
2068 The CA certificate, chassis certificate and private key are required to
2070 be installed in each chassis before enabling IPsec. Please see
2071 ovs-vswitchd.conf.db(5) for setting up CA based IPsec authentication.
2072
2074 Tunnel Encapsulations
2075 In general, OVN annotates logical network packets that it sends from
2076 one hypervisor to another with the following three pieces of metadata,
2077 which are encoded in an encapsulation-specific fashion:
2078 • 24-bit logical datapath identifier, from the tunnel_key
2080 column in the OVN Southbound Datapath_Binding table.
2081 • 15-bit logical ingress port identifier. ID 0 is reserved
2083 for internal use within OVN. IDs 1 through 32767, inclu‐
2084 sive, may be assigned to logical ports (see the tun‐
2085 nel_key column in the OVN Southbound Port_Binding table).
2086 • 16-bit logical egress port identifier. IDs 0 through
2088 32767 have the same meaning as for logical ingress ports.
2089 IDs 32768 through 65535, inclusive, may be assigned to
2090 logical multicast groups (see the tunnel_key column in
2091 the OVN Southbound Multicast_Group table).
2092 When VXLAN is enabled on any hypervisor in a cluster, datapath and
2094 egress port identifier ranges are reduced to 12-bits. This is done be‐
2095 cause only STT and Geneve provide the large space for metadata (over 32
2096 bits per packet). To accommodate for VXLAN, 24 bits available are split
2097 as follows:
2098 • 12-bit logical datapath identifier, derived from the tun‐
2100 nel_key column in the OVN Southbound Datapath_Binding ta‐
2101 ble.
2102 • 12-bit logical egress port identifier. IDs 0 through 2047
2104 are used for unicast output ports. IDs 2048 through 4095,
2105 inclusive, may be assigned to logical multicast groups
2106 (see the tunnel_key column in the OVN Southbound Multi‐
2107 cast_Group table). For multicast group tunnel keys, a
2108 special mapping scheme is used to internally transform
2109 from internal OVN 16-bit keys to 12-bit values before
2110 sending packets through a VXLAN tunnel, and back from
2111 12-bit tunnel keys to 16-bit values when receiving pack‐
2112 ets from a VXLAN tunnel.
2113 • No logical ingress port identifier.
2115 The limited space available for metadata when VXLAN tunnels are enabled
2117 in a cluster put the following functional limitations onto features
2118 available to users:
2119 • The maximum number of networks is reduced to 4096.
2121 • The maximum number of ports per network is reduced to
2123 2048.
2124 • ACLs matching against logical ingress port identifiers
2126 are not supported.
2127 • OVN interconnection feature is not supported.
2129 In addition to functional limitations described above, the following
2131 should be considered before enabling it in your cluster:
2132 • STT and Geneve use randomized UDP or TCP source ports
2134 that allows efficient distribution among multiple paths
2135 in environments that use ECMP in their underlay.
2136 • NICs are available to offload STT and Geneve encapsula‐
2138 tion and decapsulation.
2139 Due to its flexibility, the preferred encapsulation between hypervisors
2141 is Geneve. For Geneve encapsulation, OVN transmits the logical datapath
2142 identifier in the Geneve VNI. OVN transmits the logical ingress and
2143 logical egress ports in a TLV with class 0x0102, type 0x80, and a
2144 32-bit value encoded as follows, from MSB to LSB:
2145 1 15 16
2147 +---+------------+-----------+
2148 |rsv|ingress port|egress port|
2149 +---+------------+-----------+
2150 0
2151 Environments whose NICs lack Geneve offload may prefer STT encapsula‐
2154 tion for performance reasons. For STT encapsulation, OVN encodes all
2155 three pieces of logical metadata in the STT 64-bit tunnel ID as fol‐
2156 lows, from MSB to LSB:
2157 9 15 16 24
2159 +--------+------------+-----------+--------+
2160 |reserved|ingress port|egress port|datapath|
2161 +--------+------------+-----------+--------+
2162 0
2163 For connecting to gateways, in addition to Geneve and STT, OVN supports
2166 VXLAN, because only VXLAN support is common on top-of-rack (ToR)
2167 switches. Currently, gateways have a feature set that matches the capa‐
2168 bilities as defined by the VTEP schema, so fewer bits of metadata are
2169 necessary. In the future, gateways that do not support encapsulations
2170 with large amounts of metadata may continue to have a reduced feature
2171 set.
2172OVN 23.09.2 OVN Architecture ovn-architecture(7)