1ovn-architecture(7) OVN Manual ovn-architecture(7)
2
6 ovn-architecture - Open Virtual Network architecture
7
9 OVN, the Open Virtual Network, is a system to support logical network
10 abstraction in virtual machine and container environments. OVN comple‐
11 ments the existing capabilities of OVS to add native support for logi‐
12 cal network abstractions, such as logical L2 and L3 overlays and secu‐
13 rity groups. Services such as DHCP are also desirable features. Just
14 like OVS, OVN’s design goal is to have a production-quality implementa‐
15 tion that can operate at significant scale.
16 A physical network comprises physical wires, switches, and routers. A
18 virtual network extends a physical network into a hypervisor or con‐
19 tainer platform, bridging VMs or containers into the physical network.
20 An OVN logical network is a network implemented in software that is in‐
21 sulated from physical (and thus virtual) networks by tunnels or other
22 encapsulations. This allows IP and other address spaces used in logical
23 networks to overlap with those used on physical networks without caus‐
24 ing conflicts. Logical network topologies can be arranged without re‐
25 gard for the topologies of the physical networks on which they run.
26 Thus, VMs that are part of a logical network can migrate from one phys‐
27 ical machine to another without network disruption. See Logical Net‐
28 works, below, for more information.
29 The encapsulation layer prevents VMs and containers connected to a log‐
31 ical network from communicating with nodes on physical networks. For
32 clustering VMs and containers, this can be acceptable or even desir‐
33 able, but in many cases VMs and containers do need connectivity to
34 physical networks. OVN provides multiple forms of gateways for this
35 purpose. See Gateways, below, for more information.
36 An OVN deployment consists of several components:
38 • A Cloud Management System (CMS), which is OVN’s ultimate
40 client (via its users and administrators). OVN integra‐
41 tion requires installing a CMS-specific plugin and re‐
42 lated software (see below). OVN initially targets Open‐
43 Stack as CMS.
44 We generally speak of ``the’’ CMS, but one can imagine
46 scenarios in which multiple CMSes manage different parts
47 of an OVN deployment.
48 • An OVN Database physical or virtual node (or, eventually,
50 cluster) installed in a central location.
51 • One or more (usually many) hypervisors. Hypervisors must
53 run Open vSwitch and implement the interface described in
54 Documentation/topics/integration.rst in the Open vSwitch
55 source tree. Any hypervisor platform supported by Open
56 vSwitch is acceptable.
57 • Zero or more gateways. A gateway extends a tunnel-based
59 logical network into a physical network by bidirection‐
60 ally forwarding packets between tunnels and a physical
61 Ethernet port. This allows non-virtualized machines to
62 participate in logical networks. A gateway may be a phys‐
63 ical host, a virtual machine, or an ASIC-based hardware
64 switch that supports the vtep(5) schema.
65 Hypervisors and gateways are together called transport
67 node or chassis.
68 The diagram below shows how the major components of OVN and related
70 software interact. Starting at the top of the diagram, we have:
71 • The Cloud Management System, as defined above.
73 • The OVN/CMS Plugin is the component of the CMS that in‐
75 terfaces to OVN. In OpenStack, this is a Neutron plugin.
76 The plugin’s main purpose is to translate the CMS’s no‐
77 tion of logical network configuration, stored in the
78 CMS’s configuration database in a CMS-specific format,
79 into an intermediate representation understood by OVN.
80 This component is necessarily CMS-specific, so a new
82 plugin needs to be developed for each CMS that is inte‐
83 grated with OVN. All of the components below this one in
84 the diagram are CMS-independent.
85 • The OVN Northbound Database receives the intermediate
87 representation of logical network configuration passed
88 down by the OVN/CMS Plugin. The database schema is meant
89 to be ``impedance matched’’ with the concepts used in a
90 CMS, so that it directly supports notions of logical
91 switches, routers, ACLs, and so on. See ovn-nb(5) for de‐
92 tails.
93 The OVN Northbound Database has only two clients: the
95 OVN/CMS Plugin above it and ovn-northd below it.
96 • ovn-northd(8) connects to the OVN Northbound Database
98 above it and the OVN Southbound Database below it. It
99 translates the logical network configuration in terms of
100 conventional network concepts, taken from the OVN North‐
101 bound Database, into logical datapath flows in the OVN
102 Southbound Database below it.
103 • The OVN Southbound Database is the center of the system.
105 Its clients are ovn-northd(8) above it and ovn-con‐
106 troller(8) on every transport node below it.
107 The OVN Southbound Database contains three kinds of data:
109 Physical Network (PN) tables that specify how to reach
110 hypervisor and other nodes, Logical Network (LN) tables
111 that describe the logical network in terms of ``logical
112 datapath flows,’’ and Binding tables that link logical
113 network components’ locations to the physical network.
114 The hypervisors populate the PN and Port_Binding tables,
115 whereas ovn-northd(8) populates the LN tables.
116 OVN Southbound Database performance must scale with the
118 number of transport nodes. This will likely require some
119 work on ovsdb-server(1) as we encounter bottlenecks.
120 Clustering for availability may be needed.
121 The remaining components are replicated onto each hypervisor:
123 • ovn-controller(8) is OVN’s agent on each hypervisor and
125 software gateway. Northbound, it connects to the OVN
126 Southbound Database to learn about OVN configuration and
127 status and to populate the PN table and the Chassis col‐
128 umn in Binding table with the hypervisor’s status. South‐
129 bound, it connects to ovs-vswitchd(8) as an OpenFlow con‐
130 troller, for control over network traffic, and to the lo‐
131 cal ovsdb-server(1) to allow it to monitor and control
132 Open vSwitch configuration.
133 • ovs-vswitchd(8) and ovsdb-server(1) are conventional com‐
135 ponents of Open vSwitch.
136 CMS
138 |
139 |
140 +-----------|-----------+
141 | | |
142 | OVN/CMS Plugin |
143 | | |
144 | | |
145 | OVN Northbound DB |
146 | | |
147 | | |
148 | ovn-northd |
149 | | |
150 +-----------|-----------+
151 |
152 |
153 +-------------------+
154 | OVN Southbound DB |
155 +-------------------+
156 |
157 |
158 +------------------+------------------+
159 | | |
160 HV 1 | | HV n |
161 +---------------|---------------+ . +---------------|---------------+
162 | | | . | | |
163 | ovn-controller | . | ovn-controller |
164 | | | | . | | | |
165 | | | | | | | |
166 | ovs-vswitchd ovsdb-server | | ovs-vswitchd ovsdb-server |
167 | | | |
168 +-------------------------------+ +-------------------------------+
169 Information Flow in OVN
172 Configuration data in OVN flows from north to south. The CMS, through
173 its OVN/CMS plugin, passes the logical network configuration to
174 ovn-northd via the northbound database. In turn, ovn-northd compiles
175 the configuration into a lower-level form and passes it to all of the
176 chassis via the southbound database.
177 Status information in OVN flows from south to north. OVN currently pro‐
179 vides only a few forms of status information. First, ovn-northd popu‐
180 lates the up column in the northbound Logical_Switch_Port table: if a
181 logical port’s chassis column in the southbound Port_Binding table is
182 nonempty, it sets up to true, otherwise to false. This allows the CMS
183 to detect when a VM’s networking has come up.
184 Second, OVN provides feedback to the CMS on the realization of its con‐
186 figuration, that is, whether the configuration provided by the CMS has
187 taken effect. This feature requires the CMS to participate in a se‐
188 quence number protocol, which works the following way:
189 1. When the CMS updates the configuration in the northbound
191 database, as part of the same transaction, it increments the
192 value of the nb_cfg column in the NB_Global table. (This is
193 only necessary if the CMS wants to know when the configura‐
194 tion has been realized.)
195 2. When ovn-northd updates the southbound database based on a
197 given snapshot of the northbound database, it copies nb_cfg
198 from northbound NB_Global into the southbound database
199 SB_Global table, as part of the same transaction. (Thus, an
200 observer monitoring both databases can determine when the
201 southbound database is caught up with the northbound.)
202 3. After ovn-northd receives confirmation from the southbound
204 database server that its changes have committed, it updates
205 sb_cfg in the northbound NB_Global table to the nb_cfg ver‐
206 sion that was pushed down. (Thus, the CMS or another ob‐
207 server can determine when the southbound database is caught
208 up without a connection to the southbound database.)
209 4. The ovn-controller process on each chassis receives the up‐
211 dated southbound database, with the updated nb_cfg. This
212 process in turn updates the physical flows installed in the
213 chassis’s Open vSwitch instances. When it receives confirma‐
214 tion from Open vSwitch that the physical flows have been up‐
215 dated, it updates nb_cfg in its own Chassis record in the
216 southbound database.
217 5. ovn-northd monitors the nb_cfg column in all of the Chassis
219 records in the southbound database. It keeps track of the
220 minimum value among all the records and copies it into the
221 hv_cfg column in the northbound NB_Global table. (Thus, the
222 CMS or another observer can determine when all of the hyper‐
223 visors have caught up to the northbound configuration.)
224 Chassis Setup
226 Each chassis in an OVN deployment must be configured with an Open
227 vSwitch bridge dedicated for OVN’s use, called the integration bridge.
228 System startup scripts may create this bridge prior to starting
229 ovn-controller if desired. If this bridge does not exist when ovn-con‐
230 troller starts, it will be created automatically with the default con‐
231 figuration suggested below. The ports on the integration bridge in‐
232 clude:
233 • On any chassis, tunnel ports that OVN uses to maintain
235 logical network connectivity. ovn-controller adds, up‐
236 dates, and removes these tunnel ports.
237 • On a hypervisor, any VIFs that are to be attached to log‐
239 ical networks. For instances connected through software
240 emulated ports such as TUN/TAP or VETH pairs, the hyper‐
241 visor itself will normally create ports and plug them
242 into the integration bridge. For instances connected
243 through representor ports, typically used with hardware
244 offload, the ovn-controller may on CMS direction consult
245 a VIF plug provider for representor port lookup and plug
246 them into the integration bridge (please refer to Docu‐
247 mentation/top‐
248 ics/vif-plug-providers/vif-plug-providers.rst
249 for more information). In both cases the conventions de‐
250 scribed in Documentation/topics/integration.rst in the
251 Open vSwitch source tree is followed to ensure mapping
252 between OVN logical port and VIF. (This is pre-existing
253 integration work that has already been done on hypervi‐
254 sors that support OVS.)
255 • On a gateway, the physical port used for logical network
257 connectivity. System startup scripts add this port to the
258 bridge prior to starting ovn-controller. This can be a
259 patch port to another bridge, instead of a physical port,
260 in more sophisticated setups.
261 Other ports should not be attached to the integration bridge. In par‐
263 ticular, physical ports attached to the underlay network (as opposed to
264 gateway ports, which are physical ports attached to logical networks)
265 must not be attached to the integration bridge. Underlay physical ports
266 should instead be attached to a separate Open vSwitch bridge (they need
267 not be attached to any bridge at all, in fact).
268 The integration bridge should be configured as described below. The ef‐
270 fect of each of these settings is documented in
271 ovs-vswitchd.conf.db(5):
272 fail-mode=secure
274 Avoids switching packets between isolated logical net‐
275 works before ovn-controller starts up. See Controller
276 Failure Settings in ovs-vsctl(8) for more information.
277 other-config:disable-in-band=true
279 Suppresses in-band control flows for the integration
280 bridge. It would be unusual for such flows to show up
281 anyway, because OVN uses a local controller (over a Unix
282 domain socket) instead of a remote controller. It’s pos‐
283 sible, however, for some other bridge in the same system
284 to have an in-band remote controller, and in that case
285 this suppresses the flows that in-band control would or‐
286 dinarily set up. Refer to the documentation for more in‐
287 formation.
288 The customary name for the integration bridge is br-int, but another
290 name may be used.
291 Logical Networks
293 Logical network concepts in OVN include logical switches and logical
294 routers, the logical version of Ethernet switches and IP routers, re‐
295 spectively. Like their physical cousins, logical switches and routers
296 can be connected into sophisticated topologies. Logical switches and
297 routers are ordinarily purely logical entities, that is, they are not
298 associated or bound to any physical location, and they are implemented
299 in a distributed manner at each hypervisor that participates in OVN.
300 Logical switch ports (LSPs) are points of connectivity into and out of
302 logical switches. There are many kinds of logical switch ports. The
303 most ordinary kind represent VIFs, that is, attachment points for VMs
304 or containers. A VIF logical port is associated with the physical loca‐
305 tion of its VM, which might change as the VM migrates. (A VIF logical
306 port can be associated with a VM that is powered down or suspended.
307 Such a logical port has no location and no connectivity.)
308 Logical router ports (LRPs) are points of connectivity into and out of
310 logical routers. A LRP connects a logical router either to a logical
311 switch or to another logical router. Logical routers only connect to
312 VMs, containers, and other network nodes indirectly, through logical
313 switches.
314 Logical switches and logical routers have distinct kinds of logical
316 ports, so properly speaking one should usually talk about logical
317 switch ports or logical router ports. However, an unqualified ``logical
318 port’’ usually refers to a logical switch port.
319 When a VM sends a packet to a VIF logical switch port, the Open vSwitch
321 flow tables simulate the packet’s journey through that logical switch
322 and any other logical routers and logical switches that it might en‐
323 counter. This happens without transmitting the packet across any physi‐
324 cal medium: the flow tables implement all of the switching and routing
325 decisions and behavior. If the flow tables ultimately decide to output
326 the packet at a logical port attached to another hypervisor (or another
327 kind of transport node), then that is the time at which the packet is
328 encapsulated for physical network transmission and sent.
329 Logical Switch Port Types
331 OVN supports a number of kinds of logical switch ports. VIF ports that
333 connect to VMs or containers, described above, are the most ordinary
334 kind of LSP. In the OVN northbound database, VIF ports have an empty
335 string for their type. This section describes some of the additional
336 port types.
337 A router logical switch port connects a logical switch to a logical
339 router, designating a particular LRP as its peer.
340 A localnet logical switch port bridges a logical switch to a physical
342 VLAN. A logical switch may have one or more localnet ports. Such a log‐
343 ical switch is used in two scenarios:
344 • With one or more router logical switch ports, to attach
346 L3 gateway routers and distributed gateways to a physical
347 network.
348 • With one or more VIF logical switch ports, to attach VMs
350 or containers directly to a physical network. In this
351 case, the logical switch is not really logical, since it
352 is bridged to the physical network rather than insulated
353 from it, and therefore cannot have independent but over‐
354 lapping IP address namespaces, etc. A deployment might
355 nevertheless choose such a configuration to take advan‐
356 tage of the OVN control plane and features such as port
357 security and ACLs.
358 When a logical switch contains multiple localnet ports, the following
360 is assumed.
361 • Each chassis has a bridge mapping for one of the localnet
363 physical networks only.
364 • To facilitate interconnectivity between VIF ports of the
366 switch that are located on different chassis with differ‐
367 ent physical network connectivity, the fabric implements
368 L3 routing between these adjacent physical network seg‐
369 ments.
370 Note: nothing said above implies that a chassis cannot be plugged to
372 multiple physical networks as long as they belong to different
373 switches.
374 A localport logical switch port is a special kind of VIF logical switch
376 port. These ports are present in every chassis, not bound to any par‐
377 ticular one. Traffic to such a port will never be forwarded through a
378 tunnel, and traffic from such a port is expected to be destined only to
379 the same chassis, typically in response to a request it received. Open‐
380 Stack Neutron uses a localport port to serve metadata to VMs. A meta‐
381 data proxy process is attached to this port on every host and all VMs
382 within the same network will reach it at the same IP/MAC address with‐
383 out any traffic being sent over a tunnel. For further details, see the
384 OpenStack documentation for networking-ovn.
385 LSP types vtep and l2gateway are used for gateways. See Gateways, be‐
387 low, for more information.
388 Implementation Details
390 These concepts are details of how OVN is implemented internally. They
392 might still be of interest to users and administrators.
393 Logical datapaths are an implementation detail of logical networks in
395 the OVN southbound database. ovn-northd translates each logical switch
396 or router in the northbound database into a logical datapath in the
397 southbound database Datapath_Binding table.
398 For the most part, ovn-northd also translates each logical switch port
400 in the OVN northbound database into a record in the southbound database
401 Port_Binding table. The latter table corresponds roughly to the north‐
402 bound Logical_Switch_Port table. It has multiple types of logical port
403 bindings, of which many types correspond directly to northbound LSP
404 types. LSP types handled this way include VIF (empty string), localnet,
405 localport, vtep, and l2gateway.
406 The Port_Binding table has some types of port binding that do not cor‐
408 respond directly to logical switch port types. The common is patch port
409 bindings, known as logical patch ports. These port bindings always oc‐
410 cur in pairs, and a packet that enters on either side comes out on the
411 other. ovn-northd connects logical switches and logical routers to‐
412 gether using logical patch ports.
413 Port bindings with types vtep, l2gateway, l3gateway, and chassisredi‐
415 rect are used for gateways. These are explained in Gateways, below.
416 Gateways
418 Gateways provide limited connectivity between logical networks and
419 physical ones. They can also provide connectivity between different OVN
420 deployments. This section will focus on the former, and the latter will
421 be described in details in section OVN Deployments Interconnection.
422 OVN support multiple kinds of gateways.
424 VTEP Gateways
426 A ``VTEP gateway’’ connects an OVN logical network to a physical (or
428 virtual) switch that implements the OVSDB VTEP schema that accompanies
429 Open vSwitch. (The ``VTEP gateway’’ term is a misnomer, since a VTEP is
430 just a VXLAN Tunnel Endpoint, but it is a well established name.) See
431 Life Cycle of a VTEP gateway, below, for more information.
432 The main intended use case for VTEP gateways is to attach physical
434 servers to an OVN logical network using a physical top-of-rack switch
435 that supports the OVSDB VTEP schema.
436 L2 Gateways
438 A L2 gateway simply attaches a designated physical L2 segment available
440 on some chassis to a logical network. The physical network effectively
441 becomes part of the logical network.
442 To set up a L2 gateway, the CMS adds an l2gateway LSP to an appropriate
444 logical switch, setting LSP options to name the chassis on which it
445 should be bound. ovn-northd copies this configuration into a southbound
446 Port_Binding record. On the designated chassis, ovn-controller forwards
447 packets appropriately to and from the physical segment.
448 L2 gateway ports have features in common with localnet ports. However,
450 with a localnet port, the physical network becomes the transport be‐
451 tween hypervisors. With an L2 gateway, packets are still transported
452 between hypervisors over tunnels and the l2gateway port is only used
453 for the packets that are on the physical network. The application for
454 L2 gateways is similar to that for VTEP gateways, e.g. to add non-vir‐
455 tualized machines to a logical network, but L2 gateways do not require
456 special support from top-of-rack hardware switches.
457 L3 Gateway Routers
459 As described above under Logical Networks, ordinary OVN logical routers
461 are distributed: they are not implemented in a single place but rather
462 in every hypervisor chassis. This is a problem for stateful services
463 such as SNAT and DNAT, which need to be implemented in a centralized
464 manner.
465 To allow for this kind of functionality, OVN supports L3 gateway
467 routers, which are OVN logical routers that are implemented in a desig‐
468 nated chassis. Gateway routers are typically used between distributed
469 logical routers and physical networks. The distributed logical router
470 and the logical switches behind it, to which VMs and containers attach,
471 effectively reside on each hypervisor. The distributed router and the
472 gateway router are connected by another logical switch, sometimes re‐
473 ferred to as a ``join’’ logical switch. (OVN logical routers may be
474 connected to one another directly, without an intervening switch, but
475 the OVN implementation only supports gateway logical routers that are
476 connected to logical switches. Using a join logical switch also reduces
477 the number of IP addresses needed on the distributed router.) On the
478 other side, the gateway router connects to another logical switch that
479 has a localnet port connecting to the physical network.
480 The following diagram shows a typical situation. One or more logical
482 switches LS1, ..., LSn connect to distributed logical router LR1, which
483 in turn connects through LSjoin to gateway logical router GLR, which
484 also connects to logical switch LSlocal, which includes a localnet port
485 to attach to the physical network.
486 LSlocal
488 |
489 GLR
490 |
491 LSjoin
492 |
493 LR1
494 |
495 +----+----+
496 | | |
497 LS1 ... LSn
498 To configure an L3 gateway router, the CMS sets options:chassis in the
501 router’s northbound Logical_Router to the chassis’s name. In response,
502 ovn-northd uses a special l3gateway port binding (instead of a patch
503 binding) in the southbound database to connect the logical router to
504 its neighbors. In turn, ovn-controller tunnels packets to this port
505 binding to the designated L3 gateway chassis, instead of processing
506 them locally.
507 DNAT and SNAT rules may be associated with a gateway router, which pro‐
509 vides a central location that can handle one-to-many SNAT (aka IP mas‐
510 querading). Distributed gateway ports, described below, also support
511 NAT.
512 Distributed Gateway Ports
514 A distributed gateway port is a logical router port that is specially
516 configured to designate one distinguished chassis, called the gateway
517 chassis, for centralized processing. A distributed gateway port should
518 connect to a logical switch that has an LSP that connects externally,
519 that is, either a localnet LSP or a connection to another OVN deploy‐
520 ment (see OVN Deployments Interconnection). Packets that traverse the
521 distributed gateway port are processed without involving the gateway
522 chassis when they can be, but when needed they do take an extra hop
523 through it.
524 The following diagram illustrates the use of a distributed gateway
526 port. A number of logical switches LS1, ..., LSn connect to distributed
527 logical router LR1, which in turn connects through the distributed
528 gateway port to logical switch LSlocal that includes a localnet port to
529 attach to the physical network.
530 LSlocal
532 |
533 LR1
534 |
535 +----+----+
536 | | |
537 LS1 ... LSn
538 ovn-northd creates two southbound Port_Binding records to represent a
541 distributed gateway port, instead of the usual one. One of these is a
542 patch port binding named for the LRP, which is used for as much traffic
543 as it can. The other one is a port binding with type chassisredirect,
544 named cr-port. The chassisredirect port binding has one specialized
545 job: when a packet is output to it, the flow table causes it to be tun‐
546 neled to the gateway chassis, at which point it is automatically output
547 to the patch port binding. Thus, the flow table can output to this port
548 binding in cases where a particular task has to happen on the gateway
549 chassis. The chassisredirect port binding is not otherwise used (for
550 example, it never receives packets).
551 The CMS may configure distributed gateway ports three different ways.
553 See Distributed Gateway Ports in the documentation for Logi‐
554 cal_Router_Port in ovn-nb(5) for details.
555 Distributed gateway ports support high availability. When more than one
557 chassis is specified, OVN only uses one at a time as the gateway chas‐
558 sis. OVN uses BFD to monitor gateway connectivity, preferring the high‐
559 est-priority gateway that is online.
560 A logical router can have multiple distributed gateway ports, each con‐
562 necting different external networks. Load balancing is not yet sup‐
563 ported for logical routers with more than one distributed gateway port
564 configured.
565 Physical VLAN MTU Issues
567 Consider the preceding diagram again:
569 LSlocal
571 |
572 LR1
573 |
574 +----+----+
575 | | |
576 LS1 ... LSn
577 Suppose that each logical switch LS1, ..., LSn is bridged to a physical
580 VLAN-tagged network attached to a localnet port on LSlocal, over a dis‐
581 tributed gateway port on LR1. If a packet originating on LSi is des‐
582 tined to the external network, OVN sends it to the gateway chassis over
583 a tunnel. There, the packet traverses LR1’s logical router pipeline,
584 possibly undergoes NAT, and eventually ends up at LSlocal’s localnet
585 port. If all of the physical links in the network have the same MTU,
586 then the packet’s transit across a tunnel causes an MTU problem: tunnel
587 overhead prevents a packet that uses the full physical MTU from cross‐
588 ing the tunnel to the gateway chassis (without fragmentation).
589 OVN offers two solutions to this problem, the reside-on-redirect-chas‐
591 sis and redirect-type options. Both solutions require each logical
592 switch LS1, ..., LSn to include a localnet logical switch port LN1,
593 ..., LNn respectively, that is present on each chassis. Both cause
594 packets to be sent over the localnet ports instead of tunnels. They
595 differ in which packets-some or all-are sent this way. The most promi‐
596 nent tradeoff between these options is that reside-on-redirect-chassis
597 is easier to configure and that redirect-type performs better for east-
598 west traffic.
599 The first solution is the reside-on-redirect-chassis option for logical
601 router ports. Setting this option on a LRP from (e.g.) LS1 to LR1 dis‐
602 ables forwarding from LS1 to LR1 except on the gateway chassis. On
603 chassis other than the gateway chassis, this single change means that
604 packets that would otherwise have been forwarded to LR1 are instead
605 forwarded to LN1. The instance of LN1 on the gateway chassis then re‐
606 ceives the packet and forwards it to LR1. The packet traverses the LR1
607 logical router pipeline, possibly undergoes NAT, and eventually ends up
608 at LSlocal’s localnet port. The packet never traverses a tunnel, avoid‐
609 ing the MTU issue.
610 This option has the further consequence of centralizing ``distributed’’
612 logical router LR1, since no packets are forwarded from LS1 to LR1 on
613 any chassis other than the gateway chassis. Therefore, east-west traf‐
614 fic passes through the gateway chassis, not just north-south. (The
615 naive ``fix’’ of allowing east-west traffic to flow directly between
616 chassis over LN1 does not work because routing sets the Ethernet source
617 address to LR1’s source address. Seeing this single Ethernet source ad‐
618 dress originate from all of the chassis will confuse the physical
619 switch.)
620 Do not set the reside-on-redirect-chassis option on a distributed gate‐
622 way port. In the diagram above, it would be set on the LRPs connecting
623 LS1, ..., LSn to LR1.
624 The second solution is the redirect-type option for distributed gateway
626 ports. Setting this option to bridged causes packets that are redi‐
627 rected to the gateway chassis to go over the localnet ports instead of
628 being tunneled. This option does not change how OVN treats packets not
629 redirected to the gateway chassis.
630 The redirect-type option requires the administrator or the CMS to con‐
632 figure each participating chassis with a unique Ethernet address for
633 the logical router by setting ovn-chassis-mac-mappings in the Open
634 vSwitch database, for use by ovn-controller. This makes it more diffi‐
635 cult to configure than reside-on-redirect-chassis.
636 Set the redirect-type option on a distributed gateway port.
638 Using Distributed Gateway Ports For Scalability
640 Although the primary goal of distributed gateway ports is to provide
642 connectivity to external networks, there is a special use case for
643 scalability.
644 In some deployments, such as the ones using ovn-kubernetes, logical
646 switches are bound to individual chassises, and are connected by a dis‐
647 tributed logical router. In such deployments, the chassis level logical
648 switches are centralized on the chassis instead of distributed, which
649 means the ovn-controller on each chassis doesn’t need to process flows
650 and ports of logical switches on other chassises. However, without any
651 specific hint, ovn-controller would still process all the logical
652 switches as if they are fully distributed. In this case, distributed
653 gateway port can be very useful. The chassis level logical switches can
654 be connected to the distributed router using distributed gateway ports,
655 by setting the gateway chassis (or HA chassis groups with only a single
656 chassis in it) to the chassis that each logical switch is bound to.
657 ovn-controller would then skip processing the logical switches on all
658 the other chassises, largely improving the scalability, especially when
659 there are a big number of chassises.
660 Life Cycle of a VIF
662 Tables and their schemas presented in isolation are difficult to under‐
663 stand. Here’s an example.
664 A VIF on a hypervisor is a virtual network interface attached either to
666 a VM or a container running directly on that hypervisor (This is dif‐
667 ferent from the interface of a container running inside a VM).
668 The steps in this example refer often to details of the OVN and OVN
670 Northbound database schemas. Please see ovn-sb(5) and ovn-nb(5), re‐
671 spectively, for the full story on these databases.
672 1. A VIF’s life cycle begins when a CMS administrator creates a
674 new VIF using the CMS user interface or API and adds it to a
675 switch (one implemented by OVN as a logical switch). The CMS
676 updates its own configuration. This includes associating
677 unique, persistent identifier vif-id and Ethernet address
678 mac with the VIF.
679 2. The CMS plugin updates the OVN Northbound database to in‐
681 clude the new VIF, by adding a row to the Logi‐
682 cal_Switch_Port table. In the new row, name is vif-id, mac
683 is mac, switch points to the OVN logical switch’s Logi‐
684 cal_Switch record, and other columns are initialized appro‐
685 priately.
686 3. ovn-northd receives the OVN Northbound database update. In
688 turn, it makes the corresponding updates to the OVN South‐
689 bound database, by adding rows to the OVN Southbound data‐
690 base Logical_Flow table to reflect the new port, e.g. add a
691 flow to recognize that packets destined to the new port’s
692 MAC address should be delivered to it, and update the flow
693 that delivers broadcast and multicast packets to include the
694 new port. It also creates a record in the Binding table and
695 populates all its columns except the column that identifies
696 the chassis.
697 4. On every hypervisor, ovn-controller receives the Logi‐
699 cal_Flow table updates that ovn-northd made in the previous
700 step. As long as the VM that owns the VIF is powered off,
701 ovn-controller cannot do much; it cannot, for example, ar‐
702 range to send packets to or receive packets from the VIF,
703 because the VIF does not actually exist anywhere.
704 5. Eventually, a user powers on the VM that owns the VIF. On
706 the hypervisor where the VM is powered on, the integration
707 between the hypervisor and Open vSwitch (described in Docu‐
708 mentation/topics/integration.rst in the Open vSwitch source
709 tree) adds the VIF to the OVN integration bridge and stores
710 vif-id in external_ids:iface-id to indicate that the inter‐
711 face is an instantiation of the new VIF. (None of this code
712 is new in OVN; this is pre-existing integration work that
713 has already been done on hypervisors that support OVS.)
714 6. On the hypervisor where the VM is powered on, ovn-controller
716 notices external_ids:iface-id in the new Interface. In re‐
717 sponse, in the OVN Southbound DB, it updates the Binding ta‐
718 ble’s chassis column for the row that links the logical port
719 from external_ids: iface-id to the hypervisor. Afterward,
720 ovn-controller updates the local hypervisor’s OpenFlow ta‐
721 bles so that packets to and from the VIF are properly han‐
722 dled.
723 7. Some CMS systems, including OpenStack, fully start a VM only
725 when its networking is ready. To support this, ovn-northd
726 notices the chassis column updated for the row in Binding
727 table and pushes this upward by updating the up column in
728 the OVN Northbound database’s Logical_Switch_Port table to
729 indicate that the VIF is now up. The CMS, if it uses this
730 feature, can then react by allowing the VM’s execution to
731 proceed.
732 8. On every hypervisor but the one where the VIF resides,
734 ovn-controller notices the completely populated row in the
735 Binding table. This provides ovn-controller the physical lo‐
736 cation of the logical port, so each instance updates the
737 OpenFlow tables of its switch (based on logical datapath
738 flows in the OVN DB Logical_Flow table) so that packets to
739 and from the VIF can be properly handled via tunnels.
740 9. Eventually, a user powers off the VM that owns the VIF. On
742 the hypervisor where the VM was powered off, the VIF is
743 deleted from the OVN integration bridge.
744 10. On the hypervisor where the VM was powered off, ovn-con‐
746 troller notices that the VIF was deleted. In response, it
747 removes the Chassis column content in the Binding table for
748 the logical port.
749 11. On every hypervisor, ovn-controller notices the empty Chas‐
751 sis column in the Binding table’s row for the logical port.
752 This means that ovn-controller no longer knows the physical
753 location of the logical port, so each instance updates its
754 OpenFlow table to reflect that.
755 12. Eventually, when the VIF (or its entire VM) is no longer
757 needed by anyone, an administrator deletes the VIF using the
758 CMS user interface or API. The CMS updates its own configu‐
759 ration.
760 13. The CMS plugin removes the VIF from the OVN Northbound data‐
762 base, by deleting its row in the Logical_Switch_Port table.
763 14. ovn-northd receives the OVN Northbound update and in turn
765 updates the OVN Southbound database accordingly, by removing
766 or updating the rows from the OVN Southbound database Logi‐
767 cal_Flow table and Binding table that were related to the
768 now-destroyed VIF.
769 15. On every hypervisor, ovn-controller receives the Logi‐
771 cal_Flow table updates that ovn-northd made in the previous
772 step. ovn-controller updates OpenFlow tables to reflect the
773 update, although there may not be much to do, since the VIF
774 had already become unreachable when it was removed from the
775 Binding table in a previous step.
776 Life Cycle of a Container Interface Inside a VM
778 OVN provides virtual network abstractions by converting information
779 written in OVN_NB database to OpenFlow flows in each hypervisor. Secure
780 virtual networking for multi-tenants can only be provided if OVN con‐
781 troller is the only entity that can modify flows in Open vSwitch. When
782 the Open vSwitch integration bridge resides in the hypervisor, it is a
783 fair assumption to make that tenant workloads running inside VMs cannot
784 make any changes to Open vSwitch flows.
785 If the infrastructure provider trusts the applications inside the con‐
787 tainers not to break out and modify the Open vSwitch flows, then con‐
788 tainers can be run in hypervisors. This is also the case when contain‐
789 ers are run inside the VMs and Open vSwitch integration bridge with
790 flows added by OVN controller resides in the same VM. For both the
791 above cases, the workflow is the same as explained with an example in
792 the previous section ("Life Cycle of a VIF").
793 This section talks about the life cycle of a container interface (CIF)
795 when containers are created in the VMs and the Open vSwitch integration
796 bridge resides inside the hypervisor. In this case, even if a container
797 application breaks out, other tenants are not affected because the con‐
798 tainers running inside the VMs cannot modify the flows in the Open
799 vSwitch integration bridge.
800 When multiple containers are created inside a VM, there are multiple
802 CIFs associated with them. The network traffic associated with these
803 CIFs need to reach the Open vSwitch integration bridge running in the
804 hypervisor for OVN to support virtual network abstractions. OVN should
805 also be able to distinguish network traffic coming from different CIFs.
806 There are two ways to distinguish network traffic of CIFs.
807 One way is to provide one VIF for every CIF (1:1 model). This means
809 that there could be a lot of network devices in the hypervisor. This
810 would slow down OVS because of all the additional CPU cycles needed for
811 the management of all the VIFs. It would also mean that the entity cre‐
812 ating the containers in a VM should also be able to create the corre‐
813 sponding VIFs in the hypervisor.
814 The second way is to provide a single VIF for all the CIFs (1:many
816 model). OVN could then distinguish network traffic coming from differ‐
817 ent CIFs via a tag written in every packet. OVN uses this mechanism and
818 uses VLAN as the tagging mechanism.
819 1. A CIF’s life cycle begins when a container is spawned inside
821 a VM by the either the same CMS that created the VM or a
822 tenant that owns that VM or even a container Orchestration
823 System that is different than the CMS that initially created
824 the VM. Whoever the entity is, it will need to know the vif-
825 id that is associated with the network interface of the VM
826 through which the container interface’s network traffic is
827 expected to go through. The entity that creates the con‐
828 tainer interface will also need to choose an unused VLAN in‐
829 side that VM.
830 2. The container spawning entity (either directly or through
832 the CMS that manages the underlying infrastructure) updates
833 the OVN Northbound database to include the new CIF, by
834 adding a row to the Logical_Switch_Port table. In the new
835 row, name is any unique identifier, parent_name is the vif-
836 id of the VM through which the CIF’s network traffic is ex‐
837 pected to go through and the tag is the VLAN tag that iden‐
838 tifies the network traffic of that CIF.
839 3. ovn-northd receives the OVN Northbound database update. In
841 turn, it makes the corresponding updates to the OVN South‐
842 bound database, by adding rows to the OVN Southbound data‐
843 base’s Logical_Flow table to reflect the new port and also
844 by creating a new row in the Binding table and populating
845 all its columns except the column that identifies the chas‐
846 sis.
847 4. On every hypervisor, ovn-controller subscribes to the
849 changes in the Binding table. When a new row is created by
850 ovn-northd that includes a value in parent_port column of
851 Binding table, the ovn-controller in the hypervisor whose
852 OVN integration bridge has that same value in vif-id in ex‐
853 ternal_ids:iface-id updates the local hypervisor’s OpenFlow
854 tables so that packets to and from the VIF with the particu‐
855 lar VLAN tag are properly handled. Afterward it updates the
856 chassis column of the Binding to reflect the physical loca‐
857 tion.
858 5. One can only start the application inside the container af‐
860 ter the underlying network is ready. To support this,
861 ovn-northd notices the updated chassis column in Binding ta‐
862 ble and updates the up column in the OVN Northbound data‐
863 base’s Logical_Switch_Port table to indicate that the CIF is
864 now up. The entity responsible to start the container appli‐
865 cation queries this value and starts the application.
866 6. Eventually the entity that created and started the con‐
868 tainer, stops it. The entity, through the CMS (or directly)
869 deletes its row in the Logical_Switch_Port table.
870 7. ovn-northd receives the OVN Northbound update and in turn
872 updates the OVN Southbound database accordingly, by removing
873 or updating the rows from the OVN Southbound database Logi‐
874 cal_Flow table that were related to the now-destroyed CIF.
875 It also deletes the row in the Binding table for that CIF.
876 8. On every hypervisor, ovn-controller receives the Logi‐
878 cal_Flow table updates that ovn-northd made in the previous
879 step. ovn-controller updates OpenFlow tables to reflect the
880 update.
881 Architectural Physical Life Cycle of a Packet
883 This section describes how a packet travels from one virtual machine or
884 container to another through OVN. This description focuses on the phys‐
885 ical treatment of a packet; for a description of the logical life cycle
886 of a packet, please refer to the Logical_Flow table in ovn-sb(5).
887 This section mentions several data and metadata fields, for clarity
889 summarized here:
890 tunnel key
892 When OVN encapsulates a packet in Geneve or another tun‐
893 nel, it attaches extra data to it to allow the receiving
894 OVN instance to process it correctly. This takes differ‐
895 ent forms depending on the particular encapsulation, but
896 in each case we refer to it here as the ``tunnel key.’’
897 See Tunnel Encapsulations, below, for details.
898 logical datapath field
900 A field that denotes the logical datapath through which a
901 packet is being processed. OVN uses the field that Open‐
902 Flow 1.1+ simply (and confusingly) calls ``metadata’’ to
903 store the logical datapath. (This field is passed across
904 tunnels as part of the tunnel key.)
905 logical input port field
907 A field that denotes the logical port from which the
908 packet entered the logical datapath. OVN stores this in
909 Open vSwitch extension register number 14.
910 Geneve and STT tunnels pass this field as part of the
912 tunnel key. Ramp switch VXLAN tunnels do not explicitly
913 carry a logical input port, but since they are used to
914 communicate with gateways that from OVN’s perspective
915 consist of only a single logical port, so that OVN can
916 set the logical input port field to this one on ingress
917 to the OVN logical pipeline. As for regular VXLAN tun‐
918 nels, they don’t carry input port field at all. This puts
919 additional limitations on cluster capabilities that are
920 described in Tunnel Encapsulations section.
921 logical output port field
923 A field that denotes the logical port from which the
924 packet will leave the logical datapath. This is initial‐
925 ized to 0 at the beginning of the logical ingress pipe‐
926 line. OVN stores this in Open vSwitch extension register
927 number 15.
928 Geneve, STT and regular VXLAN tunnels pass this field as
930 part of the tunnel key. Ramp switch VXLAN tunnels do not
931 transmit the logical output port field, and since they do
932 not carry a logical output port field in the tunnel key,
933 when a packet is received from ramp switch VXLAN tunnel
934 by an OVN hypervisor, the packet is resubmitted to table
935 8 to determine the output port(s); when the packet
936 reaches table 37, these packets are resubmitted to table
937 38 for local delivery by checking a MLF_RCV_FROM_RAMP
938 flag, which is set when the packet arrives from a ramp
939 tunnel.
940 conntrack zone field for logical ports
942 A field that denotes the connection tracking zone for
943 logical ports. The value only has local significance and
944 is not meaningful between chassis. This is initialized to
945 0 at the beginning of the logical ingress pipeline. OVN
946 stores this in Open vSwitch extension register number 13.
947 conntrack zone fields for routers
949 Fields that denote the connection tracking zones for
950 routers. These values only have local significance and
951 are not meaningful between chassis. OVN stores the zone
952 information for north to south traffic (for DNATting or
953 ECMP symmetric replies) in Open vSwitch extension regis‐
954 ter number 11 and zone information for south to north
955 traffic (for SNATing) in Open vSwitch extension register
956 number 12.
957 logical flow flags
959 The logical flags are intended to handle keeping context
960 between tables in order to decide which rules in subse‐
961 quent tables are matched. These values only have local
962 significance and are not meaningful between chassis. OVN
963 stores the logical flags in Open vSwitch extension regis‐
964 ter number 10.
965 VLAN ID
967 The VLAN ID is used as an interface between OVN and con‐
968 tainers nested inside a VM (see Life Cycle of a container
969 interface inside a VM, above, for more information).
970 Initially, a VM or container on the ingress hypervisor sends a packet
972 on a port attached to the OVN integration bridge. Then:
973 1. OpenFlow table 0 performs physical-to-logical translation.
975 It matches the packet’s ingress port. Its actions annotate
976 the packet with logical metadata, by setting the logical
977 datapath field to identify the logical datapath that the
978 packet is traversing and the logical input port field to
979 identify the ingress port. Then it resubmits to table 8 to
980 enter the logical ingress pipeline.
981 Packets that originate from a container nested within a VM
983 are treated in a slightly different way. The originating
984 container can be distinguished based on the VIF-specific
985 VLAN ID, so the physical-to-logical translation flows addi‐
986 tionally match on VLAN ID and the actions strip the VLAN
987 header. Following this step, OVN treats packets from con‐
988 tainers just like any other packets.
989 Table 0 also processes packets that arrive from other chas‐
991 sis. It distinguishes them from other packets by ingress
992 port, which is a tunnel. As with packets just entering the
993 OVN pipeline, the actions annotate these packets with logi‐
994 cal datapath metadata. For tunnel types that support it,
995 they are also annotated with logical ingress port metadata.
996 In addition, the actions set the logical output port field,
997 which is available because in OVN tunneling occurs after the
998 logical output port is known. These pieces of information
999 are obtained from the tunnel encapsulation metadata (see
1000 Tunnel Encapsulations for encoding details). Then the ac‐
1001 tions resubmit to table 33 to enter the logical egress pipe‐
1002 line.
1003 2. OpenFlow tables 8 through 31 execute the logical ingress
1005 pipeline from the Logical_Flow table in the OVN Southbound
1006 database. These tables are expressed entirely in terms of
1007 logical concepts like logical ports and logical datapaths. A
1008 big part of ovn-controller’s job is to translate them into
1009 equivalent OpenFlow (in particular it translates the table
1010 numbers: Logical_Flow tables 0 through 23 become OpenFlow
1011 tables 8 through 31).
1012 Each logical flow maps to one or more OpenFlow flows. An ac‐
1014 tual packet ordinarily matches only one of these, although
1015 in some cases it can match more than one of these flows
1016 (which is not a problem because all of them have the same
1017 actions). ovn-controller uses the first 32 bits of the logi‐
1018 cal flow’s UUID as the cookie for its OpenFlow flow or
1019 flows. (This is not necessarily unique, since the first 32
1020 bits of a logical flow’s UUID is not necessarily unique.)
1021 Some logical flows can map to the Open vSwitch ``conjunctive
1023 match’’ extension (see ovs-fields(7)). Flows with a conjunc‐
1024 tion action use an OpenFlow cookie of 0, because they can
1025 correspond to multiple logical flows. The OpenFlow flow for
1026 a conjunctive match includes a match on conj_id.
1027 Some logical flows may not be represented in the OpenFlow
1029 tables on a given hypervisor, if they could not be used on
1030 that hypervisor. For example, if no VIF in a logical switch
1031 resides on a given hypervisor, and the logical switch is not
1032 otherwise reachable on that hypervisor (e.g. over a series
1033 of hops through logical switches and routers starting from a
1034 VIF on the hypervisor), then the logical flow may not be
1035 represented there.
1036 Most OVN actions have fairly obvious implementations in
1038 OpenFlow (with OVS extensions), e.g. next; is implemented as
1039 resubmit, field = constant; as set_field. A few are worth
1040 describing in more detail:
1041 output:
1043 Implemented by resubmitting the packet to table 37.
1044 If the pipeline executes more than one output action,
1045 then each one is separately resubmitted to table 37.
1046 This can be used to send multiple copies of the
1047 packet to multiple ports. (If the packet was not mod‐
1048 ified between the output actions, and some of the
1049 copies are destined to the same hypervisor, then us‐
1050 ing a logical multicast output port would save band‐
1051 width between hypervisors.)
1052 get_arp(P, A);
1054 get_nd(P, A);
1055 Implemented by storing arguments into OpenFlow fields,
1056 then resubmitting to table 66, which ovn-controller
1057 populates with flows generated from the MAC_Binding ta‐
1058 ble in the OVN Southbound database. If there is a match
1059 in table 66, then its actions store the bound MAC in
1060 the Ethernet destination address field.
1061 (The OpenFlow actions save and restore the OpenFlow
1063 fields used for the arguments, so that the OVN actions
1064 do not have to be aware of this temporary use.)
1065 put_arp(P, A, E);
1067 put_nd(P, A, E);
1068 Implemented by storing the arguments into OpenFlow
1069 fields, then outputting a packet to ovn-controller,
1070 which updates the MAC_Binding table.
1071 (The OpenFlow actions save and restore the OpenFlow
1073 fields used for the arguments, so that the OVN actions
1074 do not have to be aware of this temporary use.)
1075 R = lookup_arp(P, A, M);
1077 R = lookup_nd(P, A, M);
1078 Implemented by storing arguments into OpenFlow fields,
1079 then resubmitting to table 67, which ovn-controller
1080 populates with flows generated from the MAC_Binding ta‐
1081 ble in the OVN Southbound database. If there is a match
1082 in table 67, then its actions set the logical flow flag
1083 MLF_LOOKUP_MAC.
1084 (The OpenFlow actions save and restore the OpenFlow
1086 fields used for the arguments, so that the OVN actions
1087 do not have to be aware of this temporary use.)
1088 3. OpenFlow tables 37 through 39 implement the output action in
1090 the logical ingress pipeline. Specifically, table 37 handles
1091 packets to remote hypervisors, table 38 handles packets to
1092 the local hypervisor, and table 39 checks whether packets
1093 whose logical ingress and egress port are the same should be
1094 discarded.
1095 Logical patch ports are a special case. Logical patch ports
1097 do not have a physical location and effectively reside on
1098 every hypervisor. Thus, flow table 38, for output to ports
1099 on the local hypervisor, naturally implements output to uni‐
1100 cast logical patch ports too. However, applying the same
1101 logic to a logical patch port that is part of a logical mul‐
1102 ticast group yields packet duplication, because each hyper‐
1103 visor that contains a logical port in the multicast group
1104 will also output the packet to the logical patch port. Thus,
1105 multicast groups implement output to logical patch ports in
1106 table 37.
1107 Each flow in table 37 matches on a logical output port for
1109 unicast or multicast logical ports that include a logical
1110 port on a remote hypervisor. Each flow’s actions implement
1111 sending a packet to the port it matches. For unicast logical
1112 output ports on remote hypervisors, the actions set the tun‐
1113 nel key to the correct value, then send the packet on the
1114 tunnel port to the correct hypervisor. (When the remote hy‐
1115 pervisor receives the packet, table 0 there will recognize
1116 it as a tunneled packet and pass it along to table 38.) For
1117 multicast logical output ports, the actions send one copy of
1118 the packet to each remote hypervisor, in the same way as for
1119 unicast destinations. If a multicast group includes a logi‐
1120 cal port or ports on the local hypervisor, then its actions
1121 also resubmit to table 38. Table 37 also includes:
1122 • A higher-priority rule to match packets received from
1124 ramp switch tunnels, based on flag MLF_RCV_FROM_RAMP,
1125 and resubmit these packets to table 38 for local de‐
1126 livery. Packets received from ramp switch tunnels
1127 reach here because of a lack of logical output port
1128 field in the tunnel key and thus these packets needed
1129 to be submitted to table 8 to determine the output
1130 port.
1131 • A higher-priority rule to match packets received from
1133 ports of type localport, based on the logical input
1134 port, and resubmit these packets to table 38 for lo‐
1135 cal delivery. Ports of type localport exist on every
1136 hypervisor and by definition their traffic should
1137 never go out through a tunnel.
1138 • A higher-priority rule to match packets that have the
1140 MLF_LOCAL_ONLY logical flow flag set, and whose des‐
1141 tination is a multicast address. This flag indicates
1142 that the packet should not be delivered to remote hy‐
1143 pervisors, even if the multicast destination includes
1144 ports on remote hypervisors. This flag is used when
1145 ovn-controller is the originator of the multicast
1146 packet. Since each ovn-controller instance is origi‐
1147 nating these packets, the packets only need to be de‐
1148 livered to local ports.
1149 • A fallback flow that resubmits to table 38 if there
1151 is no other match.
1152 Flows in table 38 resemble those in table 37 but for logical
1154 ports that reside locally rather than remotely. For unicast
1155 logical output ports on the local hypervisor, the actions
1156 just resubmit to table 39. For multicast output ports that
1157 include one or more logical ports on the local hypervisor,
1158 for each such logical port P, the actions change the logical
1159 output port to P, then resubmit to table 39.
1160 A special case is that when a localnet port exists on the
1162 datapath, remote port is connected by switching to the lo‐
1163 calnet port. In this case, instead of adding a flow in table
1164 37 to reach the remote port, a flow is added in table 38 to
1165 switch the logical outport to the localnet port, and resub‐
1166 mit to table 38 as if it were unicasted to a logical port on
1167 the local hypervisor.
1168 Table 39 matches and drops packets for which the logical in‐
1170 put and output ports are the same and the MLF_ALLOW_LOOPBACK
1171 flag is not set. It also drops MLF_LOCAL_ONLY packets di‐
1172 rected to a localnet port. It resubmits other packets to ta‐
1173 ble 40.
1174 4. OpenFlow tables 40 through 63 execute the logical egress
1176 pipeline from the Logical_Flow table in the OVN Southbound
1177 database. The egress pipeline can perform a final stage of
1178 validation before packet delivery. Eventually, it may exe‐
1179 cute an output action, which ovn-controller implements by
1180 resubmitting to table 64. A packet for which the pipeline
1181 never executes output is effectively dropped (although it
1182 may have been transmitted through a tunnel across a physical
1183 network).
1184 The egress pipeline cannot change the logical output port or
1186 cause further tunneling.
1187 5. Table 64 bypasses OpenFlow loopback when MLF_ALLOW_LOOPBACK
1189 is set. Logical loopback was handled in table 39, but Open‐
1190 Flow by default also prevents loopback to the OpenFlow
1191 ingress port. Thus, when MLF_ALLOW_LOOPBACK is set, OpenFlow
1192 table 64 saves the OpenFlow ingress port, sets it to zero,
1193 resubmits to table 65 for logical-to-physical transforma‐
1194 tion, and then restores the OpenFlow ingress port, effec‐
1195 tively disabling OpenFlow loopback prevents. When MLF_AL‐
1196 LOW_LOOPBACK is unset, table 64 flow simply resubmits to ta‐
1197 ble 65.
1198 6. OpenFlow table 65 performs logical-to-physical translation,
1200 the opposite of table 0. It matches the packet’s logical
1201 egress port. Its actions output the packet to the port at‐
1202 tached to the OVN integration bridge that represents that
1203 logical port. If the logical egress port is a container
1204 nested with a VM, then before sending the packet the actions
1205 push on a VLAN header with an appropriate VLAN ID.
1206 Logical Routers and Logical Patch Ports
1208 Typically logical routers and logical patch ports do not have a physi‐
1209 cal location and effectively reside on every hypervisor. This is the
1210 case for logical patch ports between logical routers and logical
1211 switches behind those logical routers, to which VMs (and VIFs) attach.
1212 Consider a packet sent from one virtual machine or container to another
1214 VM or container that resides on a different subnet. The packet will
1215 traverse tables 0 to 65 as described in the previous section Architec‐
1216 tural Physical Life Cycle of a Packet, using the logical datapath rep‐
1217 resenting the logical switch that the sender is attached to. At table
1218 37, the packet will use the fallback flow that resubmits locally to ta‐
1219 ble 38 on the same hypervisor. In this case, all of the processing from
1220 table 0 to table 65 occurs on the hypervisor where the sender resides.
1221 When the packet reaches table 65, the logical egress port is a logical
1223 patch port. ovn-controller implements output to the logical patch is
1224 packet by cloning and resubmitting directly to the first OpenFlow flow
1225 table in the ingress pipeline, setting the logical ingress port to the
1226 peer logical patch port, and using the peer logical patch port’s logi‐
1227 cal datapath (that represents the logical router).
1228 The packet re-enters the ingress pipeline in order to traverse tables 8
1230 to 65 again, this time using the logical datapath representing the log‐
1231 ical router. The processing continues as described in the previous sec‐
1232 tion Architectural Physical Life Cycle of a Packet. When the packet
1233 reachs table 65, the logical egress port will once again be a logical
1234 patch port. In the same manner as described above, this logical patch
1235 port will cause the packet to be resubmitted to OpenFlow tables 8 to
1236 65, this time using the logical datapath representing the logical
1237 switch that the destination VM or container is attached to.
1238 The packet traverses tables 8 to 65 a third and final time. If the des‐
1240 tination VM or container resides on a remote hypervisor, then table 37
1241 will send the packet on a tunnel port from the sender’s hypervisor to
1242 the remote hypervisor. Finally table 65 will output the packet directly
1243 to the destination VM or container.
1244 The following sections describe two exceptions, where logical routers
1246 and/or logical patch ports are associated with a physical location.
1247 Gateway Routers
1249 A gateway router is a logical router that is bound to a physical loca‐
1251 tion. This includes all of the logical patch ports of the logical
1252 router, as well as all of the peer logical patch ports on logical
1253 switches. In the OVN Southbound database, the Port_Binding entries for
1254 these logical patch ports use the type l3gateway rather than patch, in
1255 order to distinguish that these logical patch ports are bound to a
1256 chassis.
1257 When a hypervisor processes a packet on a logical datapath representing
1259 a logical switch, and the logical egress port is a l3gateway port rep‐
1260 resenting connectivity to a gateway router, the packet will match a
1261 flow in table 37 that sends the packet on a tunnel port to the chassis
1262 where the gateway router resides. This processing in table 37 is done
1263 in the same manner as for VIFs.
1264 Distributed Gateway Ports
1266 This section provides additional details on distributed gateway ports,
1268 outlined earlier.
1269 The primary design goal of distributed gateway ports is to allow as
1271 much traffic as possible to be handled locally on the hypervisor where
1272 a VM or container resides. Whenever possible, packets from the VM or
1273 container to the outside world should be processed completely on that
1274 VM’s or container’s hypervisor, eventually traversing a localnet port
1275 instance or a tunnel to the physical network or a different OVN deploy‐
1276 ment. Whenever possible, packets from the outside world to a VM or con‐
1277 tainer should be directed through the physical network directly to the
1278 VM’s or container’s hypervisor.
1279 In order to allow for the distributed processing of packets described
1281 in the paragraph above, distributed gateway ports need to be logical
1282 patch ports that effectively reside on every hypervisor, rather than
1283 l3gateway ports that are bound to a particular chassis. However, the
1284 flows associated with distributed gateway ports often need to be asso‐
1285 ciated with physical locations, for the following reasons:
1286 • The physical network that the localnet port is attached
1288 to typically uses L2 learning. Any Ethernet address used
1289 over the distributed gateway port must be restricted to a
1290 single physical location so that upstream L2 learning is
1291 not confused. Traffic sent out the distributed gateway
1292 port towards the localnet port with a specific Ethernet
1293 address must be sent out one specific instance of the
1294 distributed gateway port on one specific chassis. Traffic
1295 received from the localnet port (or from a VIF on the
1296 same logical switch as the localnet port) with a specific
1297 Ethernet address must be directed to the logical switch’s
1298 patch port instance on that specific chassis.
1299 Due to the implications of L2 learning, the Ethernet ad‐
1301 dress and IP address of the distributed gateway port need
1302 to be restricted to a single physical location. For this
1303 reason, the user must specify one chassis associated with
1304 the distributed gateway port. Note that traffic travers‐
1305 ing the distributed gateway port using other Ethernet ad‐
1306 dresses and IP addresses (e.g. one-to-one NAT) is not re‐
1307 stricted to this chassis.
1308 Replies to ARP and ND requests must be restricted to a
1310 single physical location, where the Ethernet address in
1311 the reply resides. This includes ARP and ND replies for
1312 the IP address of the distributed gateway port, which are
1313 restricted to the chassis that the user associated with
1314 the distributed gateway port.
1315 • In order to support one-to-many SNAT (aka IP masquerad‐
1317 ing), where multiple logical IP addresses spread across
1318 multiple chassis are mapped to a single external IP ad‐
1319 dress, it will be necessary to handle some of the logical
1320 router processing on a specific chassis in a centralized
1321 manner. Since the SNAT external IP address is typically
1322 the distributed gateway port IP address, and for simplic‐
1323 ity, the same chassis associated with the distributed
1324 gateway port is used.
1325 The details of flow restrictions to specific chassis are described in
1327 the ovn-northd documentation.
1328 While most of the physical location dependent aspects of distributed
1330 gateway ports can be handled by restricting some flows to specific
1331 chassis, one additional mechanism is required. When a packet leaves the
1332 ingress pipeline and the logical egress port is the distributed gateway
1333 port, one of two different sets of actions is required at table 37:
1334 • If the packet can be handled locally on the sender’s hy‐
1336 pervisor (e.g. one-to-one NAT traffic), then the packet
1337 should just be resubmitted locally to table 38, in the
1338 normal manner for distributed logical patch ports.
1339 • However, if the packet needs to be handled on the chassis
1341 associated with the distributed gateway port (e.g. one-
1342 to-many SNAT traffic or non-NAT traffic), then table 37
1343 must send the packet on a tunnel port to that chassis.
1344 In order to trigger the second set of actions, the chassisredirect type
1346 of southbound Port_Binding has been added. Setting the logical egress
1347 port to the type chassisredirect logical port is simply a way to indi‐
1348 cate that although the packet is destined for the distributed gateway
1349 port, it needs to be redirected to a different chassis. At table 37,
1350 packets with this logical egress port are sent to a specific chassis,
1351 in the same way that table 37 directs packets whose logical egress port
1352 is a VIF or a type l3gateway port to different chassis. Once the packet
1353 arrives at that chassis, table 38 resets the logical egress port to the
1354 value representing the distributed gateway port. For each distributed
1355 gateway port, there is one type chassisredirect port, in addition to
1356 the distributed logical patch port representing the distributed gateway
1357 port.
1358 High Availability for Distributed Gateway Ports
1360 OVN allows you to specify a prioritized list of chassis for a distrib‐
1362 uted gateway port. This is done by associating multiple Gateway_Chassis
1363 rows with a Logical_Router_Port in the OVN_Northbound database.
1364 When multiple chassis have been specified for a gateway, all chassis
1366 that may send packets to that gateway will enable BFD on tunnels to all
1367 configured gateway chassis. The current master chassis for the gateway
1368 is the highest priority gateway chassis that is currently viewed as ac‐
1369 tive based on BFD status.
1370 For more information on L3 gateway high availability, please refer to
1372 http://docs.ovn.org/en/latest/topics/high-availability.
1373 Restrictions of Distributed Gateway Ports
1375 Distributed gateway ports are used to connect to an external network,
1377 which can be a physical network modeled by a logical switch with a lo‐
1378 calnet port, and can also be a logical switch that interconnects dif‐
1379 ferent OVN deployments (see OVN Deployments Interconnection). Usually
1380 there can be many logical routers connected to the same external logi‐
1381 cal switch, as shown in below diagram.
1382 +--LS-EXT-+
1384 | | |
1385 | | |
1386 LR1 ... LRn
1387 In this diagram, there are n logical routers connected to a logical
1390 switch LS-EXT, each with a distributed gateway port, so that traffic
1391 sent to external world is redirected to the gateway chassis that is as‐
1392 signed to the distributed gateway port of respective logical router.
1393 In the logical topology, nothing can prevent an user to add a route be‐
1395 tween the logical routers via the connected distributed gateway ports
1396 on LS-EXT. However, the route works only if the LS-EXT is a physical
1397 network (modeled by a logical switch with a localnet port). In that
1398 case the packet will be delivered between the gateway chassises through
1399 the localnet port via physical network. If the LS-EXT is a regular log‐
1400 ical switch (backed by tunneling only, as in the use case of OVN inter‐
1401 connection), then the packet will be dropped on the source gateway
1402 chassis. The limitation is due the fact that distributed gateway ports
1403 are tied to physical location, and without physical network connection,
1404 we will end up with either dropping the packet or transferring it over
1405 the tunnels which could cause bigger problems such as broadcast packets
1406 being redirect repeatedly by different gateway chassises.
1407 With the limitation in mind, if a user do want the direct connectivity
1409 between the logical routers, it is better to create an internal logical
1410 switch connected to the logical routers with regular logical router
1411 ports, which are completely distributed and the packets don’t have to
1412 leave a chassis unless necessary, which is more optimal than routing
1413 via the distributed gateway ports.
1414 ARP request and ND NS packet processing
1416 Due to the fact that ARP requests and ND NA packets are usually broad‐
1418 cast packets, for performance reasons, OVN deals with requests that
1419 target OVN owned IP addresses (i.e., IP addresses configured on the
1420 router ports, VIPs, NAT IPs) in a specific way and only forwards them
1421 to the logical router that owns the target IP address. This behavior is
1422 different than that of traditional switches and implies that other
1423 routers/hosts connected to the logical switch will not learn the MAC/IP
1424 binding from the request packet.
1425 All other ARP and ND packets are flooded in the L2 broadcast domain and
1427 to all attached logical patch ports.
1428 VIFs on the logical switch connected by a distributed gateway port
1430 Typically the logical switch connected by a distributed gateway port is
1432 for external connectivity, usually to a physical network through a lo‐
1433 calnet port on the logical switch, or to a remote OVN deployment
1434 through OVN Interconnection. In these cases there is no VIF ports re‐
1435 quired on the logical switch.
1436 While not very common, it is still possible to create VIF ports on the
1438 logical switch connected by a distributed gateway port, but there is a
1439 limitation that the logical ports need to reside on the gateway chassis
1440 where the distributed gateway port resides to get connectivity to other
1441 logical switches through the distributed gateway port. There is no lim‐
1442 itation for the VIFs to connect within the logical switch, or beyond
1443 the logical switch through other regular distributed logical router
1444 ports.
1445 A special case is when using distributed gateway ports for scalability
1447 purpose, as mentioned earlier in this document. The logical switches
1448 connected by distributed gateway ports are not for connectivity but
1449 just for regular VIFs. However, the above limitation usually does not
1450 matter because in this use case all the VIFs on the logical switch are
1451 located on the same chassis with the distributed gateway port that con‐
1452 nects the logical switch.
1453 Multiple localnet logical switches connected to a Logical Router
1455 It is possible to have multiple logical switches each with a localnet
1456 port (representing physical networks) connected to a logical router, in
1457 which one localnet logical switch may provide the external connectivity
1458 via a distributed gateway port and rest of the localnet logical
1459 switches use VLAN tagging in the physical network. It is expected that
1460 ovn-bridge-mappings is configured appropriately on the chassis for all
1461 these localnet networks.
1462 East West routing
1464 East-West routing between these localnet VLAN tagged logical switches
1466 work almost the same way as normal logical switches. When the VM sends
1467 such a packet, then:
1468 1. It first enters the ingress pipeline, and then egress pipe‐
1470 line of the source localnet logical switch datapath. It then
1471 enters the ingress pipeline of the logical router datapath
1472 via the logical router port in the source chassis.
1473 2. Routing decision is taken.
1475 3. From the router datapath, packet enters the ingress pipeline
1477 and then egress pipeline of the destination localnet logical
1478 switch datapath and goes out of the integration bridge to
1479 the provider bridge ( belonging to the destination logical
1480 switch) via the localnet port. While sending the packet to
1481 provider bridge, we also replace router port MAC as source
1482 MAC with a chassis unique MAC.
1483 This chassis unique MAC is configured as global ovs config
1485 on each chassis (eg. via "ovs-vsctl set open . external-ids:
1486 ovn-chassis-mac-mappings="phys:aa:bb:cc:dd:ee:$i$i""). For
1487 more details, see ovn-controller(8).
1488 If the above is not configured, then source MAC would be the
1490 router port MAC. This could create problem if we have more
1491 than one chassis. This is because, since the router port is
1492 distributed, the same (MAC,VLAN) tuple will seen by physical
1493 network from other chassis as well, which could cause these
1494 issues:
1495 • Continuous MAC moves in top-of-rack switch (ToR).
1497 • ToR dropping the traffic, which is causing continuous
1499 MAC moves.
1500 • ToR blocking the ports from which MAC moves are hap‐
1502 pening.
1503 4. The destination chassis receives the packet via the localnet
1505 port and sends it to the integration bridge. Before entering
1506 the integration bridge the source mac of the packet will be
1507 replaced with router port mac again. The packet enters the
1508 ingress pipeline and then egress pipeline of the destination
1509 localnet logical switch and finally gets delivered to the
1510 destination VM port.
1511 External traffic
1513 The following happens when a VM sends an external traffic (which re‐
1515 quires NATting) and the chassis hosting the VM doesn’t have a distrib‐
1516 uted gateway port.
1517 1. The packet first enters the ingress pipeline, and then
1519 egress pipeline of the source localnet logical switch data‐
1520 path. It then enters the ingress pipeline of the logical
1521 router datapath via the logical router port in the source
1522 chassis.
1523 2. Routing decision is taken. Since the gateway router or the
1525 distributed gateway port doesn’t reside in the source chas‐
1526 sis, the traffic is redirected to the gateway chassis via
1527 the tunnel port.
1528 3. The gateway chassis receives the packet via the tunnel port
1530 and the packet enters the egress pipeline of the logical
1531 router datapath. NAT rules are applied here. The packet then
1532 enters the ingress pipeline and then egress pipeline of the
1533 localnet logical switch datapath which provides external
1534 connectivity and finally goes out via the localnet port of
1535 the logical switch which provides external connectivity.
1536 Although this works, the VM traffic is tunnelled when sent from the
1538 compute chassis to the gateway chassis. In order for it to work prop‐
1539 erly, the MTU of the localnet logical switches must be lowered to ac‐
1540 count for the tunnel encapsulation.
1541 Centralized routing for localnet VLAN tagged logical switches connected to
1543 a Logical Router
1544 To overcome the tunnel encapsulation problem described in the previous
1545 section, OVN supports the option of enabling centralized routing for
1546 localnet VLAN tagged logical switches. CMS can configure the option op‐
1547 tions:reside-on-redirect-chassis to true for each Logical_Router_Port
1548 which connects to the localnet VLAN tagged logical switches. This
1549 causes the gateway chassis (hosting the distributed gateway port) to
1550 handle all the routing for these networks, making it centralized. It
1551 will reply to the ARP requests for the logical router port IPs.
1552 If the logical router doesn’t have a distributed gateway port connect‐
1554 ing to the localnet logical switch which provides external connectiv‐
1555 ity, or if it has more than one distributed gateway ports, then this
1556 option is ignored by OVN.
1557 The following happens when a VM sends an east-west traffic which needs
1559 to be routed:
1560 1. The packet first enters the ingress pipeline, and then
1562 egress pipeline of the source localnet logical switch data‐
1563 path and is sent out via a localnet port of the source lo‐
1564 calnet logical switch (instead of sending it to router pipe‐
1565 line).
1566 2. The gateway chassis receives the packet via a localnet port
1568 of the source localnet logical switch and sends it to the
1569 integration bridge. The packet then enters the ingress pipe‐
1570 line, and then egress pipeline of the source localnet logi‐
1571 cal switch datapath and enters the ingress pipeline of the
1572 logical router datapath.
1573 3. Routing decision is taken.
1575 4. From the router datapath, packet enters the ingress pipeline
1577 and then egress pipeline of the destination localnet logical
1578 switch datapath. It then goes out of the integration bridge
1579 to the provider bridge ( belonging to the destination logi‐
1580 cal switch) via a localnet port.
1581 5. The destination chassis receives the packet via a localnet
1583 port and sends it to the integration bridge. The packet en‐
1584 ters the ingress pipeline and then egress pipeline of the
1585 destination localnet logical switch and finally delivered to
1586 the destination VM port.
1587 The following happens when a VM sends an external traffic which re‐
1589 quires NATting:
1590 1. The packet first enters the ingress pipeline, and then
1592 egress pipeline of the source localnet logical switch data‐
1593 path and is sent out via a localnet port of the source lo‐
1594 calnet logical switch (instead of sending it to router pipe‐
1595 line).
1596 2. The gateway chassis receives the packet via a localnet port
1598 of the source localnet logical switch and sends it to the
1599 integration bridge. The packet then enters the ingress pipe‐
1600 line, and then egress pipeline of the source localnet logi‐
1601 cal switch datapath and enters the ingress pipeline of the
1602 logical router datapath.
1603 3. Routing decision is taken and NAT rules are applied.
1605 4. From the router datapath, packet enters the ingress pipeline
1607 and then egress pipeline of the localnet logical switch
1608 datapath which provides external connectivity. It then goes
1609 out of the integration bridge to the provider bridge (be‐
1610 longing to the logical switch which provides external con‐
1611 nectivity) via a localnet port.
1612 The following happens for the reverse external traffic.
1614 1. The gateway chassis receives the packet from a localnet port
1616 of the logical switch which provides external connectivity.
1617 The packet then enters the ingress pipeline and then egress
1618 pipeline of the localnet logical switch (which provides ex‐
1619 ternal connectivity). The packet then enters the ingress
1620 pipeline of the logical router datapath.
1621 2. The ingress pipeline of the logical router datapath applies
1623 the unNATting rules. The packet then enters the ingress
1624 pipeline and then egress pipeline of the source localnet
1625 logical switch. Since the source VM doesn’t reside in the
1626 gateway chassis, the packet is sent out via a localnet port
1627 of the source logical switch.
1628 3. The source chassis receives the packet via a localnet port
1630 and sends it to the integration bridge. The packet enters
1631 the ingress pipeline and then egress pipeline of the source
1632 localnet logical switch and finally gets delivered to the
1633 source VM port.
1634 As an alternative to reside-on-redirect-chassis, OVN supports VLAN-
1636 based redirection. Whereas reside-on-redirect-chassis centralizes all
1637 router functionality, VLAN-based redirection only changes how OVN redi‐
1638 rects packets to the gateway chassis. By setting options:redirect-type
1639 to bridged on a distributed gateway port, OVN redirects packets to the
1640 gateway chassis using the localnet port of the router’s peer logical
1641 switch, instead of a tunnel.
1642 If the logical router doesn’t have a distributed gateway port connect‐
1644 ing to the localnet logical switch which provides external connectiv‐
1645 ity, or if it has more than one distributed gateway ports, then this
1646 option is ignored by OVN.
1647 Following happens for bridged redirection:
1649 1. On compute chassis, packet passes though logical router’s
1651 ingress pipeline.
1652 2. If logical outport is gateway chassis attached router port
1654 then packet is "redirected" to gateway chassis using peer
1655 logical switch’s localnet port.
1656 3. This redirected packet has destination mac as router port
1658 mac (the one to which gateway chassis is attached). Its VLAN
1659 id is that of localnet port (peer logical switch of the log‐
1660 ical router port).
1661 4. On the gateway chassis packet will enter the logical router
1663 pipeline again and this time it will passthrough egress
1664 pipeline as well.
1665 5. Reverse traffic packet flows stays the same.
1667 Some guidelines and expections with bridged redirection:
1669 1. Since router port mac is destination mac, hence it has to be
1671 ensured that physical network learns it on ONLY from the
1672 gateway chassis. Which means that ovn-chassis-mac-mappings
1673 should be configure on all the compute nodes, so that physi‐
1674 cal network never learn router port mac from compute nodes.
1675 2. Since packet enters logical router ingress pipeline twice
1677 (once on compute chassis and again on gateway chassis),
1678 hence ttl will be decremented twice.
1679 3. Default redirection type continues to be overlay. User can
1681 switch the redirect-type between bridged and overlay by
1682 changing the value of options:redirect-type
1683 Life Cycle of a VTEP gateway
1685 A gateway is a chassis that forwards traffic between the OVN-managed
1686 part of a logical network and a physical VLAN, extending a tunnel-based
1687 logical network into a physical network.
1688 The steps below refer often to details of the OVN and VTEP database
1690 schemas. Please see ovn-sb(5), ovn-nb(5) and vtep(5), respectively, for
1691 the full story on these databases.
1692 1. A VTEP gateway’s life cycle begins with the administrator
1694 registering the VTEP gateway as a Physical_Switch table en‐
1695 try in the VTEP database. The ovn-controller-vtep connected
1696 to this VTEP database, will recognize the new VTEP gateway
1697 and create a new Chassis table entry for it in the
1698 OVN_Southbound database.
1699 2. The administrator can then create a new Logical_Switch table
1701 entry, and bind a particular vlan on a VTEP gateway’s port
1702 to any VTEP logical switch. Once a VTEP logical switch is
1703 bound to a VTEP gateway, the ovn-controller-vtep will detect
1704 it and add its name to the vtep_logical_switches column of
1705 the Chassis table in the OVN_Southbound database. Note, the
1706 tunnel_key column of VTEP logical switch is not filled at
1707 creation. The ovn-controller-vtep will set the column when
1708 the correponding vtep logical switch is bound to an OVN log‐
1709 ical network.
1710 3. Now, the administrator can use the CMS to add a VTEP logical
1712 switch to the OVN logical network. To do that, the CMS must
1713 first create a new Logical_Switch_Port table entry in the
1714 OVN_Northbound database. Then, the type column of this entry
1715 must be set to "vtep". Next, the vtep-logical-switch and
1716 vtep-physical-switch keys in the options column must also be
1717 specified, since multiple VTEP gateways can attach to the
1718 same VTEP logical switch. Next, the addresses column of this
1719 logical port must be set to "unknown", it will add a prior‐
1720 ity 0 entry in "ls_in_l2_lkup" stage of logical switch
1721 ingress pipeline. So, traffic with unrecorded mac by OVN
1722 would go through the Logical_Switch_Port to physical net‐
1723 work.
1724 4. The newly created logical port in the OVN_Northbound data‐
1726 base and its configuration will be passed down to the
1727 OVN_Southbound database as a new Port_Binding table entry.
1728 The ovn-controller-vtep will recognize the change and bind
1729 the logical port to the corresponding VTEP gateway chassis.
1730 Configuration of binding the same VTEP logical switch to a
1731 different OVN logical networks is not allowed and a warning
1732 will be generated in the log.
1733 5. Beside binding to the VTEP gateway chassis, the ovn-con‐
1735 troller-vtep will update the tunnel_key column of the VTEP
1736 logical switch to the corresponding Datapath_Binding table
1737 entry’s tunnel_key for the bound OVN logical network.
1738 6. Next, the ovn-controller-vtep will keep reacting to the con‐
1740 figuration change in the Port_Binding in the OVN_Northbound
1741 database, and updating the Ucast_Macs_Remote table in the
1742 VTEP database. This allows the VTEP gateway to understand
1743 where to forward the unicast traffic coming from the ex‐
1744 tended external network.
1745 7. Eventually, the VTEP gateway’s life cycle ends when the ad‐
1747 ministrator unregisters the VTEP gateway from the VTEP data‐
1748 base. The ovn-controller-vtep will recognize the event and
1749 remove all related configurations (Chassis table entry and
1750 port bindings) in the OVN_Southbound database.
1751 8. When the ovn-controller-vtep is terminated, all related con‐
1753 figurations in the OVN_Southbound database and the VTEP
1754 database will be cleaned, including Chassis table entries
1755 for all registered VTEP gateways and their port bindings,
1756 and all Ucast_Macs_Remote table entries and the Logi‐
1757 cal_Switch tunnel keys.
1758 OVN Deployments Interconnection
1760 It is not uncommon for an operator to deploy multiple OVN clusters, for
1761 two main reasons. Firstly, an operator may prefer to deploy one OVN
1762 cluster for each availability zone, e.g. in different physical regions,
1763 to avoid single point of failure. Secondly, there is always an upper
1764 limit for a single OVN control plane to scale.
1765 Although the control planes of the different availability zone (AZ)s
1767 are independent from each other, the workloads from different AZs may
1768 need to communicate across the zones. The OVN interconnection feature
1769 provides a native way to interconnect different AZs by L3 routing
1770 through transit overlay networks between logical routers of different
1771 AZs.
1772 A global OVN Interconnection Northbound database is introduced for the
1774 operator (probably through CMS systems) to configure transit logical
1775 switches that connect logical routers from different AZs. A transit
1776 switch is similar to a regular logical switch, but it is used for in‐
1777 terconnection purpose only. Typically, each transit switch can be used
1778 to connect all logical routers that belong to same tenant across all
1779 AZs.
1780 A dedicated daemon process ovn-ic, OVN interconnection controller, in
1782 each AZ will consume this data and populate corresponding logical
1783 switches to their own northbound databases for each AZ, so that logical
1784 routers can be connected to the transit switch by creating patch port
1785 pairs in their northbound databases. Any router ports connected to the
1786 transit switches are considered interconnection ports, which will be
1787 exchanged between AZs.
1788 Physically, when workloads from different AZs communicate, packets need
1790 to go through multiple hops: source chassis, source gateway, destina‐
1791 tion gateway and destination chassis. All these hops are connected
1792 through tunnels so that the packets never leave overlay networks. A
1793 distributed gateway port is required to connect the logical router to a
1794 transit switch, with a gateway chassis specified, so that the traffic
1795 can be forwarded through the gateway chassis.
1796 A global OVN Interconnection Southbound database is introduced for ex‐
1798 changing control plane information between the AZs. The data in this
1799 database is populated and consumed by the ovn-ic, of each AZ. The main
1800 information in this database includes:
1801 • Datapath bindings for transit switches, which mainly con‐
1803 tains the tunnel keys generated for each transit switch.
1804 Separate key ranges are reserved for transit switches so
1805 that they will never conflict with any tunnel keys lo‐
1806 cally assigned for datapaths within each AZ.
1807 • Availability zones, which are registerd by ovn-ic from
1809 each AZ.
1810 • Gateways. Each AZ specifies chassises that are supposed
1812 to work as interconnection gateways, and the ovn-ic will
1813 populate this information to the interconnection south‐
1814 bound DB. The ovn-ic from all the other AZs will learn
1815 the gateways and populate to their own southbound DB as a
1816 chassis.
1817 • Port bindings for logical switch ports created on the
1819 transit switch. Each AZ maintains their logical router to
1820 transit switch connections independently, but ovn-ic au‐
1821 tomatically populates local port bindings on transit
1822 switches to the global interconnection southbound DB, and
1823 learns remote port bindings from other AZs back to its
1824 own northbound and southbound DBs, so that logical flows
1825 can be produced and then translated to OVS flows locally,
1826 which finally enables data plane communication.
1827 • Routes that are advertised between different AZs. If en‐
1829 abled, routes are automatically exchanged by ovn-ic. Both
1830 static routes and directly connected subnets are adver‐
1831 tised. Options in options column of the NB_Global table
1832 of OVN_NB database control the behavior of route adver‐
1833 tisement, such as enable/disable the advertising/learning
1834 routes, whether default routes are advertised/learned,
1835 and blacklisted CIDRs. See ovn-nb(5) for more details.
1836 The tunnel keys for transit switch datapaths and related port bindings
1838 must be agreed across all AZs. This is ensured by generating and stor‐
1839 ing the keys in the global interconnection southbound database. Any
1840 ovn-ic from any AZ can allocate the key, but race conditions are solved
1841 by enforcing unique index for the column in the database.
1842 Once each AZ’s NB and SB databases are populated with interconnection
1844 switches and ports, and agreed upon the tunnel keys, data plane commu‐
1845 nication between the AZs are established.
1846 When VXLAN tunneling is enabled in an OVN cluster, due to the limited
1848 range available for VNIs, Interconnection feature is not supported.
1849 A day in the life of a packet crossing AZs
1851 1. An IP packet is sent out from a VIF on a hypervisor (HV1) of
1853 AZ1, with destination IP belonging to a VIF in AZ2.
1854 2. In HV1’s OVS flow tables, the packet goes through logical
1856 switch and logical router pipelines, and in a logical router
1857 pipeline, the routing stage finds out the next hop for the
1858 destination IP, which belongs to a remote logical router
1859 port in AZ2, and the output port, which is a chassis-redi‐
1860 rect port located on an interconnection gateway (GW1 in
1861 AZ1), so HV1 sends the packet to GW1 through tunnel.
1862 3. On GW1, it continues with the logical router pipe line and
1864 switches to the transit switch’s pipeline through the peer
1865 port of the chassis redirect port. In the transit switch’s
1866 pipeline it outputs to the remote logical port which is lo‐
1867 cated on a gateway (GW2) in AZ2, so the GW1 sends the packet
1868 to GW2 in tunnel.
1869 4. On GW2, it continues with the transit switch pipeline and
1871 switches to the logical router pipeline through the peer
1872 port, which is a chassis redirect port that is located on
1873 GW2. The logical router pipeline then forwards the packet to
1874 relevant logical pipelines according to the destination IP
1875 address, and figures out the MAC and location of the desti‐
1876 nation VIF port - a hypervisor (HV2). The GW2 then sends the
1877 packet to HV2 in tunnel.
1878 5. On HV2, the packet is delivered to the final destination VIF
1880 port by the logical switch egress pipeline, just the same
1881 way as for intra-AZ communications.
1882 Native OVN services for external logical ports
1884 To support OVN native services (like DHCP/IPv6 RA/DNS lookup) to the
1885 cloud resources which are external, OVN supports external logical
1886 ports.
1887 Below are some of the use cases where external ports can be used.
1889 • VMs connected to SR-IOV nics - Traffic from these VMs by
1891 passes the kernel stack and local ovn-controller do not
1892 bind these ports and cannot serve the native services.
1893 • When CMS supports provisioning baremetal servers.
1895 OVN will provide the native services if CMS has done the below configu‐
1897 ration in the OVN Northbound Database.
1898 • A row is created in Logical_Switch_Port, configuring the
1900 addresses column and setting the type to external.
1901 • ha_chassis_group column is configured.
1903 • The HA chassis which belongs to the HA chassis group has
1905 the ovn-bridge-mappings configured and has proper L2 con‐
1906 nectivity so that it can receive the DHCP and other re‐
1907 lated request packets from these external resources.
1908 • The Logical_Switch of this port has a localnet port.
1910 • Native OVN services are enabled by configuring the DHCP
1912 and other options like the way it is done for the normal
1913 logical ports.
1914 It is recommended to use the same HA chassis group for all the external
1916 ports of a logical switch. Otherwise, the physical switch might see MAC
1917 flap issue when different chassis provide the native services. For ex‐
1918 ample when supporting native DHCPv4 service, DHCPv4 server mac (config‐
1919 ured in options:server_mac column in table DHCP_Options) originating
1920 from different ports can cause MAC flap issue. The MAC of the logical
1921 router IP(s) can also flap if the same HA chassis group is not set for
1922 all the external ports of a logical switch.
1923
1925 Role-Based Access Controls for the Southbound DB
1926 In order to provide additional security against the possibility of an
1927 OVN chassis becoming compromised in such a way as to allow rogue soft‐
1928 ware to make arbitrary modifications to the southbound database state
1929 and thus disrupt the OVN network, role-based access controls (see
1930 ovsdb-server(1) for additional details) are provided for the southbound
1931 database.
1932 The implementation of role-based access controls (RBAC) requires the
1934 addition of two tables to an OVSDB schema: the RBAC_Role table, which
1935 is indexed by role name and maps the the names of the various tables
1936 that may be modifiable for a given role to individual rows in a permis‐
1937 sions table containing detailed permission information for that role,
1938 and the permission table itself which consists of rows containing the
1939 following information:
1940 Table Name
1942 The name of the associated table. This column exists pri‐
1943 marily as an aid for humans reading the contents of this
1944 table.
1945 Auth Criteria
1947 A set of strings containing the names of columns (or col‐
1948 umn:key pairs for columns containing string:string maps).
1949 The contents of at least one of the columns or column:key
1950 values in a row to be modified, inserted, or deleted must
1951 be equal to the ID of the client attempting to act on the
1952 row in order for the authorization check to pass. If the
1953 authorization criteria is empty, authorization checking
1954 is disabled and all clients for the role will be treated
1955 as authorized.
1956 Insert/Delete
1958 Row insertion/deletion permission; boolean value indicat‐
1959 ing whether insertion and deletion of rows is allowed for
1960 the associated table. If true, insertion and deletion of
1961 rows is allowed for authorized clients.
1962 Updatable Columns
1964 A set of strings containing the names of columns or col‐
1965 umn:key pairs that may be updated or mutated by autho‐
1966 rized clients. Modifications to columns within a row are
1967 only permitted when the authorization check for the
1968 client passes and all columns to be modified are included
1969 in this set of modifiable columns.
1970 RBAC configuration for the OVN southbound database is maintained by
1972 ovn-northd. With RBAC enabled, modifications are only permitted for the
1973 Chassis, Encap, Port_Binding, and MAC_Binding tables, and are re‐
1974 stricted as follows:
1975 Chassis
1977 Authorization: client ID must match the chassis name.
1978 Insert/Delete: authorized row insertion and deletion are
1980 permitted.
1981 Update: The columns nb_cfg, external_ids, encaps, and
1983 vtep_logical_switches may be modified when authorized.
1984 Encap Authorization: client ID must match the chassis name.
1986 Insert/Delete: row insertion and row deletion are permit‐
1988 ted.
1989 Update: The columns type, options, and ip can be modi‐
1991 fied.
1992 Port_Binding
1994 Authorization: disabled (all clients are considered au‐
1995 thorized. A future enhancement may add columns (or keys
1996 to external_ids) in order to control which chassis are
1997 allowed to bind each port.
1998 Insert/Delete: row insertion/deletion are not permitted
2000 (ovn-northd maintains rows in this table.
2001 Update: Only modifications to the chassis column are per‐
2003 mitted.
2004 MAC_Binding
2006 Authorization: disabled (all clients are considered to be
2007 authorized).
2008 Insert/Delete: row insertion/deletion are permitted.
2010 Update: The columns logical_port, ip, mac, and datapath
2012 may be modified by ovn-controller.
2013 IGMP_Group
2015 Authorization: disabled (all clients are considered to be
2016 authorized).
2017 Insert/Delete: row insertion/deletion are permitted.
2019 Update: The columns address, chassis, datapath, and ports
2021 may be modified by ovn-controller.
2022 Enabling RBAC for ovn-controller connections to the southbound database
2024 requires the following steps:
2025 1. Creating SSL certificates for each chassis with the certifi‐
2027 cate CN field set to the chassis name (e.g. for a chassis
2028 with external-ids:system-id=chassis-1, via the command
2029 "ovs-pki -u req+sign chassis-1 switch").
2030 2. Configuring each ovn-controller to use SSL when connecting
2032 to the southbound database (e.g. via "ovs-vsctl set open .
2033 external-ids:ovn-remote=ssl:x.x.x.x:6642").
2034 3. Configuring a southbound database SSL remote with "ovn-con‐
2036 troller" role (e.g. via "ovn-sbctl set-connection
2037 role=ovn-controller pssl:6642").
2038 Encrypt Tunnel Traffic with IPsec
2040 OVN tunnel traffic goes through physical routers and switches. These
2041 physical devices could be untrusted (devices in public network) or
2042 might be compromised. Enabling encryption to the tunnel traffic can
2043 prevent the traffic data from being monitored and manipulated.
2044 The tunnel traffic is encrypted with IPsec. The CMS sets the ipsec col‐
2046 umn in the northbound NB_Global table to enable or disable IPsec encry‐
2047 tion. If ipsec is true, all OVN tunnels will be encrypted. If ipsec is
2048 false, no OVN tunnels will be encrypted.
2049 When CMS updates the ipsec column in the northbound NB_Global table,
2051 ovn-northd copies the value to the ipsec column in the southbound
2052 SB_Global table. ovn-controller in each chassis monitors the southbound
2053 database and sets the options of the OVS tunnel interface accordingly.
2054 OVS tunnel interface options are monitored by the ovs-monitor-ipsec
2055 daemon which configures IKE daemon to set up IPsec connections.
2056 Chassis authenticates each other by using certificate. The authentica‐
2058 tion succeeds if the other end in tunnel presents a certificate signed
2059 by a trusted CA and the common name (CN) matches the expected chassis
2060 name. The SSL certificates used in role-based access controls (RBAC)
2061 can be used in IPsec. Or use ovs-pki to create different certificates.
2062 The certificate is required to be x.509 version 3, and with CN field
2063 and subjectAltName field being set to the chassis name.
2064 The CA certificate, chassis certificate and private key are required to
2066 be installed in each chassis before enabling IPsec. Please see
2067 ovs-vswitchd.conf.db(5) for setting up CA based IPsec authentication.
2068
2070 Tunnel Encapsulations
2071 In general, OVN annotates logical network packets that it sends from
2072 one hypervisor to another with the following three pieces of metadata,
2073 which are encoded in an encapsulation-specific fashion:
2074 • 24-bit logical datapath identifier, from the tunnel_key
2076 column in the OVN Southbound Datapath_Binding table.
2077 • 15-bit logical ingress port identifier. ID 0 is reserved
2079 for internal use within OVN. IDs 1 through 32767, inclu‐
2080 sive, may be assigned to logical ports (see the tun‐
2081 nel_key column in the OVN Southbound Port_Binding table).
2082 • 16-bit logical egress port identifier. IDs 0 through
2084 32767 have the same meaning as for logical ingress ports.
2085 IDs 32768 through 65535, inclusive, may be assigned to
2086 logical multicast groups (see the tunnel_key column in
2087 the OVN Southbound Multicast_Group table).
2088 When VXLAN is enabled on any hypervisor in a cluster, datapath and
2090 egress port identifier ranges are reduced to 12-bits. This is done be‐
2091 cause only STT and Geneve provide the large space for metadata (over 32
2092 bits per packet). To accommodate for VXLAN, 24 bits available are split
2093 as follows:
2094 • 12-bit logical datapath identifier, derived from the tun‐
2096 nel_key column in the OVN Southbound Datapath_Binding ta‐
2097 ble.
2098 • 12-bit logical egress port identifier. IDs 0 through 2047
2100 are used for unicast output ports. IDs 2048 through 4095,
2101 inclusive, may be assigned to logical multicast groups
2102 (see the tunnel_key column in the OVN Southbound Multi‐
2103 cast_Group table). For multicast group tunnel keys, a
2104 special mapping scheme is used to internally transform
2105 from internal OVN 16-bit keys to 12-bit values before
2106 sending packets through a VXLAN tunnel, and back from
2107 12-bit tunnel keys to 16-bit values when receiving pack‐
2108 ets from a VXLAN tunnel.
2109 • No logical ingress port identifier.
2111 The limited space available for metadata when VXLAN tunnels are enabled
2113 in a cluster put the following functional limitations onto features
2114 available to users:
2115 • The maximum number of networks is reduced to 4096.
2117 • The maximum number of ports per network is reduced to
2119 4096. (Including multicast group ports.)
2120 • ACLs matching against logical ingress port identifiers
2122 are not supported.
2123 • OVN interconnection feature is not supported.
2125 In addition to functional limitations described above, the following
2127 should be considered before enabling it in your cluster:
2128 • STT and Geneve use randomized UDP or TCP source ports
2130 that allows efficient distribution among multiple paths
2131 in environments that use ECMP in their underlay.
2132 • NICs are available to offload STT and Geneve encapsula‐
2134 tion and decapsulation.
2135 Due to its flexibility, the preferred encapsulation between hypervisors
2137 is Geneve. For Geneve encapsulation, OVN transmits the logical datapath
2138 identifier in the Geneve VNI. OVN transmits the logical ingress and
2139 logical egress ports in a TLV with class 0x0102, type 0x80, and a
2140 32-bit value encoded as follows, from MSB to LSB:
2141 1 15 16
2143 +---+------------+-----------+
2144 |rsv|ingress port|egress port|
2145 +---+------------+-----------+
2146 0
2147 Environments whose NICs lack Geneve offload may prefer STT encapsula‐
2150 tion for performance reasons. For STT encapsulation, OVN encodes all
2151 three pieces of logical metadata in the STT 64-bit tunnel ID as fol‐
2152 lows, from MSB to LSB:
2153 9 15 16 24
2155 +--------+------------+-----------+--------+
2156 |reserved|ingress port|egress port|datapath|
2157 +--------+------------+-----------+--------+
2158 0
2159 For connecting to gateways, in addition to Geneve and STT, OVN supports
2162 VXLAN, because only VXLAN support is common on top-of-rack (ToR)
2163 switches. Currently, gateways have a feature set that matches the capa‐
2164 bilities as defined by the VTEP schema, so fewer bits of metadata are
2165 necessary. In the future, gateways that do not support encapsulations
2166 with large amounts of metadata may continue to have a reduced feature
2167 set.
2168OVN 22.06.1 OVN Architecture ovn-architecture(7)