1ovn-architecture(7)               OVN Manual               ovn-architecture(7)
2
3
4

NAME

6       ovn-architecture - Open Virtual Network architecture
7

DESCRIPTION

9       OVN,  the  Open Virtual Network, is a system to support logical network
10       abstraction in virtual machine and container environments. OVN  comple‐
11       ments  the existing capabilities of OVS to add native support for logi‐
12       cal network abstractions, such as logical L2 and L3 overlays and  secu‐
13       rity  groups.  Services  such as DHCP are also desirable features. Just
14       like OVS, OVN’s design goal is to have a production-quality implementa‐
15       tion that can operate at significant scale.
16
17       A  physical  network comprises physical wires, switches, and routers. A
18       virtual network extends a physical network into a  hypervisor  or  con‐
19       tainer  platform, bridging VMs or containers into the physical network.
20       An OVN logical network is a network implemented in software that is in‐
21       sulated  from  physical (and thus virtual) networks by tunnels or other
22       encapsulations. This allows IP and other address spaces used in logical
23       networks  to overlap with those used on physical networks without caus‐
24       ing conflicts. Logical network topologies can be arranged  without  re‐
25       gard  for  the  topologies  of the physical networks on which they run.
26       Thus, VMs that are part of a logical network can migrate from one phys‐
27       ical  machine  to  another without network disruption. See Logical Net‐
28       works, below, for more information.
29
30       The encapsulation layer prevents VMs and containers connected to a log‐
31       ical  network  from  communicating with nodes on physical networks. For
32       clustering VMs and containers, this can be acceptable  or  even  desir‐
33       able,  but  in  many  cases  VMs and containers do need connectivity to
34       physical networks. OVN provides multiple forms  of  gateways  for  this
35       purpose. See Gateways, below, for more information.
36
37       An OVN deployment consists of several components:
38
39              •      A  Cloud Management System (CMS), which is OVN’s ultimate
40                     client (via its users and administrators).  OVN  integra‐
41                     tion  requires  installing  a CMS-specific plugin and re‐
42                     lated software (see below). OVN initially  targets  Open‐
43                     Stack as CMS.
44
45                     We  generally  speak  of ``the’’ CMS, but one can imagine
46                     scenarios in which multiple CMSes manage different  parts
47                     of an OVN deployment.
48
49              •      An OVN Database physical or virtual node (or, eventually,
50                     cluster) installed in a central location.
51
52              •      One or more (usually many) hypervisors. Hypervisors  must
53                     run Open vSwitch and implement the interface described in
54                     Documentation/topics/integration.rst in the Open  vSwitch
55                     source  tree.  Any  hypervisor platform supported by Open
56                     vSwitch is acceptable.
57
58              •      Zero or more gateways. A gateway extends  a  tunnel-based
59                     logical  network  into a physical network by bidirection‐
60                     ally forwarding packets between tunnels  and  a  physical
61                     Ethernet  port.  This  allows non-virtualized machines to
62                     participate in logical networks. A gateway may be a phys‐
63                     ical  host,  a virtual machine, or an ASIC-based hardware
64                     switch that supports the vtep(5) schema.
65
66                     Hypervisors and gateways are  together  called  transport
67                     node or chassis.
68
69       The  diagram  below  shows  how the major components of OVN and related
70       software interact. Starting at the top of the diagram, we have:
71
72              •      The Cloud Management System, as defined above.
73
74              •      The OVN/CMS Plugin is the component of the CMS  that  in‐
75                     terfaces  to OVN. In OpenStack, this is a Neutron plugin.
76                     The plugin’s main purpose is to translate the  CMS’s  no‐
77                     tion  of  logical  network  configuration,  stored in the
78                     CMS’s configuration database in  a  CMS-specific  format,
79                     into an intermediate representation understood by OVN.
80
81                     This  component  is  necessarily  CMS-specific,  so a new
82                     plugin needs to be developed for each CMS that  is  inte‐
83                     grated  with OVN. All of the components below this one in
84                     the diagram are CMS-independent.
85
86              •      The OVN Northbound  Database  receives  the  intermediate
87                     representation  of  logical  network configuration passed
88                     down by the OVN/CMS Plugin. The database schema is  meant
89                     to  be  ``impedance matched’’ with the concepts used in a
90                     CMS, so that it  directly  supports  notions  of  logical
91                     switches, routers, ACLs, and so on. See ovn-nb(5) for de‐
92                     tails.
93
94                     The OVN Northbound Database has  only  two  clients:  the
95                     OVN/CMS Plugin above it and ovn-northd below it.
96
97ovn-northd(8)  connects  to  the  OVN Northbound Database
98                     above it and the OVN Southbound  Database  below  it.  It
99                     translates  the logical network configuration in terms of
100                     conventional network concepts, taken from the OVN  North‐
101                     bound  Database,  into  logical datapath flows in the OVN
102                     Southbound Database below it.
103
104              •      The OVN Southbound Database is the center of the  system.
105                     Its  clients  are  ovn-northd(8)  above  it  and ovn-con‐
106                     troller(8) on every transport node below it.
107
108                     The OVN Southbound Database contains three kinds of data:
109                     Physical  Network  (PN)  tables that specify how to reach
110                     hypervisor and other nodes, Logical Network  (LN)  tables
111                     that  describe  the logical network in terms of ``logical
112                     datapath flows,’’ and Binding tables  that  link  logical
113                     network  components’  locations  to the physical network.
114                     The hypervisors populate the PN and Port_Binding  tables,
115                     whereas ovn-northd(8) populates the LN tables.
116
117                     OVN  Southbound  Database performance must scale with the
118                     number of transport nodes. This will likely require  some
119                     work  on  ovsdb-server(1)  as  we  encounter bottlenecks.
120                     Clustering for availability may be needed.
121
122       The remaining components are replicated onto each hypervisor:
123
124ovn-controller(8) is OVN’s agent on each  hypervisor  and
125                     software  gateway.  Northbound,  it  connects  to the OVN
126                     Southbound Database to learn about OVN configuration  and
127                     status  and to populate the PN table and the Chassis col‐
128                     umn in Binding table with the hypervisor’s status. South‐
129                     bound, it connects to ovs-vswitchd(8) as an OpenFlow con‐
130                     troller, for control over network traffic, and to the lo‐
131                     cal  ovsdb-server(1)  to  allow it to monitor and control
132                     Open vSwitch configuration.
133
134ovs-vswitchd(8) and ovsdb-server(1) are conventional com‐
135                     ponents of Open vSwitch.
136
137                                         CMS
138                                          |
139                                          |
140                              +-----------|-----------+
141                              |           |           |
142                              |     OVN/CMS Plugin    |
143                              |           |           |
144                              |           |           |
145                              |   OVN Northbound DB   |
146                              |           |           |
147                              |           |           |
148                              |       ovn-northd      |
149                              |           |           |
150                              +-----------|-----------+
151                                          |
152                                          |
153                                +-------------------+
154                                | OVN Southbound DB |
155                                +-------------------+
156                                          |
157                                          |
158                       +------------------+------------------+
159                       |                  |                  |
160         HV 1          |                  |    HV n          |
161       +---------------|---------------+  .  +---------------|---------------+
162       |               |               |  .  |               |               |
163       |        ovn-controller         |  .  |        ovn-controller         |
164       |         |          |          |  .  |         |          |          |
165       |         |          |          |     |         |          |          |
166       |  ovs-vswitchd   ovsdb-server  |     |  ovs-vswitchd   ovsdb-server  |
167       |                               |     |                               |
168       +-------------------------------+     +-------------------------------+
169
170
171   Information Flow in OVN
172       Configuration  data  in OVN flows from north to south. The CMS, through
173       its  OVN/CMS  plugin,  passes  the  logical  network  configuration  to
174       ovn-northd  via  the  northbound database. In turn, ovn-northd compiles
175       the configuration into a lower-level form and passes it to all  of  the
176       chassis via the southbound database.
177
178       Status information in OVN flows from south to north. OVN currently pro‐
179       vides only a few forms of status information. First,  ovn-northd  popu‐
180       lates  the  up column in the northbound Logical_Switch_Port table: if a
181       logical port’s chassis column in the southbound Port_Binding  table  is
182       nonempty,  it  sets up to true, otherwise to false. This allows the CMS
183       to detect when a VM’s networking has come up.
184
185       Second, OVN provides feedback to the CMS on the realization of its con‐
186       figuration,  that is, whether the configuration provided by the CMS has
187       taken effect. This feature requires the CMS to  participate  in  a  se‐
188       quence number protocol, which works the following way:
189
190              1.  When  the  CMS  updates  the configuration in the northbound
191                  database, as part of the same transaction, it increments the
192                  value  of the nb_cfg column in the NB_Global table. (This is
193                  only necessary if the CMS wants to know when the  configura‐
194                  tion has been realized.)
195
196              2.  When  ovn-northd  updates the southbound database based on a
197                  given snapshot of the northbound database, it copies  nb_cfg
198                  from  northbound  NB_Global  into  the  southbound  database
199                  SB_Global table, as part of the same transaction. (Thus,  an
200                  observer  monitoring  both  databases can determine when the
201                  southbound database is caught up with the northbound.)
202
203              3.  After ovn-northd receives confirmation from  the  southbound
204                  database  server that its changes have committed, it updates
205                  sb_cfg in the northbound NB_Global table to the nb_cfg  ver‐
206                  sion  that  was  pushed  down. (Thus, the CMS or another ob‐
207                  server can determine when the southbound database is  caught
208                  up without a connection to the southbound database.)
209
210              4.  The  ovn-controller process on each chassis receives the up‐
211                  dated southbound database, with  the  updated  nb_cfg.  This
212                  process  in turn updates the physical flows installed in the
213                  chassis’s Open vSwitch instances. When it receives confirma‐
214                  tion from Open vSwitch that the physical flows have been up‐
215                  dated, it updates nb_cfg in its own Chassis  record  in  the
216                  southbound database.
217
218              5.  ovn-northd  monitors the nb_cfg column in all of the Chassis
219                  records in the southbound database. It keeps  track  of  the
220                  minimum  value  among all the records and copies it into the
221                  hv_cfg column in the northbound NB_Global table. (Thus,  the
222                  CMS or another observer can determine when all of the hyper‐
223                  visors have caught up to the northbound configuration.)
224
225   Chassis Setup
226       Each chassis in an OVN deployment  must  be  configured  with  an  Open
227       vSwitch  bridge dedicated for OVN’s use, called the integration bridge.
228       System startup  scripts  may  create  this  bridge  prior  to  starting
229       ovn-controller  if desired. If this bridge does not exist when ovn-con‐
230       troller starts, it will be created automatically with the default  con‐
231       figuration  suggested  below.  The  ports on the integration bridge in‐
232       clude:
233
234              •      On any chassis, tunnel ports that OVN  uses  to  maintain
235                     logical  network  connectivity.  ovn-controller adds, up‐
236                     dates, and removes these tunnel ports.
237
238              •      On a hypervisor, any VIFs that are to be attached to log‐
239                     ical  networks.  For instances connected through software
240                     emulated ports such as TUN/TAP or VETH pairs, the  hyper‐
241                     visor  itself  will  normally  create ports and plug them
242                     into the  integration  bridge.  For  instances  connected
243                     through  representor  ports, typically used with hardware
244                     offload, the ovn-controller may on CMS direction  consult
245                     a  VIF plug provider for representor port lookup and plug
246                     them into the integration bridge (please refer  to  Docu‐
247                     mentation/top‐
248                     ics/vif-plug-providers/vif-plug-providers.rst
249                      for more information). In both cases the conventions de‐
250                     scribed  in  Documentation/topics/integration.rst  in the
251                     Open vSwitch source tree is followed  to  ensure  mapping
252                     between  OVN  logical port and VIF. (This is pre-existing
253                     integration work that has already been done  on  hypervi‐
254                     sors that support OVS.)
255
256              •      On  a gateway, the physical port used for logical network
257                     connectivity. System startup scripts add this port to the
258                     bridge  prior  to  starting ovn-controller. This can be a
259                     patch port to another bridge, instead of a physical port,
260                     in more sophisticated setups.
261
262       Other  ports  should not be attached to the integration bridge. In par‐
263       ticular, physical ports attached to the underlay network (as opposed to
264       gateway  ports,  which are physical ports attached to logical networks)
265       must not be attached to the integration bridge. Underlay physical ports
266       should instead be attached to a separate Open vSwitch bridge (they need
267       not be attached to any bridge at all, in fact).
268
269       The integration bridge should be configured as described below. The ef‐
270       fect    of    each    of    these    settings    is    documented    in
271       ovs-vswitchd.conf.db(5):
272
273              fail-mode=secure
274                     Avoids switching packets between  isolated  logical  net‐
275                     works  before  ovn-controller  starts  up. See Controller
276                     Failure Settings in ovs-vsctl(8) for more information.
277
278              other-config:disable-in-band=true
279                     Suppresses in-band  control  flows  for  the  integration
280                     bridge.  It  would  be  unusual for such flows to show up
281                     anyway, because OVN uses a local controller (over a  Unix
282                     domain  socket) instead of a remote controller. It’s pos‐
283                     sible, however, for some other bridge in the same  system
284                     to  have  an  in-band remote controller, and in that case
285                     this suppresses the flows that in-band control would  or‐
286                     dinarily  set up. Refer to the documentation for more in‐
287                     formation.
288
289       The customary name for the integration bridge is  br-int,  but  another
290       name may be used.
291
292   Logical Networks
293       Logical  network  concepts  in OVN include logical switches and logical
294       routers, the logical version of Ethernet switches and IP  routers,  re‐
295       spectively.  Like  their physical cousins, logical switches and routers
296       can be connected into sophisticated topologies.  Logical  switches  and
297       routers  are  ordinarily purely logical entities, that is, they are not
298       associated or bound to any physical location, and they are  implemented
299       in a distributed manner at each hypervisor that participates in OVN.
300
301       Logical  switch ports (LSPs) are points of connectivity into and out of
302       logical switches. There are many kinds of  logical  switch  ports.  The
303       most  ordinary  kind represent VIFs, that is, attachment points for VMs
304       or containers. A VIF logical port is associated with the physical loca‐
305       tion  of  its VM, which might change as the VM migrates. (A VIF logical
306       port can be associated with a VM that is  powered  down  or  suspended.
307       Such a logical port has no location and no connectivity.)
308
309       Logical  router ports (LRPs) are points of connectivity into and out of
310       logical routers. A LRP connects a logical router either  to  a  logical
311       switch  or  to  another logical router. Logical routers only connect to
312       VMs, containers, and other network nodes  indirectly,  through  logical
313       switches.
314
315       Logical  switches  and  logical  routers have distinct kinds of logical
316       ports, so properly speaking  one  should  usually  talk  about  logical
317       switch ports or logical router ports. However, an unqualified ``logical
318       port’’ usually refers to a logical switch port.
319
320       When a VM sends a packet to a VIF logical switch port, the Open vSwitch
321       flow  tables  simulate the packet’s journey through that logical switch
322       and any other logical routers and logical switches that  it  might  en‐
323       counter. This happens without transmitting the packet across any physi‐
324       cal medium: the flow tables implement all of the switching and  routing
325       decisions  and behavior. If the flow tables ultimately decide to output
326       the packet at a logical port attached to another hypervisor (or another
327       kind  of  transport node), then that is the time at which the packet is
328       encapsulated for physical network transmission and sent.
329
330     Logical Switch Port Types
331
332       OVN supports a number of kinds of logical switch ports. VIF ports  that
333       connect  to  VMs  or containers, described above, are the most ordinary
334       kind of LSP. In the OVN northbound database, VIF ports  have  an  empty
335       string  for  their  type. This section describes some of the additional
336       port types.
337
338       A router logical switch port connects a logical  switch  to  a  logical
339       router, designating a particular LRP as its peer.
340
341       A  localnet  logical switch port bridges a logical switch to a physical
342       VLAN. A logical switch may have one or more localnet ports. Such a log‐
343       ical switch is used in two scenarios:
344
345              •      With  one  or more router logical switch ports, to attach
346                     L3 gateway routers and distributed gateways to a physical
347                     network.
348
349              •      With  one or more VIF logical switch ports, to attach VMs
350                     or containers directly to a  physical  network.  In  this
351                     case,  the logical switch is not really logical, since it
352                     is bridged to the physical network rather than  insulated
353                     from  it, and therefore cannot have independent but over‐
354                     lapping IP address namespaces, etc.  A  deployment  might
355                     nevertheless  choose  such a configuration to take advan‐
356                     tage of the OVN control plane and features such  as  port
357                     security and ACLs.
358
359       When  a  logical switch contains multiple localnet ports, the following
360       is assumed.
361
362              •      Each chassis has a bridge mapping for one of the localnet
363                     physical networks only.
364
365              •      To  facilitate interconnectivity between VIF ports of the
366                     switch that are located on different chassis with differ‐
367                     ent  physical network connectivity, the fabric implements
368                     L3 routing between these adjacent physical  network  seg‐
369                     ments.
370
371       Note:  nothing  said  above implies that a chassis cannot be plugged to
372       multiple  physical  networks  as  long  as  they  belong  to  different
373       switches.
374
375       A localport logical switch port is a special kind of VIF logical switch
376       port. These ports are present in every chassis, not bound to  any  par‐
377       ticular  one.  Traffic to such a port will never be forwarded through a
378       tunnel, and traffic from such a port is expected to be destined only to
379       the same chassis, typically in response to a request it received. Open‐
380       Stack Neutron uses a localport port to serve metadata to VMs.  A  meta‐
381       data  proxy  process is attached to this port on every host and all VMs
382       within the same network will reach it at the same IP/MAC address  with‐
383       out  any traffic being sent over a tunnel. For further details, see the
384       OpenStack documentation for networking-ovn.
385
386       LSP types vtep and l2gateway are used for gateways. See  Gateways,  be‐
387       low, for more information.
388
389     Implementation Details
390
391       These  concepts  are details of how OVN is implemented internally. They
392       might still be of interest to users and administrators.
393
394       Logical datapaths are an implementation detail of logical  networks  in
395       the  OVN southbound database. ovn-northd translates each logical switch
396       or router in the northbound database into a  logical  datapath  in  the
397       southbound database Datapath_Binding table.
398
399       For  the most part, ovn-northd also translates each logical switch port
400       in the OVN northbound database into a record in the southbound database
401       Port_Binding  table. The latter table corresponds roughly to the north‐
402       bound Logical_Switch_Port table. It has multiple types of logical  port
403       bindings,  of  which  many  types correspond directly to northbound LSP
404       types. LSP types handled this way include VIF (empty string), localnet,
405       localport, vtep, and l2gateway.
406
407       The  Port_Binding table has some types of port binding that do not cor‐
408       respond directly to logical switch port types. The common is patch port
409       bindings,  known as logical patch ports. These port bindings always oc‐
410       cur in pairs, and a packet that enters on either side comes out on  the
411       other.  ovn-northd  connects  logical  switches and logical routers to‐
412       gether using logical patch ports.
413
414       Port bindings with types vtep, l2gateway, l3gateway,  and  chassisredi‐
415       rect are used for gateways. These are explained in Gateways, below.
416
417   Gateways
418       Gateways  provide  limited  connectivity  between  logical networks and
419       physical ones. They can also provide connectivity between different OVN
420       deployments. This section will focus on the former, and the latter will
421       be described in details in section OVN Deployments Interconnection.
422
423       OVN support multiple kinds of gateways.
424
425     VTEP Gateways
426
427       A ``VTEP gateway’’ connects an OVN logical network to  a  physical  (or
428       virtual)  switch that implements the OVSDB VTEP schema that accompanies
429       Open vSwitch. (The ``VTEP gateway’’ term is a misnomer, since a VTEP is
430       just  a  VXLAN Tunnel Endpoint, but it is a well established name.) See
431       Life Cycle of a VTEP gateway, below, for more information.
432
433       The main intended use case for VTEP  gateways  is  to  attach  physical
434       servers  to  an OVN logical network using a physical top-of-rack switch
435       that supports the OVSDB VTEP schema.
436
437     L2 Gateways
438
439       A L2 gateway simply attaches a designated physical L2 segment available
440       on  some chassis to a logical network. The physical network effectively
441       becomes part of the logical network.
442
443       To set up a L2 gateway, the CMS adds an l2gateway LSP to an appropriate
444       logical  switch,  setting  LSP  options to name the chassis on which it
445       should be bound. ovn-northd copies this configuration into a southbound
446       Port_Binding record. On the designated chassis, ovn-controller forwards
447       packets appropriately to and from the physical segment.
448
449       L2 gateway ports have features in common with localnet ports.  However,
450       with  a  localnet  port, the physical network becomes the transport be‐
451       tween hypervisors. With an L2 gateway, packets  are  still  transported
452       between  hypervisors  over  tunnels and the l2gateway port is only used
453       for the packets that are on the physical network. The  application  for
454       L2  gateways is similar to that for VTEP gateways, e.g. to add non-vir‐
455       tualized machines to a logical network, but L2 gateways do not  require
456       special support from top-of-rack hardware switches.
457
458     L3 Gateway Routers
459
460       As described above under Logical Networks, ordinary OVN logical routers
461       are distributed: they are not implemented in a single place but  rather
462       in  every  hypervisor  chassis. This is a problem for stateful services
463       such as SNAT and DNAT, which need to be implemented  in  a  centralized
464       manner.
465
466       To  allow  for  this  kind  of  functionality,  OVN supports L3 gateway
467       routers, which are OVN logical routers that are implemented in a desig‐
468       nated  chassis.  Gateway routers are typically used between distributed
469       logical routers and physical networks. The distributed  logical  router
470       and the logical switches behind it, to which VMs and containers attach,
471       effectively reside on each hypervisor. The distributed router  and  the
472       gateway  router  are connected by another logical switch, sometimes re‐
473       ferred to as a ``join’’ logical switch. (OVN  logical  routers  may  be
474       connected  to  one another directly, without an intervening switch, but
475       the OVN implementation only supports gateway logical routers  that  are
476       connected to logical switches. Using a join logical switch also reduces
477       the number of IP addresses needed on the distributed  router.)  On  the
478       other  side, the gateway router connects to another logical switch that
479       has a localnet port connecting to the physical network.
480
481       The following diagram shows a typical situation. One  or  more  logical
482       switches LS1, ..., LSn connect to distributed logical router LR1, which
483       in turn connects through LSjoin to gateway logical  router  GLR,  which
484       also connects to logical switch LSlocal, which includes a localnet port
485       to attach to the physical network.
486
487                                       LSlocal
488                                          |
489                                         GLR
490                                          |
491                                       LSjoin
492                                          |
493                                         LR1
494                                          |
495                                     +----+----+
496                                     |    |    |
497                                    LS1  ...  LSn
498
499
500       To configure an L3 gateway router, the CMS sets options:chassis in  the
501       router’s  northbound Logical_Router to the chassis’s name. In response,
502       ovn-northd uses a special l3gateway port binding (instead  of  a  patch
503       binding)  in  the  southbound database to connect the logical router to
504       its neighbors. In turn, ovn-controller tunnels  packets  to  this  port
505       binding  to  the  designated  L3 gateway chassis, instead of processing
506       them locally.
507
508       DNAT and SNAT rules may be associated with a gateway router, which pro‐
509       vides  a central location that can handle one-to-many SNAT (aka IP mas‐
510       querading). Distributed gateway ports, described  below,  also  support
511       NAT.
512
513     Distributed Gateway Ports
514
515       A  distributed  gateway port is a logical router port that is specially
516       configured to designate one distinguished chassis, called  the  gateway
517       chassis,  for centralized processing. A distributed gateway port should
518       connect to a logical switch that has an LSP that  connects  externally,
519       that  is,  either a localnet LSP or a connection to another OVN deploy‐
520       ment (see OVN Deployments Interconnection). Packets that  traverse  the
521       distributed  gateway  port  are processed without involving the gateway
522       chassis when they can be, but when needed they do  take  an  extra  hop
523       through it.
524
525       The  following  diagram  illustrates  the  use of a distributed gateway
526       port. A number of logical switches LS1, ..., LSn connect to distributed
527       logical  router  LR1,  which  in  turn connects through the distributed
528       gateway port to logical switch LSlocal that includes a localnet port to
529       attach to the physical network.
530
531                                       LSlocal
532                                          |
533                                         LR1
534                                          |
535                                     +----+----+
536                                     |    |    |
537                                    LS1  ...  LSn
538
539
540       ovn-northd  creates  two southbound Port_Binding records to represent a
541       distributed gateway port, instead of the usual one. One of these  is  a
542       patch port binding named for the LRP, which is used for as much traffic
543       as it can. The other one is a port binding with  type  chassisredirect,
544       named  cr-port.  The  chassisredirect  port binding has one specialized
545       job: when a packet is output to it, the flow table causes it to be tun‐
546       neled to the gateway chassis, at which point it is automatically output
547       to the patch port binding. Thus, the flow table can output to this port
548       binding  in  cases where a particular task has to happen on the gateway
549       chassis. The chassisredirect port binding is not  otherwise  used  (for
550       example, it never receives packets).
551
552       The  CMS  may configure distributed gateway ports three different ways.
553       See  Distributed  Gateway  Ports  in  the   documentation   for   Logi‐
554       cal_Router_Port in ovn-nb(5) for details.
555
556       Distributed gateway ports support high availability. When more than one
557       chassis is specified, OVN only uses one at a time as the gateway  chas‐
558       sis. OVN uses BFD to monitor gateway connectivity, preferring the high‐
559       est-priority gateway that is online.
560
561       A logical router can have multiple distributed gateway ports, each con‐
562       necting  different  external  networks.  Load balancing is not yet sup‐
563       ported for logical routers with more than one distributed gateway  port
564       configured.
565
566       Physical VLAN MTU Issues
567
568       Consider the preceding diagram again:
569
570                                       LSlocal
571                                          |
572                                         LR1
573                                          |
574                                     +----+----+
575                                     |    |    |
576                                    LS1  ...  LSn
577
578
579       Suppose that each logical switch LS1, ..., LSn is bridged to a physical
580       VLAN-tagged network attached to a localnet port on LSlocal, over a dis‐
581       tributed  gateway  port  on LR1. If a packet originating on LSi is des‐
582       tined to the external network, OVN sends it to the gateway chassis over
583       a  tunnel.  There,  the packet traverses LR1’s logical router pipeline,
584       possibly undergoes NAT, and eventually ends up  at  LSlocal’s  localnet
585       port.  If  all  of the physical links in the network have the same MTU,
586       then the packet’s transit across a tunnel causes an MTU problem: tunnel
587       overhead  prevents a packet that uses the full physical MTU from cross‐
588       ing the tunnel to the gateway chassis (without fragmentation).
589
590       OVN offers two solutions to this problem, the  reside-on-redirect-chas‐
591       sis  and  redirect-type  options.  Both  solutions require each logical
592       switch LS1, ..., LSn to include a localnet  logical  switch  port  LN1,
593       ...,  LNn  respectively,  that  is  present on each chassis. Both cause
594       packets to be sent over the localnet ports  instead  of  tunnels.  They
595       differ  in which packets-some or all-are sent this way. The most promi‐
596       nent tradeoff between these options is that  reside-on-redirect-chassis
597       is easier to configure and that redirect-type performs better for east-
598       west traffic.
599
600       The first solution is the reside-on-redirect-chassis option for logical
601       router  ports. Setting this option on a LRP from (e.g.) LS1 to LR1 dis‐
602       ables forwarding from LS1 to LR1 except  on  the  gateway  chassis.  On
603       chassis  other  than the gateway chassis, this single change means that
604       packets that would otherwise have been forwarded  to  LR1  are  instead
605       forwarded  to  LN1. The instance of LN1 on the gateway chassis then re‐
606       ceives the packet and forwards it to LR1. The packet traverses the  LR1
607       logical router pipeline, possibly undergoes NAT, and eventually ends up
608       at LSlocal’s localnet port. The packet never traverses a tunnel, avoid‐
609       ing the MTU issue.
610
611       This option has the further consequence of centralizing ``distributed’’
612       logical router LR1, since no packets are forwarded from LS1 to  LR1  on
613       any  chassis other than the gateway chassis. Therefore, east-west traf‐
614       fic passes through the gateway  chassis,  not  just  north-south.  (The
615       naive  ``fix’’  of  allowing east-west traffic to flow directly between
616       chassis over LN1 does not work because routing sets the Ethernet source
617       address to LR1’s source address. Seeing this single Ethernet source ad‐
618       dress originate from all of  the  chassis  will  confuse  the  physical
619       switch.)
620
621       Do not set the reside-on-redirect-chassis option on a distributed gate‐
622       way port. In the diagram above, it would be set on the LRPs  connecting
623       LS1, ..., LSn to LR1.
624
625       The second solution is the redirect-type option for distributed gateway
626       ports. Setting this option to bridged causes  packets  that  are  redi‐
627       rected  to the gateway chassis to go over the localnet ports instead of
628       being tunneled. This option does not change how OVN treats packets  not
629       redirected to the gateway chassis.
630
631       The  redirect-type option requires the administrator or the CMS to con‐
632       figure each participating chassis with a unique  Ethernet  address  for
633       the  logical  router  by  setting  ovn-chassis-mac-mappings in the Open
634       vSwitch database, for use by ovn-controller. This makes it more  diffi‐
635       cult to configure than reside-on-redirect-chassis.
636
637       Set the redirect-type option on a distributed gateway port.
638
639       Using Distributed Gateway Ports For Scalability
640
641       Although  the  primary  goal of distributed gateway ports is to provide
642       connectivity to external networks, there is  a  special  use  case  for
643       scalability.
644
645       In  some  deployments,  such  as the ones using ovn-kubernetes, logical
646       switches are bound to individual chassises, and are connected by a dis‐
647       tributed logical router. In such deployments, the chassis level logical
648       switches are centralized on the chassis instead of  distributed,  which
649       means  the ovn-controller on each chassis doesn’t need to process flows
650       and ports of logical switches on other chassises. However, without  any
651       specific  hint,  ovn-controller  would  still  process  all the logical
652       switches as if they are fully distributed. In  this  case,  distributed
653       gateway port can be very useful. The chassis level logical switches can
654       be connected to the distributed router using distributed gateway ports,
655       by setting the gateway chassis (or HA chassis groups with only a single
656       chassis in it) to the chassis that each logical  switch  is  bound  to.
657       ovn-controller  would  then skip processing the logical switches on all
658       the other chassises, largely improving the scalability, especially when
659       there are a big number of chassises.
660
661   Life Cycle of a VIF
662       Tables and their schemas presented in isolation are difficult to under‐
663       stand. Here’s an example.
664
665       A VIF on a hypervisor is a virtual network interface attached either to
666       a  VM  or a container running directly on that hypervisor (This is dif‐
667       ferent from the interface of a container running inside a VM).
668
669       The steps in this example refer often to details of  the  OVN  and  OVN
670       Northbound  database  schemas.  Please see ovn-sb(5) and ovn-nb(5), re‐
671       spectively, for the full story on these databases.
672
673              1.  A VIF’s life cycle begins when a CMS administrator creates a
674                  new VIF using the CMS user interface or API and adds it to a
675                  switch (one implemented by OVN as a logical switch). The CMS
676                  updates  its  own  configuration.  This includes associating
677                  unique, persistent identifier vif-id  and  Ethernet  address
678                  mac with the VIF.
679
680              2.  The  CMS  plugin  updates the OVN Northbound database to in‐
681                  clude  the  new  VIF,  by  adding  a  row   to   the   Logi‐
682                  cal_Switch_Port  table.  In the new row, name is vif-id, mac
683                  is mac, switch points to  the  OVN  logical  switch’s  Logi‐
684                  cal_Switch  record, and other columns are initialized appro‐
685                  priately.
686
687              3.  ovn-northd receives the OVN Northbound database  update.  In
688                  turn,  it  makes the corresponding updates to the OVN South‐
689                  bound database, by adding rows to the OVN  Southbound  data‐
690                  base  Logical_Flow table to reflect the new port, e.g. add a
691                  flow to recognize that packets destined to  the  new  port’s
692                  MAC  address  should be delivered to it, and update the flow
693                  that delivers broadcast and multicast packets to include the
694                  new  port. It also creates a record in the Binding table and
695                  populates all its columns except the column that  identifies
696                  the chassis.
697
698              4.  On  every  hypervisor,  ovn-controller  receives  the  Logi‐
699                  cal_Flow table updates that ovn-northd made in the  previous
700                  step.  As  long  as the VM that owns the VIF is powered off,
701                  ovn-controller cannot do much; it cannot, for  example,  ar‐
702                  range  to  send  packets to or receive packets from the VIF,
703                  because the VIF does not actually exist anywhere.
704
705              5.  Eventually, a user powers on the VM that owns  the  VIF.  On
706                  the  hypervisor  where the VM is powered on, the integration
707                  between the hypervisor and Open vSwitch (described in  Docu‐
708                  mentation/topics/integration.rst  in the Open vSwitch source
709                  tree) adds the VIF to the OVN integration bridge and  stores
710                  vif-id  in external_ids:iface-id to indicate that the inter‐
711                  face is an instantiation of the new VIF. (None of this  code
712                  is  new  in  OVN; this is pre-existing integration work that
713                  has already been done on hypervisors that support OVS.)
714
715              6.  On the hypervisor where the VM is powered on, ovn-controller
716                  notices  external_ids:iface-id  in the new Interface. In re‐
717                  sponse, in the OVN Southbound DB, it updates the Binding ta‐
718                  ble’s chassis column for the row that links the logical port
719                  from external_ids: iface-id to  the  hypervisor.  Afterward,
720                  ovn-controller  updates  the local hypervisor’s OpenFlow ta‐
721                  bles so that packets to and from the VIF are  properly  han‐
722                  dled.
723
724              7.  Some CMS systems, including OpenStack, fully start a VM only
725                  when its networking is ready. To  support  this,  ovn-northd
726                  notices  the  chassis  column updated for the row in Binding
727                  table and pushes this upward by updating the  up  column  in
728                  the  OVN  Northbound database’s Logical_Switch_Port table to
729                  indicate that the VIF is now up. The CMS, if  it  uses  this
730                  feature,  can  then  react by allowing the VM’s execution to
731                  proceed.
732
733              8.  On every hypervisor but  the  one  where  the  VIF  resides,
734                  ovn-controller  notices  the completely populated row in the
735                  Binding table. This provides ovn-controller the physical lo‐
736                  cation  of  the  logical  port, so each instance updates the
737                  OpenFlow tables of its switch  (based  on  logical  datapath
738                  flows  in  the OVN DB Logical_Flow table) so that packets to
739                  and from the VIF can be properly handled via tunnels.
740
741              9.  Eventually, a user powers off the VM that owns the  VIF.  On
742                  the  hypervisor  where  the  VM  was powered off, the VIF is
743                  deleted from the OVN integration bridge.
744
745              10. On the hypervisor where the VM  was  powered  off,  ovn-con‐
746                  troller  notices  that  the VIF was deleted. In response, it
747                  removes the Chassis column content in the Binding table  for
748                  the logical port.
749
750              11. On  every hypervisor, ovn-controller notices the empty Chas‐
751                  sis column in the Binding table’s row for the logical  port.
752                  This  means that ovn-controller no longer knows the physical
753                  location of the logical port, so each instance  updates  its
754                  OpenFlow table to reflect that.
755
756              12. Eventually,  when  the  VIF  (or its entire VM) is no longer
757                  needed by anyone, an administrator deletes the VIF using the
758                  CMS  user interface or API. The CMS updates its own configu‐
759                  ration.
760
761              13. The CMS plugin removes the VIF from the OVN Northbound data‐
762                  base, by deleting its row in the Logical_Switch_Port table.
763
764              14. ovn-northd  receives  the  OVN Northbound update and in turn
765                  updates the OVN Southbound database accordingly, by removing
766                  or  updating the rows from the OVN Southbound database Logi‐
767                  cal_Flow table and Binding table that were  related  to  the
768                  now-destroyed VIF.
769
770              15. On  every  hypervisor,  ovn-controller  receives  the  Logi‐
771                  cal_Flow table updates that ovn-northd made in the  previous
772                  step.  ovn-controller updates OpenFlow tables to reflect the
773                  update, although there may not be much to do, since the  VIF
774                  had  already become unreachable when it was removed from the
775                  Binding table in a previous step.
776
777   Life Cycle of a Container Interface Inside a VM
778       OVN provides virtual network  abstractions  by  converting  information
779       written in OVN_NB database to OpenFlow flows in each hypervisor. Secure
780       virtual networking for multi-tenants can only be provided if  OVN  con‐
781       troller  is the only entity that can modify flows in Open vSwitch. When
782       the Open vSwitch integration bridge resides in the hypervisor, it is  a
783       fair assumption to make that tenant workloads running inside VMs cannot
784       make any changes to Open vSwitch flows.
785
786       If the infrastructure provider trusts the applications inside the  con‐
787       tainers  not  to break out and modify the Open vSwitch flows, then con‐
788       tainers can be run in hypervisors. This is also the case when  contain‐
789       ers  are  run  inside  the VMs and Open vSwitch integration bridge with
790       flows added by OVN controller resides in the  same  VM.  For  both  the
791       above  cases,  the workflow is the same as explained with an example in
792       the previous section ("Life Cycle of a VIF").
793
794       This section talks about the life cycle of a container interface  (CIF)
795       when containers are created in the VMs and the Open vSwitch integration
796       bridge resides inside the hypervisor. In this case, even if a container
797       application breaks out, other tenants are not affected because the con‐
798       tainers running inside the VMs cannot modify  the  flows  in  the  Open
799       vSwitch integration bridge.
800
801       When  multiple  containers  are created inside a VM, there are multiple
802       CIFs associated with them. The network traffic  associated  with  these
803       CIFs  need  to reach the Open vSwitch integration bridge running in the
804       hypervisor for OVN to support virtual network abstractions. OVN  should
805       also be able to distinguish network traffic coming from different CIFs.
806       There are two ways to distinguish network traffic of CIFs.
807
808       One way is to provide one VIF for every CIF  (1:1  model).  This  means
809       that  there  could  be a lot of network devices in the hypervisor. This
810       would slow down OVS because of all the additional CPU cycles needed for
811       the management of all the VIFs. It would also mean that the entity cre‐
812       ating the containers in a VM should also be able to create  the  corre‐
813       sponding VIFs in the hypervisor.
814
815       The  second  way  is  to  provide a single VIF for all the CIFs (1:many
816       model). OVN could then distinguish network traffic coming from  differ‐
817       ent CIFs via a tag written in every packet. OVN uses this mechanism and
818       uses VLAN as the tagging mechanism.
819
820              1.  A CIF’s life cycle begins when a container is spawned inside
821                  a  VM  by  the  either the same CMS that created the VM or a
822                  tenant that owns that VM or even a  container  Orchestration
823                  System that is different than the CMS that initially created
824                  the VM. Whoever the entity is, it will need to know the vif-
825                  id  that  is associated with the network interface of the VM
826                  through which the container interface’s network  traffic  is
827                  expected  to  go  through.  The entity that creates the con‐
828                  tainer interface will also need to choose an unused VLAN in‐
829                  side that VM.
830
831              2.  The  container  spawning  entity (either directly or through
832                  the CMS that manages the underlying infrastructure)  updates
833                  the  OVN  Northbound  database  to  include  the new CIF, by
834                  adding a row to the Logical_Switch_Port table.  In  the  new
835                  row,  name is any unique identifier, parent_name is the vif-
836                  id of the VM through which the CIF’s network traffic is  ex‐
837                  pected  to go through and the tag is the VLAN tag that iden‐
838                  tifies the network traffic of that CIF.
839
840              3.  ovn-northd receives the OVN Northbound database  update.  In
841                  turn,  it  makes the corresponding updates to the OVN South‐
842                  bound database, by adding rows to the OVN  Southbound  data‐
843                  base’s  Logical_Flow  table to reflect the new port and also
844                  by creating a new row in the Binding  table  and  populating
845                  all  its columns except the column that identifies the chas‐
846                  sis.
847
848              4.  On  every  hypervisor,  ovn-controller  subscribes  to   the
849                  changes  in  the Binding table. When a new row is created by
850                  ovn-northd that includes a value in  parent_port  column  of
851                  Binding  table,  the  ovn-controller in the hypervisor whose
852                  OVN integration bridge has that same value in vif-id in  ex‐
853                  ternal_ids:iface-id  updates the local hypervisor’s OpenFlow
854                  tables so that packets to and from the VIF with the particu‐
855                  lar  VLAN tag are properly handled. Afterward it updates the
856                  chassis column of the Binding to reflect the physical  loca‐
857                  tion.
858
859              5.  One  can only start the application inside the container af‐
860                  ter the  underlying  network  is  ready.  To  support  this,
861                  ovn-northd notices the updated chassis column in Binding ta‐
862                  ble and updates the up column in the  OVN  Northbound  data‐
863                  base’s Logical_Switch_Port table to indicate that the CIF is
864                  now up. The entity responsible to start the container appli‐
865                  cation queries this value and starts the application.
866
867              6.  Eventually  the  entity  that  created  and started the con‐
868                  tainer, stops it. The entity, through the CMS (or  directly)
869                  deletes its row in the Logical_Switch_Port table.
870
871              7.  ovn-northd  receives  the  OVN Northbound update and in turn
872                  updates the OVN Southbound database accordingly, by removing
873                  or  updating the rows from the OVN Southbound database Logi‐
874                  cal_Flow table that were related to the  now-destroyed  CIF.
875                  It also deletes the row in the Binding table for that CIF.
876
877              8.  On  every  hypervisor,  ovn-controller  receives  the  Logi‐
878                  cal_Flow table updates that ovn-northd made in the  previous
879                  step.  ovn-controller updates OpenFlow tables to reflect the
880                  update.
881
882   Architectural Physical Life Cycle of a Packet
883       This section describes how a packet travels from one virtual machine or
884       container to another through OVN. This description focuses on the phys‐
885       ical treatment of a packet; for a description of the logical life cycle
886       of a packet, please refer to the Logical_Flow table in ovn-sb(5).
887
888       This  section  mentions  several  data and metadata fields, for clarity
889       summarized here:
890
891              tunnel key
892                     When OVN encapsulates a packet in Geneve or another  tun‐
893                     nel,  it attaches extra data to it to allow the receiving
894                     OVN instance to process it correctly. This takes  differ‐
895                     ent  forms depending on the particular encapsulation, but
896                     in each case we refer to it here as the  ``tunnel  key.’’
897                     See Tunnel Encapsulations, below, for details.
898
899              logical datapath field
900                     A field that denotes the logical datapath through which a
901                     packet is being processed. OVN uses the field that  Open‐
902                     Flow  1.1+ simply (and confusingly) calls ``metadata’’ to
903                     store the logical datapath. (This field is passed  across
904                     tunnels as part of the tunnel key.)
905
906              logical input port field
907                     A  field  that  denotes  the  logical port from which the
908                     packet entered the logical datapath. OVN stores  this  in
909                     Open vSwitch extension register number 14.
910
911                     Geneve  and  STT  tunnels  pass this field as part of the
912                     tunnel key. Ramp switch VXLAN tunnels do  not  explicitly
913                     carry  a  logical  input port, but since they are used to
914                     communicate with gateways  that  from  OVN’s  perspective
915                     consist  of  only  a single logical port, so that OVN can
916                     set the logical input port field to this one  on  ingress
917                     to  the  OVN  logical pipeline. As for regular VXLAN tun‐
918                     nels, they don’t carry input port field at all. This puts
919                     additional  limitations  on cluster capabilities that are
920                     described in Tunnel Encapsulations section.
921
922              logical output port field
923                     A field that denotes the  logical  port  from  which  the
924                     packet  will leave the logical datapath. This is initial‐
925                     ized to 0 at the beginning of the logical  ingress  pipe‐
926                     line.  OVN stores this in Open vSwitch extension register
927                     number 15.
928
929                     Geneve, STT and regular VXLAN tunnels pass this field  as
930                     part  of the tunnel key. Ramp switch VXLAN tunnels do not
931                     transmit the logical output port field, and since they do
932                     not  carry a logical output port field in the tunnel key,
933                     when a packet is received from ramp switch  VXLAN  tunnel
934                     by  an OVN hypervisor, the packet is resubmitted to table
935                     8 to  determine  the  output  port(s);  when  the  packet
936                     reaches  table 37, these packets are resubmitted to table
937                     38 for local delivery  by  checking  a  MLF_RCV_FROM_RAMP
938                     flag,  which  is  set when the packet arrives from a ramp
939                     tunnel.
940
941              conntrack zone field for logical ports
942                     A field that denotes the  connection  tracking  zone  for
943                     logical  ports. The value only has local significance and
944                     is not meaningful between chassis. This is initialized to
945                     0  at  the beginning of the logical ingress pipeline. OVN
946                     stores this in Open vSwitch extension register number 13.
947
948              conntrack zone fields for routers
949                     Fields that denote  the  connection  tracking  zones  for
950                     routers.  These  values  only have local significance and
951                     are not meaningful between chassis. OVN stores  the  zone
952                     information  for  north to south traffic (for DNATting or
953                     ECMP symmetric replies) in Open vSwitch extension  regis‐
954                     ter  number  11  and  zone information for south to north
955                     traffic (for SNATing) in Open vSwitch extension  register
956                     number 12.
957
958              logical flow flags
959                     The  logical flags are intended to handle keeping context
960                     between tables in order to decide which rules  in  subse‐
961                     quent  tables  are  matched. These values only have local
962                     significance and are not meaningful between chassis.  OVN
963                     stores the logical flags in Open vSwitch extension regis‐
964                     ter number 10.
965
966              VLAN ID
967                     The VLAN ID is used as an interface between OVN and  con‐
968                     tainers nested inside a VM (see Life Cycle of a container
969                     interface inside a VM, above, for more information).
970
971       Initially, a VM or container on the ingress hypervisor sends  a  packet
972       on a port attached to the OVN integration bridge. Then:
973
974              1.  OpenFlow  table  0 performs physical-to-logical translation.
975                  It matches the packet’s ingress port. Its  actions  annotate
976                  the  packet  with  logical  metadata, by setting the logical
977                  datapath field to identify the  logical  datapath  that  the
978                  packet  is  traversing  and  the logical input port field to
979                  identify the ingress port. Then it resubmits to table  8  to
980                  enter the logical ingress pipeline.
981
982                  Packets  that  originate from a container nested within a VM
983                  are treated in a slightly  different  way.  The  originating
984                  container  can  be  distinguished  based on the VIF-specific
985                  VLAN ID, so the physical-to-logical translation flows  addi‐
986                  tionally  match  on  VLAN  ID and the actions strip the VLAN
987                  header. Following this step, OVN treats  packets  from  con‐
988                  tainers just like any other packets.
989
990                  Table  0 also processes packets that arrive from other chas‐
991                  sis. It distinguishes them from  other  packets  by  ingress
992                  port,  which  is a tunnel. As with packets just entering the
993                  OVN pipeline, the actions annotate these packets with  logi‐
994                  cal  datapath  metadata.  For  tunnel types that support it,
995                  they are also annotated with logical ingress port  metadata.
996                  In  addition, the actions set the logical output port field,
997                  which is available because in OVN tunneling occurs after the
998                  logical  output  port  is known. These pieces of information
999                  are obtained from the  tunnel  encapsulation  metadata  (see
1000                  Tunnel  Encapsulations  for  encoding details). Then the ac‐
1001                  tions resubmit to table 33 to enter the logical egress pipe‐
1002                  line.
1003
1004              2.  OpenFlow  tables  8  through  31 execute the logical ingress
1005                  pipeline from the Logical_Flow table in the  OVN  Southbound
1006                  database.  These  tables  are expressed entirely in terms of
1007                  logical concepts like logical ports and logical datapaths. A
1008                  big  part  of ovn-controller’s job is to translate them into
1009                  equivalent OpenFlow (in particular it translates  the  table
1010                  numbers:  Logical_Flow  tables  0 through 23 become OpenFlow
1011                  tables 8 through 31).
1012
1013                  Each logical flow maps to one or more OpenFlow flows. An ac‐
1014                  tual  packet  ordinarily matches only one of these, although
1015                  in some cases it can match more  than  one  of  these  flows
1016                  (which  is  not  a problem because all of them have the same
1017                  actions). ovn-controller uses the first 32 bits of the logi‐
1018                  cal  flow’s  UUID  as  the  cookie  for its OpenFlow flow or
1019                  flows. (This is not necessarily unique, since the  first  32
1020                  bits of a logical flow’s UUID is not necessarily unique.)
1021
1022                  Some logical flows can map to the Open vSwitch ``conjunctive
1023                  match’’ extension (see ovs-fields(7)). Flows with a conjunc‐
1024                  tion  action  use  an OpenFlow cookie of 0, because they can
1025                  correspond to multiple logical flows. The OpenFlow flow  for
1026                  a conjunctive match includes a match on conj_id.
1027
1028                  Some  logical  flows  may not be represented in the OpenFlow
1029                  tables on a given hypervisor, if they could not be  used  on
1030                  that  hypervisor. For example, if no VIF in a logical switch
1031                  resides on a given hypervisor, and the logical switch is not
1032                  otherwise  reachable  on that hypervisor (e.g. over a series
1033                  of hops through logical switches and routers starting from a
1034                  VIF  on  the  hypervisor),  then the logical flow may not be
1035                  represented there.
1036
1037                  Most OVN actions  have  fairly  obvious  implementations  in
1038                  OpenFlow (with OVS extensions), e.g. next; is implemented as
1039                  resubmit, field = constant; as set_field. A  few  are  worth
1040                  describing in more detail:
1041
1042                  output:
1043                         Implemented  by  resubmitting the packet to table 37.
1044                         If the pipeline executes more than one output action,
1045                         then  each one is separately resubmitted to table 37.
1046                         This can be used  to  send  multiple  copies  of  the
1047                         packet to multiple ports. (If the packet was not mod‐
1048                         ified between the output actions,  and  some  of  the
1049                         copies  are destined to the same hypervisor, then us‐
1050                         ing a logical multicast output port would save  band‐
1051                         width between hypervisors.)
1052
1053                  get_arp(P, A);
1054                  get_nd(P, A);
1055                       Implemented  by storing arguments into OpenFlow fields,
1056                       then resubmitting to  table  66,  which  ovn-controller
1057                       populates with flows generated from the MAC_Binding ta‐
1058                       ble in the OVN Southbound database. If there is a match
1059                       in  table  66,  then its actions store the bound MAC in
1060                       the Ethernet destination address field.
1061
1062                       (The OpenFlow actions save  and  restore  the  OpenFlow
1063                       fields  used for the arguments, so that the OVN actions
1064                       do not have to be aware of this temporary use.)
1065
1066                  put_arp(P, A, E);
1067                  put_nd(P, A, E);
1068                       Implemented by  storing  the  arguments  into  OpenFlow
1069                       fields,  then  outputting  a  packet to ovn-controller,
1070                       which updates the MAC_Binding table.
1071
1072                       (The OpenFlow actions save  and  restore  the  OpenFlow
1073                       fields  used for the arguments, so that the OVN actions
1074                       do not have to be aware of this temporary use.)
1075
1076                  R = lookup_arp(P, A, M);
1077                  R = lookup_nd(P, A, M);
1078                       Implemented by storing arguments into OpenFlow  fields,
1079                       then  resubmitting  to  table  67, which ovn-controller
1080                       populates with flows generated from the MAC_Binding ta‐
1081                       ble in the OVN Southbound database. If there is a match
1082                       in table 67, then its actions set the logical flow flag
1083                       MLF_LOOKUP_MAC.
1084
1085                       (The  OpenFlow  actions  save  and restore the OpenFlow
1086                       fields used for the arguments, so that the OVN  actions
1087                       do not have to be aware of this temporary use.)
1088
1089              3.  OpenFlow tables 37 through 39 implement the output action in
1090                  the logical ingress pipeline. Specifically, table 37 handles
1091                  packets  to  remote hypervisors, table 38 handles packets to
1092                  the local hypervisor, and table 39  checks  whether  packets
1093                  whose logical ingress and egress port are the same should be
1094                  discarded.
1095
1096                  Logical patch ports are a special case. Logical patch  ports
1097                  do  not  have  a physical location and effectively reside on
1098                  every hypervisor. Thus, flow table 38, for output  to  ports
1099                  on the local hypervisor, naturally implements output to uni‐
1100                  cast logical patch ports too.  However,  applying  the  same
1101                  logic to a logical patch port that is part of a logical mul‐
1102                  ticast group yields packet duplication, because each  hyper‐
1103                  visor  that  contains  a logical port in the multicast group
1104                  will also output the packet to the logical patch port. Thus,
1105                  multicast  groups implement output to logical patch ports in
1106                  table 37.
1107
1108                  Each flow in table 37 matches on a logical output  port  for
1109                  unicast  or  multicast  logical ports that include a logical
1110                  port on a remote hypervisor. Each flow’s  actions  implement
1111                  sending a packet to the port it matches. For unicast logical
1112                  output ports on remote hypervisors, the actions set the tun‐
1113                  nel  key  to  the correct value, then send the packet on the
1114                  tunnel port to the correct hypervisor. (When the remote  hy‐
1115                  pervisor  receives  the packet, table 0 there will recognize
1116                  it as a tunneled packet and pass it along to table 38.)  For
1117                  multicast logical output ports, the actions send one copy of
1118                  the packet to each remote hypervisor, in the same way as for
1119                  unicast  destinations. If a multicast group includes a logi‐
1120                  cal port or ports on the local hypervisor, then its  actions
1121                  also resubmit to table 38. Table 37 also includes:
1122
1123                  •      A higher-priority rule to match packets received from
1124                         ramp switch tunnels, based on flag MLF_RCV_FROM_RAMP,
1125                         and  resubmit these packets to table 38 for local de‐
1126                         livery. Packets received  from  ramp  switch  tunnels
1127                         reach  here  because of a lack of logical output port
1128                         field in the tunnel key and thus these packets needed
1129                         to  be  submitted  to table 8 to determine the output
1130                         port.
1131
1132                  •      A higher-priority rule to match packets received from
1133                         ports  of  type localport, based on the logical input
1134                         port, and resubmit these packets to table 38 for  lo‐
1135                         cal  delivery. Ports of type localport exist on every
1136                         hypervisor and by  definition  their  traffic  should
1137                         never go out through a tunnel.
1138
1139                  •      A higher-priority rule to match packets that have the
1140                         MLF_LOCAL_ONLY logical flow flag set, and whose  des‐
1141                         tination  is a multicast address. This flag indicates
1142                         that the packet should not be delivered to remote hy‐
1143                         pervisors, even if the multicast destination includes
1144                         ports on remote hypervisors. This flag is  used  when
1145                         ovn-controller  is  the  originator  of the multicast
1146                         packet. Since each ovn-controller instance is  origi‐
1147                         nating these packets, the packets only need to be de‐
1148                         livered to local ports.
1149
1150                  •      A fallback flow that resubmits to table 38  if  there
1151                         is no other match.
1152
1153                  Flows in table 38 resemble those in table 37 but for logical
1154                  ports that reside locally rather than remotely. For  unicast
1155                  logical  output  ports  on the local hypervisor, the actions
1156                  just resubmit to table 39. For multicast output  ports  that
1157                  include  one  or more logical ports on the local hypervisor,
1158                  for each such logical port P, the actions change the logical
1159                  output port to P, then resubmit to table 39.
1160
1161                  A  special  case  is that when a localnet port exists on the
1162                  datapath, remote port is connected by switching to  the  lo‐
1163                  calnet port. In this case, instead of adding a flow in table
1164                  37 to reach the remote port, a flow is added in table 38  to
1165                  switch  the logical outport to the localnet port, and resub‐
1166                  mit to table 38 as if it were unicasted to a logical port on
1167                  the local hypervisor.
1168
1169                  Table 39 matches and drops packets for which the logical in‐
1170                  put and output ports are the same and the MLF_ALLOW_LOOPBACK
1171                  flag  is  not  set. It also drops MLF_LOCAL_ONLY packets di‐
1172                  rected to a localnet port. It resubmits other packets to ta‐
1173                  ble 40.
1174
1175              4.  OpenFlow  tables  40  through  63 execute the logical egress
1176                  pipeline from the Logical_Flow table in the  OVN  Southbound
1177                  database.  The  egress pipeline can perform a final stage of
1178                  validation before packet delivery. Eventually, it  may  exe‐
1179                  cute  an  output  action, which ovn-controller implements by
1180                  resubmitting to table 64. A packet for  which  the  pipeline
1181                  never  executes  output  is effectively dropped (although it
1182                  may have been transmitted through a tunnel across a physical
1183                  network).
1184
1185                  The egress pipeline cannot change the logical output port or
1186                  cause further tunneling.
1187
1188              5.  Table 64 bypasses OpenFlow loopback when  MLF_ALLOW_LOOPBACK
1189                  is  set. Logical loopback was handled in table 39, but Open‐
1190                  Flow by default  also  prevents  loopback  to  the  OpenFlow
1191                  ingress port. Thus, when MLF_ALLOW_LOOPBACK is set, OpenFlow
1192                  table 64 saves the OpenFlow ingress port, sets it  to  zero,
1193                  resubmits  to  table  65 for logical-to-physical transforma‐
1194                  tion, and then restores the OpenFlow  ingress  port,  effec‐
1195                  tively  disabling  OpenFlow  loopback prevents. When MLF_AL‐
1196                  LOW_LOOPBACK is unset, table 64 flow simply resubmits to ta‐
1197                  ble 65.
1198
1199              6.  OpenFlow  table 65 performs logical-to-physical translation,
1200                  the opposite of table 0. It  matches  the  packet’s  logical
1201                  egress  port.  Its actions output the packet to the port at‐
1202                  tached to the OVN integration bridge  that  represents  that
1203                  logical  port.  If  the  logical  egress port is a container
1204                  nested with a VM, then before sending the packet the actions
1205                  push on a VLAN header with an appropriate VLAN ID.
1206
1207   Logical Routers and Logical Patch Ports
1208       Typically  logical routers and logical patch ports do not have a physi‐
1209       cal location and effectively reside on every hypervisor.  This  is  the
1210       case  for  logical  patch  ports  between  logical  routers and logical
1211       switches behind those logical routers, to which VMs (and VIFs) attach.
1212
1213       Consider a packet sent from one virtual machine or container to another
1214       VM  or  container  that  resides on a different subnet. The packet will
1215       traverse tables 0 to 65 as described in the previous section  Architec‐
1216       tural  Physical Life Cycle of a Packet, using the logical datapath rep‐
1217       resenting the logical switch that the sender is attached to.  At  table
1218       37, the packet will use the fallback flow that resubmits locally to ta‐
1219       ble 38 on the same hypervisor. In this case, all of the processing from
1220       table 0 to table 65 occurs on the hypervisor where the sender resides.
1221
1222       When  the packet reaches table 65, the logical egress port is a logical
1223       patch port. ovn-controller implements output to the  logical  patch  is
1224       packet  by cloning and resubmitting directly to the first OpenFlow flow
1225       table in the ingress pipeline, setting the logical ingress port to  the
1226       peer  logical patch port, and using the peer logical patch port’s logi‐
1227       cal datapath (that represents the logical router).
1228
1229       The packet re-enters the ingress pipeline in order to traverse tables 8
1230       to 65 again, this time using the logical datapath representing the log‐
1231       ical router. The processing continues as described in the previous sec‐
1232       tion  Architectural  Physical  Life  Cycle of a Packet. When the packet
1233       reachs table 65, the logical egress port will once again be  a  logical
1234       patch  port.  In the same manner as described above, this logical patch
1235       port will cause the packet to be resubmitted to OpenFlow  tables  8  to
1236       65,  this  time  using  the  logical  datapath representing the logical
1237       switch that the destination VM or container is attached to.
1238
1239       The packet traverses tables 8 to 65 a third and final time. If the des‐
1240       tination  VM or container resides on a remote hypervisor, then table 37
1241       will send the packet on a tunnel port from the sender’s  hypervisor  to
1242       the remote hypervisor. Finally table 65 will output the packet directly
1243       to the destination VM or container.
1244
1245       The following sections describe two exceptions, where  logical  routers
1246       and/or logical patch ports are associated with a physical location.
1247
1248     Gateway Routers
1249
1250       A  gateway router is a logical router that is bound to a physical loca‐
1251       tion. This includes all of the  logical  patch  ports  of  the  logical
1252       router,  as  well  as  all  of  the peer logical patch ports on logical
1253       switches. In the OVN Southbound database, the Port_Binding entries  for
1254       these  logical patch ports use the type l3gateway rather than patch, in
1255       order to distinguish that these logical patch  ports  are  bound  to  a
1256       chassis.
1257
1258       When a hypervisor processes a packet on a logical datapath representing
1259       a logical switch, and the logical egress port is a l3gateway port  rep‐
1260       resenting  connectivity  to  a  gateway router, the packet will match a
1261       flow in table 37 that sends the packet on a tunnel port to the  chassis
1262       where  the  gateway router resides. This processing in table 37 is done
1263       in the same manner as for VIFs.
1264
1265     Distributed Gateway Ports
1266
1267       This section provides additional details on distributed gateway  ports,
1268       outlined earlier.
1269
1270       The  primary  design  goal  of distributed gateway ports is to allow as
1271       much traffic as possible to be handled locally on the hypervisor  where
1272       a  VM  or  container resides. Whenever possible, packets from the VM or
1273       container to the outside world should be processed completely  on  that
1274       VM’s  or  container’s hypervisor, eventually traversing a localnet port
1275       instance or a tunnel to the physical network or a different OVN deploy‐
1276       ment. Whenever possible, packets from the outside world to a VM or con‐
1277       tainer should be directed through the physical network directly to  the
1278       VM’s or container’s hypervisor.
1279
1280       In  order  to allow for the distributed processing of packets described
1281       in the paragraph above, distributed gateway ports need  to  be  logical
1282       patch  ports  that  effectively reside on every hypervisor, rather than
1283       l3gateway ports that are bound to a particular  chassis.  However,  the
1284       flows  associated with distributed gateway ports often need to be asso‐
1285       ciated with physical locations, for the following reasons:
1286
1287              •      The physical network that the localnet port  is  attached
1288                     to  typically uses L2 learning. Any Ethernet address used
1289                     over the distributed gateway port must be restricted to a
1290                     single  physical location so that upstream L2 learning is
1291                     not confused. Traffic sent out  the  distributed  gateway
1292                     port  towards  the localnet port with a specific Ethernet
1293                     address must be sent out one  specific  instance  of  the
1294                     distributed gateway port on one specific chassis. Traffic
1295                     received from the localnet port (or from  a  VIF  on  the
1296                     same logical switch as the localnet port) with a specific
1297                     Ethernet address must be directed to the logical switch’s
1298                     patch port instance on that specific chassis.
1299
1300                     Due  to the implications of L2 learning, the Ethernet ad‐
1301                     dress and IP address of the distributed gateway port need
1302                     to  be restricted to a single physical location. For this
1303                     reason, the user must specify one chassis associated with
1304                     the  distributed gateway port. Note that traffic travers‐
1305                     ing the distributed gateway port using other Ethernet ad‐
1306                     dresses and IP addresses (e.g. one-to-one NAT) is not re‐
1307                     stricted to this chassis.
1308
1309                     Replies to ARP and ND requests must be  restricted  to  a
1310                     single  physical  location, where the Ethernet address in
1311                     the reply resides. This includes ARP and ND  replies  for
1312                     the IP address of the distributed gateway port, which are
1313                     restricted to the chassis that the user  associated  with
1314                     the distributed gateway port.
1315
1316              •      In  order  to support one-to-many SNAT (aka IP masquerad‐
1317                     ing), where multiple logical IP addresses  spread  across
1318                     multiple  chassis  are mapped to a single external IP ad‐
1319                     dress, it will be necessary to handle some of the logical
1320                     router  processing on a specific chassis in a centralized
1321                     manner. Since the SNAT external IP address  is  typically
1322                     the distributed gateway port IP address, and for simplic‐
1323                     ity, the same chassis  associated  with  the  distributed
1324                     gateway port is used.
1325
1326       The  details  of flow restrictions to specific chassis are described in
1327       the ovn-northd documentation.
1328
1329       While most of the physical location dependent  aspects  of  distributed
1330       gateway  ports  can  be  handled  by restricting some flows to specific
1331       chassis, one additional mechanism is required. When a packet leaves the
1332       ingress pipeline and the logical egress port is the distributed gateway
1333       port, one of two different sets of actions is required at table 37:
1334
1335              •      If the packet can be handled locally on the sender’s  hy‐
1336                     pervisor  (e.g.  one-to-one NAT traffic), then the packet
1337                     should just be resubmitted locally to table  38,  in  the
1338                     normal manner for distributed logical patch ports.
1339
1340              •      However, if the packet needs to be handled on the chassis
1341                     associated with the distributed gateway port  (e.g.  one-
1342                     to-many  SNAT  traffic or non-NAT traffic), then table 37
1343                     must send the packet on a tunnel port to that chassis.
1344
1345       In order to trigger the second set of actions, the chassisredirect type
1346       of  southbound  Port_Binding has been added. Setting the logical egress
1347       port to the type chassisredirect logical port is simply a way to  indi‐
1348       cate  that  although the packet is destined for the distributed gateway
1349       port, it needs to be redirected to a different chassis.  At  table  37,
1350       packets  with  this logical egress port are sent to a specific chassis,
1351       in the same way that table 37 directs packets whose logical egress port
1352       is a VIF or a type l3gateway port to different chassis. Once the packet
1353       arrives at that chassis, table 38 resets the logical egress port to the
1354       value  representing  the distributed gateway port. For each distributed
1355       gateway port, there is one type chassisredirect port,  in  addition  to
1356       the distributed logical patch port representing the distributed gateway
1357       port.
1358
1359     High Availability for Distributed Gateway Ports
1360
1361       OVN allows you to specify a prioritized list of chassis for a  distrib‐
1362       uted gateway port. This is done by associating multiple Gateway_Chassis
1363       rows with a Logical_Router_Port in the OVN_Northbound database.
1364
1365       When multiple chassis have been specified for a  gateway,  all  chassis
1366       that may send packets to that gateway will enable BFD on tunnels to all
1367       configured gateway chassis. The current master chassis for the  gateway
1368       is the highest priority gateway chassis that is currently viewed as ac‐
1369       tive based on BFD status.
1370
1371       For more information on L3 gateway high availability, please  refer  to
1372       http://docs.ovn.org/en/latest/topics/high-availability.
1373
1374     Restrictions of Distributed Gateway Ports
1375
1376       Distributed  gateway  ports are used to connect to an external network,
1377       which can be a physical network modeled by a logical switch with a  lo‐
1378       calnet  port,  and can also be a logical switch that interconnects dif‐
1379       ferent OVN deployments (see OVN Deployments  Interconnection).  Usually
1380       there  can be many logical routers connected to the same external logi‐
1381       cal switch, as shown in below diagram.
1382
1383                                     +--LS-EXT-+
1384                                     |    |    |
1385                                     |    |    |
1386                                    LR1  ...  LRn
1387
1388
1389       In this diagram, there are n logical routers  connected  to  a  logical
1390       switch  LS-EXT,  each  with a distributed gateway port, so that traffic
1391       sent to external world is redirected to the gateway chassis that is as‐
1392       signed to the distributed gateway port of respective logical router.
1393
1394       In the logical topology, nothing can prevent an user to add a route be‐
1395       tween the logical routers via the connected distributed  gateway  ports
1396       on  LS-EXT.  However,  the route works only if the LS-EXT is a physical
1397       network (modeled by a logical switch with a  localnet  port).  In  that
1398       case the packet will be delivered between the gateway chassises through
1399       the localnet port via physical network. If the LS-EXT is a regular log‐
1400       ical switch (backed by tunneling only, as in the use case of OVN inter‐
1401       connection), then the packet will be  dropped  on  the  source  gateway
1402       chassis.  The limitation is due the fact that distributed gateway ports
1403       are tied to physical location, and without physical network connection,
1404       we  will end up with either dropping the packet or transferring it over
1405       the tunnels which could cause bigger problems such as broadcast packets
1406       being redirect repeatedly by different gateway chassises.
1407
1408       With  the limitation in mind, if a user do want the direct connectivity
1409       between the logical routers, it is better to create an internal logical
1410       switch  connected  to  the  logical routers with regular logical router
1411       ports, which are completely distributed and the packets don’t  have  to
1412       leave  a  chassis  unless necessary, which is more optimal than routing
1413       via the distributed gateway ports.
1414
1415     ARP request and ND NS packet processing
1416
1417       Due to the fact that ARP requests and ND NA packets are usually  broad‐
1418       cast  packets,  for  performance  reasons, OVN deals with requests that
1419       target OVN owned IP addresses (i.e., IP  addresses  configured  on  the
1420       router  ports,  VIPs, NAT IPs) in a specific way and only forwards them
1421       to the logical router that owns the target IP address. This behavior is
1422       different  than  that  of  traditional  switches and implies that other
1423       routers/hosts connected to the logical switch will not learn the MAC/IP
1424       binding from the request packet.
1425
1426       All other ARP and ND packets are flooded in the L2 broadcast domain and
1427       to all attached logical patch ports.
1428
1429     VIFs on the logical switch connected by a distributed gateway port
1430
1431       Typically the logical switch connected by a distributed gateway port is
1432       for  external connectivity, usually to a physical network through a lo‐
1433       calnet port on the logical  switch,  or  to  a  remote  OVN  deployment
1434       through  OVN  Interconnection. In these cases there is no VIF ports re‐
1435       quired on the logical switch.
1436
1437       While not very common, it is still possible to create VIF ports on  the
1438       logical  switch connected by a distributed gateway port, but there is a
1439       limitation that the logical ports need to reside on the gateway chassis
1440       where the distributed gateway port resides to get connectivity to other
1441       logical switches through the distributed gateway port. There is no lim‐
1442       itation  for  the  VIFs to connect within the logical switch, or beyond
1443       the logical switch through other  regular  distributed  logical  router
1444       ports.
1445
1446       A  special case is when using distributed gateway ports for scalability
1447       purpose, as mentioned earlier in this document.  The  logical  switches
1448       connected  by  distributed  gateway  ports are not for connectivity but
1449       just for regular VIFs. However, the above limitation usually  does  not
1450       matter  because in this use case all the VIFs on the logical switch are
1451       located on the same chassis with the distributed gateway port that con‐
1452       nects the logical switch.
1453
1454   Multiple localnet logical switches connected to a Logical Router
1455       It  is  possible to have multiple logical switches each with a localnet
1456       port (representing physical networks) connected to a logical router, in
1457       which one localnet logical switch may provide the external connectivity
1458       via a distributed  gateway  port  and  rest  of  the  localnet  logical
1459       switches  use VLAN tagging in the physical network. It is expected that
1460       ovn-bridge-mappings is configured appropriately on the chassis for  all
1461       these localnet networks.
1462
1463     East West routing
1464
1465       East-West  routing  between these localnet VLAN tagged logical switches
1466       work almost the same way as normal logical switches. When the VM  sends
1467       such a packet, then:
1468
1469              1.  It  first enters the ingress pipeline, and then egress pipe‐
1470                  line of the source localnet logical switch datapath. It then
1471                  enters  the  ingress pipeline of the logical router datapath
1472                  via the logical router port in the source chassis.
1473
1474              2.  Routing decision is taken.
1475
1476              3.  From the router datapath, packet enters the ingress pipeline
1477                  and then egress pipeline of the destination localnet logical
1478                  switch datapath and goes out of the  integration  bridge  to
1479                  the  provider  bridge ( belonging to the destination logical
1480                  switch) via the localnet port. While sending the  packet  to
1481                  provider  bridge,  we also replace router port MAC as source
1482                  MAC with a chassis unique MAC.
1483
1484                  This chassis unique MAC is configured as global  ovs  config
1485                  on each chassis (eg. via "ovs-vsctl set open . external-ids:
1486                  ovn-chassis-mac-mappings="phys:aa:bb:cc:dd:ee:$i$i"").   For
1487                  more details, see ovn-controller(8).
1488
1489                  If the above is not configured, then source MAC would be the
1490                  router port MAC. This could create problem if we  have  more
1491                  than  one chassis. This is because, since the router port is
1492                  distributed, the same (MAC,VLAN) tuple will seen by physical
1493                  network  from other chassis as well, which could cause these
1494                  issues:
1495
1496                  •      Continuous MAC moves in top-of-rack switch (ToR).
1497
1498                  •      ToR dropping the traffic, which is causing continuous
1499                         MAC moves.
1500
1501                  •      ToR  blocking the ports from which MAC moves are hap‐
1502                         pening.
1503
1504              4.  The destination chassis receives the packet via the localnet
1505                  port and sends it to the integration bridge. Before entering
1506                  the integration bridge the source mac of the packet will  be
1507                  replaced  with  router port mac again. The packet enters the
1508                  ingress pipeline and then egress pipeline of the destination
1509                  localnet  logical  switch  and finally gets delivered to the
1510                  destination VM port.
1511
1512     External traffic
1513
1514       The following happens when a VM sends an external  traffic  (which  re‐
1515       quires  NATting) and the chassis hosting the VM doesn’t have a distrib‐
1516       uted gateway port.
1517
1518              1.  The packet first  enters  the  ingress  pipeline,  and  then
1519                  egress  pipeline of the source localnet logical switch data‐
1520                  path. It then enters the ingress  pipeline  of  the  logical
1521                  router  datapath  via  the logical router port in the source
1522                  chassis.
1523
1524              2.  Routing decision is taken. Since the gateway router  or  the
1525                  distributed  gateway port doesn’t reside in the source chas‐
1526                  sis, the traffic is redirected to the  gateway  chassis  via
1527                  the tunnel port.
1528
1529              3.  The  gateway chassis receives the packet via the tunnel port
1530                  and the packet enters the egress  pipeline  of  the  logical
1531                  router datapath. NAT rules are applied here. The packet then
1532                  enters the ingress pipeline and then egress pipeline of  the
1533                  localnet  logical  switch  datapath  which provides external
1534                  connectivity and finally goes out via the localnet  port  of
1535                  the logical switch which provides external connectivity.
1536
1537       Although  this  works,  the  VM traffic is tunnelled when sent from the
1538       compute chassis to the gateway chassis. In order for it to  work  prop‐
1539       erly,  the  MTU of the localnet logical switches must be lowered to ac‐
1540       count for the tunnel encapsulation.
1541
1542   Centralized routing for localnet VLAN tagged logical switches connected  to
1543       a Logical Router
1544       To  overcome the tunnel encapsulation problem described in the previous
1545       section, OVN supports the option of enabling  centralized  routing  for
1546       localnet VLAN tagged logical switches. CMS can configure the option op‐
1547       tions:reside-on-redirect-chassis to true for  each  Logical_Router_Port
1548       which  connects  to  the  localnet  VLAN  tagged logical switches. This
1549       causes the gateway chassis (hosting the distributed  gateway  port)  to
1550       handle  all  the  routing for these networks, making it centralized. It
1551       will reply to the ARP requests for the logical router port IPs.
1552
1553       If the logical router doesn’t have a distributed gateway port  connect‐
1554       ing  to  the localnet logical switch which provides external connectiv‐
1555       ity, or if it has more than one distributed gateway  ports,  then  this
1556       option is ignored by OVN.
1557
1558       The  following happens when a VM sends an east-west traffic which needs
1559       to be routed:
1560
1561              1.  The packet first  enters  the  ingress  pipeline,  and  then
1562                  egress  pipeline of the source localnet logical switch data‐
1563                  path and is sent out via a localnet port of the  source  lo‐
1564                  calnet logical switch (instead of sending it to router pipe‐
1565                  line).
1566
1567              2.  The gateway chassis receives the packet via a localnet  port
1568                  of  the  source  localnet logical switch and sends it to the
1569                  integration bridge. The packet then enters the ingress pipe‐
1570                  line,  and then egress pipeline of the source localnet logi‐
1571                  cal switch datapath and enters the ingress pipeline  of  the
1572                  logical router datapath.
1573
1574              3.  Routing decision is taken.
1575
1576              4.  From the router datapath, packet enters the ingress pipeline
1577                  and then egress pipeline of the destination localnet logical
1578                  switch  datapath. It then goes out of the integration bridge
1579                  to the provider bridge ( belonging to the destination  logi‐
1580                  cal switch) via a localnet port.
1581
1582              5.  The  destination  chassis receives the packet via a localnet
1583                  port and sends it to the integration bridge. The packet  en‐
1584                  ters  the  ingress  pipeline and then egress pipeline of the
1585                  destination localnet logical switch and finally delivered to
1586                  the destination VM port.
1587
1588       The  following  happens  when  a VM sends an external traffic which re‐
1589       quires NATting:
1590
1591              1.  The packet first  enters  the  ingress  pipeline,  and  then
1592                  egress  pipeline of the source localnet logical switch data‐
1593                  path and is sent out via a localnet port of the  source  lo‐
1594                  calnet logical switch (instead of sending it to router pipe‐
1595                  line).
1596
1597              2.  The gateway chassis receives the packet via a localnet  port
1598                  of  the  source  localnet logical switch and sends it to the
1599                  integration bridge. The packet then enters the ingress pipe‐
1600                  line,  and then egress pipeline of the source localnet logi‐
1601                  cal switch datapath and enters the ingress pipeline  of  the
1602                  logical router datapath.
1603
1604              3.  Routing decision is taken and NAT rules are applied.
1605
1606              4.  From the router datapath, packet enters the ingress pipeline
1607                  and then egress pipeline  of  the  localnet  logical  switch
1608                  datapath  which provides external connectivity. It then goes
1609                  out of the integration bridge to the  provider  bridge  (be‐
1610                  longing  to  the logical switch which provides external con‐
1611                  nectivity) via a localnet port.
1612
1613       The following happens for the reverse external traffic.
1614
1615              1.  The gateway chassis receives the packet from a localnet port
1616                  of  the logical switch which provides external connectivity.
1617                  The packet then enters the ingress pipeline and then  egress
1618                  pipeline  of the localnet logical switch (which provides ex‐
1619                  ternal connectivity). The packet  then  enters  the  ingress
1620                  pipeline of the logical router datapath.
1621
1622              2.  The  ingress pipeline of the logical router datapath applies
1623                  the unNATting rules. The  packet  then  enters  the  ingress
1624                  pipeline  and  then  egress  pipeline of the source localnet
1625                  logical switch. Since the source VM doesn’t  reside  in  the
1626                  gateway  chassis, the packet is sent out via a localnet port
1627                  of the source logical switch.
1628
1629              3.  The source chassis receives the packet via a  localnet  port
1630                  and  sends  it  to the integration bridge. The packet enters
1631                  the ingress pipeline and then egress pipeline of the  source
1632                  localnet  logical  switch  and finally gets delivered to the
1633                  source VM port.
1634
1635       As an alternative to  reside-on-redirect-chassis,  OVN  supports  VLAN-
1636       based  redirection.  Whereas reside-on-redirect-chassis centralizes all
1637       router functionality, VLAN-based redirection only changes how OVN redi‐
1638       rects  packets to the gateway chassis. By setting options:redirect-type
1639       to bridged on a distributed gateway port, OVN redirects packets to  the
1640       gateway  chassis  using  the localnet port of the router’s peer logical
1641       switch, instead of a tunnel.
1642
1643       If the logical router doesn’t have a distributed gateway port  connect‐
1644       ing  to  the localnet logical switch which provides external connectiv‐
1645       ity, or if it has more than one distributed gateway  ports,  then  this
1646       option is ignored by OVN.
1647
1648       Following happens for bridged redirection:
1649
1650              1.  On  compute  chassis,  packet passes though logical router’s
1651                  ingress pipeline.
1652
1653              2.  If logical outport is gateway chassis attached  router  port
1654                  then  packet  is  "redirected" to gateway chassis using peer
1655                  logical switch’s localnet port.
1656
1657              3.  This redirected packet has destination mac  as  router  port
1658                  mac (the one to which gateway chassis is attached). Its VLAN
1659                  id is that of localnet port (peer logical switch of the log‐
1660                  ical router port).
1661
1662              4.  On  the gateway chassis packet will enter the logical router
1663                  pipeline again and this  time  it  will  passthrough  egress
1664                  pipeline as well.
1665
1666              5.  Reverse traffic packet flows stays the same.
1667
1668       Some guidelines and expections with bridged redirection:
1669
1670              1.  Since router port mac is destination mac, hence it has to be
1671                  ensured that physical network learns it  on  ONLY  from  the
1672                  gateway  chassis.  Which means that ovn-chassis-mac-mappings
1673                  should be configure on all the compute nodes, so that physi‐
1674                  cal network never learn router port mac from compute nodes.
1675
1676              2.  Since  packet  enters  logical router ingress pipeline twice
1677                  (once on compute chassis  and  again  on  gateway  chassis),
1678                  hence ttl will be decremented twice.
1679
1680              3.  Default  redirection  type continues to be overlay. User can
1681                  switch the redirect-type  between  bridged  and  overlay  by
1682                  changing the value of options:redirect-type
1683
1684   Life Cycle of a VTEP gateway
1685       A  gateway  is  a chassis that forwards traffic between the OVN-managed
1686       part of a logical network and a physical VLAN, extending a tunnel-based
1687       logical network into a physical network.
1688
1689       The  steps  below  refer  often to details of the OVN and VTEP database
1690       schemas. Please see ovn-sb(5), ovn-nb(5) and vtep(5), respectively, for
1691       the full story on these databases.
1692
1693              1.  A  VTEP  gateway’s  life cycle begins with the administrator
1694                  registering the VTEP gateway as a Physical_Switch table  en‐
1695                  try  in the VTEP database. The ovn-controller-vtep connected
1696                  to this VTEP database, will recognize the new  VTEP  gateway
1697                  and  create  a  new  Chassis  table  entry  for  it  in  the
1698                  OVN_Southbound database.
1699
1700              2.  The administrator can then create a new Logical_Switch table
1701                  entry,  and  bind a particular vlan on a VTEP gateway’s port
1702                  to any VTEP logical switch. Once a VTEP  logical  switch  is
1703                  bound to a VTEP gateway, the ovn-controller-vtep will detect
1704                  it and add its name to the vtep_logical_switches  column  of
1705                  the  Chassis table in the OVN_Southbound database. Note, the
1706                  tunnel_key column of VTEP logical switch is  not  filled  at
1707                  creation.  The  ovn-controller-vtep will set the column when
1708                  the correponding vtep logical switch is bound to an OVN log‐
1709                  ical network.
1710
1711              3.  Now, the administrator can use the CMS to add a VTEP logical
1712                  switch to the OVN logical network. To do that, the CMS  must
1713                  first  create  a  new Logical_Switch_Port table entry in the
1714                  OVN_Northbound database. Then, the type column of this entry
1715                  must  be  set  to  "vtep". Next, the vtep-logical-switch and
1716                  vtep-physical-switch keys in the options column must also be
1717                  specified,  since  multiple  VTEP gateways can attach to the
1718                  same VTEP logical switch. Next, the addresses column of this
1719                  logical  port must be set to "unknown", it will add a prior‐
1720                  ity 0 entry  in  "ls_in_l2_lkup"  stage  of  logical  switch
1721                  ingress  pipeline.  So,  traffic  with unrecorded mac by OVN
1722                  would go through the Logical_Switch_Port  to  physical  net‐
1723                  work.
1724
1725              4.  The  newly  created logical port in the OVN_Northbound data‐
1726                  base and its  configuration  will  be  passed  down  to  the
1727                  OVN_Southbound  database  as a new Port_Binding table entry.
1728                  The ovn-controller-vtep will recognize the change  and  bind
1729                  the  logical port to the corresponding VTEP gateway chassis.
1730                  Configuration of binding the same VTEP logical switch  to  a
1731                  different  OVN logical networks is not allowed and a warning
1732                  will be generated in the log.
1733
1734              5.  Beside binding to the VTEP  gateway  chassis,  the  ovn-con‐
1735                  troller-vtep  will  update the tunnel_key column of the VTEP
1736                  logical switch to the corresponding  Datapath_Binding  table
1737                  entry’s tunnel_key for the bound OVN logical network.
1738
1739              6.  Next, the ovn-controller-vtep will keep reacting to the con‐
1740                  figuration change in the Port_Binding in the  OVN_Northbound
1741                  database,  and  updating  the Ucast_Macs_Remote table in the
1742                  VTEP database. This allows the VTEP  gateway  to  understand
1743                  where  to  forward  the  unicast traffic coming from the ex‐
1744                  tended external network.
1745
1746              7.  Eventually, the VTEP gateway’s life cycle ends when the  ad‐
1747                  ministrator unregisters the VTEP gateway from the VTEP data‐
1748                  base. The ovn-controller-vtep will recognize the  event  and
1749                  remove  all  related configurations (Chassis table entry and
1750                  port bindings) in the OVN_Southbound database.
1751
1752              8.  When the ovn-controller-vtep is terminated, all related con‐
1753                  figurations  in  the  OVN_Southbound  database  and the VTEP
1754                  database will be cleaned, including  Chassis  table  entries
1755                  for  all  registered  VTEP gateways and their port bindings,
1756                  and  all  Ucast_Macs_Remote  table  entries  and  the  Logi‐
1757                  cal_Switch tunnel keys.
1758
1759   OVN Deployments Interconnection
1760       It is not uncommon for an operator to deploy multiple OVN clusters, for
1761       two main reasons. Firstly, an operator may prefer  to  deploy  one  OVN
1762       cluster for each availability zone, e.g. in different physical regions,
1763       to avoid single point of failure. Secondly, there is  always  an  upper
1764       limit for a single OVN control plane to scale.
1765
1766       Although  the  control  planes of the different availability zone (AZ)s
1767       are independent from each other, the workloads from different  AZs  may
1768       need  to  communicate across the zones. The OVN interconnection feature
1769       provides a native way to  interconnect  different  AZs  by  L3  routing
1770       through  transit  overlay networks between logical routers of different
1771       AZs.
1772
1773       A global OVN Interconnection Northbound database is introduced for  the
1774       operator  (probably  through  CMS systems) to configure transit logical
1775       switches that connect logical routers from  different  AZs.  A  transit
1776       switch  is  similar to a regular logical switch, but it is used for in‐
1777       terconnection purpose only. Typically, each transit switch can be  used
1778       to  connect  all  logical routers that belong to same tenant across all
1779       AZs.
1780
1781       A dedicated daemon process ovn-ic, OVN interconnection  controller,  in
1782       each  AZ  will  consume  this  data  and populate corresponding logical
1783       switches to their own northbound databases for each AZ, so that logical
1784       routers  can  be connected to the transit switch by creating patch port
1785       pairs in their northbound databases. Any router ports connected to  the
1786       transit  switches  are  considered interconnection ports, which will be
1787       exchanged between AZs.
1788
1789       Physically, when workloads from different AZs communicate, packets need
1790       to  go  through multiple hops: source chassis, source gateway, destina‐
1791       tion gateway and destination chassis.  All  these  hops  are  connected
1792       through  tunnels  so  that  the packets never leave overlay networks. A
1793       distributed gateway port is required to connect the logical router to a
1794       transit  switch,  with a gateway chassis specified, so that the traffic
1795       can be forwarded through the gateway chassis.
1796
1797       A global OVN Interconnection Southbound database is introduced for  ex‐
1798       changing  control  plane  information between the AZs. The data in this
1799       database is populated and consumed by the ovn-ic, of each AZ. The  main
1800       information in this database includes:
1801
1802              •      Datapath bindings for transit switches, which mainly con‐
1803                     tains the tunnel keys generated for each transit  switch.
1804                     Separate  key ranges are reserved for transit switches so
1805                     that they will never conflict with any  tunnel  keys  lo‐
1806                     cally assigned for datapaths within each AZ.
1807
1808              •      Availability  zones,  which  are registerd by ovn-ic from
1809                     each AZ.
1810
1811              •      Gateways. Each AZ specifies chassises that  are  supposed
1812                     to  work as interconnection gateways, and the ovn-ic will
1813                     populate this information to the  interconnection  south‐
1814                     bound  DB.  The  ovn-ic from all the other AZs will learn
1815                     the gateways and populate to their own southbound DB as a
1816                     chassis.
1817
1818              •      Port  bindings  for  logical  switch ports created on the
1819                     transit switch. Each AZ maintains their logical router to
1820                     transit  switch connections independently, but ovn-ic au‐
1821                     tomatically populates  local  port  bindings  on  transit
1822                     switches to the global interconnection southbound DB, and
1823                     learns remote port bindings from other AZs  back  to  its
1824                     own  northbound and southbound DBs, so that logical flows
1825                     can be produced and then translated to OVS flows locally,
1826                     which finally enables data plane communication.
1827
1828              •      Routes  that are advertised between different AZs. If en‐
1829                     abled, routes are automatically exchanged by ovn-ic. Both
1830                     static  routes  and directly connected subnets are adver‐
1831                     tised. Options in options column of the  NB_Global  table
1832                     of  OVN_NB  database control the behavior of route adver‐
1833                     tisement, such as enable/disable the advertising/learning
1834                     routes,  whether  default  routes are advertised/learned,
1835                     and blacklisted CIDRs. See ovn-nb(5) for more details.
1836
1837       The tunnel keys for transit switch datapaths and related port  bindings
1838       must  be agreed across all AZs. This is ensured by generating and stor‐
1839       ing the keys in the global  interconnection  southbound  database.  Any
1840       ovn-ic from any AZ can allocate the key, but race conditions are solved
1841       by enforcing unique index for the column in the database.
1842
1843       Once each AZ’s NB and SB databases are populated  with  interconnection
1844       switches  and ports, and agreed upon the tunnel keys, data plane commu‐
1845       nication between the AZs are established.
1846
1847       When VXLAN tunneling is enabled in an OVN cluster, due to  the  limited
1848       range available for VNIs, Interconnection feature is not supported.
1849
1850     A day in the life of a packet crossing AZs
1851
1852              1.  An IP packet is sent out from a VIF on a hypervisor (HV1) of
1853                  AZ1, with destination IP belonging to a VIF in AZ2.
1854
1855              2.  In HV1’s OVS flow tables, the packet  goes  through  logical
1856                  switch and logical router pipelines, and in a logical router
1857                  pipeline, the routing stage finds out the next hop  for  the
1858                  destination  IP,  which  belongs  to a remote logical router
1859                  port in AZ2, and the output port, which is  a  chassis-redi‐
1860                  rect  port  located  on  an  interconnection gateway (GW1 in
1861                  AZ1), so HV1 sends the packet to GW1 through tunnel.
1862
1863              3.  On GW1, it continues with the logical router pipe  line  and
1864                  switches  to  the transit switch’s pipeline through the peer
1865                  port of the chassis redirect port. In the  transit  switch’s
1866                  pipeline  it outputs to the remote logical port which is lo‐
1867                  cated on a gateway (GW2) in AZ2, so the GW1 sends the packet
1868                  to GW2 in tunnel.
1869
1870              4.  On  GW2,  it  continues with the transit switch pipeline and
1871                  switches to the logical router  pipeline  through  the  peer
1872                  port,  which  is  a chassis redirect port that is located on
1873                  GW2. The logical router pipeline then forwards the packet to
1874                  relevant  logical  pipelines according to the destination IP
1875                  address, and figures out the MAC and location of the  desti‐
1876                  nation VIF port - a hypervisor (HV2). The GW2 then sends the
1877                  packet to HV2 in tunnel.
1878
1879              5.  On HV2, the packet is delivered to the final destination VIF
1880                  port  by  the  logical switch egress pipeline, just the same
1881                  way as for intra-AZ communications.
1882
1883   Native OVN services for external logical ports
1884       To support OVN native services (like DHCP/IPv6 RA/DNS  lookup)  to  the
1885       cloud  resources  which  are  external,  OVN  supports external logical
1886       ports.
1887
1888       Below are some of the use cases where external ports can be used.
1889
1890              •      VMs connected to SR-IOV nics - Traffic from these VMs  by
1891                     passes  the  kernel stack and local ovn-controller do not
1892                     bind these ports and cannot serve the native services.
1893
1894              •      When CMS supports provisioning baremetal servers.
1895
1896       OVN will provide the native services if CMS has done the below configu‐
1897       ration in the OVN Northbound Database.
1898
1899              •      A  row is created in Logical_Switch_Port, configuring the
1900                     addresses column and setting the type to external.
1901
1902ha_chassis_group column is configured.
1903
1904              •      The HA chassis which belongs to the HA chassis group  has
1905                     the ovn-bridge-mappings configured and has proper L2 con‐
1906                     nectivity so that it can receive the DHCP and  other  re‐
1907                     lated request packets from these external resources.
1908
1909              •      The Logical_Switch of this port has a localnet port.
1910
1911              •      Native  OVN  services are enabled by configuring the DHCP
1912                     and other options like the way it is done for the  normal
1913                     logical ports.
1914
1915       It is recommended to use the same HA chassis group for all the external
1916       ports of a logical switch. Otherwise, the physical switch might see MAC
1917       flap  issue when different chassis provide the native services. For ex‐
1918       ample when supporting native DHCPv4 service, DHCPv4 server mac (config‐
1919       ured  in  options:server_mac  column in table DHCP_Options) originating
1920       from different ports can cause MAC flap issue. The MAC of  the  logical
1921       router  IP(s) can also flap if the same HA chassis group is not set for
1922       all the external ports of a logical switch.
1923

SECURITY

1925   Role-Based Access Controls for the Southbound DB
1926       In order to provide additional security against the possibility  of  an
1927       OVN  chassis becoming compromised in such a way as to allow rogue soft‐
1928       ware to make arbitrary modifications to the southbound  database  state
1929       and  thus  disrupt  the  OVN  network,  role-based access controls (see
1930       ovsdb-server(1) for additional details) are provided for the southbound
1931       database.
1932
1933       The  implementation  of  role-based access controls (RBAC) requires the
1934       addition of two tables to an OVSDB schema: the RBAC_Role  table,  which
1935       is  indexed  by  role name and maps the the names of the various tables
1936       that may be modifiable for a given role to individual rows in a permis‐
1937       sions  table  containing detailed permission information for that role,
1938       and the permission table itself which consists of rows  containing  the
1939       following information:
1940
1941              Table Name
1942                     The name of the associated table. This column exists pri‐
1943                     marily as an aid for humans reading the contents of  this
1944                     table.
1945
1946              Auth Criteria
1947                     A set of strings containing the names of columns (or col‐
1948                     umn:key pairs for columns containing string:string maps).
1949                     The contents of at least one of the columns or column:key
1950                     values in a row to be modified, inserted, or deleted must
1951                     be equal to the ID of the client attempting to act on the
1952                     row in order for the authorization check to pass. If  the
1953                     authorization  criteria  is empty, authorization checking
1954                     is disabled and all clients for the role will be  treated
1955                     as authorized.
1956
1957              Insert/Delete
1958                     Row insertion/deletion permission; boolean value indicat‐
1959                     ing whether insertion and deletion of rows is allowed for
1960                     the  associated table. If true, insertion and deletion of
1961                     rows is allowed for authorized clients.
1962
1963              Updatable Columns
1964                     A set of strings containing the names of columns or  col‐
1965                     umn:key  pairs  that  may be updated or mutated by autho‐
1966                     rized clients. Modifications to columns within a row  are
1967                     only  permitted  when  the  authorization  check  for the
1968                     client passes and all columns to be modified are included
1969                     in this set of modifiable columns.
1970
1971       RBAC  configuration  for  the  OVN southbound database is maintained by
1972       ovn-northd. With RBAC enabled, modifications are only permitted for the
1973       Chassis,  Encap,  Port_Binding,  and  MAC_Binding  tables,  and are re‐
1974       stricted as follows:
1975
1976              Chassis
1977                     Authorization: client ID must match the chassis name.
1978
1979                     Insert/Delete: authorized row insertion and deletion  are
1980                     permitted.
1981
1982                     Update:  The  columns  nb_cfg,  external_ids, encaps, and
1983                     vtep_logical_switches may be modified when authorized.
1984
1985              Encap  Authorization: client ID must match the chassis name.
1986
1987                     Insert/Delete: row insertion and row deletion are permit‐
1988                     ted.
1989
1990                     Update:  The  columns  type, options, and ip can be modi‐
1991                     fied.
1992
1993              Port_Binding
1994                     Authorization: disabled (all clients are  considered  au‐
1995                     thorized.  A  future enhancement may add columns (or keys
1996                     to external_ids) in order to control  which  chassis  are
1997                     allowed to bind each port.
1998
1999                     Insert/Delete:  row  insertion/deletion are not permitted
2000                     (ovn-northd maintains rows in this table.
2001
2002                     Update: Only modifications to the chassis column are per‐
2003                     mitted.
2004
2005              MAC_Binding
2006                     Authorization: disabled (all clients are considered to be
2007                     authorized).
2008
2009                     Insert/Delete: row insertion/deletion are permitted.
2010
2011                     Update: The columns logical_port, ip, mac,  and  datapath
2012                     may be modified by ovn-controller.
2013
2014              IGMP_Group
2015                     Authorization: disabled (all clients are considered to be
2016                     authorized).
2017
2018                     Insert/Delete: row insertion/deletion are permitted.
2019
2020                     Update: The columns address, chassis, datapath, and ports
2021                     may be modified by ovn-controller.
2022
2023       Enabling RBAC for ovn-controller connections to the southbound database
2024       requires the following steps:
2025
2026              1.  Creating SSL certificates for each chassis with the certifi‐
2027                  cate  CN  field  set to the chassis name (e.g. for a chassis
2028                  with  external-ids:system-id=chassis-1,  via   the   command
2029                  "ovs-pki -u req+sign chassis-1 switch").
2030
2031              2.  Configuring  each  ovn-controller to use SSL when connecting
2032                  to the southbound database (e.g. via "ovs-vsctl set  open  .
2033                  external-ids:ovn-remote=ssl:x.x.x.x:6642").
2034
2035              3.  Configuring  a southbound database SSL remote with "ovn-con‐
2036                  troller"   role   (e.g.   via   "ovn-sbctl    set-connection
2037                  role=ovn-controller pssl:6642").
2038
2039   Encrypt Tunnel Traffic with IPsec
2040       OVN  tunnel  traffic  goes through physical routers and switches. These
2041       physical devices could be untrusted  (devices  in  public  network)  or
2042       might  be  compromised.  Enabling  encryption to the tunnel traffic can
2043       prevent the traffic data from being monitored and manipulated.
2044
2045       The tunnel traffic is encrypted with IPsec. The CMS sets the ipsec col‐
2046       umn in the northbound NB_Global table to enable or disable IPsec encry‐
2047       tion. If ipsec is true, all OVN tunnels will be encrypted. If ipsec  is
2048       false, no OVN tunnels will be encrypted.
2049
2050       When  CMS  updates  the ipsec column in the northbound NB_Global table,
2051       ovn-northd copies the value to  the  ipsec  column  in  the  southbound
2052       SB_Global table. ovn-controller in each chassis monitors the southbound
2053       database and sets the options of the OVS tunnel interface  accordingly.
2054       OVS  tunnel  interface  options  are monitored by the ovs-monitor-ipsec
2055       daemon which configures IKE daemon to set up IPsec connections.
2056
2057       Chassis authenticates each other by using certificate. The  authentica‐
2058       tion  succeeds if the other end in tunnel presents a certificate signed
2059       by a trusted CA and the common name (CN) matches the  expected  chassis
2060       name.  The  SSL  certificates used in role-based access controls (RBAC)
2061       can be used in IPsec. Or use ovs-pki to create different  certificates.
2062       The  certificate  is  required to be x.509 version 3, and with CN field
2063       and subjectAltName field being set to the chassis name.
2064
2065       The CA certificate, chassis certificate and private key are required to
2066       be  installed  in  each  chassis  before  enabling  IPsec.  Please  see
2067       ovs-vswitchd.conf.db(5) for setting up CA based IPsec authentication.
2068

DESIGN DECISIONS

2070   Tunnel Encapsulations
2071       In general, OVN annotates logical network packets that  it  sends  from
2072       one  hypervisor to another with the following three pieces of metadata,
2073       which are encoded in an encapsulation-specific fashion:
2074
2075              •      24-bit logical datapath identifier, from  the  tunnel_key
2076                     column in the OVN Southbound Datapath_Binding table.
2077
2078              •      15-bit  logical ingress port identifier. ID 0 is reserved
2079                     for internal use within OVN. IDs 1 through 32767,  inclu‐
2080                     sive,  may  be  assigned  to  logical ports (see the tun‐
2081                     nel_key column in the OVN Southbound Port_Binding table).
2082
2083              •      16-bit logical egress  port  identifier.  IDs  0  through
2084                     32767 have the same meaning as for logical ingress ports.
2085                     IDs 32768 through 65535, inclusive, may  be  assigned  to
2086                     logical  multicast  groups  (see the tunnel_key column in
2087                     the OVN Southbound Multicast_Group table).
2088
2089       When VXLAN is enabled on any hypervisor  in  a  cluster,  datapath  and
2090       egress  port identifier ranges are reduced to 12-bits. This is done be‐
2091       cause only STT and Geneve provide the large space for metadata (over 32
2092       bits per packet). To accommodate for VXLAN, 24 bits available are split
2093       as follows:
2094
2095              •      12-bit logical datapath identifier, derived from the tun‐
2096                     nel_key column in the OVN Southbound Datapath_Binding ta‐
2097                     ble.
2098
2099              •      12-bit logical egress port identifier. IDs 0 through 2047
2100                     are used for unicast output ports. IDs 2048 through 4095,
2101                     inclusive, may be assigned to  logical  multicast  groups
2102                     (see  the  tunnel_key column in the OVN Southbound Multi‐
2103                     cast_Group table). For multicast  group  tunnel  keys,  a
2104                     special  mapping  scheme  is used to internally transform
2105                     from internal OVN 16-bit keys  to  12-bit  values  before
2106                     sending  packets  through  a  VXLAN tunnel, and back from
2107                     12-bit tunnel keys to 16-bit values when receiving  pack‐
2108                     ets from a VXLAN tunnel.
2109
2110              •      No logical ingress port identifier.
2111
2112       The limited space available for metadata when VXLAN tunnels are enabled
2113       in a cluster put the following  functional  limitations  onto  features
2114       available to users:
2115
2116              •      The maximum number of networks is reduced to 4096.
2117
2118              •      The  maximum  number  of  ports per network is reduced to
2119                     4096. (Including multicast group ports.)
2120
2121              •      ACLs matching against logical  ingress  port  identifiers
2122                     are not supported.
2123
2124              •      OVN interconnection feature is not supported.
2125
2126       In  addition  to  functional limitations described above, the following
2127       should be considered before enabling it in your cluster:
2128
2129              •      STT and Geneve use randomized UDP  or  TCP  source  ports
2130                     that  allows  efficient distribution among multiple paths
2131                     in environments that use ECMP in their underlay.
2132
2133              •      NICs are available to offload STT and  Geneve  encapsula‐
2134                     tion and decapsulation.
2135
2136       Due to its flexibility, the preferred encapsulation between hypervisors
2137       is Geneve. For Geneve encapsulation, OVN transmits the logical datapath
2138       identifier  in  the  Geneve  VNI. OVN transmits the logical ingress and
2139       logical egress ports in a TLV with  class  0x0102,  type  0x80,  and  a
2140       32-bit value encoded as follows, from MSB to LSB:
2141
2142         1       15          16
2143       +---+------------+-----------+
2144       |rsv|ingress port|egress port|
2145       +---+------------+-----------+
2146         0
2147
2148
2149       Environments  whose  NICs lack Geneve offload may prefer STT encapsula‐
2150       tion for performance reasons. For STT encapsulation,  OVN  encodes  all
2151       three  pieces  of  logical metadata in the STT 64-bit tunnel ID as fol‐
2152       lows, from MSB to LSB:
2153
2154           9          15          16         24
2155       +--------+------------+-----------+--------+
2156       |reserved|ingress port|egress port|datapath|
2157       +--------+------------+-----------+--------+
2158           0
2159
2160
2161       For connecting to gateways, in addition to Geneve and STT, OVN supports
2162       VXLAN,  because  only  VXLAN  support  is  common  on top-of-rack (ToR)
2163       switches. Currently, gateways have a feature set that matches the capa‐
2164       bilities  as  defined by the VTEP schema, so fewer bits of metadata are
2165       necessary. In the future, gateways that do not  support  encapsulations
2166       with  large  amounts of metadata may continue to have a reduced feature
2167       set.
2168
2169
2170
2171OVN 22.06.1                    OVN Architecture            ovn-architecture(7)
Impressum