1EVS_OVERVIEW(8) Openais Programmer's Manual EVS_OVERVIEW(8)
2
6 The EVS library is delivered with the openais project. This library is
7 used to create distributed applications that operate properly during
8 partitions, merges, and faults.
9 The library provides a mechanism to: * handle abstraction for multiple
11 instances of an EVS library in one application * Deliver messages *
12 Deliver configuration changes * join one or more groups * leave one or
13 more groups * send messages to one or more groups * send messages to
14 currently joined groups
15 The EVS library implements a messaging model known as Extended Virtual
17 Synchrony. This model allows one sender to transmit to many receivers
18 using standard UDP/IP. UDP/IP is unreliable and unordered, so the EVS
19 library applies ordering and reliability to messages. Hardware multi‐
20 cast is used to avoid duplicated packets with two or more receivers.
21 Erroneous messages are corrected automatically by the library.
22 Certain gaurantees are provided by the EVS library. These guarantees
24 are related to message delivery and configuration change delivery.
25
27 multicast
28 A multicast occurs when a network interface card sends a UDP
29 packet to multiple receivers simulatenously.
30 processor
32 A processor is the entity that executes the extended virtual
33 synchrony algorithms.
34 configuration
36 A configuration is the current description of the processors
37 executing the extended virtual syncrhony algorithm.
38 configuration change
40 A configuration change occurs when a new configuration is deliv‐
41 ered.
42 partition
44 A partition occurs when a configuration splits into two or more
45 configurations, or a processor fails or is stopped and leaves
46 the configuration.
47 merge A merge occurs when two or more configurations join into a
49 larger new configuration. When a new processor starts up, it is
50 treated as a configuration with only one processor and a merge
51 occurs.
52 fifo ordering
54 A message is FIFO ordered when one sender and one receiver agree
55 on the order of the messages sent.
56 agreed ordering
58 A message is AGREED ordered when all processors agree on the
59 order of the messages sent.
60 safe ordering
62 A message is SAFE ordered when all processors agree on the order
63 of messages sent and those messages are not delivered until all
64 processors have a copy of the message to deliver.
65 virtual syncrhony
67 Virtual syncrhony is obtained when all processors agree on the
68 order of messages sent and configuration changes sent for each
69 new configuration.
70
72 The virtual synchrony messaging model has many benefits for developing
73 distributed applications. Applications designed using replication have
74 the most benefits. Applications that must be able to partition and
75 merge also benefit from the virtual synchrony messaging model.
76 All applications receive a copy of transmitted messages even if there
78 are errors on the transmission media. This allows optimiziations when
79 every processor must receive a copy of the message for replication.
80 All messages are ordered according to agreed ordering. This mechanism
82 allows the avoidance of race conditions. Consider a lock service
83 implemented over several processors. Two requests occur at the same
84 time on two seperate processors. The requests are ordered for every
85 processor in the same order and delivered to the processors. Then all
86 processors will get request A before request B and can reject request
87 B. Any type of creation or deletion of a shared data structure can
88 benefit from this mechanism.
89 Self delivery ensures that messages that are sent by a processor are
91 also delivered back to that processor. This allows the processor send‐
92 ing the message to execute logic when the message is self delivered
93 according to agreed ordering and the virtual synchrony rules. It also
94 permits all logic to be placed in one message handler instead of two
95 seperate places.
96 Virtual Synchrony allows the current configuration to be used to make
98 decisions in partitions and merges. Since the configuration is sent in
99 the stream of messages to the application, the application can alter
100 its behavior based upon the configuration changes.
101
103 The EVS library is a thin IPC interface to the openais executive. The
104 openais executive provides services for the SA Forum AIS libraries as
105 well as the EVS library.
106 The openais executive uses a ring protocol and membership protocol to
108 send messages according to the semantics required by extended virtual
109 synchrony. The ring protocol creates a virtual ring of processors. A
110 token is rotated around the ring of processors. When the token is pos‐
111 sessed by a processor, that processor may multicast messages to other
112 processors in the system.
113 The token is called the ORF token (for ordering, reliability, flow con‐
115 trol). The ORF token orders all messages by increasing a sequence num‐
116 ber every time a message is multicasted. In this way, an ordering is
117 placed on all messages that all processors agree to. The token also
118 contains a retransmission list. If a token is received by a processor
119 that has not yet received a message it should have, a message sequence
120 number is added to the retransmission list. A processor that has a
121 copy of the message then retransmits the message. The ORF token pro‐
122 vides configuration-wide flow control by tracking the number of mes‐
123 sages sent and limiting the number of messages that may be sent by one
124 processor on each posession of the token.
125 The membership protocol is responsible for ring formation and detecting
127 when a processor within a ring has failed. If the token fails to make
128 a rotation within a timeout period known as the token rotation timeout,
129 the membership protocol will form a new ring. If a new processor
130 starts, it will also form a new ring. Two or more configurations may
131 be used to form a new ring, allowing many partitions to merge together
132 into one new configuration.
133
135 The EVS library obtains 8.5MB/sec throughput on 100 mbit network links
136 with many processors. Larger messages obtain better throughput results
137 because the time to access Ethernet is about the same for a small mes‐
138 sage as it is for a larger message. Smaller messages obtain better
139 messages per second, because the time to send a message is not exactly
140 the same.
141 80% of CPU utilization occurs because of encryption and authentication.
143 The openais can be built without encryption and authentication for
144 those with no security requirements and low CPU utilization require‐
145 ments. Even without encryption or authentication, under heavy load,
146 processor utilization can reach 25% on 1.5 GHZ CPU processors.
147 The current openais executive supports 16 processors, however, support
149 for more processors is possible by changing defines in the openais
150 executive. This is untested, however.
151
153 The EVS library encrypts all messages sent over the network using the
154 SOBER-128 stream cipher. The EVS library uses HMAC and SHA1 to authen‐
155 ticate all messages. The EVS library uses SOBER-128 as a pseudo random
156 number generator. The EVS library feeds the PRNG using the /dev/random
157 Linux device.
158
160 This software is not yet production, so there may still be some bugs.
161 But it appears there are very few since nobody reports any unknown bugs
162 at this point.
163
165 evs_initialize(3), evs_finalize(3), evs_fd_get(3), evs_dispatch(3),
166 evs_join(3), evs_leave(3), evs_mcast_joined(3), evs_mcast_groups(3),
167 evs_mmembership_get(3)
168openais Man Page 2004-08-31 EVS_OVERVIEW(8)