1COROSYNC-QDEVICE(8) System Manager's Manual COROSYNC-QDEVICE(8)
2
6 corosync-qdevice - QDevice daemon
7
9 corosync-qdevice [-dfh] [-S option=value[,option2=value2,...]]
10
13 corosync-qdevice is a daemon running on each node of a cluster. It pro‐
14 vides a configured number of votes to the quorum subsystem based on a
15 third-party arbitrator's decision. Its primary use is to allow a clus‐
16 ter to sustain more node failures than standard quorum rules allow. It
17 is recommended for clusters with an even number of nodes and highly
18 recommended for 2 node clusters.
19
21 -d Forcefully turn on debug information without the need to change
22 corosync.conf. For bumping syslog messages priority to info,
23 use this parameter twice.
24 -f Do not daemonize, run in the foreground.
26 -h Show short help text
28 -S Set advanced settings described in its own section below. This
30 option shouldn't be generally used because most of the options
31 are not safe to change.
32
34 corosync-qdevice reads its configuration from corosync.conf file.
35 The main configuration is within quorum.device sub-key. Each model also
37 has its own configuration within a similarly named sub-key.
38 model Specifies the model to be used. This parameter is required.
40 corosync-qdevice is modular and is able to support multiple dif‐
41 ferent models. The model basically defines what type of arbitra‐
42 tor is used. Currently only net is supported.
43 timeout
45 Specifies how often corosync-qdevice should call the votequo‐
46 rum_qdevice_poll function. It is also used by the net model to
47 adjust its hearbeat timeout. It is recommended that you don't
48 change this value. Default is 10000.
49 sync_timeout
51 Specifies how often corosync-qdevice should call the votequo‐
52 rum_qdevice_poll function during a sync phase. It is recommended
53 that you don't change this value. Default is 30000.
54 votes The number of votes provided to the cluster by qdevice. Default
56 is (number_of_nodes - 1) or generally sum(votes_per_node) - 1.
57 quorum.device.heuristics subkey holds the configuration of the heuris‐
60 tics. Heuristics are set of commands executed locally on startup, clus‐
61 ter membership change, successful connect to corosync-qnetd and option‐
62 ally also at regular times. Commands are executed in parallel. When
63 all commands finish successfully (their return error code is zero) on
64 time, heuristics have passed, otherwise they have failed. The heuris‐
65 tics result is sent to corosync-qnetd and there it's used in calcula‐
66 tions to determine which partition should be quorate.
67 timeout
69 Specifies maximum time in milliseconds how long corosync-qdevice
70 waits till the heuristics commands finish. If some command
71 doesn't finish before the timeout, it's killed and heuristics
72 fail. This timeout is used for heuristics executed at regular
73 times. Default value is half of the quorum.device.timeout, so
74 5000.
75 sync_timeout
77 Similar to quorum.device.heuristics.timeout but used during mem‐
78 bership changes. Default value is half of the quorum.de‐
79 vice.sync_timeout, so 15000.
80 interval
82 Specifies interval between two regular heuristics execution. De‐
83 fault value is 3 * quorum.device.timeout, so 30000.
84 mode Can be one of on, sync or off and specifies mode of operation of
86 heuristics. Default is off, which means heuristics are disabled.
87 When sync is set, heuristics are executed only during startup,
88 membership change and when connection to corosync-qnetd is es‐
89 tablished. When heuristics should be running also on regular ba‐
90 sis, this option should be set to on value.
91 exec_NAME
93 defines executables. NAME can be arbitrary valid cmap key name
94 string and it has no special meaning. The value of this vari‐
95 able must contain a command to execute. The value is parsed
96 (split) into arguments similarly as Bourne shell would do. Quot‐
97 ing is possible by using backslash and double quotes.
98 quorum.device.net subkey holds the configuration for model net.
101 tls Can be one of on, off or required and specifies if tls should be
103 used. on means a connection with TLS is attempted first, but if
104 the server doesn't advertise TLS support then non-TLS will be
105 used. off is used then TLS is not required and it's then not
106 even tried. This mode is the only one which doesn't need a prop‐
107 erly initialized NSS database. required means TLS is required
108 and if the server doesn't support TLS, qdevice will exit with
109 error message. Default is on.
110 host Specifies the IP address or host name of the qnetd server to be
112 used. This parameter is required.
113 port Specifies TCP port of qnetd server. Default is 5403.
115 algorithm
117 Decision algorithm. Can be one of the ffsplit or lms. (actually
118 there are also test and 2nodelms, both of which are mainly for
119 developers and shouldn't be used for production clusters). For
120 a description of what each algorithm means and how the algo‐
121 rithms differ see their individual sections. Default value is
122 ffsplit.
123 tie_breaker
125 can be one of lowest, highest or valid_node_id (number) values.
126 It's used as a fallback if qdevice has to decide between two or
127 more equal partitions. lowest means the partition with the low‐
128 est node id is chosen. highest means the partition with highest
129 node id is chosen. And valid_node_id means that the partition
130 containing the node with the given node id is chosen. Default
131 is lowest.
132 connect_timeout
134 Timeout when corosync-qdevice is trying to connect to corosync-
135 qnetd host. Default is 0.8 * quorum.device.timeout.
136 force_ip_version
138 can be one of 0|4|6 and forces the software to use the given IP
139 version. 0 (default value) means IPv6 is preferred and IPv4
140 should be used as a fallback.
141 keep_active_partition_tie_breaker
143 Can be one of on or off and specifies if keep active partition
144 tie breaker should be used. When this option is enabled and tie
145 happens QNetd will prefer partition with members of previously
146 active (quorate) partition. This is hard-coded behavior of LMS
147 algorithm so this setting affects only FFSplit algorithm. De‐
148 fault is on.
149 Logging configuration is within the logging directive. corosync-qde‐
152 vice parses and supports only debug option. The logger_subsys sub-di‐
153 rective can be also used if subsys is set to QDEVICE.
154 For corosync-qdevice to work correctly, the nodelist directive has to
157 be used and properly configured. Also the net model requires that
158 totem.cluster_name option is set.
159
162 For model net to work using TLS, it's necessary to create the NSS data‐
163 base, import Qnetd CA certificate, and get/distribute a valid client
164 certificate.
165 If pcs is used (recommended) the following steps are not needed because
167 pcs does them automatically.
168 corosync-qdevice-net-certutil is the tool to perform required actions
170 semi-automatically. Please consult the help output of it and its man
171 page. For a first time configuration it may make sense to start with
172 the -Q option.
173 If TLS is not required just edit corosync.conf file and set quorum.de‐
175 vice.net.tls to off.
176 Depending on configuration of NSS (stored in nss.config file usually in
178 /etc/crypto-policies/back-ends/ directory) disabled ciphers or too
179 short keys may be rejected. Proper solution is to regenerate NSS data‐
180 bases for both corosync-qnetd and corosync-qdevice daemons. As a quick
181 workaround it's also possible to set environment variable NSS_IG‐
182 NORE_SYSTEM_POLICY=1 before running corosync-qdevice daemon.
183 When NSS is updated it may also be needed to upgrade database into new
185 format. There is no consensus on recommended way, but following command
186 seems to work just fine (if qdevice sysconfdir is set to /etc)
187 # certutil -N -d /etc/corosync/qdevice/net/nssdb -f /etc/corosync/qdevice/net/nssdb/pwdfile.txt
189
192 Algorithms are used to change behavior of how corosync-qnetd provides
193 votes to a given node/partition. Currently there are two algorithms
194 supported.
195 ffsplit
197 This one makes sense only for clusters with an even number of
198 nodes. It provides exactly one vote to the partition with the
199 highest number of active nodes. If there are two exactly similar
200 partitions, it provides its vote to the partition with higher
201 score. The score is computed as (number_of_connected_nodes +
202 number_of_connected_nodes_with_passed_heuristics - num‐
203 ber_of_connected_nodes_with_failed_heuristics) If the scores are
204 equal, the vote is provided to partition with the most clients
205 connected to the qnetd server. If this number is also equal,
206 then the tie_breaker is used. It is able to transition its vote
207 if the currently active partition becomes partitioned and a non-
208 active partition still has at least 50% of the active nodes. Be‐
209 cause of this, a vote is not provided if the qnetd connection is
210 not active.
211 To use this algorithm it's required to set the number of votes
213 per node to 1 (default) and the qdevice number of votes has to
214 be also 1. This is achieved by setting quorum.device.votes key
215 in corosync.conf file to 1.
216 lms Last-man-standing. If the node is the only one left in the clus‐
218 ter that can see the qnetd server then we return a vote.
219 If more than one node can see the qnetd server but some nodes
221 can't see each other then the cluster is divided up into 'parti‐
222 tions' based on their ring_id and this algorithm returns a vote
223 to the partition with highest heuristics score (computed the
224 same way as for the ffsplit algorithm), or if there is more than
225 1 partition with equal scores, the largest active partition or,
226 if there is more than 1 equal partition, the partition that con‐
227 tains the tie_breaker node (lowest, highest, etc). For LMS to
228 work, the number of qdevice votes has to be set to default (so
229 just delete quorum.device.votes key from corosync.conf).
230
233 Set by using -S option. The default value is shown in parentheses) Op‐
234 tions beginning with net_ prefix are specific to model net.
235 lock_file
237 Lock file location. (/var/run/corosync-qdevice/corosync-qde‐
238 vice.pid)
239 local_socket_file
241 Internal IPC socket file location. (/var/run/corosync-qde‐
242 vice/corosync-qdevice.sock)
243 local_socket_backlog
245 Parameter passed to listen syscall. (10)
246 max_cs_try_again
248 How many times to retry the call to a corosync function which
249 has returned CS_ERR_TRY_AGAIN. (10)
250 votequorum_device_name
252 Name used for qdevice registration. (Qdevice)
253 ipc_max_clients
255 Maximum allowed simultaneous IPC clients. (10)
256 ipc_max_receive_size
258 Maximum size of a message received by IPC client. (4096)
259 ipc_max_send_size
261 Maximum size of a message allowed to be sent to an IPC client.
262 (65536)
263 master_wins
265 Force enable/disable master wins. (default is model)
266 heuristics_ipc_max_send_buffers
268 Maximum number of heuristics worker send buffers. (128)
269 heuristics_ipc_max_send_receive_size
271 Maximum size of a message allowed to be send to, or received
272 from heuristics worker. (4096)
273 heuristics_min_timeout
275 Minimum heuristics timeout accepted by client in ms. (1000)
276 heuristics_max_timeout
278 Maximum heuristics timeout accepted by client in ms. (120000)
279 heuristics_min_interval
281 Minimum heuristics interval accepted by client in ms. (1000)
282 heuristics_max_interval
284 Maximum heuristics interval accepted by client in ms. (3600000)
285 heuristics_max_execs
287 Maximum number of exec_ commands. (32)
288 heuristics_use_execvp
290 Use execvp instead of execv for executing commands. (off)
291 heuristics_max_processes
293 Maximum number of processes running at one time. (160)
294 heuristics_kill_list_interval
296 Interval between status is gathered and eventually signal is
297 sent to processes which didn't finished on time in ms. (5000)
298 net_nss_db_dir
300 NSS database directory. (/etc/corosync/qdevice/net/nssdb)
301 net_initial_msg_receive_size
303 Initial (used during connection parameters negotiation) maximum
304 size of the receive buffer for message (maximum allowed message
305 size received from qnetd). (32768)
306 net_initial_msg_send_size
308 Initial (used during connection parameter negotiation) maximum
309 size of one send buffer (message) to be sent to server. (32768)
310 net_min_msg_send_size
312 Minimum required size of one send buffer (message) to be sent to
313 server. (32768)
314 net_max_msg_receive_size
316 Maximum allowed size of receive buffer for a message sent by
317 server. (16777216)
318 net_max_send_buffers
320 Maximum number of send buffers. (10)
321 net_nss_qnetd_cn
323 Canonical name of qnetd server certificate. (Qnetd Server)
324 net_nss_client_cert_nickname
326 NSS nickname of qdevice client certificate. (Cluster Cert)
327 net_heartbeat_interval_min
329 Minimum heartbeat timeout accepted by client in ms. (1000)
330 net_heartbeat_interval_max
332 Maximum heartbeat timeout accepted by client in ms. (120000)
333 net_min_connect_timeout
335 Minimum connection timeout accepted by client in ms. (1000)
336 net_max_connect_timeout
338 Maximum connection timeout accepted by client in ms. (120000)
339 net_test_algorithm_enabled
341 Enable test algorithm. (if built with --enable-debug on, other‐
342 wise off)
343
346 Define qdevice with net model connecting to qnetd running on qnetd.ex‐
347 ample.org host, using ffsplit algorithm. Heuristics is set to sync
348 mode and executes two commands.
349 quorum {
351 provider: corosync_votequorum
352 device {
353 votes: 1
354 model: net
355 net {
356 tls: on
357 host: qnetd.example.org
358 algorithm: ffsplit
359 }
360 heuristics {
361 mode: sync
362 exec_ping: /bin/ping -q -c 1 "www.example.org"
363 exec_test_txt_exists: /usr/bin/test -f /tmp/test.txt
364 }
365 }
366
368 corosync-qdevice-tool(8) corosync-qdevice-net-certutil(8) corosync-
369 qnetd(8) corosync.conf(5) votequorum_qdevice_poll(3)
370
372 Jan Friesse
373 2020-10-27 COROSYNC-QDEVICE(8)