1BOOTHD(8) BOOTHD(8)
2
6 boothd - The Booth Cluster Ticket Manager.
7
9 boothd daemon [-SD] [-c config] [-l lockfile]
10 booth list [-s site] [-c config]
12 booth grant [-s site] [-c config] [-FCw] ticket
14 booth revoke [-s site] [-c config] [-w] ticket
16 booth peers [-s site] [-c config]
18 booth status [-D] [-c config]
20
22 Booth manages tickets which authorizes one of the cluster sites located
23 in geographically dispersed distances to run certain resources. It is
24 designed to be extend Pacemaker to support geographically distributed
25 clustering.
26 It is based on the RAFT protocol, see eg.
28 https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf
29 for details.
30
32 # boothd daemon -D
33 # booth list
35 # booth grant ticket-nfs
37 # booth revoke ticket-nfs
39
41 -c configfile
42 Configuration to use.
43 Can be a full path to a configuration file, or a short name; in the
45 latter case, the directory /etc/booth and suffix .conf are added.
46 Per default booth is used, which results in the path
47 /etc/booth/booth.conf.
48 The configuration name also determines the name of the PID file -
50 for the defaults, /var/run/booth/booth.pid.
51 -s
53 Site address or name.
54 The special value other can be used to specify the other site.
56 Obviously, in that case, the booth configuration must have exactly
57 two sites defined.
58 -F
60 immediate grant: Don’t wait for unreachable sites to relinquish the
61 ticket. See the Booth ticket management section below for more
62 details. For manual tickets this option allows one to grant a
63 ticket which is currently granted. See the Manual tickets section
64 below for more details.
65 This option may be DANGEROUS. It makes booth grant the ticket even
67 though it cannot ascertain that unreachable sites don’t hold the
68 same ticket. It is up to the user to make sure that unreachable
69 sites don’t have this ticket as granted.
70 -w
72 wait for the request outcome: The client waits for the final
73 outcome of grant or revoke request.
74 -C
76 wait for ticket commit to CIB: The client waits for the ticket
77 commit to CIB (only for grant requests). If one or more sites are
78 unreachable, this takes the ticket expire time (plus, if defined,
79 the acquire-after time).
80 -h, --help
82 Give a short usage output.
83 --version
85 Report version information.
86 -S
88 systemd mode: don’t fork. Disables daemonizing, the process will
89 remain in the foreground.
90 -D
92 Increases the debug output level.
93 -l lockfile
95 Use another lock file. By default, the lock file name is inferred
96 from the configuration file name. Normally not needed.
97
99 Whether the binary is called as boothd or booth doesn’t matter; the
100 first argument determines the mode of operation.
101 daemon
103 Tells boothd to serve a site. The locally configured interfaces are
104 searched for an IP address that is defined in the configuration.
105 booth then runs in either /arbitrator/ or /site/ mode.
106 client
108 Booth clients can list the ticket information (see also crm_ticket
109 -L), and revoke or grant tickets to a site.
110 The grant and, under certain circumstances, revoke operations may
112 take a while to return a definite operation’s outcome. The client
113 will wait up to the network timeout value (by default 5 seconds)
114 for the result. Unless the -w option was set, in which case the
115 client waits indefinitely.
116 In this mode the configuration file is searched for an IP address
118 that is locally reachable, ie. matches a configured subnet. This
119 allows one to run the client commands on another node in the same
120 cluster, as long as the config file and the service IP is locally
121 reachable.
122 For instance, if the booth service IP is 192.168.55.200, and the
124 local node has 192.168.55.15 configured on one of its network
125 interfaces, it knows which site it belongs to.
126 Use -s to direct client to connect to a different site.
128 status
130 boothd looks for the (locked) PID file and the UDP socket, prints
131 some output to stdout (for use in shell scripts) and returns an
132 OCF-compatible return code. With -D, a human-readable message is
133 printed to STDERR as well.
134 peers
136 List the other boothd servers we know about.
137 In addition to the type, name (IP address), and the last time the
139 server was heard from, network statistics are also printed. The
140 statistics are split into two rows, the first one consists of
141 counters for the sent packets and the second one for the received
142 packets. The first counter is the total number of packets and
143 descriptions of the other counters follows:
144 resends
146 Packets which had to be resent because the recipient didn’t
147 acknowledge a message. This usually means that either the
148 message or the acknowledgement got lost. The number of resends
149 usually reflect the network reliability.
150 error
152 Packets which either couldn’t be sent, got truncated, or were
153 badly formed. Should be zero.
154 invalid
156 These packets contain either invalid or non-existing ticket
157 name or refer to a non-existing ticket leader. Should be zero.
158 authfail
160 Packets which couldn’t be authenticated. Should be zero.
161
163 The configuration file must be identical on all sites and arbitrators.
164 A minimal file may look like this:
166 site="192.168.201.100"
168 site="192.168.202.100"
169 arbitrator="192.168.203.100"
170 ticket="ticket-db8"
171 Comments start with a hash-sign ('#'). Whitespace at the start and end
173 of the line, and around the '=', are ignored.
174 The following key/value pairs are defined:
176 port
178 The UDP/TCP port to use. Default is 9929.
179 transport
181 The transport protocol to use for Raft exchanges. Currently only
182 UDP is supported.
183 Clients use TCP to communicate with a daemon; Booth will always
185 bind and listen to both UDP and TCP ports.
186 authfile
188 File containing the authentication key. The key can be either
189 binary or text. If the latter, then both leading and trailing white
190 space, including new lines, is ignored. This key is a shared secret
191 and used to authenticate both clients and servers. The key must be
192 between 8 and 64 characters long and be readable only by the file
193 owner.
194 maxtimeskew
196 As protection against replay attacks, packets contain generation
197 timestamps. Such a timestamp is not allowed to be too old. Just how
198 old can be specified with this parameter. The value is in seconds
199 and the default is 600 (10 minutes). If clocks vary more than this
200 default between sites and nodes (which is definitely something you
201 should fix) then set this parameter to a higher value. The time
202 skew test is performed only in concert with authentication.
203 debug
205 Specifies the debug output level. Alternative to command line
206 argument. Effective only for daemon mode of operation.
207 site
209 Defines a site Raft member with the given IP. Sites can acquire
210 tickets. The sites' IP should be managed by the cluster.
211 arbitrator
213 Defines an arbitrator Raft member with the given IP. Arbitrators
214 help reach consensus in elections and cannot hold tickets.
215 Booth needs at least three members for normal operation. Odd number of
217 members provides more redundancy.
218 site-user, site-group, arbitrator-user, arbitrator-group
220 These define the credentials boothd will be running with.
221 On a (Pacemaker) site the booth process will have to call
223 crm_ticket, so the default is to use hacluster:'haclient'; for an
224 arbitrator this user and group might not exists, so there we
225 default to nobody:'nobody'.
226 ticket
228 Registers a ticket. Multiple tickets can be handled by single Booth
229 instance.
230 Use the special ticket name __defaults__ to modify the defaults.
232 The __defaults__ stanza must precede all the other ticket
233 specifications.
234 All times are in seconds.
236 expire
238 The lease time for a ticket. After that time the ticket can be
239 acquired by another site if the ticket holder is not reachable.
240 The default is 600.
242 acquire-after
244 Once a ticket is lost, wait this time in addition before acquiring
245 the ticket.
246 This is to allow for the site that lost the ticket to relinquish
248 the resources, by either stopping them or fencing a node.
249 A typical delay might be 60 seconds, but ultimately it depends on
251 the protected resources and the fencing configuration.
252 The default is 0.
254 renewal-freq
256 Set the ticket renewal frequency period.
257 If the network reliability is often reduced over prolonged periods,
259 it is advisable to try to renew more often.
260 Before every renewal, if defined, the command or commands specified
262 in before-acquire-handler is run. In that case the renewal-freq
263 parameter is effectively also the local cluster monitoring
264 interval.
265 timeout
267 After that time booth will re-send packets if there was an
268 insufficient number of replies. This should be long enough to allow
269 packets to reach other members.
270 The default is 5.
272 retries
274 Defines how many times to retry sending packets before giving up
275 waiting for acks from other members.
276 Default is 10. Values lower than 3 are illegal.
278 Ticket renewals should allow for this number of retries. Hence, the
280 total retry time must be shorter than the renewal time (either half
281 the expire time or renewal-freq):
282 timeout*(retries+1) < renewal
284 weights
286 A comma-separated list of integers that define the weight of
287 individual Raft members, in the same order as the site and
288 arbitrator lines.
289 Default is 0 for all; this means that the order in the
291 configuration file defines priority for conflicting requests.
292 before-acquire-handler
294 If set, this parameter specifies either a file containing a program
295 to be run or a directory where a number of programs can reside.
296 They are invoked before boothd tries to acquire or renew a ticket.
297 If any of them exits with a code other than 0, boothd relinquishes
298 the ticket.
299 Thus it is possible to ensure whether the services and its
301 dependencies protected by the ticket are in good shape at this
302 site. For instance, if a service in the dependency-chain has a
303 failcount of INFINITY on all available nodes, the service will be
304 unable to run. In that case, it is of no use to claim the ticket.
305 One or more arguments may follow the program or directory location.
307 Typically, there is at least the name of one of the resources which
308 depend on this ticket.
309 See below for details about booth specific environment variables.
311 The distributed service-runnable script is an example which may be
312 used to test whether a pacemaker resource can be started.
313 attr-prereq
315 Sites can have GEO attributes managed with the geostore(8) program.
316 Attributes are within ticket’s scope and may be tested by boothd
317 for additional control of ticket failover (automatic) or ticket
318 acquire (manual).
319 Attributes are typically used to convey extra information about
321 resources, for instance database replication status. The attributes
322 are commonly updated by resource agents.
323 Attribute values are referenced in expressions and may be tested
325 for equality with the eq binary operator or inequality with the ne
326 operator. The usage is as follows:
327 attr-prereq = <grant_type> <name> <op> <value>
329 <grant_type>: "auto" | "manual"
331 <name>: attribute name
332 <op>: "eq" | "ne"
333 <value>: attribute value
334 The two grant types are auto for ticket failover and manual for
336 grants using the booth client. Only in case the expression
337 evaluates to true can the ticket be granted.
338 It is not clear whether the manual grant type has any practical use
340 because, obviously, this operation is anyway controlled by a human.
341 Note that there can be no guarantee on whether an attribute value
343 is up to date, i.e. if it actually reflects the current state.
344 mode
346 Specifies if the ticket is manual or automatic.
347 By default all tickets are automatic (that is, they are fully
349 controlled by Raft algorithm). Assign the strings "manual" or
350 "MANUAL" to define the ticket as manually controlled.
351 One example of a booth configuration file:
353 transport = udp
355 port = 9930
356 # D-85774
358 site="192.168.201.100"
359 # D-90409
360 site="::ffff:192.168.202.100"
361 # A-1120
362 arbitrator="192.168.203.100"
363 ticket="ticket-db8"
365 expire = 600
366 acquire-after = 60
367 timeout = 10
368 retries = 5
369 renewal-freq = 60
370 before-acquire-handler = /usr/share/booth/service-runnable db8
371 attr-prereq = auto repl_state eq ACTIVE
372
374 The booth cluster guarantees that every ticket is owned by only one
375 site at the time.
376 Tickets must be initially granted with the booth client grant command.
378 Once it gets granted, the ticket is managed by the booth cluster.
379 Hence, only granted tickets are managed by booth.
380 If the ticket gets lost, i.e. that the other members of the booth
382 cluster do not hear from the ticket owner in a sufficiently long time,
383 one of the remaining sites will acquire the ticket. This is what is
384 called ticket failover.
385 If the remaining members cannot form a majority, then the ticket cannot
387 fail over.
388 A ticket may be revoked at any time with the booth client revoke
390 command. For revoke to succeed, the site holding the ticket must be
391 reachable.
392 Once the ticket is administratively revoked, it is not managed by the
394 booth cluster anymore. For the booth cluster to start managing the
395 ticket again, it must be again granted to a site.
396 The grant operation, in case not all sites are reachable, may get
398 delayed for the ticket expire time (and, if defined, the acquire-after
399 time). The reason is that the other booth members may not know if the
400 ticket is currently granted at the unreachable site.
401 This delay may be disabled with the -F option. In that case, it is up
403 to the administrator to make sure that the unreachable site is not
404 holding the ticket.
405 When the ticket is managed by booth, it is dangerous to modify it
407 manually using either crm_ticket command or crm site ticket. Neither of
408 these tools is aware of booth and, consequently, booth itself may not
409 be aware of any ticket status changes. A notable exception is setting
410 the ticket to standby which is typically done before a planned
411 failover.
412
414 Tickets are not meant to be moved around quickly, the default expire
415 time is 600 seconds (10 minutes).
416 booth works with both IPv4 and IPv6 addresses.
418 booth renews a ticket before it expires, to account for possible
420 transmission delays. The renewal time, unless explicitly set, is set to
421 half the expire time.
422
424 Currently, there’s only one external handler defined (see the
425 before-acquire-handler configuration item above).
426 The following environment variables are exported to the handler:
428 BOOTH_TICKET
430 The ticket name, as given in the configuration file. (See ticket
431 item above.)
432 BOOTH_LOCAL
434 The local site name, as defined in site.
435 BOOTH_CONF_PATH
437 The path to the active configuration file.
438 BOOTH_CONF_NAME
440 The configuration name, as used by the -c commandline argument.
441 BOOTH_TICKET_EXPIRES
443 When the ticket expires (in seconds since 1.1.1970), or 0.
444 The handler is invoked with positional arguments specified after it.
446
448 /etc/booth/booth.conf
449 The default configuration file name. See also the -c argument.
450 /etc/booth/authkey
452 There is no default, but this is a typical location for the shared
453 secret (authentication key).
454 /var/run/booth/
456 Directory that holds PID/lock files. See also the status command.
457
459 In essence, every ticket corresponds to a separate Raft cluster.
460 A ticket is granted to the Raft Leader which then owns (or keeps) the
462 ticket.
463
465 The booth daemon for an arbitrator which typically doesn’t run the
466 cluster stack, may be started through systemd or with
467 /etc/init.d/booth-arbitrator, depending on which init system the
468 platform supports.
469 The SysV init script starts a booth arbitrator for every configuration
471 file found in /etc/booth.
472 Platforms running systemd can enable and start every configuration
474 separately using systemctl:
475 # systemctl enable booth@<configurationname>
477 # systemctl start booth@<configurationname>
478 systemctl requires the configuration name, even for the default name
480 booth.
481 MANUAL TICKETS
483 Manual tickets allow users to create and manage tickets which are
485 subsequently handled by booth without using the Raft algorithm.
486 Granting and revoking manual tickets is fully controlled by the
487 administrator. It is possible to define a number of manual and normal
488 tickets in one GEO cluster.
489 Automatic ticket management provided by Raft algorithm isn’t applied to
491 manually controlled tickets. In particular, there is no elections,
492 automatic failover procedures, and term expiration.
493 However, booth controls if a ticket is currently being granted to any
495 site and warns the user approprietly.
496 Tickets which were manually granted to a site, will remain there until
498 they are manually revoked. Even if a site becomes offline, the ticket
499 will not be moved to another site. This behavior allows administrators
500 to make sure that some services will remain in a particular site and
501 will not be moved to another site, possibly located in a different
502 geographical location.
503 Also, configuring only manual tickets in a GEO cluster, allows one to
505 have just two sites in a cluster, without a need of having an
506 arbitrator. This is possible because there is no automatic elections
507 and no voting performed for manual tickets.
508 Manual tickets are defined in a configuration files by adding a mode
510 ticket parameter and setting it to manual or MANUAL:
511 ticket="manual-ticket"
513 [...]
514 mode = manual
515 [...]
516 Manual tickets can be granted and revoked by using normal grant and
518 revoke commands, with the usual flags and parameters. The only
519 difference is that specyfiyng -F flag during grant command, forced a
520 site to become a leader of the specified ticket, even if the ticket is
521 granted to another site.
522
524 0
525 Success. For the status command: Daemon running.
526 1 (PCMK_OCF_UNKNOWN_ERROR)
528 General error code.
529 7 (PCMK_OCF_NOT_RUNNING)
531 No daemon process for that configuration active.
532
534 Booth is tested regularly. See the README-testing file for more
535 information.
536 Please report any bugs either at GitHub:
538 https://github.com/ClusterLabs/booth/issues
539 Or, if you prefer bugzilla, at openSUSE bugzilla (component "High
541 Availability"):
542 https://bugzilla.opensuse.org/enter_bug.cgi?product=openSUSE%20Factory
543
545 boothd was originally written (mostly) by Jiaju Zhang.
546 In 2013 and 2014 Philipp Marek took over maintainership.
548 Since April 2014 it has been mainly developed by Dejan Muhamedagic.
550 Many people contributed (see the AUTHORS file).
552
554 GitHub: https://github.com/ClusterLabs/booth
555 Documentation:
557 http://doc.opensuse.org/products/draft/SLE-HA/SLE-ha-guide_sd_draft/cha.ha.geo.html
558
560 Copyright © 2011 Jiaju Zhang jjzhang@suse.de
561 Copyright © 2013-2014 Philipp Marek philipp.marek@linbit.com
563 Copyright © 2014 Dejan Muhamedagic dmuhamedagic@suse.com
565 Free use of this software is granted under the terms of the GNU General
567 Public License (GPL) as of version 2 (see COPYING file) or later.
568 2022-11-21 BOOTHD(8)