1BOOTHD(8) BOOTHD(8)
2
6 boothd - The Booth Cluster Ticket Manager.
7
9 boothd daemon [-SD] [-c config] [-l lockfile]
10 booth list [-s site] [-c config]
12 booth grant [-s site] [-c config] [-FCw] ticket
14 booth revoke [-s site] [-c config] [-w] ticket
16 booth peers [-s site] [-c config]
18 booth status [-D] [-c config]
20
22 Booth manages tickets which authorizes one of the cluster sites located
23 in geographically dispersed distances to run certain resources. It is
24 designed to be extend Pacemaker to support geographically distributed
25 clustering.
26 It is based on the RAFT protocol, see eg.
28 https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf
29 for details.
30
32 # boothd daemon -D
33 # booth list
35 # booth grant ticket-nfs
37 # booth revoke ticket-nfs
39
41 -c configfile
42 Configuration to use.
43 Can be a full path to a configuration file, or a short name; in the
45 latter case, the directory /etc/booth and suffix .conf are added.
46 Per default booth is used, which results in the path
47 /etc/booth/booth.conf.
48 The configuration name also determines the name of the PID file -
50 for the defaults, /var/run/booth/booth.pid.
51 -s
53 Site address or name.
54 The special value 'other' can be used to specify the other
56 site. Obviously, in that case, the booth configuration must
57 have exactly two sites defined.
58 -F
60 immediate grant: Don’t wait for unreachable sites to relinquish the
61 ticket. See the Booth ticket management section below for more
62 details. For manual tickets this option allows one to grant a
63 ticket which is currently granted. See the Manual tickets section
64 below for more details.
65 This option may be DANGEROUS. It makes booth grant the ticket
67 even though it cannot ascertain that unreachable sites don't
68 hold the same ticket. It is up to the user to make sure that
69 unreachable sites don't have this ticket as granted.
70 -w
72 wait for the request outcome: The client waits for the final
73 outcome of grant or revoke request.
74 -C
76 wait for ticket commit to CIB: The client waits for the ticket
77 commit to CIB (only for grant requests). If one or more sites are
78 unreachable, this takes the ticket expire time (plus, if defined,
79 the acquire-after time).
80 -h, --help
82 Give a short usage output.
83 --version
85 Report version information.
86 -S
88 systemd mode: don’t fork. Disables daemonizing, the process will
89 remain in the foreground.
90 -D
92 Increases the debug output level.
93 -l lockfile
95 Use another lock file. By default, the lock file name is inferred
96 from the configuration file name. Normally not needed.
97
99 Whether the binary is called as boothd or booth doesn’t matter; the
100 first argument determines the mode of operation.
101 daemon
103 Tells boothd to serve a site. The locally configured interfaces are
104 searched for an IP address that is defined in the configuration.
105 booth then runs in either /arbitrator/ or /site/ mode.
106 client
108 Booth clients can list the ticket information (see also crm_ticket
109 -L), and revoke or grant tickets to a site.
110 The grant and, under certain circumstances, revoke operations may
112 take a while to return a definite operation’s outcome. The client
113 will wait up to the network timeout value (by default 5 seconds)
114 for the result. Unless the -w option was set, in which case the
115 client waits indefinitely.
116 In this mode the configuration file is searched for an IP address
118 that is locally reachable, ie. matches a configured subnet. This
119 allows one to run the client commands on another node in the same
120 cluster, as long as the config file and the service IP is locally
121 reachable.
122 For instance, if the booth service IP is 192.168.55.200, and the
124 local node has 192.168.55.15 configured on one of its network
125 interfaces, it knows which site it belongs to.
126 Use -s to direct client to connect to a different site.
128 status
130 boothd looks for the (locked) PID file and the UDP socket, prints
131 some output to stdout (for use in shell scripts) and returns an
132 OCF-compatible return code. With -D, a human-readable message is
133 printed to STDERR as well.
134 peers
136 List the other boothd servers we know about.
137 In addition to the type, name (IP address), and the last time the
139 server was heard from, network statistics are also printed. The
140 statistics are split into two rows, the first one consists of
141 counters for the sent packets and the second one for the received
142 packets. The first counter is the total number of packets and
143 descriptions of the other counters follows:
144 resends
146 Packets which had to be resent because the recipient didn’t
147 acknowledge a message. This usually means that either the message
148 or the acknowledgement got lost. The number of resends usually
149 reflect the network reliability.
150 error
152 Packets which either couldn’t be sent, got truncated, or were badly
153 formed. Should be zero.
154 invalid
156 These packets contain either invalid or non-existing ticket name or
157 refer to a non-existing ticket leader. Should be zero.
158 authfail
160 Packets which couldn’t be authenticated. Should be zero.
161
163 The configuration file must be identical on all sites and arbitrators.
164 A minimal file may look like this:
166 site="192.168.201.100"
168 site="192.168.202.100"
169 arbitrator="192.168.203.100"
170 ticket="ticket-db8"
171 Comments start with a hash-sign ('#'). Whitespace at the start and end
173 of the line, and around the '=', are ignored.
174 The following key/value pairs are defined:
176 port
178 The UDP/TCP port to use. Default is 9929.
179 transport
181 The transport protocol to use for Raft exchanges. Currently only
182 UDP is supported.
183 Clients use TCP to communicate with a daemon; Booth will always
185 bind and listen to both UDP and TCP ports.
186 authfile
188 File containing the authentication key. The key can be either
189 binary or text. If the latter, then both leading and trailing white
190 space, including new lines, is ignored. This key is a shared secret
191 and used to authenticate both clients and servers. The key must be
192 between 8 and 64 characters long and be readable only by the file
193 owner.
194 maxtimeskew
196 As protection against replay attacks, packets contain generation
197 timestamps. Such a timestamp is not allowed to be too old. Just how
198 old can be specified with this parameter. The value is in seconds
199 and the default is 600 (10 minutes). If clocks vary more than this
200 default between sites and nodes (which is definitely something you
201 should fix) then set this parameter to a higher value. The time
202 skew test is performed only in concert with authentication.
203 site
205 Defines a site Raft member with the given IP. Sites can acquire
206 tickets. The sites' IP should be managed by the cluster.
207 arbitrator
209 Defines an arbitrator Raft member with the given IP. Arbitrators
210 help reach consensus in elections and cannot hold tickets.
211 Booth needs at least three members for normal operation. Odd number of
213 members provides more redundancy.
214 site-user, site-group, arbitrator-user, arbitrator-group
216 These define the credentials boothd will be running with.
217 On a (Pacemaker) site the booth process will have to call
219 crm_ticket, so the default is to use hacluster:'haclient'; for an
220 arbitrator this user and group might not exists, so there we
221 default to nobody:'nobody'.
222 ticket
224 Registers a ticket. Multiple tickets can be handled by single Booth
225 instance.
226 Use the special ticket name defaults to modify the defaults. The
228 defaults stanza must precede all the other ticket specifications.
229 All times are in seconds.
231 expire
233 The lease time for a ticket. After that time the ticket can be
234 acquired by another site if the ticket holder is not reachable.
235 The default is 600.
237 acquire-after
239 Once a ticket is lost, wait this time in addition before acquiring
240 the ticket.
241 This is to allow for the site that lost the ticket to relinquish
243 the resources, by either stopping them or fencing a node.
244 A typical delay might be 60 seconds, but ultimately it depends on
246 the protected resources and the fencing configuration.
247 The default is 0.
249 renewal-freq
251 Set the ticket renewal frequency period.
252 If the network reliability is often reduced over prolonged periods,
254 it is advisable to try to renew more often.
255 Before every renewal, if defined, the command or commands specified
257 in before-acquire-handler is run. In that case the renewal-freq
258 parameter is effectively also the local cluster monitoring
259 interval.
260 timeout
262 After that time booth will re-send packets if there was an
263 insufficient number of replies. This should be long enough to allow
264 packets to reach other members.
265 The default is 5.
267 retries
269 Defines how many times to retry sending packets before giving up
270 waiting for acks from other members.
271 Default is 10. Values lower than 3 are illegal.
273 Ticket renewals should allow for this number of retries. Hence, the
275 total retry time must be shorter than the renewal time (either half
276 the expire time or renewal-freq):
277 timeout*(retries+1) < renewal
279 weights
281 A comma-separated list of integers that define the weight of
282 individual Raft members, in the same order as the site and
283 arbitrator lines.
284 Default is 0 for all; this means that the order in the
286 configuration file defines priority for conflicting requests.
287 before-acquire-handler
289 If set, this parameter specifies either a file containing a program
290 to be run or a directory where a number of programs can reside.
291 They are invoked before boothd tries to acquire or renew a ticket.
292 If any of them exits with a code other than 0, boothd relinquishes
293 the ticket.
294 Thus it is possible to ensure whether the services and its
296 dependencies protected by the ticket are in good shape at this
297 site. For instance, if a service in the dependency-chain has a
298 failcount of INFINITY on all available nodes, the service will be
299 unable to run. In that case, it is of no use to claim the ticket.
300 One or more arguments may follow the program or directory location.
302 Typically, there is at least the name of one of the resources which
303 depend on this ticket.
304 See below for details about booth specific environment variables.
306 The distributed service-runnable script is an example which may be
307 used to test whether a pacemaker resource can be started.
308 attr-prereq
310 Sites can have GEO attributes managed with the geostore(8) program.
311 Attributes are within ticket’s scope and may be tested by boothd
312 for additional control of ticket failover (automatic) or ticket
313 acquire (manual).
314 Attributes are typically used to convey extra information about
316 resources, for instance database replication status. The attributes
317 are commonly updated by resource agents.
318 Attribute values are referenced in expressions and may be tested
320 for equality with the eq binary operator or inequality with the ne
321 operator. The usage is as follows:
322 attr-prereq = <grant_type> <name> <op> <value>
324 <grant_type>: "auto" | "manual"
326 <name>: attribute name
327 <op>: "eq" | "ne"
328 <value>: attribute value
329 The two grant types are auto for ticket failover and manual for
331 grants using the booth client. Only in case the expression
332 evaluates to true can the ticket be granted.
333 It is not clear whether the manual grant type has any practical use
335 because, obviously, this operation is anyway controlled by a human.
336 Note that there can be no guarantee on whether an attribute value
338 is up to date, i.e. if it actually reflects the current state.
339 mode
341 Specifies if the ticket is manual or automatic.
342 By default all tickets are automatic (that is, they are fully
344 controlled by Raft algorithm). Assign the strings "manual" or
345 "MANUAL" to define the ticket as manually controlled.
346 One example of a booth configuration file:
348 transport = udp
350 port = 9930
351 # D-85774
353 site="192.168.201.100"
354 # D-90409
355 site="::ffff:192.168.202.100"
356 # A-1120
357 arbitrator="192.168.203.100"
358 ticket="ticket-db8"
360 expire = 600
361 acquire-after = 60
362 timeout = 10
363 retries = 5
364 renewal-freq = 60
365 before-acquire-handler = /usr/share/booth/service-runnable db8
366 attr-prereq = auto repl_state eq ACTIVE
367
369 The booth cluster guarantees that every ticket is owned by only one
370 site at the time.
371 Tickets must be initially granted with the booth client grant command.
373 Once it gets granted, the ticket is managed by the booth cluster.
374 Hence, only granted tickets are managed by booth.
375 If the ticket gets lost, i.e. that the other members of the booth
377 cluster do not hear from the ticket owner in a sufficiently long time,
378 one of the remaining sites will acquire the ticket. This is what is
379 called ticket failover.
380 If the remaining members cannot form a majority, then the ticket cannot
382 fail over.
383 A ticket may be revoked at any time with the booth client revoke
385 command. For revoke to succeed, the site holding the ticket must be
386 reachable.
387 Once the ticket is administratively revoked, it is not managed by the
389 booth cluster anymore. For the booth cluster to start managing the
390 ticket again, it must be again granted to a site.
391 The grant operation, in case not all sites are reachable, may get
393 delayed for the ticket expire time (and, if defined, the acquire-after
394 time). The reason is that the other booth members may not know if the
395 ticket is currently granted at the unreachable site.
396 This delay may be disabled with the -F option. In that case, it is up
398 to the administrator to make sure that the unreachable site is not
399 holding the ticket.
400 When the ticket is managed by booth, it is dangerous to modify it
402 manually using either crm_ticket command or crm site ticket. Neither of
403 these tools is aware of booth and, consequently, booth itself may not
404 be aware of any ticket status changes. A notable exception is setting
405 the ticket to standby which is typically done before a planned
406 failover.
407
409 Tickets are not meant to be moved around quickly, the default expire
410 time is 600 seconds (10 minutes).
411 booth works with both IPv4 and IPv6 addresses.
413 booth renews a ticket before it expires, to account for possible
415 transmission delays. The renewal time, unless explicitly set, is set to
416 half the expire time.
417
419 Currently, there’s only one external handler defined (see the
420 before-acquire-handler configuration item above).
421 The following environment variables are exported to the handler:
423 *BOOTH_TICKET
425 The ticket name, as given in the configuration file. (See ticket
426 item above.)
427 *BOOTH_LOCAL
429 The local site name, as defined in site.
430 *BOOTH_CONF_PATH
432 The path to the active configuration file.
433 *BOOTH_CONF_NAME
435 The configuration name, as used by the -c commandline argument.
436 *BOOTH_TICKET_EXPIRES
438 When the ticket expires (in seconds since 1.1.1970), or 0.
439 The handler is invoked with positional arguments specified after it.
441
443 /etc/booth/booth.conf
444 The default configuration file name. See also the -c argument.
445 /etc/booth/authkey
447 There is no default, but this is a typical location for the shared
448 secret (authentication key).
449 /var/run/booth/
451 Directory that holds PID/lock files. See also the status command.
452
454 In essence, every ticket corresponds to a separate Raft cluster.
455 A ticket is granted to the Raft Leader which then owns (or keeps) the
457 ticket.
458
460 The booth daemon for an arbitrator which typically doesn’t run the
461 cluster stack, may be started through systemd or with
462 /etc/init.d/booth-arbitrator, depending on which init system the
463 platform supports.
464 The SysV init script starts a booth arbitrator for every configuration
466 file found in /etc/booth.
467 Platforms running systemd can enable and start every configuration
469 separately using systemctl:
470 # systemctl enable booth@<configurationname>
472 # systemctl start booth@<configurationname>
473 systemctl requires the configuration name, even for the default name
475 booth.
476 MANUAL TICKETS
478 Manual tickets allow users to create and manage tickets which are
480 subsequently handled by booth without using the Raft algorithm.
481 Granting and revoking manual tickets is fully controlled by the
482 administrator. It is possible to define a number of manual and normal
483 tickets in one GEO cluster.
484 Automatic ticket management provided by Raft algorithm isn’t applied to
486 manually controlled tickets. In particular, there is no elections,
487 automatic failover procedures, and term expiration.
488 However, booth controls if a ticket is currently being granted to any
490 site and warns the user approprietly.
491 Tickets which were manually granted to a site, will remain there until
493 they are manually revoked. Even if a site becomes offline, the ticket
494 will not be moved to another site. This behavior allows administrators
495 to make sure that some services will remain in a particular site and
496 will not be moved to another site, possibly located in a different
497 geographical location.
498 Also, configuring only manual tickets in a GEO cluster, allows one to
500 have just two sites in a cluster, without a need of having an
501 arbitrator. This is possible because there is no automatic elections
502 and no voting performed for manual tickets.
503 Manual tickets are defined in a configuration files by adding a mode
505 ticket parameter and setting it to manual or MANUAL:
506 ticket="manual-ticket"
508 [...]
509 mode = manual
510 [...]
511 Manual tickets can be granted and revoked by using normal grant and
513 revoke commands, with the usual flags and parameters. The only
514 difference is that specyfiyng -F flag during grant command, forced a
515 site to become a leader of the specified ticket, even if the ticket is
516 granted to another site.
517
519 0
520 Success. For the status command: Daemon running.
521 1 (PCMK_OCF_UNKNOWN_ERROR)
523 General error code.
524 7 (PCMK_OCF_NOT_RUNNING)
526 No daemon process for that configuration active.
527
529 Booth is tested regularly. See the README-testing file for more
530 information.
531 Please report any bugs either at GitHub:
533 https://github.com/ClusterLabs/booth/issues
534 Or, if you prefer bugzilla, at openSUSE bugzilla (component "High
536 Availability"):
537 https://bugzilla.opensuse.org/enter_bug.cgi?product=openSUSE%20Factory
538
540 boothd was originally written (mostly) by Jiaju Zhang.
541 In 2013 and 2014 Philipp Marek took over maintainership.
543 Since April 2014 it has been mainly developed by Dejan Muhamedagic.
545 Many people contributed (see the AUTHORS file).
547
549 GitHub: https://github.com/ClusterLabs/booth
550 Documentation:
552 http://doc.opensuse.org/products/draft/SLE-HA/SLE-ha-guide_sd_draft/cha.ha.geo.html
553
555 Copyright © 2011 Jiaju Zhang <jjzhang@suse.de>
556 Copyright © 2013-2014 Philipp Marek <philipp.marek@linbit.com>
558 Copyright © 2014 Dejan Muhamedagic <dmuhamedagic@suse.com>
560 Free use of this software is granted under the terms of the GNU General
562 Public License (GPL) as of version 2 (see COPYING file) or later.
563 2020-07-27 BOOTHD(8)