1BOOTHD(8) BOOTHD(8)
2
6 boothd - The Booth Cluster Ticket Manager.
7
9 boothd daemon [-SD] [-c config] [-l lockfile]
10 booth list [-s site] [-c config]
12 booth grant [-s site] [-c config] [-FCw] ticket
14 booth revoke [-s site] [-c config] [-w] ticket
16 booth peers [-s site] [-c config]
18 booth status [-D] [-c config]
20
22 Booth manages tickets which authorizes one of the cluster sites located
23 in geographically dispersed distances to run certain resources. It is
24 designed to be extend Pacemaker to support geographically distributed
25 clustering.
26 It is based on the RAFT protocol, see eg.
28 https://ramcloud.stanford.edu/wiki/download/attachments/11370504/raft.pdf
29 for details.
30
32 # boothd daemon -D
33 # booth list
35 # booth grant ticket-nfs
37 # booth revoke ticket-nfs
39
41 -c configfile
42 Configuration to use.
43 Can be a full path to a configuration file, or a short name; in the
45 latter case, the directory /etc/booth and suffix .conf are added.
46 Per default booth is used, which results in the path
47 /etc/booth/booth.conf.
48 The configuration name also determines the name of the PID file -
50 for the defaults, /var/run/booth/booth.pid.
51 -s
53 Site address or name.
54 The special value 'other' can be used to specify the other
56 site. Obviously, in that case, the booth configuration must
57 have exactly two sites defined.
58 -F
60 immediate grant: Don’t wait for unreachable sites to relinquish the
61 ticket. See the Booth ticket management section below for more
62 details. For manual tickets this option allows one to grant a
63 ticket which is currently granted. See the Manual tickets section
64 below for more details.
65 This option may be DANGEROUS. It makes booth grant the ticket
67 even though it cannot ascertain that unreachable sites don't
68 hold the same ticket. It is up to the user to make sure that
69 unreachable sites don't have this ticket as granted.
70 -w
72 wait for the request outcome: The client waits for the final
73 outcome of grant or revoke request.
74 -C
76 wait for ticket commit to CIB: The client waits for the ticket
77 commit to CIB (only for grant requests). If one or more sites are
78 unreachable, this takes the ticket expire time (plus, if defined,
79 the acquire-after time).
80 -h, --help
82 Give a short usage output.
83 --version
85 Report version information.
86 -S
88 systemd mode: don’t fork. Disables daemonizing, the process will
89 remain in the foreground.
90 -D
92 Increases the debug output level.
93 -l lockfile
95 Use another lock file. By default, the lock file name is inferred
96 from the configuration file name. Normally not needed.
97
99 Whether the binary is called as boothd or booth doesn’t matter; the
100 first argument determines the mode of operation.
101 daemon
103 Tells boothd to serve a site. The locally configured interfaces are
104 searched for an IP address that is defined in the configuration.
105 booth then runs in either /arbitrator/ or /site/ mode.
106 client
108 Booth clients can list the ticket information (see also crm_ticket
109 -L), and revoke or grant tickets to a site.
110 The grant and, under certain circumstances, revoke operations may
112 take a while to return a definite operation’s outcome. The client
113 will wait up to the network timeout value (by default 5 seconds)
114 for the result. Unless the -w option was set, in which case the
115 client waits indefinitely.
116 In this mode the configuration file is searched for an IP address
118 that is locally reachable, ie. matches a configured subnet. This
119 allows one to run the client commands on another node in the same
120 cluster, as long as the config file and the service IP is locally
121 reachable.
122 For instance, if the booth service IP is 192.168.55.200, and the
124 local node has 192.168.55.15 configured on one of its network
125 interfaces, it knows which site it belongs to.
126 Use -s to direct client to connect to a different site.
128 status
130 boothd looks for the (locked) PID file and the UDP socket, prints
131 some output to stdout (for use in shell scripts) and returns an
132 OCF-compatible return code. With -D, a human-readable message is
133 printed to STDERR as well.
134 peers
136 List the other boothd servers we know about.
137 In addition to the type, name (IP address), and the last time the
139 server was heard from, network statistics are also printed. The
140 statistics are split into two rows, the first one consists of
141 counters for the sent packets and the second one for the received
142 packets. The first counter is the total number of packets and
143 descriptions of the other counters follows:
144 resends
146 Packets which had to be resent because the recipient didn’t
147 acknowledge a message. This usually means that either the message
148 or the acknowledgement got lost. The number of resends usually
149 reflect the network reliability.
150 error
152 Packets which either couldn’t be sent, got truncated, or were badly
153 formed. Should be zero.
154 invalid
156 These packets contain either invalid or non-existing ticket name or
157 refer to a non-existing ticket leader. Should be zero.
158 authfail
160 Packets which couldn’t be authenticated. Should be zero.
161
163 The configuration file must be identical on all sites and arbitrators.
164 A minimal file may look like this:
166 site="192.168.201.100"
168 site="192.168.202.100"
169 arbitrator="192.168.203.100"
170 ticket="ticket-db8"
171 Comments start with a hash-sign ('#'). Whitespace at the start and end
173 of the line, and around the '=', are ignored.
174 The following key/value pairs are defined:
176 port
178 The UDP/TCP port to use. Default is 9929.
179 transport
181 The transport protocol to use for Raft exchanges. Currently only
182 UDP is supported.
183 Clients use TCP to communicate with a daemon; Booth will always
185 bind and listen to both UDP and TCP ports.
186 authfile
188 File containing the authentication key. The key can be either
189 binary or text. If the latter, then both leading and trailing white
190 space, including new lines, is ignored. This key is a shared secret
191 and used to authenticate both clients and servers. The key must be
192 between 8 and 64 characters long and be readable only by the file
193 owner.
194 maxtimeskew
196 As protection against replay attacks, packets contain generation
197 timestamps. Such a timestamp is not allowed to be too old. Just how
198 old can be specified with this parameter. The value is in seconds
199 and the default is 600 (10 minutes). If clocks vary more than this
200 default between sites and nodes (which is definitely something you
201 should fix) then set this parameter to a higher value. The time
202 skew test is performed only in concert with authentication.
203 site
205 Defines a site Raft member with the given IP. Sites can acquire
206 tickets. The sites' IP should be managed by the cluster.
207 arbitrator
209 Defines an arbitrator Raft member with the given IP. Arbitrators
210 help reach consensus in elections and cannot hold tickets.
211 Booth needs at least three members for normal operation. Odd number of
213 members provides more redundancy.
214 site-user, site-group, arbitrator-user, arbitrator-group
216 These define the credentials boothd will be running with.
217 On a (Pacemaker) site the booth process will have to call
219 crm_ticket, so the default is to use hacluster:'haclient'; for an
220 arbitrator this user and group might not exists, so there we
221 default to nobody:'nobody'.
222 ticket
224 Registers a ticket. Multiple tickets can be handled by single Booth
225 instance.
226 Use the special ticket name defaults to modify the defaults. The
228 defaults stanza must precede all the other ticket specifications.
229 All times are in seconds.
231 expire
233 The lease time for a ticket. After that time the ticket can be
234 acquired by another site if the ticket holder is not reachable.
235 The default is 600.
237 acquire-after
239 Once a ticket is lost, wait this time in addition before acquiring
240 the ticket.
241 This is to allow for the site that lost the ticket to relinquish
243 the resources, by either stopping them or fencing a node.
244 A typical delay might be 60 seconds, but ultimately it depends on
246 the protected resources and the fencing configuration.
247 The default is 0.
249 renewal-freq
251 Set the ticket renewal frequency period.
252 If the network reliability is often reduced over prolonged periods,
254 it is advisable to try to renew more often.
255 Before every renewal, if defined, the command or commands specified
257 in before-acquire-handler is run. In that case the renewal-freq
258 parameter is effectively also the local cluster monitoring
259 interval.
260 timeout
262 After that time booth will re-send packets if there was an
263 insufficient number of replies. This should be long enough to allow
264 packets to reach other members.
265 The default is 5.
267 retries
269 Defines how many times to retry sending packets before giving up
270 waiting for acks from other members.
271 Default is 10. Values lower than 3 are illegal.
273 Ticket renewals should allow for this number of retries. Hence, the
275 total retry time must be shorter than the renewal time (either half
276 the expire time or renewal-freq):
277 timeout*(retries+1) < renewal
279 weights
281 A comma-separated list of integers that define the weight of
282 individual Raft members, in the same order as the site and
283 arbitrator lines.
284 Default is 0 for all; this means that the order in the
286 configuration file defines priority for conflicting requests.
287 before-acquire-handler
289 If set, this parameter specifies either a file containing a program
290 to be run or a directory where a number of programs can reside.
291 They are invoked before boothd tries to acquire or renew a ticket.
292 If any of them exits with a code other than 0, boothd relinquishes
293 the ticket.
294 Thus it is possible to ensure whether the services and its
296 dependencies protected by the ticket are in good shape at this
297 site. For instance, if a service in the dependency-chain has a
298 failcount of INFINITY on all available nodes, the service will be
299 unable to run. In that case, it is of no use to claim the ticket.
300 One or more arguments may follow the program or directory location.
302 Typically, there is at least the name of one of the resources which
303 depend on this ticket.
304 See below for details about booth specific environment variables.
306 The distributed service-runnable script is an example which may be
307 used to test whether a pacemaker resource can be started.
308 attr-prereq
310 Sites can have GEO attributes managed with the geostore(8) program.
311 Attributes are within ticket’s scope and may be tested by boothd
312 for additional control of ticket failover (automatic) or ticket
313 acquire (manual).
314 Attributes are typically used to convey extra information about
316 resources, for instance database replication status. The attributes
317 are commonly updated by resource agents.
318 Attribute values are referenced in expressions and may be tested
320 for equality with the eq binary operator or inequality with the ne
321 operator. The usage is as follows:
322 attr-prereq = <grant_type> <name> <op> <value>
324 <grant_type>: "auto" | "manual"
326 <name>: attribute name
327 <op>: "eq" | "ne"
328 <value>: attribute value
329 The two grant types are auto for ticket failover and manual for
331 grants using the booth client. Only in case the expression
332 evaluates to true can the ticket be granted.
333 It is not clear whether the manual grant type has any practical use
335 because, obviously, this operation is anyway controlled by a human.
336 Note that there can be no guarantee on whether an attribute value
338 is up to date, i.e. if it actually reflects the current state.
339 mode
341 Specifies if the ticket is manual or automatic.
342 By default all tickets are automatic (that is, they are fully
344 controlled by Raft algorithm). Assign the strings "manual" or
345 "MANUAL" to define the ticket as manually controlled.
346 debug
348 Specifies the debug output level. Alternative to command line
349 argument. Effective only for daemon mode of operation.
350 One example of a booth configuration file:
352 transport = udp
354 port = 9930
355 # D-85774
357 site="192.168.201.100"
358 # D-90409
359 site="::ffff:192.168.202.100"
360 # A-1120
361 arbitrator="192.168.203.100"
362 ticket="ticket-db8"
364 expire = 600
365 acquire-after = 60
366 timeout = 10
367 retries = 5
368 renewal-freq = 60
369 before-acquire-handler = /usr/share/booth/service-runnable db8
370 attr-prereq = auto repl_state eq ACTIVE
371
373 The booth cluster guarantees that every ticket is owned by only one
374 site at the time.
375 Tickets must be initially granted with the booth client grant command.
377 Once it gets granted, the ticket is managed by the booth cluster.
378 Hence, only granted tickets are managed by booth.
379 If the ticket gets lost, i.e. that the other members of the booth
381 cluster do not hear from the ticket owner in a sufficiently long time,
382 one of the remaining sites will acquire the ticket. This is what is
383 called ticket failover.
384 If the remaining members cannot form a majority, then the ticket cannot
386 fail over.
387 A ticket may be revoked at any time with the booth client revoke
389 command. For revoke to succeed, the site holding the ticket must be
390 reachable.
391 Once the ticket is administratively revoked, it is not managed by the
393 booth cluster anymore. For the booth cluster to start managing the
394 ticket again, it must be again granted to a site.
395 The grant operation, in case not all sites are reachable, may get
397 delayed for the ticket expire time (and, if defined, the acquire-after
398 time). The reason is that the other booth members may not know if the
399 ticket is currently granted at the unreachable site.
400 This delay may be disabled with the -F option. In that case, it is up
402 to the administrator to make sure that the unreachable site is not
403 holding the ticket.
404 When the ticket is managed by booth, it is dangerous to modify it
406 manually using either crm_ticket command or crm site ticket. Neither of
407 these tools is aware of booth and, consequently, booth itself may not
408 be aware of any ticket status changes. A notable exception is setting
409 the ticket to standby which is typically done before a planned
410 failover.
411
413 Tickets are not meant to be moved around quickly, the default expire
414 time is 600 seconds (10 minutes).
415 booth works with both IPv4 and IPv6 addresses.
417 booth renews a ticket before it expires, to account for possible
419 transmission delays. The renewal time, unless explicitly set, is set to
420 half the expire time.
421
423 Currently, there’s only one external handler defined (see the
424 before-acquire-handler configuration item above).
425 The following environment variables are exported to the handler:
427 *BOOTH_TICKET
429 The ticket name, as given in the configuration file. (See ticket
430 item above.)
431 *BOOTH_LOCAL
433 The local site name, as defined in site.
434 *BOOTH_CONF_PATH
436 The path to the active configuration file.
437 *BOOTH_CONF_NAME
439 The configuration name, as used by the -c commandline argument.
440 *BOOTH_TICKET_EXPIRES
442 When the ticket expires (in seconds since 1.1.1970), or 0.
443 The handler is invoked with positional arguments specified after it.
445
447 /etc/booth/booth.conf
448 The default configuration file name. See also the -c argument.
449 /etc/booth/authkey
451 There is no default, but this is a typical location for the shared
452 secret (authentication key).
453 /var/run/booth/
455 Directory that holds PID/lock files. See also the status command.
456
458 In essence, every ticket corresponds to a separate Raft cluster.
459 A ticket is granted to the Raft Leader which then owns (or keeps) the
461 ticket.
462
464 The booth daemon for an arbitrator which typically doesn’t run the
465 cluster stack, may be started through systemd or with
466 /etc/init.d/booth-arbitrator, depending on which init system the
467 platform supports.
468 The SysV init script starts a booth arbitrator for every configuration
470 file found in /etc/booth.
471 Platforms running systemd can enable and start every configuration
473 separately using systemctl:
474 # systemctl enable booth@<configurationname>
476 # systemctl start booth@<configurationname>
477 systemctl requires the configuration name, even for the default name
479 booth.
480 MANUAL TICKETS
482 Manual tickets allow users to create and manage tickets which are
484 subsequently handled by booth without using the Raft algorithm.
485 Granting and revoking manual tickets is fully controlled by the
486 administrator. It is possible to define a number of manual and normal
487 tickets in one GEO cluster.
488 Automatic ticket management provided by Raft algorithm isn’t applied to
490 manually controlled tickets. In particular, there is no elections,
491 automatic failover procedures, and term expiration.
492 However, booth controls if a ticket is currently being granted to any
494 site and warns the user approprietly.
495 Tickets which were manually granted to a site, will remain there until
497 they are manually revoked. Even if a site becomes offline, the ticket
498 will not be moved to another site. This behavior allows administrators
499 to make sure that some services will remain in a particular site and
500 will not be moved to another site, possibly located in a different
501 geographical location.
502 Also, configuring only manual tickets in a GEO cluster, allows one to
504 have just two sites in a cluster, without a need of having an
505 arbitrator. This is possible because there is no automatic elections
506 and no voting performed for manual tickets.
507 Manual tickets are defined in a configuration files by adding a mode
509 ticket parameter and setting it to manual or MANUAL:
510 ticket="manual-ticket"
512 [...]
513 mode = manual
514 [...]
515 Manual tickets can be granted and revoked by using normal grant and
517 revoke commands, with the usual flags and parameters. The only
518 difference is that specyfiyng -F flag during grant command, forced a
519 site to become a leader of the specified ticket, even if the ticket is
520 granted to another site.
521
523 0
524 Success. For the status command: Daemon running.
525 1 (PCMK_OCF_UNKNOWN_ERROR)
527 General error code.
528 7 (PCMK_OCF_NOT_RUNNING)
530 No daemon process for that configuration active.
531
533 Booth is tested regularly. See the README-testing file for more
534 information.
535 Please report any bugs either at GitHub:
537 https://github.com/ClusterLabs/booth/issues
538 Or, if you prefer bugzilla, at openSUSE bugzilla (component "High
540 Availability"):
541 https://bugzilla.opensuse.org/enter_bug.cgi?product=openSUSE%20Factory
542
544 boothd was originally written (mostly) by Jiaju Zhang.
545 In 2013 and 2014 Philipp Marek took over maintainership.
547 Since April 2014 it has been mainly developed by Dejan Muhamedagic.
549 Many people contributed (see the AUTHORS file).
551
553 GitHub: https://github.com/ClusterLabs/booth
554 Documentation:
556 http://doc.opensuse.org/products/draft/SLE-HA/SLE-ha-guide_sd_draft/cha.ha.geo.html
557
559 Copyright © 2011 Jiaju Zhang jjzhang@suse.de
560 Copyright © 2013-2014 Philipp Marek philipp.marek@linbit.com
562 Copyright © 2014 Dejan Muhamedagic dmuhamedagic@suse.com
564 Free use of this software is granted under the terms of the GNU General
566 Public License (GPL) as of version 2 (see COPYING file) or later.
567 2022-05-19 BOOTHD(8)