1SBD(8) STONITH Block Device SBD(8)
2
6 sbd - STONITH Block Device daemon
7
9 sbd <-d /dev/...> [options] "command"
10
12 SBD provides a node fencing mechanism (Shoot the other node in the
13 head, STONITH) for Pacemaker-based clusters through the exchange of
14 messages via shared block storage such as for example a SAN, iSCSI,
15 FCoE. This isolates the fencing mechanism from changes in firmware
16 version or dependencies on specific firmware controllers, and it can be
17 used as a STONITH mechanism in all configurations that have reliable
18 shared storage.
19 SBD can also be used without any shared storage. In this mode, the
21 watchdog device will be used to reset the node if it loses quorum, if
22 any monitored daemon is lost and not recovered or if Pacemaker decides
23 that the node requires fencing.
24 The sbd binary implements both the daemon that watches the message
26 slots as well as the management tool for interacting with the block
27 storage device(s). This mode of operation is specified via the
28 "command" parameter; some of these modes take additional parameters.
29 To use SBD with shared storage, you must first "create" the messaging
31 layout on one to three block devices. Second, configure
32 /etc/sysconfig/sbd to list those devices (and possibly adjust other
33 options), and restart the cluster stack on each node to ensure that
34 "sbd" is started. Third, configure the "external/sbd" fencing resource
35 in the Pacemaker CIB.
36 Each of these steps is documented in more detail below the description
38 of the command options.
39 "sbd" can only be used as root.
41 GENERAL OPTIONS
43 -d /dev/...
44 Specify the block device(s) to be used. If you have more than one,
45 specify this option up to three times. This parameter is mandatory
46 for all modes, since SBD always needs a block device to interact
47 with.
48 This man page uses /dev/sda1, /dev/sdb1, and /dev/sdc1 as example
50 device names for brevity. However, in your production environment,
51 you should instead always refer to them by using the long, stable
52 device name (e.g.,
53 /dev/disk/by-id/dm-uuid-part1-mpath-3600508b400105b5a0001500000250000).
54 -v Enable some verbose debug logging.
56 -h Display a concise summary of "sbd" options.
58 -n node
60 Set local node name; defaults to "uname -n". This should not need
61 to be set.
62 -R Do not enable realtime priority. By default, "sbd" runs at realtime
64 priority, locks itself into memory, and also acquires highest IO
65 priority to protect itself against interference from other
66 processes on the system. This is a debugging-only option.
67 -I N
69 Async IO timeout (defaults to 3 seconds, optional). You should not
70 need to adjust this unless your IO setup is really very slow.
71 (In daemon mode, the watchdog is refreshed when the majority of
73 devices could be read within this time.)
74 create
76 Example usage:
77 sbd -d /dev/sdc2 -d /dev/sdd3 create
79 If you specify the create command, sbd will write a metadata header to
81 the device(s) specified and also initialize the messaging slots for up
82 to 255 nodes.
83 Warning: This command will not prompt for confirmation. Roughly the
85 first megabyte of the specified block device(s) will be overwritten
86 immediately and without backup.
87 This command accepts a few options to adjust the default timings that
89 are written to the metadata (to ensure they are identical across all
90 nodes accessing the device).
91 -1 N
93 Set watchdog timeout to N seconds. This depends mostly on your
94 storage latency; the majority of devices must be successfully read
95 within this time, or else the node will self-fence.
96 If your sbd device(s) reside on a multipath setup or iSCSI, this
98 should be the time required to detect a path failure. You may be
99 able to reduce this if your device outages are independent, or if
100 you are using the Pacemaker integration.
101 -2 N
103 Set slot allocation timeout to N seconds. You should not need to
104 tune this.
105 -3 N
107 Set daemon loop timeout to N seconds. You should not need to tune
108 this.
109 -4 N
111 Set msgwait timeout to N seconds. This should be twice the watchdog
112 timeout. This is the time after which a message written to a node's
113 slot will be considered delivered. (Or long enough for the node to
114 detect that it needed to self-fence.)
115 This also affects the stonith-timeout in Pacemaker's CIB; see
117 below.
118 list
120 Example usage:
121 # sbd -d /dev/sda1 list
123 0 hex-0 clear
124 1 hex-7 clear
125 2 hex-9 clear
126 List all allocated slots on device, and messages. You should see all
128 cluster nodes that have ever been started against this device. Nodes
129 that are currently running should have a clear state; nodes that have
130 been fenced, but not yet restarted, will show the appropriate fencing
131 message.
132 dump
134 Example usage:
135 # sbd -d /dev/sda1 dump
137 ==Dumping header on disk /dev/sda1
138 Header version : 2
139 Number of slots : 255
140 Sector size : 512
141 Timeout (watchdog) : 15
142 Timeout (allocate) : 2
143 Timeout (loop) : 1
144 Timeout (msgwait) : 30
145 ==Header on disk /dev/sda1 is dumped
146 Dump meta-data header from device.
148 watch
150 Example usage:
151 sbd -d /dev/sdc2 -d /dev/sdd3 -P watch
153 This command will make "sbd" start in daemon mode. It will constantly
155 monitor the message slot of the local node for incoming messages,
156 reachability, and optionally take Pacemaker's state into account.
157 "sbd" must be started on boot before the cluster stack! See below for
159 enabling this according to your boot environment.
160 The options for this mode are rarely specified directly on the
162 commandline directly, but most frequently set via /etc/sysconfig/sbd.
163 It also constantly monitors connectivity to the storage device, and
165 self-fences in case the partition becomes unreachable, guaranteeing
166 that it does not disconnect from fencing messages.
167 A node slot is automatically allocated on the device(s) the first time
169 the daemon starts watching the device; hence, manual allocation is not
170 usually required.
171 If a watchdog is used together with the "sbd" as is strongly
173 recommended, the watchdog is activated at initial start of the sbd
174 daemon. The watchdog is refreshed every time the majority of SBD
175 devices has been successfully read. Using a watchdog provides
176 additional protection against "sbd" crashing.
177 If the Pacemaker integration is activated, "sbd" will not self-fence if
179 device majority is lost, if:
180 1. The partition the node is in is still quorate according to the CIB;
182 2. it is still quorate according to Corosync's node count;
184 3. the node itself is considered online and healthy by Pacemaker.
186 This allows "sbd" to survive temporary outages of the majority of
188 devices. However, while the cluster is in such a degraded state, it can
189 neither successfully fence nor be shutdown cleanly (as taking the
190 cluster below the quorum threshold will immediately cause all remaining
191 nodes to self-fence). In short, it will not tolerate any further
192 faults. Please repair the system before continuing.
193 There is one "sbd" process that acts as a master to which all watchers
195 report; one per device to monitor the node's slot; and, optionally, one
196 that handles the Pacemaker integration.
197 -W Enable or disable use of the system watchdog to protect against the
199 sbd processes failing and the node being left in an undefined
200 state. Specify this once to enable, twice to disable.
201 Defaults to enabled.
203 -w /dev/watchdog
205 This can be used to override the default watchdog device used and
206 should not usually be necessary.
207 -p /var/run/sbd.pid
209 This option can be used to specify a pidfile for the main sbd
210 process.
211 -F N
213 Number of failures before a failing servant process will not be
214 restarted immediately until the dampening delay has expired. If set
215 to zero, servants will be restarted immediately and indefinitely.
216 If set to one, a failed servant will be restarted once every -t
217 seconds. If set to a different value, the servant will be restarted
218 that many times within the dampening period and then delay.
219 Defaults to 1.
221 -t N
223 Dampening delay before faulty servants are restarted. Combined with
224 "-F 1", the most logical way to tune the restart frequency of
225 servant processes. Default is 5 seconds.
226 If set to zero, processes will be restarted indefinitely and
228 immediately.
229 -P Enable Pacemaker integration which checks Pacemaker quorum and node
231 health. Specify this once to enable, twice to disable.
232 Defaults to enabled.
234 -S N
236 Set the start mode. (Defaults to 0.)
237 If this is set to zero, sbd will always start up unconditionally,
239 regardless of whether the node was previously fenced or not.
240 If set to one, sbd will only start if the node was previously
242 shutdown cleanly (as indicated by an exit request message in the
243 slot), or if the slot is empty. A reset, crashdump, or power-off
244 request in any slot will halt the start up.
245 This is useful to prevent nodes from rejoining if they were faulty.
247 The node must be manually "unfenced" by sending an empty message to
248 it:
249 sbd -d /dev/sda1 message node1 clear
251 -s N
253 Set the start-up wait time for devices. (Defaults to 120.)
254 Dynamic block devices such as iSCSI might not be fully initialized
256 and present yet. This allows to set a timeout for waiting for
257 devices to appear on start-up. If set to 0, start-up will be
258 aborted immediately if no devices are available.
259 -Z Enable trace mode. Warning: this is unsafe for production, use at
261 your own risk! Specifying this once will turn all reboots or power-
262 offs, be they caused by self-fence decisions or messages, into a
263 crashdump. Specifying this twice will just log them but not
264 continue running.
265 -T By default, the daemon will set the watchdog timeout as specified
267 in the device metadata. However, this does not work for every
268 watchdog device. In this case, you must manually ensure that the
269 watchdog timeout used by the system correctly matches the SBD
270 settings, and then specify this option to allow "sbd" to continue
271 with start-up.
272 -5 N
274 Warn if the time interval for tickling the watchdog exceeds this
275 many seconds. Since the node is unable to log the watchdog expiry
276 (it reboots immediately without a chance to write its logs to
277 disk), this is very useful for getting an indication that the
278 watchdog timeout is too short for the IO load of the system.
279 Default is 3 seconds, set to zero to disable.
281 -C N
283 Watchdog timeout to set before crashdumping. If SBD is set to
284 crashdump instead of reboot - either via the trace mode settings or
285 the external/sbd fencing agent's parameter -, SBD will adjust the
286 watchdog timeout to this setting before triggering the dump.
287 Otherwise, the watchdog might trigger and prevent a successful
288 crashdump from ever being written.
289 Defaults to 240 seconds. Set to zero to disable.
291 -r N
293 Actions to be executed when the watchers don't timely report to the
294 sbd master process or one of the watchers detects that the master
295 process has died.
296 Set timeout-action to comma-separated combination of noflush|flush
298 plus reboot|crashdump|off. If just one of both is given the other
299 stays at the default.
300 This doesn't affect actions like off, crashdump, reboot explicitly
302 triggered via message slots. And it does as well not configure the
303 action a watchdog would trigger should it run off (there is no
304 generic interface).
305 Defaults to flush,reboot.
307 allocate
309 Example usage:
310 sbd -d /dev/sda1 allocate node1
312 Explicitly allocates a slot for the specified node name. This should
314 rarely be necessary, as every node will automatically allocate itself a
315 slot the first time it starts up on watch mode.
316 message
318 Example usage:
319 sbd -d /dev/sda1 message node1 test
321 Writes the specified message to node's slot. This is rarely done
323 directly, but rather abstracted via the "external/sbd" fencing agent
324 configured as a cluster resource.
325 Supported message types are:
327 test
329 This only generates a log message on the receiving node and can be
330 used to check if SBD is seeing the device. Note that this could
331 overwrite a fencing request send by the cluster, so should not be
332 used during production.
333 reset
335 Reset the target upon receipt of this message.
336 off Power-off the target.
338 crashdump
340 Cause the target node to crashdump.
341 exit
343 This will make the "sbd" daemon exit cleanly on the target. You
344 should not send this message manually; this is handled properly
345 during shutdown of the cluster stack. Manually stopping the daemon
346 means the node is unprotected!
347 clear
349 This message indicates that no real message has been sent to the
350 node. You should not set this manually; "sbd" will clear the
351 message slot automatically during start-up, and setting this
352 manually could overwrite a fencing message by the cluster.
353 query-watchdog
355 Example usage:
356 sbd query-watchdog
358 Check for available watchdog devices and print some info.
360 Warning: This command will arm the watchdog during query, and if your
362 watchdog refuses disarming (for example, if its kernel module has the
363 'nowayout' parameter set) this will reset your system.
364 test-watchdog
366 Example usage:
367 sbd test-watchdog [-w /dev/watchdog3]
369 Test specified watchdog device (/dev/watchdog by default).
371 Warning: This command will arm the watchdog and have your system reset
373 in case your watchdog is working properly! If issued from an
374 interactive session, it will prompt for confirmation.
375
377 Configure a watchdog
378 It is highly recommended that you configure your Linux system to load a
379 watchdog driver with hardware assistance (as is available on most
380 modern systems), such as hpwdt, iTCO_wdt, or others. As a fall-back,
381 you can use the softdog module.
382 No other software must access the watchdog timer; it can only be
384 accessed by one process at any given time. Some hardware vendors ship
385 systems management software that use the watchdog for system resets
386 (f.e. HP ASR daemon). Such software has to be disabled if the watchdog
387 is to be used by SBD.
388 Choosing and initializing the block device(s)
390 First, you have to decide if you want to use one, two, or three
391 devices.
392 If you are using multiple ones, they should reside on independent
394 storage setups. Putting all three of them on the same logical unit for
395 example would not provide any additional redundancy.
396 The SBD device can be connected via Fibre Channel, Fibre Channel over
398 Ethernet, or even iSCSI. Thus, an iSCSI target can become a sort-of
399 network-based quorum server; the advantage is that it does not require
400 a smart host at your third location, just block storage.
401 The SBD partitions themselves must not be mirrored (via MD, DRBD, or
403 the storage layer itself), since this could result in a split-mirror
404 scenario. Nor can they reside on cLVM2 volume groups, since they must
405 be accessed by the cluster stack before it has started the cLVM2
406 daemons; hence, these should be either raw partitions or logical units
407 on (multipath) storage.
408 The block device(s) must be accessible from all nodes. (While it is not
410 necessary that they share the same path name on all nodes, this is
411 considered a very good idea.)
412 SBD will only use about one megabyte per device, so you can easily
414 create a small partition, or very small logical units. (The size of
415 the SBD device depends on the block size of the underlying device.
416 Thus, 1MB is fine on plain SCSI devices and SAN storage with 512 byte
417 blocks. On the IBM s390x architecture in particular, disks default to
418 4k blocks, and thus require roughly 4MB.)
419 The number of devices will affect the operation of SBD as follows:
421 One device
423 In its most simple implementation, you use one device only. This is
424 appropriate for clusters where all your data is on the same shared
425 storage (with internal redundancy) anyway; the SBD device does not
426 introduce an additional single point of failure then.
427 If the SBD device is not accessible, the daemon will fail to start
429 and inhibit openais startup.
430 Two devices
432 This configuration is a trade-off, primarily aimed at environments
433 where host-based mirroring is used, but no third storage device is
434 available.
435 SBD will not commit suicide if it loses access to one mirror leg;
437 this allows the cluster to continue to function even in the face of
438 one outage.
439 However, SBD will not fence the other side while only one mirror
441 leg is available, since it does not have enough knowledge to detect
442 an asymmetric split of the storage. So it will not be able to
443 automatically tolerate a second failure while one of the storage
444 arrays is down. (Though you can use the appropriate crm command to
445 acknowledge the fence manually.)
446 It will not start unless both devices are accessible on boot.
448 Three devices
450 In this most reliable and recommended configuration, SBD will only
451 self-fence if more than one device is lost; hence, this
452 configuration is resilient against temporary single device outages
453 (be it due to failures or maintenance). Fencing messages can still
454 be successfully relayed if at least two devices remain accessible.
455 This configuration is appropriate for more complex scenarios where
457 storage is not confined to a single array. For example, host-based
458 mirroring solutions could have one SBD per mirror leg (not mirrored
459 itself), and an additional tie-breaker on iSCSI.
460 It will only start if at least two devices are accessible on boot.
462 After you have chosen the devices and created the appropriate
464 partitions and perhaps multipath alias names to ease management, use
465 the "sbd create" command described above to initialize the SBD metadata
466 on them.
467 Sharing the block device(s) between multiple clusters
469 It is possible to share the block devices between multiple clusters,
471 provided the total number of nodes accessing them does not exceed 255
472 nodes, and they all must share the same SBD timeouts (since these are
473 part of the metadata).
474 If you are using multiple devices this can reduce the setup overhead
476 required. However, you should not share devices between clusters in
477 different security domains.
478 Configure SBD to start on boot
480 On systems using "sysvinit", the "openais" or "corosync" system start-
481 up scripts must handle starting or stopping "sbd" as required before
482 starting the rest of the cluster stack.
483 For "systemd", sbd simply has to be enabled using
485 systemctl enable sbd.service
487 The daemon is brought online on each node before corosync and Pacemaker
489 are started, and terminated only after all other cluster components
490 have been shut down - ensuring that cluster resources are never
491 activated without SBD supervision.
492 Configuration via sysconfig
494 The system instance of "sbd" is configured via /etc/sysconfig/sbd. In
495 this file, you must specify the device(s) used, as well as any options
496 to pass to the daemon:
497 SBD_DEVICE="/dev/sda1;/dev/sdb1;/dev/sdc1"
499 SBD_PACEMAKER="true"
500 "sbd" will fail to start if no "SBD_DEVICE" is specified. See the
502 installed template for more options that can be configured here. In
503 general configuration done via parameters takes precedence over the
504 configuration from the configuration file.
505 Testing the sbd installation
507 After a restart of the cluster stack on this node, you can now try
508 sending a test message to it as root, from this or any other node:
509 sbd -d /dev/sda1 message node1 test
511 The node will acknowledge the receipt of the message in the system
513 logs:
514 Aug 29 14:10:00 node1 sbd: [13412]: info: Received command test from node2
516 This confirms that SBD is indeed up and running on the node, and that
518 it is ready to receive messages.
519 Make sure that /etc/sysconfig/sbd is identical on all cluster nodes,
521 and that all cluster nodes are running the daemon.
522
524 Fencing resource
525 Pacemaker can only interact with SBD to issue a node fence if there is
526 a configure fencing resource. This should be a primitive, not a clone,
527 as follows:
528 primitive fencing-sbd stonith:external/sbd \
530 params pcmk_delay_max=30
531 This will automatically use the same devices as configured in
533 /etc/sysconfig/sbd.
534 While you should not configure this as a clone (as Pacemaker will
536 register the fencing device on each node automatically), the
537 pcmk_delay_max setting enables random fencing delay which ensures, in a
538 scenario where a split-brain scenario did occur in a two node cluster,
539 that one of the nodes has a better chance to survive to avoid double
540 fencing.
541 SBD also supports turning the reset request into a crash request, which
543 may be helpful for debugging if you have kernel crashdumping
544 configured; then, every fence request will cause the node to dump core.
545 You can enable this via the "crashdump="true"" parameter on the fencing
546 resource. This is not recommended for production use, but only for
547 debugging phases.
548 General cluster properties
550 You must also enable STONITH in general, and set the STONITH timeout to
551 be at least twice the msgwait timeout you have configured, to allow
552 enough time for the fencing message to be delivered. If your msgwait
553 timeout is 60 seconds, this is a possible configuration:
554 property stonith-enabled="true"
556 property stonith-timeout="120s"
557 Caution: if stonith-timeout is too low for msgwait and the system
559 overhead, sbd will never be able to successfully complete a fence
560 request. This will create a fencing loop.
561 Note that the sbd fencing agent will try to detect this and
563 automatically extend the stonith-timeout setting to a reasonable value,
564 on the assumption that sbd modifying your configuration is preferable
565 to not fencing.
566
568 Recovering from temporary SBD device outage
569 If you have multiple devices, failure of a single device is not
570 immediately fatal. "sbd" will retry to restart the monitor for the
571 device every 5 seconds by default. However, you can tune this via the
572 options to the watch command.
573 In case you wish the immediately force a restart of all currently
575 disabled monitor processes, you can send a SIGUSR1 to the SBD
576 inquisitor process.
577
579 Copyright (C) 2008-2013 Lars Marowsky-Bree
580 This program is free software; you can redistribute it and/or modify it
582 under the terms of the GNU General Public License as published by the
583 Free Software Foundation; either version 2 of the License, or (at your
584 option) any later version.
585 This software is distributed in the hope that it will be useful, but
587 WITHOUT ANY WARRANTY; without even the implied warranty of
588 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
589 General Public License for more details.
590 For details see the GNU General Public License at
592 http://www.gnu.org/licenses/gpl-2.0.html (version 2) and/or
593 http://www.gnu.org/licenses/gpl.html (the newest as per "any later").
594SBD 2019-10-07 SBD(8)