1CTDB(7) CTDB - clustered TDB database CTDB(7)
2
6 ctdb - Clustered TDB
7
9 CTDB is a clustered database component in clustered Samba that provides
10 a high-availability load-sharing CIFS server cluster.
11 The main functions of CTDB are:
13 • Provide a clustered version of the TDB database with automatic
15 rebuild/recovery of the databases upon node failures.
16 • Monitor nodes in the cluster and services running on each node.
18 • Manage a pool of public IP addresses that are used to provide
20 services to clients. Alternatively, CTDB can be used with LVS.
21 Combined with a cluster filesystem CTDB provides a full
23 high-availablity (HA) environment for services such as clustered Samba,
24 NFS and other services.
25 In addition to the CTDB manual pages there is much more information
27 available at https://wiki.samba.org/index.php/CTDB_and_Clustered_Samba.
28
30 A CTDB cluster is a collection of nodes with 2 or more network
31 interfaces. All nodes provide network (usually file/NAS) services to
32 clients. Data served by file services is stored on shared storage
33 (usually a cluster filesystem) that is accessible by all nodes.
34 CTDB provides an "all active" cluster, where services are load balanced
36 across all nodes.
37
39 CTDB uses a cluster leader and follower model of cluster management.
40 All nodes in a cluster elect one node to be the leader. The leader node
41 coordinates privileged operations such as database recovery and IP
42 address failover.
43 CTDB previously referred to the leader as the recovery master or
45 recmaster. References to these terms may still be found in
46 documentation and code.
47
49 CTDB uses a cluster lock to assert its privileged role in the cluster.
50 This node takes the cluster lock when it becomes leader and holds the
51 lock until it is no longer leader. The cluster lock helps CTDB to avoid
52 a split brain, where a cluster becomes partitioned and each partition
53 attempts to operate independently. Issues that can result from a split
54 brain include file data corruption, because file locking metadata may
55 not be tracked correctly.
56 CTDB previously referred to the cluster lock as the recovery lock. The
58 abbreviation reclock is still used - just "clock" would be confusing.
59 CTDB is unable configure a default cluster lock, because this would
61 depend on factors such as cluster filesystem mountpoints. However,
62 running CTDB without a cluster lock is not recommended as there will be
63 no split brain protection.
64 When a cluster lock is configured it is used as the election mechanism.
66 Nodes race to take the cluster lock and the winner is the cluster
67 leader. This avoids problems when a node wins an election but is unable
68 to take the lock - this can occur if a cluster becomes partitioned (for
69 example, due to a communication failure) and a different leader is
70 elected by the nodes in each partition, or if the cluster filesystem
71 has a high failover latency.
72 By default, the cluster lock is implemented using a file (specified by
74 cluster lock in the [cluster] section of ctdb.conf(5)) residing in
75 shared storage (usually) on a cluster filesystem. To support a cluster
76 lock the cluster filesystem must support lock coherence. See
77 ping_pong(1) for more details.
78 The cluster lock can also be implemented using an arbitrary cluster
80 mutex helper (or call-out). This is indicated by using an exclamation
81 point ('!') as the first character of the cluster lock parameter. For
82 example, a value of !/usr/bin/myhelper cluster would run the given
83 helper with the specified arguments. The helper will continue to run as
84 long as it holds its mutex. See ctdb/doc/cluster_mutex_helper.txt in
85 the source tree, and related code, for clues about writing helpers.
86 When a file is specified for the cluster lock parameter (i.e. no
88 leading '!') the file lock is implemented by a default helper
89 (/usr/libexec/ctdb/ctdb_mutex_fcntl_helper). This helper has arguments
90 as follows:
91 ctdb_mutex_fcntl_helper FILE [RECHECK-INTERVAL]
94 ctdb_mutex_fcntl_helper will take a lock on FILE and then check every
98 RECHECK-INTERVAL seconds to ensure that FILE still exists and that its
99 inode number is unchanged from when the lock was taken. The default
100 value for RECHECK-INTERVAL is 5.
101 CTDB does sanity checks to ensure that the cluster lock is held as
103 expected.
104
106 Each node in a CTDB cluster has multiple IP addresses assigned to it:
107 • A single private IP address that is used for communication between
109 nodes.
110 • One or more public IP addresses that are used to provide NAS or
112 other services.
113 Private address
116 Each node is configured with a unique, permanently assigned private
117 address. This address is configured by the operating system. This
118 address uniquely identifies a physical node in the cluster and is the
119 address that CTDB daemons will use to communicate with the CTDB daemons
120 on other nodes.
121 Private addresses are listed in the file /etc/ctdb/nodes). This file
123 contains the list of private addresses for all nodes in the cluster,
124 one per line. This file must be the same on all nodes in the cluster.
125 Some users like to put this configuration file in their cluster
127 filesystem. A symbolic link should be used in this case.
128 Private addresses should not be used by clients to connect to services
130 provided by the cluster.
131 It is strongly recommended that the private addresses are configured on
133 a private network that is separate from client networks. This is
134 because the CTDB protocol is both unauthenticated and unencrypted. If
135 clients share the private network then steps need to be taken to stop
136 injection of packets to relevant ports on the private addresses. It is
137 also likely that CTDB protocol traffic between nodes could leak
138 sensitive information if it can be intercepted.
139 Example /etc/ctdb/nodes for a four node cluster:
141 192.168.1.1
143 192.168.1.2
144 192.168.1.3
145 192.168.1.4
146 Public addresses
149 Public addresses are used to provide services to clients. Public
150 addresses are not configured at the operating system level and are not
151 permanently associated with a particular node. Instead, they are
152 managed by CTDB and are assigned to interfaces on physical nodes at
153 runtime.
154 The CTDB cluster will assign/reassign these public addresses across the
156 available healthy nodes in the cluster. When one node fails, its public
157 addresses will be taken over by one or more other nodes in the cluster.
158 This ensures that services provided by all public addresses are always
159 available to clients, as long as there are nodes available capable of
160 hosting this address.
161 The public address configuration is stored in
163 /etc/ctdb/public_addresses on each node. This file contains a list of
164 the public addresses that the node is capable of hosting, one per line.
165 Each entry also contains the netmask and the interface to which the
166 address should be assigned. If this file is missing then no public
167 addresses are configured.
168 Some users who have the same public addresses on all nodes like to put
170 this configuration file in their cluster filesystem. A symbolic link
171 should be used in this case.
172 Example /etc/ctdb/public_addresses for a node that can host 4 public
174 addresses, on 2 different interfaces:
175 10.1.1.1/24 eth1
177 10.1.1.2/24 eth1
178 10.1.2.1/24 eth2
179 10.1.2.2/24 eth2
180 In many cases the public addresses file will be the same on all nodes.
183 However, it is possible to use different public address configurations
184 on different nodes.
185 Example: 4 nodes partitioned into two subgroups:
187 Node 0:/etc/ctdb/public_addresses
189 10.1.1.1/24 eth1
190 10.1.1.2/24 eth1
191 Node 1:/etc/ctdb/public_addresses
193 10.1.1.1/24 eth1
194 10.1.1.2/24 eth1
195 Node 2:/etc/ctdb/public_addresses
197 10.1.2.1/24 eth2
198 10.1.2.2/24 eth2
199 Node 3:/etc/ctdb/public_addresses
201 10.1.2.1/24 eth2
202 10.1.2.2/24 eth2
203 In this example nodes 0 and 1 host two public addresses on the 10.1.1.x
206 network while nodes 2 and 3 host two public addresses for the 10.1.2.x
207 network.
208 Public address 10.1.1.1 can be hosted by either of nodes 0 or 1 and
210 will be available to clients as long as at least one of these two nodes
211 are available.
212 If both nodes 0 and 1 become unavailable then public address 10.1.1.1
214 also becomes unavailable. 10.1.1.1 can not be failed over to nodes 2 or
215 3 since these nodes do not have this public address configured.
216 The ctdb ip command can be used to view the current assignment of
218 public addresses to physical nodes.
219
221 The current status of each node in the cluster can be viewed by the
222 ctdb status command.
223 A node can be in one of the following states:
225 OK
227 This node is healthy and fully functional. It hosts public
228 addresses to provide services.
229 DISCONNECTED
231 This node is not reachable by other nodes via the private network.
232 It is not currently participating in the cluster. It does not host
233 public addresses to provide services. It might be shut down.
234 DISABLED
236 This node has been administratively disabled. This node is
237 partially functional and participates in the cluster. However, it
238 does not host public addresses to provide services.
239 UNHEALTHY
241 A service provided by this node has failed a health check and
242 should be investigated. This node is partially functional and
243 participates in the cluster. However, it does not host public
244 addresses to provide services. Unhealthy nodes should be
245 investigated and may require an administrative action to rectify.
246 BANNED
248 CTDB is not behaving as designed on this node. For example, it may
249 have failed too many recovery attempts. Such nodes are banned from
250 participating in the cluster for a configurable time period before
251 they attempt to rejoin the cluster. A banned node does not host
252 public addresses to provide services. All banned nodes should be
253 investigated and may require an administrative action to rectify.
254 STOPPED
256 This node has been administratively exclude from the cluster. A
257 stopped node does no participate in the cluster and does not host
258 public addresses to provide services. This state can be used while
259 performing maintenance on a node.
260 PARTIALLYONLINE
262 A node that is partially online participates in a cluster like a
263 healthy (OK) node. Some interfaces to serve public addresses are
264 down, but at least one interface is up. See also ctdb ifaces.
265
267 Cluster nodes can have several different capabilities enabled. These
268 are listed below.
269 LEADER
271 Indicates that a node can become the CTDB cluster leader. The
272 current leader is decided via an election held by all active nodes
273 with this capability.
274 Default is YES.
276 LMASTER
278 Indicates that a node can be the location master (LMASTER) for
279 database records. The LMASTER always knows which node has the
280 latest copy of a record in a volatile database.
281 Default is YES.
283 The LEADER and LMASTER capabilities can be disabled when CTDB is used
285 to create a cluster spanning across WAN links. In this case CTDB acts
286 as a WAN accelerator.
287
289 LVS is a mode where CTDB presents one single IP address for the entire
290 cluster. This is an alternative to using public IP addresses and
291 round-robin DNS to loadbalance clients across the cluster.
292 This is similar to using a layer-4 loadbalancing switch but with some
294 restrictions.
295 One extra LVS public address is assigned on the public network to each
297 LVS group. Each LVS group is a set of nodes in the cluster that
298 presents the same LVS address public address to the outside world.
299 Normally there would only be one LVS group spanning an entire cluster,
300 but in situations where one CTDB cluster spans multiple physical sites
301 it might be useful to have one LVS group for each site. There can be
302 multiple LVS groups in a cluster but each node can only be member of
303 one LVS group.
304 Client access to the cluster is load-balanced across the HEALTHY nodes
306 in an LVS group. If no HEALTHY nodes exists then all nodes in the group
307 are used, regardless of health status. CTDB will, however never
308 load-balance LVS traffic to nodes that are BANNED, STOPPED, DISABLED or
309 DISCONNECTED. The ctdb lvs command is used to show which nodes are
310 currently load-balanced across.
311 In each LVS group, one of the nodes is selected by CTDB to be the LVS
313 leader. This node receives all traffic from clients coming in to the
314 LVS public address and multiplexes it across the internal network to
315 one of the nodes that LVS is using. When responding to the client, that
316 node will send the data back directly to the client, bypassing the LVS
317 leader node. The command ctdb lvs leader will show which node is the
318 current LVS leader.
319 The path used for a client I/O is:
321 1. Client sends request packet to LVS leader.
323 2. LVS leader passes the request on to one node across the internal
325 network.
326 3. Selected node processes the request.
328 4. Node responds back to client.
330 This means that all incoming traffic to the cluster will pass through
332 one physical node, which limits scalability. You can send more data to
333 the LVS address that one physical node can multiplex. This means that
334 you should not use LVS if your I/O pattern is write-intensive since you
335 will be limited in the available network bandwidth that node can
336 handle. LVS does work very well for read-intensive workloads where only
337 smallish READ requests are going through the LVS leader bottleneck and
338 the majority of the traffic volume (the data in the read replies) goes
339 straight from the processing node back to the clients. For
340 read-intensive i/o patterns you can achieve very high throughput rates
341 in this mode.
342 Note: you can use LVS and public addresses at the same time.
344 If you use LVS, you must have a permanent address configured for the
346 public interface on each node. This address must be routable and the
347 cluster nodes must be configured so that all traffic back to client
348 hosts are routed through this interface. This is also required in order
349 to allow samba/winbind on the node to talk to the domain controller.
350 This LVS IP address can not be used to initiate outgoing traffic.
351 Make sure that the domain controller and the clients are reachable from
353 a node before you enable LVS. Also ensure that outgoing traffic to
354 these hosts is routed out through the configured public interface.
355 Configuration
357 To activate LVS on a CTDB node you must specify the
358 CTDB_LVS_PUBLIC_IFACE, CTDB_LVS_PUBLIC_IP and CTDB_LVS_NODES
359 configuration variables. CTDB_LVS_NODES specifies a file containing
360 the private address of all nodes in the current node's LVS group.
361 Example:
363 CTDB_LVS_PUBLIC_IFACE=eth1
365 CTDB_LVS_PUBLIC_IP=10.1.1.237
366 CTDB_LVS_NODES=/etc/ctdb/lvs_nodes
367 Example /etc/ctdb/lvs_nodes:
370 192.168.1.2
372 192.168.1.3
373 192.168.1.4
374 Normally any node in an LVS group can act as the LVS leader. Nodes that
377 are highly loaded due to other demands maybe flagged with the
378 "follower-only" option in the CTDB_LVS_NODES file to limit the LVS
379 functionality of those nodes.
380 LVS nodes file that excludes 192.168.1.4 from being the LVS leader
382 node:
383 192.168.1.2
385 192.168.1.3
386 192.168.1.4 follower-only
387
390 CTDB tracks TCP connections from clients to public IP addresses, on
391 known ports. When an IP address moves from one node to another, all
392 existing TCP connections to that IP address are reset. The node taking
393 over this IP address will also send gratuitous ARPs (for IPv4, or
394 neighbour advertisement, for IPv6). This allows clients to reconnect
395 quickly, rather than waiting for TCP timeouts, which can be very long.
396 It is important that established TCP connections do not survive a
398 release and take of a public IP address on the same node. Such
399 connections can get out of sync with sequence and ACK numbers,
400 potentially causing a disruptive ACK storm.
401
403 NAT gateway (NATGW) is an optional feature that is used to configure
404 fallback routing for nodes. This allows cluster nodes to connect to
405 external services (e.g. DNS, AD, NIS and LDAP) when they do not host
406 any public addresses (e.g. when they are unhealthy).
407 This also applies to node startup because CTDB marks nodes as UNHEALTHY
409 until they have passed a "monitor" event. In this context, NAT gateway
410 helps to avoid a "chicken and egg" situation where a node needs to
411 access an external service to become healthy.
412 Another way of solving this type of problem is to assign an extra
414 static IP address to a public interface on every node. This is simpler
415 but it uses an extra IP address per node, while NAT gateway generally
416 uses only one extra IP address.
417 Operation
419 One extra NATGW public address is assigned on the public network to
420 each NATGW group. Each NATGW group is a set of nodes in the cluster
421 that shares the same NATGW address to talk to the outside world.
422 Normally there would only be one NATGW group spanning an entire
423 cluster, but in situations where one CTDB cluster spans multiple
424 physical sites it might be useful to have one NATGW group for each
425 site.
426 There can be multiple NATGW groups in a cluster but each node can only
428 be member of one NATGW group.
429 In each NATGW group, one of the nodes is selected by CTDB to be the
431 NATGW leader and the other nodes are consider to be NATGW followers.
432 NATGW followers establish a fallback default route to the NATGW leader
433 via the private network. When a NATGW follower hosts no public IP
434 addresses then it will use this route for outbound connections. The
435 NATGW leader hosts the NATGW public IP address and routes outgoing
436 connections from follower nodes via this IP address. It also
437 establishes a fallback default route.
438 Configuration
440 NATGW is usually configured similar to the following example
441 configuration:
442 CTDB_NATGW_NODES=/etc/ctdb/natgw_nodes
444 CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24
445 CTDB_NATGW_PUBLIC_IP=10.0.0.227/24
446 CTDB_NATGW_PUBLIC_IFACE=eth0
447 CTDB_NATGW_DEFAULT_GATEWAY=10.0.0.1
448 Normally any node in a NATGW group can act as the NATGW leader. Some
451 configurations may have special nodes that lack connectivity to a
452 public network. In such cases, those nodes can be flagged with the
453 "follower-only" option in the CTDB_NATGW_NODES file to limit the NATGW
454 functionality of those nodes.
455 See the NAT GATEWAY section in ctdb-script.options(5) for more details
457 of NATGW configuration.
458 Implementation details
460 When the NATGW functionality is used, one of the nodes is selected to
461 act as a NAT gateway for all the other nodes in the group when they
462 need to communicate with the external services. The NATGW leader is
463 selected to be a node that is most likely to have usable networks.
464 The NATGW leader hosts the NATGW public IP address CTDB_NATGW_PUBLIC_IP
466 on the configured public interfaces CTDB_NATGW_PUBLIC_IFACE and acts as
467 a router, masquerading outgoing connections from follower nodes via
468 this IP address. If CTDB_NATGW_DEFAULT_GATEWAY is set then it also
469 establishes a fallback default route to the configured this gateway
470 with a metric of 10. A metric 10 route is used so it can co-exist with
471 other default routes that may be available.
472 A NATGW follower establishes its fallback default route to the NATGW
474 leader via the private network CTDB_NATGW_PRIVATE_NETWORKwith a metric
475 of 10. This route is used for outbound connections when no other
476 default route is available because the node hosts no public addresses.
477 A metric 10 routes is used so that it can co-exist with other default
478 routes that may be available when the node is hosting public addresses.
479 CTDB_NATGW_STATIC_ROUTES can be used to have NATGW create more specific
481 routes instead of just default routes.
482 This is implemented in the 11.natgw eventscript. Please see the
484 eventscript file and the NAT GATEWAY section in ctdb-script.options(5)
485 for more details.
486
488 Policy routing is an optional CTDB feature to support complex network
489 topologies. Public addresses may be spread across several different
490 networks (or VLANs) and it may not be possible to route packets from
491 these public addresses via the system's default route. Therefore, CTDB
492 has support for policy routing via the 13.per_ip_routing eventscript.
493 This allows routing to be specified for packets sourced from each
494 public address. The routes are added and removed as CTDB moves public
495 addresses between nodes.
496 Configuration variables
498 There are 4 configuration variables related to policy routing:
499 CTDB_PER_IP_ROUTING_CONF, CTDB_PER_IP_ROUTING_RULE_PREF,
500 CTDB_PER_IP_ROUTING_TABLE_ID_LOW, CTDB_PER_IP_ROUTING_TABLE_ID_HIGH.
501 See the POLICY ROUTING section in ctdb-script.options(5) for more
502 details.
503 Configuration
505 The format of each line of CTDB_PER_IP_ROUTING_CONF is:
506 <public_address> <network> [ <gateway> ]
508 Leading whitespace is ignored and arbitrary whitespace may be used as a
511 separator. Lines that have a "public address" item that doesn't match
512 an actual public address are ignored. This means that comment lines can
513 be added using a leading character such as '#', since this will never
514 match an IP address.
515 A line without a gateway indicates a link local route.
517 For example, consider the configuration line:
519 192.168.1.99 192.168.1.0/24
521 If the corresponding public_addresses line is:
524 192.168.1.99/24 eth2,eth3
526 CTDB_PER_IP_ROUTING_RULE_PREF is 100, and CTDB adds the address to eth2
529 then the following routing information is added:
530 ip rule add from 192.168.1.99 pref 100 table ctdb.192.168.1.99
532 ip route add 192.168.1.0/24 dev eth2 table ctdb.192.168.1.99
533 This causes traffic from 192.168.1.99 to 192.168.1.0/24 go via eth2.
536 The ip rule command will show (something like - depending on other
538 public addresses and other routes on the system):
539 0: from all lookup local
541 100: from 192.168.1.99 lookup ctdb.192.168.1.99
542 32766: from all lookup main
543 32767: from all lookup default
544 ip route show table ctdb.192.168.1.99 will show:
547 192.168.1.0/24 dev eth2 scope link
549 The usual use for a line containing a gateway is to add a default route
552 corresponding to a particular source address. Consider this line of
553 configuration:
554 192.168.1.99 0.0.0.0/0 192.168.1.1
556 In the situation described above this will cause an extra routing
559 command to be executed:
560 ip route add 0.0.0.0/0 via 192.168.1.1 dev eth2 table ctdb.192.168.1.99
562 With both configuration lines, ip route show table ctdb.192.168.1.99
565 will show:
566 192.168.1.0/24 dev eth2 scope link
568 default via 192.168.1.1 dev eth2
569 Sample configuration
572 Here is a more complete example configuration.
573 /etc/ctdb/public_addresses:
575 192.168.1.98 eth2,eth3
577 192.168.1.99 eth2,eth3
578 /etc/ctdb/policy_routing:
580 192.168.1.98 192.168.1.0/24
582 192.168.1.98 192.168.200.0/24 192.168.1.254
583 192.168.1.98 0.0.0.0/0 192.168.1.1
584 192.168.1.99 192.168.1.0/24
585 192.168.1.99 192.168.200.0/24 192.168.1.254
586 192.168.1.99 0.0.0.0/0 192.168.1.1
587 The routes local packets as expected, the default route is as
590 previously discussed, but packets to 192.168.200.0/24 are routed via
591 the alternate gateway 192.168.1.254.
592
594 When certain state changes occur in CTDB, it can be configured to
595 perform arbitrary actions via notifications. For example, sending SNMP
596 traps or emails when a node becomes unhealthy or similar.
597 The notification mechanism runs all executable files ending in
599 ".script" in /etc/ctdb/events/notification/, ignoring any failures and
600 continuing to run all files.
601 CTDB currently generates notifications after CTDB changes to these
603 states:
604 init
605 setup
606 startup
607 healthy
608 unhealthy
609
611 Valid log levels, in increasing order of verbosity, are:
612 ERROR
613 WARNING
614 NOTICE
615 INFO
616 DEBUG
617
619 It is possible to have a CTDB cluster that spans across a WAN link. For
620 example where you have a CTDB cluster in your datacentre but you also
621 want to have one additional CTDB node located at a remote branch site.
622 This is similar to how a WAN accelerator works but with the difference
623 that while a WAN-accelerator often acts as a Proxy or a MitM, in the
624 ctdb remote cluster node configuration the Samba instance at the remote
625 site IS the genuine server, not a proxy and not a MitM, and thus
626 provides 100% correct CIFS semantics to clients.
627 See the cluster as one single multihomed samba server where one of the
629 NICs (the remote node) is very far away.
630 NOTE: This does require that the cluster filesystem you use can cope
632 with WAN-link latencies. Not all cluster filesystems can handle
633 WAN-link latencies! Whether this will provide very good WAN-accelerator
634 performance or it will perform very poorly depends entirely on how
635 optimized your cluster filesystem is in handling high latency for data
636 and metadata operations.
637 To activate a node as being a remote cluster node you need to set the
639 following two parameters in /etc/ctdb/ctdb.conf for the remote node:
640 [legacy]
642 lmaster capability = false
643 leader capability = false
644 Verify with the command "ctdb getcapabilities" that that node no longer
647 has the leader or the lmaster capabilities.
648
650 ctdb(1), ctdbd(1), ctdb_diagnostics(1), ltdbtool(1), onnode(1),
651 ping_pong(1), ctdb.conf(5), ctdb-script.options(5), ctdb.sysconfig(5),
652 ctdb-statistics(7), ctdb-tunables(7),
653 https://wiki.samba.org/index.php/CTDB_and_Clustered_Samba,
654 http://ctdb.samba.org/
655
657 This documentation was written by Ronnie Sahlberg, Amitay Isaacs,
658 Martin Schwenke
659
661 Copyright © 2007 Andrew Tridgell, Ronnie Sahlberg
662 This program is free software; you can redistribute it and/or modify it
664 under the terms of the GNU General Public License as published by the
665 Free Software Foundation; either version 3 of the License, or (at your
666 option) any later version.
667 This program is distributed in the hope that it will be useful, but
669 WITHOUT ANY WARRANTY; without even the implied warranty of
670 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
671 General Public License for more details.
672 You should have received a copy of the GNU General Public License along
674 with this program; if not, see http://www.gnu.org/licenses.
675ctdb 01/26/2023 CTDB(7)