1CTDB(7) CTDB - clustered TDB database CTDB(7)
2
6 ctdb - Clustered TDB
7
9 CTDB is a clustered database component in clustered Samba that provides
10 a high-availability load-sharing CIFS server cluster.
11 The main functions of CTDB are:
13 • Provide a clustered version of the TDB database with automatic
15 rebuild/recovery of the databases upon node failures.
16 • Monitor nodes in the cluster and services running on each node.
18 • Manage a pool of public IP addresses that are used to provide
20 services to clients. Alternatively, CTDB can be used with LVS.
21 Combined with a cluster filesystem CTDB provides a full
23 high-availablity (HA) environment for services such as clustered Samba,
24 NFS and other services.
25 In addition to the CTDB manual pages there is much more information
27 available at https://wiki.samba.org/index.php/CTDB_and_Clustered_Samba.
28
30 A CTDB cluster is a collection of nodes with 2 or more network
31 interfaces. All nodes provide network (usually file/NAS) services to
32 clients. Data served by file services is stored on shared storage
33 (usually a cluster filesystem) that is accessible by all nodes.
34 CTDB provides an "all active" cluster, where services are load balanced
36 across all nodes.
37
39 CTDB uses a recovery lock to avoid a split brain, where a cluster
40 becomes partitioned and each partition attempts to operate
41 independently. Issues that can result from a split brain include file
42 data corruption, because file locking metadata may not be tracked
43 correctly.
44 CTDB uses a cluster leader and follower model of cluster management.
46 All nodes in a cluster elect one node to be the leader. The leader node
47 coordinates privileged operations such as database recovery and IP
48 address failover. CTDB refers to the leader node as the recovery
49 master. This node takes and holds the recovery lock to assert its
50 privileged role in the cluster.
51 By default, the recovery lock is implemented using a file (specified by
53 recovery lock in the [cluster] section of ctdb.conf(5)) residing in
54 shared storage (usually) on a cluster filesystem. To support a recovery
55 lock the cluster filesystem must support lock coherence. See
56 ping_pong(1) for more details.
57 The recovery lock can also be implemented using an arbitrary cluster
59 mutex helper (or call-out). This is indicated by using an exclamation
60 point ('!') as the first character of the recovery lock parameter. For
61 example, a value of !/usr/bin/myhelper recovery would run the given
62 helper with the specified arguments. The helper will continue to run as
63 long as it holds its mutex. See ctdb/doc/cluster_mutex_helper.txt in
64 the source tree, and related code, for clues about writing helpers.
65 When a file is specified for the recovery lock parameter (i.e. no
67 leading '!') the file lock is implemented by a default helper
68 (/usr/libexec/ctdb/ctdb_mutex_fcntl_helper). This helper has arguments
69 as follows:
70 ctdb_mutex_fcntl_helper FILE [RECHECK-INTERVAL]
73 ctdb_mutex_fcntl_helper will take a lock on FILE and then check every
77 RECHECK-INTERVAL seconds to ensure that FILE still exists and that its
78 inode number is unchanged from when the lock was taken. The default
79 value for RECHECK-INTERVAL is 5.
80 If a cluster becomes partitioned (for example, due to a communication
82 failure) and a different recovery master is elected by the nodes in
83 each partition, then only one of these recovery masters will be able to
84 take the recovery lock. The recovery master in the "losing" partition
85 will not be able to take the recovery lock and will be excluded from
86 the cluster. The nodes in the "losing" partition will elect each node
87 in turn as their recovery master so eventually all the nodes in that
88 partition will be excluded.
89 CTDB does sanity checks to ensure that the recovery lock is held as
91 expected.
92 CTDB can run without a recovery lock but this is not recommended as
94 there will be no protection from split brains.
95
97 Each node in a CTDB cluster has multiple IP addresses assigned to it:
98 • A single private IP address that is used for communication between
100 nodes.
101 • One or more public IP addresses that are used to provide NAS or
103 other services.
104 Private address
107 Each node is configured with a unique, permanently assigned private
108 address. This address is configured by the operating system. This
109 address uniquely identifies a physical node in the cluster and is the
110 address that CTDB daemons will use to communicate with the CTDB daemons
111 on other nodes.
112 Private addresses are listed in the file /etc/ctdb/nodes). This file
114 contains the list of private addresses for all nodes in the cluster,
115 one per line. This file must be the same on all nodes in the cluster.
116 Some users like to put this configuration file in their cluster
118 filesystem. A symbolic link should be used in this case.
119 Private addresses should not be used by clients to connect to services
121 provided by the cluster.
122 It is strongly recommended that the private addresses are configured on
124 a private network that is separate from client networks. This is
125 because the CTDB protocol is both unauthenticated and unencrypted. If
126 clients share the private network then steps need to be taken to stop
127 injection of packets to relevant ports on the private addresses. It is
128 also likely that CTDB protocol traffic between nodes could leak
129 sensitive information if it can be intercepted.
130 Example /etc/ctdb/nodes for a four node cluster:
132 192.168.1.1
134 192.168.1.2
135 192.168.1.3
136 192.168.1.4
137 Public addresses
140 Public addresses are used to provide services to clients. Public
141 addresses are not configured at the operating system level and are not
142 permanently associated with a particular node. Instead, they are
143 managed by CTDB and are assigned to interfaces on physical nodes at
144 runtime.
145 The CTDB cluster will assign/reassign these public addresses across the
147 available healthy nodes in the cluster. When one node fails, its public
148 addresses will be taken over by one or more other nodes in the cluster.
149 This ensures that services provided by all public addresses are always
150 available to clients, as long as there are nodes available capable of
151 hosting this address.
152 The public address configuration is stored in
154 /etc/ctdb/public_addresses on each node. This file contains a list of
155 the public addresses that the node is capable of hosting, one per line.
156 Each entry also contains the netmask and the interface to which the
157 address should be assigned. If this file is missing then no public
158 addresses are configured.
159 Some users who have the same public addresses on all nodes like to put
161 this configuration file in their cluster filesystem. A symbolic link
162 should be used in this case.
163 Example /etc/ctdb/public_addresses for a node that can host 4 public
165 addresses, on 2 different interfaces:
166 10.1.1.1/24 eth1
168 10.1.1.2/24 eth1
169 10.1.2.1/24 eth2
170 10.1.2.2/24 eth2
171 In many cases the public addresses file will be the same on all nodes.
174 However, it is possible to use different public address configurations
175 on different nodes.
176 Example: 4 nodes partitioned into two subgroups:
178 Node 0:/etc/ctdb/public_addresses
180 10.1.1.1/24 eth1
181 10.1.1.2/24 eth1
182 Node 1:/etc/ctdb/public_addresses
184 10.1.1.1/24 eth1
185 10.1.1.2/24 eth1
186 Node 2:/etc/ctdb/public_addresses
188 10.1.2.1/24 eth2
189 10.1.2.2/24 eth2
190 Node 3:/etc/ctdb/public_addresses
192 10.1.2.1/24 eth2
193 10.1.2.2/24 eth2
194 In this example nodes 0 and 1 host two public addresses on the 10.1.1.x
197 network while nodes 2 and 3 host two public addresses for the 10.1.2.x
198 network.
199 Public address 10.1.1.1 can be hosted by either of nodes 0 or 1 and
201 will be available to clients as long as at least one of these two nodes
202 are available.
203 If both nodes 0 and 1 become unavailable then public address 10.1.1.1
205 also becomes unavailable. 10.1.1.1 can not be failed over to nodes 2 or
206 3 since these nodes do not have this public address configured.
207 The ctdb ip command can be used to view the current assignment of
209 public addresses to physical nodes.
210
212 The current status of each node in the cluster can be viewed by the
213 ctdb status command.
214 A node can be in one of the following states:
216 OK
218 This node is healthy and fully functional. It hosts public
219 addresses to provide services.
220 DISCONNECTED
222 This node is not reachable by other nodes via the private network.
223 It is not currently participating in the cluster. It does not host
224 public addresses to provide services. It might be shut down.
225 DISABLED
227 This node has been administratively disabled. This node is
228 partially functional and participates in the cluster. However, it
229 does not host public addresses to provide services.
230 UNHEALTHY
232 A service provided by this node has failed a health check and
233 should be investigated. This node is partially functional and
234 participates in the cluster. However, it does not host public
235 addresses to provide services. Unhealthy nodes should be
236 investigated and may require an administrative action to rectify.
237 BANNED
239 CTDB is not behaving as designed on this node. For example, it may
240 have failed too many recovery attempts. Such nodes are banned from
241 participating in the cluster for a configurable time period before
242 they attempt to rejoin the cluster. A banned node does not host
243 public addresses to provide services. All banned nodes should be
244 investigated and may require an administrative action to rectify.
245 STOPPED
247 This node has been administratively exclude from the cluster. A
248 stopped node does no participate in the cluster and does not host
249 public addresses to provide services. This state can be used while
250 performing maintenance on a node.
251 PARTIALLYONLINE
253 A node that is partially online participates in a cluster like a
254 healthy (OK) node. Some interfaces to serve public addresses are
255 down, but at least one interface is up. See also ctdb ifaces.
256
258 Cluster nodes can have several different capabilities enabled. These
259 are listed below.
260 RECMASTER
262 Indicates that a node can become the CTDB cluster recovery master.
263 The current recovery master is decided via an election held by all
264 active nodes with this capability.
265 Default is YES.
267 LMASTER
269 Indicates that a node can be the location master (LMASTER) for
270 database records. The LMASTER always knows which node has the
271 latest copy of a record in a volatile database.
272 Default is YES.
274 The RECMASTER and LMASTER capabilities can be disabled when CTDB is
276 used to create a cluster spanning across WAN links. In this case CTDB
277 acts as a WAN accelerator.
278
280 LVS is a mode where CTDB presents one single IP address for the entire
281 cluster. This is an alternative to using public IP addresses and
282 round-robin DNS to loadbalance clients across the cluster.
283 This is similar to using a layer-4 loadbalancing switch but with some
285 restrictions.
286 One extra LVS public address is assigned on the public network to each
288 LVS group. Each LVS group is a set of nodes in the cluster that
289 presents the same LVS address public address to the outside world.
290 Normally there would only be one LVS group spanning an entire cluster,
291 but in situations where one CTDB cluster spans multiple physical sites
292 it might be useful to have one LVS group for each site. There can be
293 multiple LVS groups in a cluster but each node can only be member of
294 one LVS group.
295 Client access to the cluster is load-balanced across the HEALTHY nodes
297 in an LVS group. If no HEALTHY nodes exists then all nodes in the group
298 are used, regardless of health status. CTDB will, however never
299 load-balance LVS traffic to nodes that are BANNED, STOPPED, DISABLED or
300 DISCONNECTED. The ctdb lvs command is used to show which nodes are
301 currently load-balanced across.
302 In each LVS group, one of the nodes is selected by CTDB to be the LVS
304 leader. This node receives all traffic from clients coming in to the
305 LVS public address and multiplexes it across the internal network to
306 one of the nodes that LVS is using. When responding to the client, that
307 node will send the data back directly to the client, bypassing the LVS
308 leader node. The command ctdb lvs leader will show which node is the
309 current LVS leader.
310 The path used for a client I/O is:
312 1. Client sends request packet to LVS leader.
314 2. LVS leader passes the request on to one node across the internal
316 network.
317 3. Selected node processes the request.
319 4. Node responds back to client.
321 This means that all incoming traffic to the cluster will pass through
323 one physical node, which limits scalability. You can send more data to
324 the LVS address that one physical node can multiplex. This means that
325 you should not use LVS if your I/O pattern is write-intensive since you
326 will be limited in the available network bandwidth that node can
327 handle. LVS does work very well for read-intensive workloads where only
328 smallish READ requests are going through the LVS leader bottleneck and
329 the majority of the traffic volume (the data in the read replies) goes
330 straight from the processing node back to the clients. For
331 read-intensive i/o patterns you can achieve very high throughput rates
332 in this mode.
333 Note: you can use LVS and public addresses at the same time.
335 If you use LVS, you must have a permanent address configured for the
337 public interface on each node. This address must be routable and the
338 cluster nodes must be configured so that all traffic back to client
339 hosts are routed through this interface. This is also required in order
340 to allow samba/winbind on the node to talk to the domain controller.
341 This LVS IP address can not be used to initiate outgoing traffic.
342 Make sure that the domain controller and the clients are reachable from
344 a node before you enable LVS. Also ensure that outgoing traffic to
345 these hosts is routed out through the configured public interface.
346 Configuration
348 To activate LVS on a CTDB node you must specify the
349 CTDB_LVS_PUBLIC_IFACE, CTDB_LVS_PUBLIC_IP and CTDB_LVS_NODES
350 configuration variables. CTDB_LVS_NODES specifies a file containing
351 the private address of all nodes in the current node's LVS group.
352 Example:
354 CTDB_LVS_PUBLIC_IFACE=eth1
356 CTDB_LVS_PUBLIC_IP=10.1.1.237
357 CTDB_LVS_NODES=/etc/ctdb/lvs_nodes
358 Example /etc/ctdb/lvs_nodes:
361 192.168.1.2
363 192.168.1.3
364 192.168.1.4
365 Normally any node in an LVS group can act as the LVS leader. Nodes that
368 are highly loaded due to other demands maybe flagged with the
369 "follower-only" option in the CTDB_LVS_NODES file to limit the LVS
370 functionality of those nodes.
371 LVS nodes file that excludes 192.168.1.4 from being the LVS leader
373 node:
374 192.168.1.2
376 192.168.1.3
377 192.168.1.4 follower-only
378
381 CTDB tracks TCP connections from clients to public IP addresses, on
382 known ports. When an IP address moves from one node to another, all
383 existing TCP connections to that IP address are reset. The node taking
384 over this IP address will also send gratuitous ARPs (for IPv4, or
385 neighbour advertisement, for IPv6). This allows clients to reconnect
386 quickly, rather than waiting for TCP timeouts, which can be very long.
387 It is important that established TCP connections do not survive a
389 release and take of a public IP address on the same node. Such
390 connections can get out of sync with sequence and ACK numbers,
391 potentially causing a disruptive ACK storm.
392
394 NAT gateway (NATGW) is an optional feature that is used to configure
395 fallback routing for nodes. This allows cluster nodes to connect to
396 external services (e.g. DNS, AD, NIS and LDAP) when they do not host
397 any public addresses (e.g. when they are unhealthy).
398 This also applies to node startup because CTDB marks nodes as UNHEALTHY
400 until they have passed a "monitor" event. In this context, NAT gateway
401 helps to avoid a "chicken and egg" situation where a node needs to
402 access an external service to become healthy.
403 Another way of solving this type of problem is to assign an extra
405 static IP address to a public interface on every node. This is simpler
406 but it uses an extra IP address per node, while NAT gateway generally
407 uses only one extra IP address.
408 Operation
410 One extra NATGW public address is assigned on the public network to
411 each NATGW group. Each NATGW group is a set of nodes in the cluster
412 that shares the same NATGW address to talk to the outside world.
413 Normally there would only be one NATGW group spanning an entire
414 cluster, but in situations where one CTDB cluster spans multiple
415 physical sites it might be useful to have one NATGW group for each
416 site.
417 There can be multiple NATGW groups in a cluster but each node can only
419 be member of one NATGW group.
420 In each NATGW group, one of the nodes is selected by CTDB to be the
422 NATGW leader and the other nodes are consider to be NATGW followers.
423 NATGW followers establish a fallback default route to the NATGW leader
424 via the private network. When a NATGW follower hosts no public IP
425 addresses then it will use this route for outbound connections. The
426 NATGW leader hosts the NATGW public IP address and routes outgoing
427 connections from follower nodes via this IP address. It also
428 establishes a fallback default route.
429 Configuration
431 NATGW is usually configured similar to the following example
432 configuration:
433 CTDB_NATGW_NODES=/etc/ctdb/natgw_nodes
435 CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24
436 CTDB_NATGW_PUBLIC_IP=10.0.0.227/24
437 CTDB_NATGW_PUBLIC_IFACE=eth0
438 CTDB_NATGW_DEFAULT_GATEWAY=10.0.0.1
439 Normally any node in a NATGW group can act as the NATGW leader. Some
442 configurations may have special nodes that lack connectivity to a
443 public network. In such cases, those nodes can be flagged with the
444 "follower-only" option in the CTDB_NATGW_NODES file to limit the NATGW
445 functionality of those nodes.
446 See the NAT GATEWAY section in ctdb-script.options(5) for more details
448 of NATGW configuration.
449 Implementation details
451 When the NATGW functionality is used, one of the nodes is selected to
452 act as a NAT gateway for all the other nodes in the group when they
453 need to communicate with the external services. The NATGW leader is
454 selected to be a node that is most likely to have usable networks.
455 The NATGW leader hosts the NATGW public IP address CTDB_NATGW_PUBLIC_IP
457 on the configured public interfaces CTDB_NATGW_PUBLIC_IFACE and acts as
458 a router, masquerading outgoing connections from follower nodes via
459 this IP address. If CTDB_NATGW_DEFAULT_GATEWAY is set then it also
460 establishes a fallback default route to the configured this gateway
461 with a metric of 10. A metric 10 route is used so it can co-exist with
462 other default routes that may be available.
463 A NATGW follower establishes its fallback default route to the NATGW
465 leader via the private network CTDB_NATGW_PRIVATE_NETWORKwith a metric
466 of 10. This route is used for outbound connections when no other
467 default route is available because the node hosts no public addresses.
468 A metric 10 routes is used so that it can co-exist with other default
469 routes that may be available when the node is hosting public addresses.
470 CTDB_NATGW_STATIC_ROUTES can be used to have NATGW create more specific
472 routes instead of just default routes.
473 This is implemented in the 11.natgw eventscript. Please see the
475 eventscript file and the NAT GATEWAY section in ctdb-script.options(5)
476 for more details.
477
479 Policy routing is an optional CTDB feature to support complex network
480 topologies. Public addresses may be spread across several different
481 networks (or VLANs) and it may not be possible to route packets from
482 these public addresses via the system's default route. Therefore, CTDB
483 has support for policy routing via the 13.per_ip_routing eventscript.
484 This allows routing to be specified for packets sourced from each
485 public address. The routes are added and removed as CTDB moves public
486 addresses between nodes.
487 Configuration variables
489 There are 4 configuration variables related to policy routing:
490 CTDB_PER_IP_ROUTING_CONF, CTDB_PER_IP_ROUTING_RULE_PREF,
491 CTDB_PER_IP_ROUTING_TABLE_ID_LOW, CTDB_PER_IP_ROUTING_TABLE_ID_HIGH.
492 See the POLICY ROUTING section in ctdb-script.options(5) for more
493 details.
494 Configuration
496 The format of each line of CTDB_PER_IP_ROUTING_CONF is:
497 <public_address> <network> [ <gateway> ]
499 Leading whitespace is ignored and arbitrary whitespace may be used as a
502 separator. Lines that have a "public address" item that doesn't match
503 an actual public address are ignored. This means that comment lines can
504 be added using a leading character such as '#', since this will never
505 match an IP address.
506 A line without a gateway indicates a link local route.
508 For example, consider the configuration line:
510 192.168.1.99 192.168.1.1/24
512 If the corresponding public_addresses line is:
515 192.168.1.99/24 eth2,eth3
517 CTDB_PER_IP_ROUTING_RULE_PREF is 100, and CTDB adds the address to eth2
520 then the following routing information is added:
521 ip rule add from 192.168.1.99 pref 100 table ctdb.192.168.1.99
523 ip route add 192.168.1.0/24 dev eth2 table ctdb.192.168.1.99
524 This causes traffic from 192.168.1.1 to 192.168.1.0/24 go via eth2.
527 The ip rule command will show (something like - depending on other
529 public addresses and other routes on the system):
530 0: from all lookup local
532 100: from 192.168.1.99 lookup ctdb.192.168.1.99
533 32766: from all lookup main
534 32767: from all lookup default
535 ip route show table ctdb.192.168.1.99 will show:
538 192.168.1.0/24 dev eth2 scope link
540 The usual use for a line containing a gateway is to add a default route
543 corresponding to a particular source address. Consider this line of
544 configuration:
545 192.168.1.99 0.0.0.0/0 192.168.1.1
547 In the situation described above this will cause an extra routing
550 command to be executed:
551 ip route add 0.0.0.0/0 via 192.168.1.1 dev eth2 table ctdb.192.168.1.99
553 With both configuration lines, ip route show table ctdb.192.168.1.99
556 will show:
557 192.168.1.0/24 dev eth2 scope link
559 default via 192.168.1.1 dev eth2
560 Sample configuration
563 Here is a more complete example configuration.
564 /etc/ctdb/public_addresses:
566 192.168.1.98 eth2,eth3
568 192.168.1.99 eth2,eth3
569 /etc/ctdb/policy_routing:
571 192.168.1.98 192.168.1.0/24
573 192.168.1.98 192.168.200.0/24 192.168.1.254
574 192.168.1.98 0.0.0.0/0 192.168.1.1
575 192.168.1.99 192.168.1.0/24
576 192.168.1.99 192.168.200.0/24 192.168.1.254
577 192.168.1.99 0.0.0.0/0 192.168.1.1
578 The routes local packets as expected, the default route is as
581 previously discussed, but packets to 192.168.200.0/24 are routed via
582 the alternate gateway 192.168.1.254.
583
585 When certain state changes occur in CTDB, it can be configured to
586 perform arbitrary actions via notifications. For example, sending SNMP
587 traps or emails when a node becomes unhealthy or similar.
588 The notification mechanism runs all executable files ending in
590 ".script" in /etc/ctdb/events/notification/, ignoring any failures and
591 continuing to run all files.
592 CTDB currently generates notifications after CTDB changes to these
594 states:
595 init
596 setup
597 startup
598 healthy
599 unhealthy
600
602 Valid log levels, in increasing order of verbosity, are:
603 ERROR
604 WARNING
605 NOTICE
606 INFO
607 DEBUG
608
610 It is possible to have a CTDB cluster that spans across a WAN link. For
611 example where you have a CTDB cluster in your datacentre but you also
612 want to have one additional CTDB node located at a remote branch site.
613 This is similar to how a WAN accelerator works but with the difference
614 that while a WAN-accelerator often acts as a Proxy or a MitM, in the
615 ctdb remote cluster node configuration the Samba instance at the remote
616 site IS the genuine server, not a proxy and not a MitM, and thus
617 provides 100% correct CIFS semantics to clients.
618 See the cluster as one single multihomed samba server where one of the
620 NICs (the remote node) is very far away.
621 NOTE: This does require that the cluster filesystem you use can cope
623 with WAN-link latencies. Not all cluster filesystems can handle
624 WAN-link latencies! Whether this will provide very good WAN-accelerator
625 performance or it will perform very poorly depends entirely on how
626 optimized your cluster filesystem is in handling high latency for data
627 and metadata operations.
628 To activate a node as being a remote cluster node you need to set the
630 following two parameters in /etc/ctdb/ctdb.conf for the remote node:
631 [legacy]
633 lmaster capability = false
634 recmaster capability = false
635 Verify with the command "ctdb getcapabilities" that that node no longer
638 has the recmaster or the lmaster capabilities.
639
641 ctdb(1), ctdbd(1), ctdbd_wrapper(1), ctdb_diagnostics(1), ltdbtool(1),
642 onnode(1), ping_pong(1), ctdb.conf(5), ctdb-script.options(5),
643 ctdb.sysconfig(5), ctdb-statistics(7), ctdb-tunables(7),
644 https://wiki.samba.org/index.php/CTDB_and_Clustered_Samba,
645 http://ctdb.samba.org/
646
648 This documentation was written by Ronnie Sahlberg, Amitay Isaacs,
649 Martin Schwenke
650
652 Copyright © 2007 Andrew Tridgell, Ronnie Sahlberg
653 This program is free software; you can redistribute it and/or modify it
655 under the terms of the GNU General Public License as published by the
656 Free Software Foundation; either version 3 of the License, or (at your
657 option) any later version.
658 This program is distributed in the hope that it will be useful, but
660 WITHOUT ANY WARRANTY; without even the implied warranty of
661 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
662 General Public License for more details.
663 You should have received a copy of the GNU General Public License along
665 with this program; if not, see http://www.gnu.org/licenses.
666ctdb 06/01/2021 CTDB(7)