1CTDB(7) CTDB - clustered TDB database CTDB(7)
2
6 ctdb - Clustered TDB
7
9 CTDB is a clustered database component in clustered Samba that provides
10 a high-availability load-sharing CIFS server cluster.
11 The main functions of CTDB are:
13 · Provide a clustered version of the TDB database with automatic
15 rebuild/recovery of the databases upon node failures.
16 · Monitor nodes in the cluster and services running on each node.
18 · Manage a pool of public IP addresses that are used to provide
20 services to clients. Alternatively, CTDB can be used with LVS.
21 Combined with a cluster filesystem CTDB provides a full
23 high-availablity (HA) environment for services such as clustered Samba,
24 NFS and other services.
25
27 A CTDB cluster is a collection of nodes with 2 or more network
28 interfaces. All nodes provide network (usually file/NAS) services to
29 clients. Data served by file services is stored on shared storage
30 (usually a cluster filesystem) that is accessible by all nodes.
31 CTDB provides an "all active" cluster, where services are load balanced
33 across all nodes.
34
36 CTDB uses a recovery lock to avoid a split brain, where a cluster
37 becomes partitioned and each partition attempts to operate
38 independently. Issues that can result from a split brain include file
39 data corruption, because file locking metadata may not be tracked
40 correctly.
41 CTDB uses a cluster leader and follower model of cluster management.
43 All nodes in a cluster elect one node to be the leader. The leader node
44 coordinates privileged operations such as database recovery and IP
45 address failover. CTDB refers to the leader node as the recovery
46 master. This node takes and holds the recovery lock to assert its
47 privileged role in the cluster.
48 By default, the recovery lock is implemented using a file (specified by
50 recovery lock in the [cluster] section of ctdb.conf(5)) residing in
51 shared storage (usually) on a cluster filesystem. To support a recovery
52 lock the cluster filesystem must support lock coherence. See
53 ping_pong(1) for more details.
54 The recovery lock can also be implemented using an arbitrary cluster
56 mutex helper (or call-out). This is indicated by using an exclamation
57 point ('!') as the first character of the recovery lock parameter. For
58 example, a value of !/usr/bin/myhelper recovery would run the given
59 helper with the specified arguments. The helper will continue to run as
60 long as it holds its mutex. See ctdb/doc/cluster_mutex_helper.txt in
61 the source tree, and related code, for clues about writing helpers.
62 When a file is specified for the recovery lock parameter (i.e. no
64 leading '!') the file lock is implemented by a default helper
65 (/usr/libexec/ctdb/ctdb_mutex_fcntl_helper). This helper has arguments
66 as follows:
67 ctdb_mutex_fcntl_helper FILE [RECHECK-INTERVAL]
70 ctdb_mutex_fcntl_helper will take a lock on FILE and then check every
74 RECHECK-INTERVAL seconds to ensure that FILE still exists and that its
75 inode number is unchanged from when the lock was taken. The default
76 value for RECHECK-INTERVAL is 5.
77 If a cluster becomes partitioned (for example, due to a communication
79 failure) and a different recovery master is elected by the nodes in
80 each partition, then only one of these recovery masters will be able to
81 take the recovery lock. The recovery master in the "losing" partition
82 will not be able to take the recovery lock and will be excluded from
83 the cluster. The nodes in the "losing" partition will elect each node
84 in turn as their recovery master so eventually all the nodes in that
85 partition will be excluded.
86 CTDB does sanity checks to ensure that the recovery lock is held as
88 expected.
89 CTDB can run without a recovery lock but this is not recommended as
91 there will be no protection from split brains.
92
94 Each node in a CTDB cluster has multiple IP addresses assigned to it:
95 · A single private IP address that is used for communication between
97 nodes.
98 · One or more public IP addresses that are used to provide NAS or
100 other services.
101 Private address
104 Each node is configured with a unique, permanently assigned private
105 address. This address is configured by the operating system. This
106 address uniquely identifies a physical node in the cluster and is the
107 address that CTDB daemons will use to communicate with the CTDB daemons
108 on other nodes.
109 Private addresses are listed in the file /etc/ctdb/nodes). This file
111 contains the list of private addresses for all nodes in the cluster,
112 one per line. This file must be the same on all nodes in the cluster.
113 Some users like to put this configuration file in their cluster
115 filesystem. A symbolic link should be used in this case.
116 Private addresses should not be used by clients to connect to services
118 provided by the cluster.
119 It is strongly recommended that the private addresses are configured on
121 a private network that is separate from client networks. This is
122 because the CTDB protocol is both unauthenticated and unencrypted. If
123 clients share the private network then steps need to be taken to stop
124 injection of packets to relevant ports on the private addresses. It is
125 also likely that CTDB protocol traffic between nodes could leak
126 sensitive information if it can be intercepted.
127 Example /etc/ctdb/nodes for a four node cluster:
129 192.168.1.1
131 192.168.1.2
132 192.168.1.3
133 192.168.1.4
134 Public addresses
137 Public addresses are used to provide services to clients. Public
138 addresses are not configured at the operating system level and are not
139 permanently associated with a particular node. Instead, they are
140 managed by CTDB and are assigned to interfaces on physical nodes at
141 runtime.
142 The CTDB cluster will assign/reassign these public addresses across the
144 available healthy nodes in the cluster. When one node fails, its public
145 addresses will be taken over by one or more other nodes in the cluster.
146 This ensures that services provided by all public addresses are always
147 available to clients, as long as there are nodes available capable of
148 hosting this address.
149 The public address configuration is stored in
151 /etc/ctdb/public_addresses on each node. This file contains a list of
152 the public addresses that the node is capable of hosting, one per line.
153 Each entry also contains the netmask and the interface to which the
154 address should be assigned. If this file is missing then no public
155 addresses are configured.
156 Some users who have the same public addresses on all nodes like to put
158 this configuration file in their cluster filesystem. A symbolic link
159 should be used in this case.
160 Example /etc/ctdb/public_addresses for a node that can host 4 public
162 addresses, on 2 different interfaces:
163 10.1.1.1/24 eth1
165 10.1.1.2/24 eth1
166 10.1.2.1/24 eth2
167 10.1.2.2/24 eth2
168 In many cases the public addresses file will be the same on all nodes.
171 However, it is possible to use different public address configurations
172 on different nodes.
173 Example: 4 nodes partitioned into two subgroups:
175 Node 0:/etc/ctdb/public_addresses
177 10.1.1.1/24 eth1
178 10.1.1.2/24 eth1
179 Node 1:/etc/ctdb/public_addresses
181 10.1.1.1/24 eth1
182 10.1.1.2/24 eth1
183 Node 2:/etc/ctdb/public_addresses
185 10.1.2.1/24 eth2
186 10.1.2.2/24 eth2
187 Node 3:/etc/ctdb/public_addresses
189 10.1.2.1/24 eth2
190 10.1.2.2/24 eth2
191 In this example nodes 0 and 1 host two public addresses on the 10.1.1.x
194 network while nodes 2 and 3 host two public addresses for the 10.1.2.x
195 network.
196 Public address 10.1.1.1 can be hosted by either of nodes 0 or 1 and
198 will be available to clients as long as at least one of these two nodes
199 are available.
200 If both nodes 0 and 1 become unavailable then public address 10.1.1.1
202 also becomes unavailable. 10.1.1.1 can not be failed over to nodes 2 or
203 3 since these nodes do not have this public address configured.
204 The ctdb ip command can be used to view the current assignment of
206 public addresses to physical nodes.
207
209 The current status of each node in the cluster can be viewed by the
210 ctdb status command.
211 A node can be in one of the following states:
213 OK
215 This node is healthy and fully functional. It hosts public
216 addresses to provide services.
217 DISCONNECTED
219 This node is not reachable by other nodes via the private network.
220 It is not currently participating in the cluster. It does not host
221 public addresses to provide services. It might be shut down.
222 DISABLED
224 This node has been administratively disabled. This node is
225 partially functional and participates in the cluster. However, it
226 does not host public addresses to provide services.
227 UNHEALTHY
229 A service provided by this node has failed a health check and
230 should be investigated. This node is partially functional and
231 participates in the cluster. However, it does not host public
232 addresses to provide services. Unhealthy nodes should be
233 investigated and may require an administrative action to rectify.
234 BANNED
236 CTDB is not behaving as designed on this node. For example, it may
237 have failed too many recovery attempts. Such nodes are banned from
238 participating in the cluster for a configurable time period before
239 they attempt to rejoin the cluster. A banned node does not host
240 public addresses to provide services. All banned nodes should be
241 investigated and may require an administrative action to rectify.
242 STOPPED
244 This node has been administratively exclude from the cluster. A
245 stopped node does no participate in the cluster and does not host
246 public addresses to provide services. This state can be used while
247 performing maintenance on a node.
248 PARTIALLYONLINE
250 A node that is partially online participates in a cluster like a
251 healthy (OK) node. Some interfaces to serve public addresses are
252 down, but at least one interface is up. See also ctdb ifaces.
253
255 Cluster nodes can have several different capabilities enabled. These
256 are listed below.
257 RECMASTER
259 Indicates that a node can become the CTDB cluster recovery master.
260 The current recovery master is decided via an election held by all
261 active nodes with this capability.
262 Default is YES.
264 LMASTER
266 Indicates that a node can be the location master (LMASTER) for
267 database records. The LMASTER always knows which node has the
268 latest copy of a record in a volatile database.
269 Default is YES.
271 The RECMASTER and LMASTER capabilities can be disabled when CTDB is
273 used to create a cluster spanning across WAN links. In this case CTDB
274 acts as a WAN accelerator.
275
277 LVS is a mode where CTDB presents one single IP address for the entire
278 cluster. This is an alternative to using public IP addresses and
279 round-robin DNS to loadbalance clients across the cluster.
280 This is similar to using a layer-4 loadbalancing switch but with some
282 restrictions.
283 One extra LVS public address is assigned on the public network to each
285 LVS group. Each LVS group is a set of nodes in the cluster that
286 presents the same LVS address public address to the outside world.
287 Normally there would only be one LVS group spanning an entire cluster,
288 but in situations where one CTDB cluster spans multiple physical sites
289 it might be useful to have one LVS group for each site. There can be
290 multiple LVS groups in a cluster but each node can only be member of
291 one LVS group.
292 Client access to the cluster is load-balanced across the HEALTHY nodes
294 in an LVS group. If no HEALTHY nodes exists then all nodes in the group
295 are used, regardless of health status. CTDB will, however never
296 load-balance LVS traffic to nodes that are BANNED, STOPPED, DISABLED or
297 DISCONNECTED. The ctdb lvs command is used to show which nodes are
298 currently load-balanced across.
299 In each LVS group, one of the nodes is selected by CTDB to be the LVS
301 master. This node receives all traffic from clients coming in to the
302 LVS public address and multiplexes it across the internal network to
303 one of the nodes that LVS is using. When responding to the client, that
304 node will send the data back directly to the client, bypassing the LVS
305 master node. The command ctdb lvs master will show which node is the
306 current LVS master.
307 The path used for a client I/O is:
309 1. Client sends request packet to LVSMASTER.
311 2. LVSMASTER passes the request on to one node across the internal
313 network.
314 3. Selected node processes the request.
316 4. Node responds back to client.
318 This means that all incoming traffic to the cluster will pass through
320 one physical node, which limits scalability. You can send more data to
321 the LVS address that one physical node can multiplex. This means that
322 you should not use LVS if your I/O pattern is write-intensive since you
323 will be limited in the available network bandwidth that node can
324 handle. LVS does work very well for read-intensive workloads where only
325 smallish READ requests are going through the LVSMASTER bottleneck and
326 the majority of the traffic volume (the data in the read replies) goes
327 straight from the processing node back to the clients. For
328 read-intensive i/o patterns you can achieve very high throughput rates
329 in this mode.
330 Note: you can use LVS and public addresses at the same time.
332 If you use LVS, you must have a permanent address configured for the
334 public interface on each node. This address must be routable and the
335 cluster nodes must be configured so that all traffic back to client
336 hosts are routed through this interface. This is also required in order
337 to allow samba/winbind on the node to talk to the domain controller.
338 This LVS IP address can not be used to initiate outgoing traffic.
339 Make sure that the domain controller and the clients are reachable from
341 a node before you enable LVS. Also ensure that outgoing traffic to
342 these hosts is routed out through the configured public interface.
343 Configuration
345 To activate LVS on a CTDB node you must specify the
346 CTDB_LVS_PUBLIC_IFACE, CTDB_LVS_PUBLIC_IP and CTDB_LVS_NODES
347 configuration variables. CTDB_LVS_NODES specifies a file containing
348 the private address of all nodes in the current node's LVS group.
349 Example:
351 CTDB_LVS_PUBLIC_IFACE=eth1
353 CTDB_LVS_PUBLIC_IP=10.1.1.237
354 CTDB_LVS_NODES=/etc/ctdb/lvs_nodes
355 Example /etc/ctdb/lvs_nodes:
358 192.168.1.2
360 192.168.1.3
361 192.168.1.4
362 Normally any node in an LVS group can act as the LVS master. Nodes that
365 are highly loaded due to other demands maybe flagged with the
366 "slave-only" option in the CTDB_LVS_NODES file to limit the LVS
367 functionality of those nodes.
368 LVS nodes file that excludes 192.168.1.4 from being the LVS master
370 node:
371 192.168.1.2
373 192.168.1.3
374 192.168.1.4 slave-only
375
378 CTDB tracks TCP connections from clients to public IP addresses, on
379 known ports. When an IP address moves from one node to another, all
380 existing TCP connections to that IP address are reset. The node taking
381 over this IP address will also send gratuitous ARPs (for IPv4, or
382 neighbour advertisement, for IPv6). This allows clients to reconnect
383 quickly, rather than waiting for TCP timeouts, which can be very long.
384 It is important that established TCP connections do not survive a
386 release and take of a public IP address on the same node. Such
387 connections can get out of sync with sequence and ACK numbers,
388 potentially causing a disruptive ACK storm.
389
391 NAT gateway (NATGW) is an optional feature that is used to configure
392 fallback routing for nodes. This allows cluster nodes to connect to
393 external services (e.g. DNS, AD, NIS and LDAP) when they do not host
394 any public addresses (e.g. when they are unhealthy).
395 This also applies to node startup because CTDB marks nodes as UNHEALTHY
397 until they have passed a "monitor" event. In this context, NAT gateway
398 helps to avoid a "chicken and egg" situation where a node needs to
399 access an external service to become healthy.
400 Another way of solving this type of problem is to assign an extra
402 static IP address to a public interface on every node. This is simpler
403 but it uses an extra IP address per node, while NAT gateway generally
404 uses only one extra IP address.
405 Operation
407 One extra NATGW public address is assigned on the public network to
408 each NATGW group. Each NATGW group is a set of nodes in the cluster
409 that shares the same NATGW address to talk to the outside world.
410 Normally there would only be one NATGW group spanning an entire
411 cluster, but in situations where one CTDB cluster spans multiple
412 physical sites it might be useful to have one NATGW group for each
413 site.
414 There can be multiple NATGW groups in a cluster but each node can only
416 be member of one NATGW group.
417 In each NATGW group, one of the nodes is selected by CTDB to be the
419 NATGW master and the other nodes are consider to be NATGW slaves. NATGW
420 slaves establish a fallback default route to the NATGW master via the
421 private network. When a NATGW slave hosts no public IP addresses then
422 it will use this route for outbound connections. The NATGW master hosts
423 the NATGW public IP address and routes outgoing connections from slave
424 nodes via this IP address. It also establishes a fallback default
425 route.
426 Configuration
428 NATGW is usually configured similar to the following example
429 configuration:
430 CTDB_NATGW_NODES=/etc/ctdb/natgw_nodes
432 CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24
433 CTDB_NATGW_PUBLIC_IP=10.0.0.227/24
434 CTDB_NATGW_PUBLIC_IFACE=eth0
435 CTDB_NATGW_DEFAULT_GATEWAY=10.0.0.1
436 Normally any node in a NATGW group can act as the NATGW master. Some
439 configurations may have special nodes that lack connectivity to a
440 public network. In such cases, those nodes can be flagged with the
441 "slave-only" option in the CTDB_NATGW_NODES file to limit the NATGW
442 functionality of those nodes.
443 See the NAT GATEWAY section in ctdb-script.options(5) for more details
445 of NATGW configuration.
446 Implementation details
448 When the NATGW functionality is used, one of the nodes is selected to
449 act as a NAT gateway for all the other nodes in the group when they
450 need to communicate with the external services. The NATGW master is
451 selected to be a node that is most likely to have usable networks.
452 The NATGW master hosts the NATGW public IP address CTDB_NATGW_PUBLIC_IP
454 on the configured public interfaces CTDB_NATGW_PUBLIC_IFACE and acts as
455 a router, masquerading outgoing connections from slave nodes via this
456 IP address. If CTDB_NATGW_DEFAULT_GATEWAY is set then it also
457 establishes a fallback default route to the configured this gateway
458 with a metric of 10. A metric 10 route is used so it can co-exist with
459 other default routes that may be available.
460 A NATGW slave establishes its fallback default route to the NATGW
462 master via the private network CTDB_NATGW_PRIVATE_NETWORKwith a metric
463 of 10. This route is used for outbound connections when no other
464 default route is available because the node hosts no public addresses.
465 A metric 10 routes is used so that it can co-exist with other default
466 routes that may be available when the node is hosting public addresses.
467 CTDB_NATGW_STATIC_ROUTES can be used to have NATGW create more specific
469 routes instead of just default routes.
470 This is implemented in the 11.natgw eventscript. Please see the
472 eventscript file and the NAT GATEWAY section in ctdb-script.options(5)
473 for more details.
474
476 Policy routing is an optional CTDB feature to support complex network
477 topologies. Public addresses may be spread across several different
478 networks (or VLANs) and it may not be possible to route packets from
479 these public addresses via the system's default route. Therefore, CTDB
480 has support for policy routing via the 13.per_ip_routing eventscript.
481 This allows routing to be specified for packets sourced from each
482 public address. The routes are added and removed as CTDB moves public
483 addresses between nodes.
484 Configuration variables
486 There are 4 configuration variables related to policy routing:
487 CTDB_PER_IP_ROUTING_CONF, CTDB_PER_IP_ROUTING_RULE_PREF,
488 CTDB_PER_IP_ROUTING_TABLE_ID_LOW, CTDB_PER_IP_ROUTING_TABLE_ID_HIGH.
489 See the POLICY ROUTING section in ctdb-script.options(5) for more
490 details.
491 Configuration
493 The format of each line of CTDB_PER_IP_ROUTING_CONF is:
494 <public_address> <network> [ <gateway> ]
496 Leading whitespace is ignored and arbitrary whitespace may be used as a
499 separator. Lines that have a "public address" item that doesn't match
500 an actual public address are ignored. This means that comment lines can
501 be added using a leading character such as '#', since this will never
502 match an IP address.
503 A line without a gateway indicates a link local route.
505 For example, consider the configuration line:
507 192.168.1.99 192.168.1.1/24
509 If the corresponding public_addresses line is:
512 192.168.1.99/24 eth2,eth3
514 CTDB_PER_IP_ROUTING_RULE_PREF is 100, and CTDB adds the address to eth2
517 then the following routing information is added:
518 ip rule add from 192.168.1.99 pref 100 table ctdb.192.168.1.99
520 ip route add 192.168.1.0/24 dev eth2 table ctdb.192.168.1.99
521 This causes traffic from 192.168.1.1 to 192.168.1.0/24 go via eth2.
524 The ip rule command will show (something like - depending on other
526 public addresses and other routes on the system):
527 0: from all lookup local
529 100: from 192.168.1.99 lookup ctdb.192.168.1.99
530 32766: from all lookup main
531 32767: from all lookup default
532 ip route show table ctdb.192.168.1.99 will show:
535 192.168.1.0/24 dev eth2 scope link
537 The usual use for a line containing a gateway is to add a default route
540 corresponding to a particular source address. Consider this line of
541 configuration:
542 192.168.1.99 0.0.0.0/0 192.168.1.1
544 In the situation described above this will cause an extra routing
547 command to be executed:
548 ip route add 0.0.0.0/0 via 192.168.1.1 dev eth2 table ctdb.192.168.1.99
550 With both configuration lines, ip route show table ctdb.192.168.1.99
553 will show:
554 192.168.1.0/24 dev eth2 scope link
556 default via 192.168.1.1 dev eth2
557 Sample configuration
560 Here is a more complete example configuration.
561 /etc/ctdb/public_addresses:
563 192.168.1.98 eth2,eth3
565 192.168.1.99 eth2,eth3
566 /etc/ctdb/policy_routing:
568 192.168.1.98 192.168.1.0/24
570 192.168.1.98 192.168.200.0/24 192.168.1.254
571 192.168.1.98 0.0.0.0/0 192.168.1.1
572 192.168.1.99 192.168.1.0/24
573 192.168.1.99 192.168.200.0/24 192.168.1.254
574 192.168.1.99 0.0.0.0/0 192.168.1.1
575 The routes local packets as expected, the default route is as
578 previously discussed, but packets to 192.168.200.0/24 are routed via
579 the alternate gateway 192.168.1.254.
580
582 When certain state changes occur in CTDB, it can be configured to
583 perform arbitrary actions via notifications. For example, sending SNMP
584 traps or emails when a node becomes unhealthy or similar.
585 The notification mechanism runs all executable files ending in
587 ".script" in /etc/ctdb/events/notification/, ignoring any failures and
588 continuing to run all files.
589 CTDB currently generates notifications after CTDB changes to these
591 states:
592 init
593 setup
594 startup
595 healthy
596 unhealthy
597
599 Valid log levels, in increasing order of verbosity, are:
600 ERROR
601 WARNING
602 NOTICE
603 INFO
604 DEBUG
605
607 It is possible to have a CTDB cluster that spans across a WAN link. For
608 example where you have a CTDB cluster in your datacentre but you also
609 want to have one additional CTDB node located at a remote branch site.
610 This is similar to how a WAN accelerator works but with the difference
611 that while a WAN-accelerator often acts as a Proxy or a MitM, in the
612 ctdb remote cluster node configuration the Samba instance at the remote
613 site IS the genuine server, not a proxy and not a MitM, and thus
614 provides 100% correct CIFS semantics to clients.
615 See the cluster as one single multihomed samba server where one of the
617 NICs (the remote node) is very far away.
618 NOTE: This does require that the cluster filesystem you use can cope
620 with WAN-link latencies. Not all cluster filesystems can handle
621 WAN-link latencies! Whether this will provide very good WAN-accelerator
622 performance or it will perform very poorly depends entirely on how
623 optimized your cluster filesystem is in handling high latency for data
624 and metadata operations.
625 To activate a node as being a remote cluster node you need to set the
627 following two parameters in /etc/ctdb/ctdb.conf for the remote node:
628 [legacy]
630 lmaster capability = false
631 recmaster capability = false
632 Verify with the command "ctdb getcapabilities" that that node no longer
635 has the recmaster or the lmaster capabilities.
636
638 ctdb(1), ctdbd(1), ctdbd_wrapper(1), ctdb_diagnostics(1), ltdbtool(1),
639 onnode(1), ping_pong(1), ctdb.conf(5), ctdb-script.options(5),
640 ctdb.sysconfig(5), ctdb-statistics(7), ctdb-tunables(7),
641 http://ctdb.samba.org/
642
644 This documentation was written by Ronnie Sahlberg, Amitay Isaacs,
645 Martin Schwenke
646
648 Copyright © 2007 Andrew Tridgell, Ronnie Sahlberg
649 This program is free software; you can redistribute it and/or modify it
651 under the terms of the GNU General Public License as published by the
652 Free Software Foundation; either version 3 of the License, or (at your
653 option) any later version.
654 This program is distributed in the hope that it will be useful, but
656 WITHOUT ANY WARRANTY; without even the implied warranty of
657 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
658 General Public License for more details.
659 You should have received a copy of the GNU General Public License along
661 with this program; if not, see http://www.gnu.org/licenses.
662ctdb 03/25/2021 CTDB(7)