1CTDB(7) CTDB - clustered TDB database CTDB(7)
2
6 ctdb - Clustered TDB
7
9 CTDB is a clustered database component in clustered Samba that provides
10 a high-availability load-sharing CIFS server cluster.
11 The main functions of CTDB are:
13 · Provide a clustered version of the TDB database with automatic
15 rebuild/recovery of the databases upon node failures.
16 · Monitor nodes in the cluster and services running on each node.
18 · Manage a pool of public IP addresses that are used to provide
20 services to clients. Alternatively, CTDB can be used with LVS.
21 Combined with a cluster filesystem CTDB provides a full
23 high-availablity (HA) environment for services such as clustered Samba,
24 NFS and other services.
25
27 A CTDB cluster is a collection of nodes with 2 or more network
28 interfaces. All nodes provide network (usually file/NAS) services to
29 clients. Data served by file services is stored on shared storage
30 (usually a cluster filesystem) that is accessible by all nodes.
31 CTDB provides an "all active" cluster, where services are load balanced
33 across all nodes.
34
36 CTDB uses a recovery lock to avoid a split brain, where a cluster
37 becomes partitioned and each partition attempts to operate
38 independently. Issues that can result from a split brain include file
39 data corruption, because file locking metadata may not be tracked
40 correctly.
41 CTDB uses a cluster leader and follower model of cluster management.
43 All nodes in a cluster elect one node to be the leader. The leader node
44 coordinates privileged operations such as database recovery and IP
45 address failover. CTDB refers to the leader node as the recovery
46 master. This node takes and holds the recovery lock to assert its
47 privileged role in the cluster.
48 By default, the recovery lock is implemented using a file (specified by
50 recovery lock in the [cluster] section of ctdb.conf(5)) residing in
51 shared storage (usually) on a cluster filesystem. To support a recovery
52 lock the cluster filesystem must support lock coherence. See
53 ping_pong(1) for more details.
54 The recovery lock can also be implemented using an arbitrary cluster
56 mutex call-out by using an exclamation point ('!') as the first
57 character of recovery lock. For example, a value of !/usr/bin/myhelper
58 recovery would run the given helper with the specified arguments. See
59 the source code relating to cluster mutexes for clues about writing
60 call-outs.
61 If a cluster becomes partitioned (for example, due to a communication
63 failure) and a different recovery master is elected by the nodes in
64 each partition, then only one of these recovery masters will be able to
65 take the recovery lock. The recovery master in the "losing" partition
66 will not be able to take the recovery lock and will be excluded from
67 the cluster. The nodes in the "losing" partition will elect each node
68 in turn as their recovery master so eventually all the nodes in that
69 partition will be excluded.
70 CTDB does sanity checks to ensure that the recovery lock is held as
72 expected.
73 CTDB can run without a recovery lock but this is not recommended as
75 there will be no protection from split brains.
76
78 Each node in a CTDB cluster has multiple IP addresses assigned to it:
79 · A single private IP address that is used for communication between
81 nodes.
82 · One or more public IP addresses that are used to provide NAS or
84 other services.
85 Private address
88 Each node is configured with a unique, permanently assigned private
89 address. This address is configured by the operating system. This
90 address uniquely identifies a physical node in the cluster and is the
91 address that CTDB daemons will use to communicate with the CTDB daemons
92 on other nodes.
93 Private addresses are listed in the file /etc/ctdb/nodes). This file
95 contains the list of private addresses for all nodes in the cluster,
96 one per line. This file must be the same on all nodes in the cluster.
97 Some users like to put this configuration file in their cluster
99 filesystem. A symbolic link should be used in this case.
100 Private addresses should not be used by clients to connect to services
102 provided by the cluster.
103 It is strongly recommended that the private addresses are configured on
105 a private network that is separate from client networks. This is
106 because the CTDB protocol is both unauthenticated and unencrypted. If
107 clients share the private network then steps need to be taken to stop
108 injection of packets to relevant ports on the private addresses. It is
109 also likely that CTDB protocol traffic between nodes could leak
110 sensitive information if it can be intercepted.
111 Example /etc/ctdb/nodes for a four node cluster:
113 192.168.1.1
115 192.168.1.2
116 192.168.1.3
117 192.168.1.4
118 Public addresses
121 Public addresses are used to provide services to clients. Public
122 addresses are not configured at the operating system level and are not
123 permanently associated with a particular node. Instead, they are
124 managed by CTDB and are assigned to interfaces on physical nodes at
125 runtime.
126 The CTDB cluster will assign/reassign these public addresses across the
128 available healthy nodes in the cluster. When one node fails, its public
129 addresses will be taken over by one or more other nodes in the cluster.
130 This ensures that services provided by all public addresses are always
131 available to clients, as long as there are nodes available capable of
132 hosting this address.
133 The public address configuration is stored in
135 /etc/ctdb/public_addresses on each node. This file contains a list of
136 the public addresses that the node is capable of hosting, one per line.
137 Each entry also contains the netmask and the interface to which the
138 address should be assigned. If this file is missing then no public
139 addresses are configured.
140 Some users who have the same public addresses on all nodes like to put
142 this configuration file in their cluster filesystem. A symbolic link
143 should be used in this case.
144 Example /etc/ctdb/public_addresses for a node that can host 4 public
146 addresses, on 2 different interfaces:
147 10.1.1.1/24 eth1
149 10.1.1.2/24 eth1
150 10.1.2.1/24 eth2
151 10.1.2.2/24 eth2
152 In many cases the public addresses file will be the same on all nodes.
155 However, it is possible to use different public address configurations
156 on different nodes.
157 Example: 4 nodes partitioned into two subgroups:
159 Node 0:/etc/ctdb/public_addresses
161 10.1.1.1/24 eth1
162 10.1.1.2/24 eth1
163 Node 1:/etc/ctdb/public_addresses
165 10.1.1.1/24 eth1
166 10.1.1.2/24 eth1
167 Node 2:/etc/ctdb/public_addresses
169 10.1.2.1/24 eth2
170 10.1.2.2/24 eth2
171 Node 3:/etc/ctdb/public_addresses
173 10.1.2.1/24 eth2
174 10.1.2.2/24 eth2
175 In this example nodes 0 and 1 host two public addresses on the 10.1.1.x
178 network while nodes 2 and 3 host two public addresses for the 10.1.2.x
179 network.
180 Public address 10.1.1.1 can be hosted by either of nodes 0 or 1 and
182 will be available to clients as long as at least one of these two nodes
183 are available.
184 If both nodes 0 and 1 become unavailable then public address 10.1.1.1
186 also becomes unavailable. 10.1.1.1 can not be failed over to nodes 2 or
187 3 since these nodes do not have this public address configured.
188 The ctdb ip command can be used to view the current assignment of
190 public addresses to physical nodes.
191
193 The current status of each node in the cluster can be viewed by the
194 ctdb status command.
195 A node can be in one of the following states:
197 OK
199 This node is healthy and fully functional. It hosts public
200 addresses to provide services.
201 DISCONNECTED
203 This node is not reachable by other nodes via the private network.
204 It is not currently participating in the cluster. It does not host
205 public addresses to provide services. It might be shut down.
206 DISABLED
208 This node has been administratively disabled. This node is
209 partially functional and participates in the cluster. However, it
210 does not host public addresses to provide services.
211 UNHEALTHY
213 A service provided by this node has failed a health check and
214 should be investigated. This node is partially functional and
215 participates in the cluster. However, it does not host public
216 addresses to provide services. Unhealthy nodes should be
217 investigated and may require an administrative action to rectify.
218 BANNED
220 CTDB is not behaving as designed on this node. For example, it may
221 have failed too many recovery attempts. Such nodes are banned from
222 participating in the cluster for a configurable time period before
223 they attempt to rejoin the cluster. A banned node does not host
224 public addresses to provide services. All banned nodes should be
225 investigated and may require an administrative action to rectify.
226 STOPPED
228 This node has been administratively exclude from the cluster. A
229 stopped node does no participate in the cluster and does not host
230 public addresses to provide services. This state can be used while
231 performing maintenance on a node.
232 PARTIALLYONLINE
234 A node that is partially online participates in a cluster like a
235 healthy (OK) node. Some interfaces to serve public addresses are
236 down, but at least one interface is up. See also ctdb ifaces.
237
239 Cluster nodes can have several different capabilities enabled. These
240 are listed below.
241 RECMASTER
243 Indicates that a node can become the CTDB cluster recovery master.
244 The current recovery master is decided via an election held by all
245 active nodes with this capability.
246 Default is YES.
248 LMASTER
250 Indicates that a node can be the location master (LMASTER) for
251 database records. The LMASTER always knows which node has the
252 latest copy of a record in a volatile database.
253 Default is YES.
255 The RECMASTER and LMASTER capabilities can be disabled when CTDB is
257 used to create a cluster spanning across WAN links. In this case CTDB
258 acts as a WAN accelerator.
259
261 LVS is a mode where CTDB presents one single IP address for the entire
262 cluster. This is an alternative to using public IP addresses and
263 round-robin DNS to loadbalance clients across the cluster.
264 This is similar to using a layer-4 loadbalancing switch but with some
266 restrictions.
267 One extra LVS public address is assigned on the public network to each
269 LVS group. Each LVS group is a set of nodes in the cluster that
270 presents the same LVS address public address to the outside world.
271 Normally there would only be one LVS group spanning an entire cluster,
272 but in situations where one CTDB cluster spans multiple physical sites
273 it might be useful to have one LVS group for each site. There can be
274 multiple LVS groups in a cluster but each node can only be member of
275 one LVS group.
276 Client access to the cluster is load-balanced across the HEALTHY nodes
278 in an LVS group. If no HEALTHY nodes exists then all nodes in the group
279 are used, regardless of health status. CTDB will, however never
280 load-balance LVS traffic to nodes that are BANNED, STOPPED, DISABLED or
281 DISCONNECTED. The ctdb lvs command is used to show which nodes are
282 currently load-balanced across.
283 In each LVS group, one of the nodes is selected by CTDB to be the LVS
285 master. This node receives all traffic from clients coming in to the
286 LVS public address and multiplexes it across the internal network to
287 one of the nodes that LVS is using. When responding to the client, that
288 node will send the data back directly to the client, bypassing the LVS
289 master node. The command ctdb lvs master will show which node is the
290 current LVS master.
291 The path used for a client I/O is:
293 1. Client sends request packet to LVSMASTER.
295 2. LVSMASTER passes the request on to one node across the internal
297 network.
298 3. Selected node processes the request.
300 4. Node responds back to client.
302 This means that all incoming traffic to the cluster will pass through
304 one physical node, which limits scalability. You can send more data to
305 the LVS address that one physical node can multiplex. This means that
306 you should not use LVS if your I/O pattern is write-intensive since you
307 will be limited in the available network bandwidth that node can
308 handle. LVS does work very well for read-intensive workloads where only
309 smallish READ requests are going through the LVSMASTER bottleneck and
310 the majority of the traffic volume (the data in the read replies) goes
311 straight from the processing node back to the clients. For
312 read-intensive i/o patterns you can achieve very high throughput rates
313 in this mode.
314 Note: you can use LVS and public addresses at the same time.
316 If you use LVS, you must have a permanent address configured for the
318 public interface on each node. This address must be routable and the
319 cluster nodes must be configured so that all traffic back to client
320 hosts are routed through this interface. This is also required in order
321 to allow samba/winbind on the node to talk to the domain controller.
322 This LVS IP address can not be used to initiate outgoing traffic.
323 Make sure that the domain controller and the clients are reachable from
325 a node before you enable LVS. Also ensure that outgoing traffic to
326 these hosts is routed out through the configured public interface.
327 Configuration
329 To activate LVS on a CTDB node you must specify the
330 CTDB_LVS_PUBLIC_IFACE, CTDB_LVS_PUBLIC_IP and CTDB_LVS_NODES
331 configuration variables. CTDB_LVS_NODES specifies a file containing
332 the private address of all nodes in the current node's LVS group.
333 Example:
335 CTDB_LVS_PUBLIC_IFACE=eth1
337 CTDB_LVS_PUBLIC_IP=10.1.1.237
338 CTDB_LVS_NODES=/etc/ctdb/lvs_nodes
339 Example /etc/ctdb/lvs_nodes:
342 192.168.1.2
344 192.168.1.3
345 192.168.1.4
346 Normally any node in an LVS group can act as the LVS master. Nodes that
349 are highly loaded due to other demands maybe flagged with the
350 "slave-only" option in the CTDB_LVS_NODES file to limit the LVS
351 functionality of those nodes.
352 LVS nodes file that excludes 192.168.1.4 from being the LVS master
354 node:
355 192.168.1.2
357 192.168.1.3
358 192.168.1.4 slave-only
359
362 CTDB tracks TCP connections from clients to public IP addresses, on
363 known ports. When an IP address moves from one node to another, all
364 existing TCP connections to that IP address are reset. The node taking
365 over this IP address will also send gratuitous ARPs (for IPv4, or
366 neighbour advertisement, for IPv6). This allows clients to reconnect
367 quickly, rather than waiting for TCP timeouts, which can be very long.
368 It is important that established TCP connections do not survive a
370 release and take of a public IP address on the same node. Such
371 connections can get out of sync with sequence and ACK numbers,
372 potentially causing a disruptive ACK storm.
373
375 NAT gateway (NATGW) is an optional feature that is used to configure
376 fallback routing for nodes. This allows cluster nodes to connect to
377 external services (e.g. DNS, AD, NIS and LDAP) when they do not host
378 any public addresses (e.g. when they are unhealthy).
379 This also applies to node startup because CTDB marks nodes as UNHEALTHY
381 until they have passed a "monitor" event. In this context, NAT gateway
382 helps to avoid a "chicken and egg" situation where a node needs to
383 access an external service to become healthy.
384 Another way of solving this type of problem is to assign an extra
386 static IP address to a public interface on every node. This is simpler
387 but it uses an extra IP address per node, while NAT gateway generally
388 uses only one extra IP address.
389 Operation
391 One extra NATGW public address is assigned on the public network to
392 each NATGW group. Each NATGW group is a set of nodes in the cluster
393 that shares the same NATGW address to talk to the outside world.
394 Normally there would only be one NATGW group spanning an entire
395 cluster, but in situations where one CTDB cluster spans multiple
396 physical sites it might be useful to have one NATGW group for each
397 site.
398 There can be multiple NATGW groups in a cluster but each node can only
400 be member of one NATGW group.
401 In each NATGW group, one of the nodes is selected by CTDB to be the
403 NATGW master and the other nodes are consider to be NATGW slaves. NATGW
404 slaves establish a fallback default route to the NATGW master via the
405 private network. When a NATGW slave hosts no public IP addresses then
406 it will use this route for outbound connections. The NATGW master hosts
407 the NATGW public IP address and routes outgoing connections from slave
408 nodes via this IP address. It also establishes a fallback default
409 route.
410 Configuration
412 NATGW is usually configured similar to the following example
413 configuration:
414 CTDB_NATGW_NODES=/etc/ctdb/natgw_nodes
416 CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24
417 CTDB_NATGW_PUBLIC_IP=10.0.0.227/24
418 CTDB_NATGW_PUBLIC_IFACE=eth0
419 CTDB_NATGW_DEFAULT_GATEWAY=10.0.0.1
420 Normally any node in a NATGW group can act as the NATGW master. Some
423 configurations may have special nodes that lack connectivity to a
424 public network. In such cases, those nodes can be flagged with the
425 "slave-only" option in the CTDB_NATGW_NODES file to limit the NATGW
426 functionality of those nodes.
427 See the NAT GATEWAY section in ctdb-script.options(5) for more details
429 of NATGW configuration.
430 Implementation details
432 When the NATGW functionality is used, one of the nodes is selected to
433 act as a NAT gateway for all the other nodes in the group when they
434 need to communicate with the external services. The NATGW master is
435 selected to be a node that is most likely to have usable networks.
436 The NATGW master hosts the NATGW public IP address CTDB_NATGW_PUBLIC_IP
438 on the configured public interfaces CTDB_NATGW_PUBLIC_IFACE and acts as
439 a router, masquerading outgoing connections from slave nodes via this
440 IP address. If CTDB_NATGW_DEFAULT_GATEWAY is set then it also
441 establishes a fallback default route to the configured this gateway
442 with a metric of 10. A metric 10 route is used so it can co-exist with
443 other default routes that may be available.
444 A NATGW slave establishes its fallback default route to the NATGW
446 master via the private network CTDB_NATGW_PRIVATE_NETWORKwith a metric
447 of 10. This route is used for outbound connections when no other
448 default route is available because the node hosts no public addresses.
449 A metric 10 routes is used so that it can co-exist with other default
450 routes that may be available when the node is hosting public addresses.
451 CTDB_NATGW_STATIC_ROUTES can be used to have NATGW create more specific
453 routes instead of just default routes.
454 This is implemented in the 11.natgw eventscript. Please see the
456 eventscript file and the NAT GATEWAY section in ctdb-script.options(5)
457 for more details.
458
460 Policy routing is an optional CTDB feature to support complex network
461 topologies. Public addresses may be spread across several different
462 networks (or VLANs) and it may not be possible to route packets from
463 these public addresses via the system's default route. Therefore, CTDB
464 has support for policy routing via the 13.per_ip_routing eventscript.
465 This allows routing to be specified for packets sourced from each
466 public address. The routes are added and removed as CTDB moves public
467 addresses between nodes.
468 Configuration variables
470 There are 4 configuration variables related to policy routing:
471 CTDB_PER_IP_ROUTING_CONF, CTDB_PER_IP_ROUTING_RULE_PREF,
472 CTDB_PER_IP_ROUTING_TABLE_ID_LOW, CTDB_PER_IP_ROUTING_TABLE_ID_HIGH.
473 See the POLICY ROUTING section in ctdb-script.options(5) for more
474 details.
475 Configuration
477 The format of each line of CTDB_PER_IP_ROUTING_CONF is:
478 <public_address> <network> [ <gateway> ]
480 Leading whitespace is ignored and arbitrary whitespace may be used as a
483 separator. Lines that have a "public address" item that doesn't match
484 an actual public address are ignored. This means that comment lines can
485 be added using a leading character such as '#', since this will never
486 match an IP address.
487 A line without a gateway indicates a link local route.
489 For example, consider the configuration line:
491 192.168.1.99 192.168.1.1/24
493 If the corresponding public_addresses line is:
496 192.168.1.99/24 eth2,eth3
498 CTDB_PER_IP_ROUTING_RULE_PREF is 100, and CTDB adds the address to eth2
501 then the following routing information is added:
502 ip rule add from 192.168.1.99 pref 100 table ctdb.192.168.1.99
504 ip route add 192.168.1.0/24 dev eth2 table ctdb.192.168.1.99
505 This causes traffic from 192.168.1.1 to 192.168.1.0/24 go via eth2.
508 The ip rule command will show (something like - depending on other
510 public addresses and other routes on the system):
511 0: from all lookup local
513 100: from 192.168.1.99 lookup ctdb.192.168.1.99
514 32766: from all lookup main
515 32767: from all lookup default
516 ip route show table ctdb.192.168.1.99 will show:
519 192.168.1.0/24 dev eth2 scope link
521 The usual use for a line containing a gateway is to add a default route
524 corresponding to a particular source address. Consider this line of
525 configuration:
526 192.168.1.99 0.0.0.0/0 192.168.1.1
528 In the situation described above this will cause an extra routing
531 command to be executed:
532 ip route add 0.0.0.0/0 via 192.168.1.1 dev eth2 table ctdb.192.168.1.99
534 With both configuration lines, ip route show table ctdb.192.168.1.99
537 will show:
538 192.168.1.0/24 dev eth2 scope link
540 default via 192.168.1.1 dev eth2
541 Sample configuration
544 Here is a more complete example configuration.
545 /etc/ctdb/public_addresses:
547 192.168.1.98 eth2,eth3
549 192.168.1.99 eth2,eth3
550 /etc/ctdb/policy_routing:
552 192.168.1.98 192.168.1.0/24
554 192.168.1.98 192.168.200.0/24 192.168.1.254
555 192.168.1.98 0.0.0.0/0 192.168.1.1
556 192.168.1.99 192.168.1.0/24
557 192.168.1.99 192.168.200.0/24 192.168.1.254
558 192.168.1.99 0.0.0.0/0 192.168.1.1
559 The routes local packets as expected, the default route is as
562 previously discussed, but packets to 192.168.200.0/24 are routed via
563 the alternate gateway 192.168.1.254.
564
566 When certain state changes occur in CTDB, it can be configured to
567 perform arbitrary actions via notifications. For example, sending SNMP
568 traps or emails when a node becomes unhealthy or similar.
569 The notification mechanism runs all executable files ending in
571 ".script" in /etc/ctdb/events/notification/, ignoring any failures and
572 continuing to run all files.
573 CTDB currently generates notifications after CTDB changes to these
575 states:
576 init
577 setup
578 startup
579 healthy
580 unhealthy
581
583 Valid log levels, in increasing order of verbosity, are:
584 ERROR
585 WARNING
586 NOTICE
587 INFO
588 DEBUG
589
591 It is possible to have a CTDB cluster that spans across a WAN link. For
592 example where you have a CTDB cluster in your datacentre but you also
593 want to have one additional CTDB node located at a remote branch site.
594 This is similar to how a WAN accelerator works but with the difference
595 that while a WAN-accelerator often acts as a Proxy or a MitM, in the
596 ctdb remote cluster node configuration the Samba instance at the remote
597 site IS the genuine server, not a proxy and not a MitM, and thus
598 provides 100% correct CIFS semantics to clients.
599 See the cluster as one single multihomed samba server where one of the
601 NICs (the remote node) is very far away.
602 NOTE: This does require that the cluster filesystem you use can cope
604 with WAN-link latencies. Not all cluster filesystems can handle
605 WAN-link latencies! Whether this will provide very good WAN-accelerator
606 performance or it will perform very poorly depends entirely on how
607 optimized your cluster filesystem is in handling high latency for data
608 and metadata operations.
609 To activate a node as being a remote cluster node you need to set the
611 following two parameters in /etc/ctdb/ctdb.conf for the remote node:
612 [legacy]
614 lmaster capability = false
615 recmaster capability = false
616 Verify with the command "ctdb getcapabilities" that that node no longer
619 has the recmaster or the lmaster capabilities.
620
622 ctdb(1), ctdbd(1), ctdbd_wrapper(1), ctdb_diagnostics(1), ltdbtool(1),
623 onnode(1), ping_pong(1), ctdb.conf(5), ctdb-script.options(5),
624 ctdb.sysconfig(5), ctdb-statistics(7), ctdb-tunables(7),
625 http://ctdb.samba.org/
626
628 This documentation was written by Ronnie Sahlberg, Amitay Isaacs,
629 Martin Schwenke
630
632 Copyright © 2007 Andrew Tridgell, Ronnie Sahlberg
633 This program is free software; you can redistribute it and/or modify it
635 under the terms of the GNU General Public License as published by the
636 Free Software Foundation; either version 3 of the License, or (at your
637 option) any later version.
638 This program is distributed in the hope that it will be useful, but
640 WITHOUT ANY WARRANTY; without even the implied warranty of
641 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
642 General Public License for more details.
643 You should have received a copy of the GNU General Public License along
645 with this program; if not, see http://www.gnu.org/licenses.
646ctdb 05/11/2019 CTDB(7)