1COROSYNC_CONF(5) Corosync Cluster Engine Programmer's Manual COROSYNC_CONF(5)
2
6 corosync.conf - corosync executive configuration file
7
10 /etc/corosync/corosync.conf
11
14 The corosync.conf instructs the corosync executive about various param‐
15 eters needed to control the corosync executive. Empty lines and lines
16 starting with # character are ignored. The configuration file consists
17 of bracketed top level directives. The possible directive choices are:
18 totem { }
21 This top level directive contains configuration options for the
22 totem protocol.
23 logging { }
25 This top level directive contains configuration options for log‐
26 ging.
27 quorum { }
29 This top level directive contains configuration options for quo‐
30 rum.
31 nodelist { }
33 This top level directive contains configuration options for
34 nodes in cluster.
35 system { }
37 This top level directive contains configuration options related
38 to system.
39 resources { }
41 This top level directive contains configuration options for re‐
42 sources.
43 nozzle { }
45 This top level directive contains configuration options for a
46 libnozzle device.
47 The interface sub-directive of totem is optional for UDP and knet
50 transports.
51 For knet, multiple interface subsections define parameters for each
53 knet link on the system.
54 For UDPU an interface section is not needed and it is recommended that
56 the nodelist is used to define cluster nodes.
57 linknumber
60 This specifies the link number for the interface. When using
61 the knet protocol, each interface should specify separate link
62 numbers to uniquely identify to the membership protocol which
63 interface to use for which link. The linknumber must start at
64 0. For UDP the only supported linknumber is 0.
65 knet_link_priority
68 This specifies the priority for the link when knet is used in
69 'passive' mode. (see link_mode below)
70 knet_ping_interval
73 This specifies the interval between knet link pings.
74 knet_ping_interval and knet_ping_timeout are a pair, if one is
75 specified the other should be too, otherwise one will be calcu‐
76 lated from the token timeout and one will be taken from the con‐
77 fig file. (default is token timeout / (knet_pong_count*2))
78 knet_ping_timeout
81 If no ping is received within this time, the knet link is de‐
82 clared dead. knet_ping_interval and knet_ping_timeout are a
83 pair, if one is specified the other should be too, otherwise one
84 will be calculated from the token timeout and one will be taken
85 from the config file. (default is token timeout /
86 knet_pong_count)
87 knet_ping_precision
90 How many values of latency are used to calculate the average
91 link latency. (default 2048 samples)
92 knet_pong_count
95 How many valid ping/pongs before a link is marked UP. (default
96 2)
97 knet_transport
100 Which IP transport knet should use. valid values are "sctp" or
101 "udp". (default: udp)
102 bindnetaddr (udp only)
105 This specifies the network address the corosync executive should
106 bind to when using udp.
107 bindnetaddr (udp only) should be an IP address configured on the
109 system, or a network address.
110 For example, if the local interface is 192.168.5.92 with netmask
112 255.255.255.0, you should set bindnetaddr to 192.168.5.92 or
113 192.168.5.0. If the local interface is 192.168.5.92 with net‐
114 mask 255.255.255.192, set bindnetaddr to 192.168.5.92 or
115 192.168.5.64, and so forth.
116 This may also be an IPV6 address, in which case IPV6 networking
118 will be used. In this case, the exact address must be specified
119 and there is no automatic selection of the network interface
120 within a specific subnet as with IPv4.
121 If IPv6 networking is used, the nodeid field in nodelist must be
123 specified.
124 broadcast (udp only)
127 This is optional and can be set to yes. If it is set to yes,
128 the broadcast address will be used for communication. If this
129 option is set, mcastaddr should not be set.
130 mcastaddr (udp only)
133 This is the multicast address used by corosync executive. The
134 default should work for most networks, but the network adminis‐
135 trator should be queried about a multicast address to use.
136 Avoid 224.x.x.x because this is a "config" multicast address.
137 This may also be an IPV6 multicast address, in which case IPV6
139 networking will be used. If IPv6 networking is used, the nodeid
140 field in nodelist must be specified.
141 It's not necessary to use this option if cluster_name option is
143 used. If both options are used, mcastaddr has higher priority.
144 mcastport (udp only)
147 This specifies the UDP port number. It is possible to use the
148 same multicast address on a network with the corosync services
149 configured for different UDP ports. Please note corosync uses
150 two UDP ports mcastport (for mcast receives) and mcastport - 1
151 (for mcast sends). If you have multiple clusters on the same
152 network using the same mcastaddr please configure the mcastports
153 with a gap.
154 ttl (udp only)
157 This specifies the Time To Live (TTL). If you run your cluster
158 on a routed network then the default of "1" will be too small.
159 This option provides a way to increase this up to 255. The valid
160 range is 0..255.
161 Within the totem directive, there are seven configuration options of
164 which one is required, five are optional, and one is required when IPV6
165 is configured in the interface subdirective. The required directive
166 controls the version of the totem configuration. The optional option
167 unless using IPV6 directive controls identification of the processor.
168 The optional options control secrecy and authentication, the network
169 mode of operation and maximum network MTU field.
170 version
173 This specifies the version of the configuration file. Currently
174 the only valid version for this directive is 2.
175 clear_node_high_bit
178 This configuration option is optional and is only relevant when
179 no nodeid is specified. Some corosync clients require a signed
180 32 bit nodeid that is greater than zero however by default
181 corosync uses all 32 bits of the IPv4 address space when gener‐
182 ating a nodeid. Set this option to yes to force the high bit to
183 be zero and therefore ensure the nodeid is a positive signed 32
184 bit integer.
185 WARNING: Cluster behavior is undefined if this option is enabled
187 on only a subset of the cluster (for example during a rolling
188 upgrade).
189 crypto_model
192 This specifies which cryptographic library should be used by
193 knet. Supported values depend on the libknet build and on the
194 installed cryptography libraries. Typically nss and openssl will
195 be available but gcrypt and others could also be allowed.
196 The default is nss.
198 crypto_hash
201 This specifies which HMAC authentication should be used to au‐
202 thenticate all messages. Valid values are none (no authentica‐
203 tion), md5, sha1, sha256, sha384 and sha512. Encrypted transmis‐
204 sion is only supported for the knet transport.
205 The default is none.
207 crypto_cipher
210 This specifies which cipher should be used to encrypt all mes‐
211 sages. Valid values are none (no encryption), aes256, aes192
212 and aes128. Enabling crypto_cipher, requires also enabling of
213 crypto_hash. Encrypted transmission is only supported for the
214 knet transport.
215 The default is none.
217 secauth
220 This implies crypto_cipher=aes256 and crypto_hash=sha256, unless
221 those options are explicitly set. Encrypted transmission is only
222 supported for the knet transport.
223 The default is off.
225 keyfile
228 This specifies the fully qualified path to the shared key used
229 to authenticate and encrypt data used within the Totem protocol.
230 The default is /etc/corosync/authkey.
232 key Shared key stored in configuration instead of authkey file. This
235 option has lower precedence than keyfile option so it's used
236 only when keyfile is not specified. Using this option is not
237 recommended for security reasons.
238 link_mode
241 This specifies the Kronosnet mode, which may be passive, active,
242 or rr (round-robin). passive: the active link with the highest
243 priority (highest number) will be used. If one or more links
244 share the same priority the one with the lowest link ID will be
245 used. active: All active links will be used simultaneously to
246 send traffic. link priority is ignored. rr: Round-Robin pol‐
247 icy. Each packet will be sent to the next active link in order.
248 If only one interface directive is specified, passive is auto‐
250 matically chosen.
251 The maximum number of interface directives that is allowed with
253 Kronosnet is 8. For other transports it is 1.
254 netmtu This specifies maximum packet length sent by corosync. It's
257 mainly for the UDPU (and UDP) transport, where it specifies the
258 network maximum transmit size, but can be used also with the
259 KNET transport, where it defines the maximum length of packets
260 passed to the knet layer. To specify the network MTU manually
261 for KNET, use the knet_mtu option.
262 For UDPU (and UDP), setting this value beyond 1500, the regular
264 frame MTU, requires ethernet devices that support large, or also
265 called jumbo, frames. If any device in the network doesn't sup‐
266 port large frames, the protocol will not operate properly. The
267 hosts must also have their mtu size set from 1500 to whatever
268 frame size is specified here.
269 Please note while some NICs or switches claim large frame sup‐
271 port, they support 9000 MTU as the maximum frame size including
272 the IP header. Setting the netmtu and host MTUs to 9000 will
273 cause totem to use the full 9000 bytes of the frame. Then Linux
274 will add a 18 byte header moving the full frame size to 9018.
275 As a result some hardware will not operate properly with this
276 size of data. A netmtu of 8982 seems to work for the few large
277 frame devices that have been tested. Some manufacturers claim
278 large frame support when in fact they support frame sizes of
279 4500 bytes.
280 When sending multicast traffic, if the network frequently recon‐
282 figures, chances are that some device in the network doesn't
283 support large frames.
284 Choose hardware carefully if intending to use large frame sup‐
286 port.
287 The default is 1500 for UDPU (and UDP) and 65536 for the KNET
289 transport.
290 transport
293 This directive controls the transport mechanism used. The de‐
294 fault is knet. The transport type can also be set to udpu or
295 udp. Only knet allows crypto or multiple interfaces per node.
296 cluster_name
299 This specifies the name of cluster and it's used for automatic
300 generating of multicast address.
301 config_version
304 This specifies version of config file. This is converted to un‐
305 signed 64-bit int. By default it's 0. Option is used to prevent
306 joining old nodes with not up-to-date configuration. If value is
307 not 0, and node is going for first time (only for first time,
308 join after split doesn't follow this rules) from single-node
309 membership to multiple nodes membership, other nodes config_ver‐
310 sions are collected. If current node config_version is not equal
311 to highest of collected versions, corosync is terminated.
312 ip_version
315 This specifies version of IP to ask DNS resolver for. The value
316 can be one of ipv4 (look only for an IPv4 address) , ipv6 (check
317 only IPv6 address) , ipv4-6 (look for all address families and
318 use first IPv4 address found in the list if there is such ad‐
319 dress, otherwise use first IPv6 address) and ipv6-4 (look for
320 all address families and use first IPv6 address found in the
321 list if there is such address, otherwise use first IPv4 ad‐
322 dress).
323 Default (if unspecified) is ipv6-4 for knet and udpu transports
325 and ipv4 for udp.
326 The knet transport supports IPv4 and IPv6 addresses concur‐
328 rently, provided they are consistent on each link.
329 Within the totem directive, there are several configuration op‐
331 tions which are used to control the operation of the protocol.
332 It is generally not recommended to change any of these values
333 without proper guidance and sufficient testing. Some networks
334 may require larger values if suffering from frequent reconfigu‐
335 rations. Some applications may require faster failure detection
336 times which can be achieved by reducing the token timeout.
337 token This timeout is used directly or as a base for real token time‐
340 out calculation (explained in token_coefficient section). Token
341 timeout specifies in milliseconds until a token loss is declared
342 after not receiving a token. This is the time spent detecting a
343 failure of a processor in the current configuration. Reforming
344 a new configuration takes about 50 milliseconds in addition to
345 this timeout.
346 For real token timeout used by totem it's possible to read cmap
348 value of runtime.config.totem.token key.
349 Be careful to use the same timeout values on each of the nodes
351 in the cluster or unpredictable results may occur.
352 The default is 3000 milliseconds.
354 token_warning
357 Specifies the interval between warnings that the token has not
358 been received. The value is a percentage of the token timeout
359 and can be set to 0 to disable warnings.
360 The default is 75%.
362 token_coefficient
365 This value is used only when nodelist section is specified and
366 contains at least 3 nodes. If so, real token timeout is then
367 computed as token + (number_of_nodes - 2) * token_coefficient.
368 This allows cluster to scale without manually changing token
369 timeout every time new node is added. This value can be set to 0
370 resulting in effective removal of this feature.
371 The default is 650 milliseconds.
373 token_retransmit
376 This timeout specifies in milliseconds after how long before re‐
377 ceiving a token the token is retransmitted. This will be auto‐
378 matically calculated if token is modified. It is not recom‐
379 mended to alter this value without guidance from the corosync
380 community.
381 The minimum is 30 milliseconds. If not set and error occur, make
383 sure token / (token_retransmits_before_loss_const + 0.2) is more
384 than 30.
385 The default is 238 milliseconds for two nodes cluster. Three or
387 more nodes reference token_coefficient.
388 knet_compression_model
391 Type of compression used by Kronosnet. Supported values depend
392 on the libknet build and on the installed compression libraries.
393 Typically zlib and lz4 will be available but bzip2 and others
394 could also be allowed. The default is 'none'.
395 knet_compression_threshold
398 Tells knet to NOT compress any packets that are smaller than the
399 value indicated. Default 100 bytes.
400 Set to 0 to reset to the default. Set to 1 to compress every‐
402 thing.
403 knet_compression_level
406 Many compression libraries allow tuning of compression parame‐
407 ters. For example 0 or 1 ... 9 are commonly used to determine
408 the level of compression. This value is passed unmodified to the
409 compression library so it is recommended to consult the li‐
410 brary's documentation for more detailed information.
411 hold This timeout specifies in milliseconds how long the token should
414 be held by the representative when the protocol is under low
415 utilization. It is not recommended to alter this value without
416 guidance from the corosync community.
417 The default is 180 milliseconds.
419 token_retransmits_before_loss_const
422 This value identifies how many token retransmits should be at‐
423 tempted before forming a new configuration. It is also used for
424 token_retransmit and hold calculations.
425 The default is 4 retransmissions.
427 join This timeout specifies in milliseconds how long to wait for join
430 messages in the membership protocol.
431 The default is 50 milliseconds.
433 send_join
436 This timeout specifies in milliseconds an upper range between 0
437 and send_join to wait before sending a join message. For con‐
438 figurations with less than 32 nodes, this parameter is not nec‐
439 essary. For larger rings, this parameter is necessary to ensure
440 the NIC is not overflowed with join messages on formation of a
441 new ring. A reasonable value for large rings (128 nodes) would
442 be 80msec. Other timer values must also change if this value is
443 changed. Seek advice from the corosync mailing list if trying
444 to run larger configurations.
445 The default is 0 milliseconds.
447 consensus
450 This timeout specifies in milliseconds how long to wait for con‐
451 sensus to be achieved before starting a new round of membership
452 configuration. The minimum value for consensus must be 1.2 *
453 token. This value will be automatically calculated at 1.2 * to‐
454 ken if the user doesn't specify a consensus value.
455 For two node clusters, a consensus larger than the join timeout
457 but less than token is safe. For three node or larger clusters,
458 consensus should be larger than token. There is an increasing
459 risk of odd membership changes, which still guarantee virtual
460 synchrony, as node count grows if consensus is less than token.
461 The default is 3600 milliseconds.
463 merge This timeout specifies in milliseconds how long to wait before
466 checking for a partition when no multicast traffic is being
467 sent. If multicast traffic is being sent, the merge detection
468 happens automatically as a function of the protocol.
469 The default is 200 milliseconds.
471 downcheck
474 This timeout specifies in milliseconds how long to wait before
475 checking that a network interface is back up after it has been
476 downed.
477 The default is 1000 milliseconds.
479 fail_recv_const
482 This constant specifies how many rotations of the token without
483 receiving any of the messages when messages should be received
484 may occur before a new configuration is formed.
485 The default is 2500 failures to receive a message.
487 seqno_unchanged_const
490 This constant specifies how many rotations of the token without
491 any multicast traffic should occur before the hold timer is
492 started.
493 The default is 30 rotations.
495 heartbeat_failures_allowed
498 [HeartBeating mechanism] Configures the optional HeartBeating
499 mechanism for faster failure detection. Keep in mind that engag‐
500 ing this mechanism in lossy networks could cause faulty loss
501 declaration as the mechanism relies on the network for heart‐
502 beating.
503 So as a rule of thumb use this mechanism if you require improved
505 failure in low to medium utilized networks.
506 This constant specifies the number of heartbeat failures the
508 system should tolerate before declaring heartbeat failure e.g 3.
509 Also if this value is not set or is 0 then the heartbeat mecha‐
510 nism is not engaged in the system and token rotation is the
511 method of failure detection
512 The default is 0 (disabled).
514 max_network_delay
517 [HeartBeating mechanism] This constant specifies in milliseconds
518 the approximate delay that your network takes to transport one
519 packet from one machine to another. This value is to be set by
520 system engineers and please don't change if not sure as this ef‐
521 fects the failure detection mechanism using heartbeat.
522 The default is 50 milliseconds.
524 window_size
527 This constant specifies the maximum number of messages that may
528 be sent on one token rotation. If all processors perform
529 equally well, this value could be large (300), which would in‐
530 troduce higher latency from origination to delivery for very
531 large rings. To reduce latency in large rings(16+), the de‐
532 faults are a safe compromise. If 1 or more slow processor(s)
533 are present among fast processors, window_size should be no
534 larger than 256000 / netmtu to avoid overflow of the kernel re‐
535 ceive buffers. The user is notified of this by the display of a
536 retransmit list in the notification logs. There is no loss of
537 data, but performance is reduced when these errors occur.
538 The default is 50 messages.
540 max_messages
543 This constant specifies the maximum number of messages that may
544 be sent by one processor on receipt of the token. The max_mes‐
545 sages parameter is limited to 256000 / netmtu to prevent over‐
546 flow of the kernel transmit buffers.
547 The default is 17 messages.
549 miss_count_const
552 This constant defines the maximum number of times on receipt of
553 a token a message is checked for retransmission before a re‐
554 transmission occurs. This parameter is useful to modify for
555 switches that delay multicast packets compared to unicast pack‐
556 ets. The default setting works well for nearly all modern
557 switches.
558 The default is 5 messages.
560 knet_pmtud_interval
563 How often the knet PMTUd runs to look for network MTU changes.
564 Value in seconds, default: 30
565 knet_mtu
568 Switch between manual and automatic MTU discovery. A value of 0
569 means automatic, other values set a manual MTU. In a setup with
570 multiple interfaces, please specify the lowest MTU of the se‐
571 lected interfaces.
572 The default value is 0.
574 block_unlisted_ips
577 Allow UDPU and KNET to drop packets from IP addresses that are
578 not known (nodes which don't exist in the nodelist) to corosync.
579 Value is yes or no.
580 This feature is mainly to protect against the joining of nodes
582 with outdated configurations after a cluster split. Another use
583 case is to allow the atomic merge of two independent clusters.
584 Changing the default value is not recommended, the overhead is
586 tiny and an existing cluster may fail if corosync is started on
587 an unlisted node with an old configuration.
588 The default value is yes.
590 cancel_token_hold_on_retransmit
593 Allows Corosync to hold token by representative when there is
594 too much retransmit messages. This allows network to process in‐
595 creased load without overloading it. Used mechanism is same as
596 described for hold directive.
597 Some deployments may prefer to never hold token when there is
599 retransmit messages. If so, option should be set to yes.
600 The default value is no.
602 Within the logging directive, there are several configuration options
605 which are all optional.
606 The following 3 options are valid only for the top level logging direc‐
609 tive:
610 timestamp
613 This specifies that a timestamp is placed on all log messages.
614 It can be one of off (no timestamp), on (second precision time‐
615 stamp) or hires (millisecond precision timestamp - only when
616 supported by LibQB).
617 The default is hires (or on if hires is not supported).
619 fileline
622 This specifies that file and line should be printed.
623 The default is off.
625 function_name
628 This specifies that the code function name should be printed.
629 The default is off.
631 blackbox
634 This specifies that blackbox functionality should be enabled.
635 The default is on.
637 The following options are valid both for top level logging directive
640 and they can be overridden in logger_subsys entries.
641 to_stderr
644 to_logfile
646 to_syslog
648 These specify the destination of logging output. Any combination
649 of these options may be specified. Valid options are yes and no.
650 The default is syslog and stderr.
652 Please note, if you are using to_logfile and want to rotate the
654 file, use logrotate(8) with the option copytruncate. eg.
655 /var/log/corosync.log {
656 missingok
657 compress
658 notifempty
659 daily
660 rotate 7
661 copytruncate
662 }
663 logfile
666 If the to_logfile directive is set to yes , this option speci‐
667 fies the pathname of the log file.
668 No default.
670 logfile_priority
673 This specifies the logfile priority for this particular subsys‐
674 tem. Ignored if debug is on. Possible values are: alert, crit,
675 debug (same as debug = on), emerg, err, info, notice, warning.
676 The default is: info.
678 syslog_facility
681 This specifies the syslog facility type that will be used for
682 any messages sent to syslog. options are daemon, local0, local1,
683 local2, local3, local4, local5, local6 & local7.
684 The default is daemon.
686 syslog_priority
689 This specifies the syslog level for this particular subsystem.
690 Ignored if debug is on. Possible values are: alert, crit, debug
691 (same as debug = on), emerg, err, info, notice, warning.
692 The default is: info.
694 debug This specifies whether debug output is logged for this particu‐
697 lar logger. Also can contain value trace, what is highest level
698 of debug information.
699 The default is off.
701 Within the logging directive, logger_subsys directives are optional.
704 Within the logger_subsys sub-directive, all of the above logging con‐
707 figuration options are valid and can be used to override the default
708 settings. The subsys entry, described below, is mandatory to identify
709 the subsystem.
710 subsys This specifies the subsystem identity (name) for which logging
713 is specified. This is the name used by a service in the
714 log_init() call. E.g. 'CPG'. This directive is required.
715 Within the quorum directive it is possible to specify the quorum algo‐
718 rithm to use with the
719 provider
722 directive. At the time of writing only corosync_votequorum is
723 supported. See votequorum(5) for configuration options.
724 Within the nodelist directive it is possible to specify specific infor‐
727 mation about nodes in cluster. Directive can contain only node sub-di‐
728 rective, which specifies every node that should be a member of the mem‐
729 bership, and where non-default options are needed. Every node must have
730 at least ring0_addr field filled.
731 Every node that should be a member of the membership must be specified.
733 Possible options are:
735 ringX_addr
737 This specifies IP or network hostname address of the particular
738 node. X is a link number.
739 nodeid This configuration option is required for each node for Kronos‐
742 net mode. It is a 32 bit value specifying the node identifier
743 delivered to the cluster membership service. The node identifier
744 value of zero is reserved and should not be used. If knet is
745 set, this field must be set.
746 name This option is used mainly with knet transport to identify local
749 node. It's also used by client software (pacemaker). Algorithm
750 for identifying local node is following:
751 1. Looks up $HOSTNAME in the nodelist
753 2. If this fails strip the domain name from $HOSTNAME and
755 looks up that in the nodelist
756 3. If this fails look in the nodelist for a fully-qualified
758 name whose short version matches the short version of
759 $HOSTNAME
760 4. If all this fails then search the interfaces list for an
762 address that matches a name in the nodelist
763 Within the system directive it is possible to specify system options.
766 Possible options are:
768 qb_ipc_type
770 This specifies type of IPC to use. Can be one of native (de‐
771 fault), shm and socket. Native means one of shm or socket, de‐
772 pending on what is supported by OS. On systems with support for
773 both, SHM is selected. SHM is generally faster, but need to al‐
774 locate ring buffer file in /dev/shm.
775 sched_rr
778 Should be set to yes (default) if corosync should try to set
779 round robin realtime scheduling with maximal priority to itself.
780 When setting of scheduler fails, fallback to set maximal prior‐
781 ity.
782 priority
785 Set priority of corosync process. Valid only when sched_rr is
786 set to no. Can be ether numeric value with similar meaning as
787 nice(1) or max / min meaning maximal / minimal priority (so min‐
788 imal / maximal nice value).
789 move_to_root_cgroup
792 Can be one of yes (Corosync always moves itself to root cgroup),
793 no (Corosync never tries to move itself to root cgroup) or auto
794 (Corosync first checks if sched_rr is enabled, and if so, it
795 tries to set round robin realtime scheduling with maximal prior‐
796 ity to itself. If setting of priority fails, corosync tries to
797 move itself to root cgroup and retries setting of priority).
798 This feature is available only for systems with cgroups v1 with
800 RT sched enabled (Linux with CONFIG_RT_GROUP_SCHED kernel op‐
801 tion) and cgroups v2.
802 It's worth noting that currently (May 3 2021) cgroup2 doesn’t
804 yet support control of realtime processes and the cpu controller
805 can only be enabled when all RT processes are in the root cgroup
806 (applies only for kernel with CONFIG_RT_GROUP_SCHED enabled). So
807 when move_to_root_cgroup is disabled, kernel is compiled with
808 CONFIG_RT_GROUP_SCHED and systemd is used, it may be impossible
809 to make systemd options like CPUQuota working correctly until
810 corosync is stopped.
811 Also when moving to root cgroup is enforced and used together
813 with cgroup2 and systemd it makes impossible (most of the time)
814 for journald to add systemd specific metadata (most importantly
815 _SYSTEMD_UNIT) properly, because corosync is moved out of cgroup
816 created by systemd. This means it is not possible to filter
817 corosync logged messages based on these metadata (for example
818 using -u or _SYSTEMD_UNIT=UNIT pattern) and also running system‐
819 ctl status doesn't display (all) corosync log messages. The
820 problem is even worse because journald caches pid for some time
821 (approx. 5 sec) so initial corosync messages have correct meta‐
822 data.
823 allow_knet_handle_fallback
826 If knet handle creation fails using privileged operations, allow
827 fallback to creating knet handle using unprivileged operations.
828 Defaults to no, meaning if privileged knet handle creation
829 fails, corosync will refuse to start.
830 The knet handle will always be created using privileged opera‐
832 tions if possible, setting this to yes only allows fallback to
833 unprivileged operations. This fallback may result in performance
834 issues, but if running in an unprivileged environment, e.g. as a
835 normal user or in unprivileged container, this may be required.
836 state_dir
839 Existing directory where corosync should chdir into. Corosync
840 stores important state files and blackboxes there.
841 The default is /var/lib/corosync.
843 Within the resources directive it is possible to specify options for
846 resources.
847 Possible option is:
849 watchdog_device
851 (Valid only if Corosync was compiled with watchdog support.)
852 Watchdog device to use, for example /dev/watchdog. If unset,
853 empty or "off", no watchdog is used.
854 In a cluster with properly configured power fencing a watchdog
856 provides no additional value. On the other hand, slow watchdog
857 communication may incur multi-second delays in the Corosync main
858 loop, potentially breaking down membership. IPMI watchdogs are
859 particularly notorious in this regard: read about kip‐
860 mid_max_busy_us in IPMI.txt in the Linux kernel documentation.
861 Within the nozzle directive it is possible to specify options for a
865 libnozzle device. This is a pseudo ethernet device that routes network
866 traffic through a channel on the corosync knet network (NOT cpg or any
867 corosync internal service) to other nodes in the cluster. This allows
868 applications to take advantage of knet features such as multipathing,
869 automatic failover, link switching etc. Note that libnozzle is not a
870 reliable transport, but you can tunnel TCP through it for reliable com‐
871 munications.
872 libnozzle also supports optional interface up/down scripts that are
873 kept under a /etc/corosync/updown.d/ directory. See the knet documenta‐
874 tion for more information.
875 Only one nozzle device is allowed.
876 The nozzle stanza takes several options:
877 name The name of the network device to be created. On Linux this may
879 be any name at all, other platforms have restrictions on the
880 name.
881 ipaddr The IP address (IPv6 or IPv4) of the interface. The bottom part
883 of this address will be replaced by the local node's nodeid in
884 conjunction with ipprefix. so, eg ipaddr: 192.168.1.0 ipprefix:
885 24 will make nodeids 1,2,5 use IP addresses 192.168.1.1,
886 192.168.1.2 & 192.168.1.5. If a prefix length of 16 is used
887 then the bottom two bytes will be filled in with nodeid numbers.
888 IPv6 addresses must end in '::', the nodeid will be added after
889 the two colons to make the local IP address. Only one IP ad‐
890 dress is currently supported in the corosync.conf file. Addi‐
891 tional IP addresses can be added in the ifup script if neces‐
892 sary.
893 ipprefix
895 specifies the IP address prefix for the nozzle device (see
896 above)
897 macaddr
899 Specifies the MAC address prefix for the nozzle device. As for
900 the IP address, the bottom part of the MAC address will be
901 filled in with the node id. In this case no prefix applies, the
902 bottom two bytes of the MAC address will always be overwritten
903 with the node id. So specifying macaddr: 54:54:12:24:12:12 on
904 nodeid 1 will result in it having a MAC address of
905 54:54:12:24:00:01
906
909 For example to add a node with address 10.24.38.108 with nodeid 3. The
910 node has the name NEW (in DNS or /etc/hosts) and is not currently run‐
911 ning corosync. The current corosync.conf nodelist looks like this:
912 nodelist {
914 node {
915 nodeid: 1
916 ring0_addr: 10.24.38.101
917 name: node1
918 }
919 node {
920 nodeid: 2
921 ring0_addr: 10.24.38.102
922 name: node2
923 }
925 }
926 Add a new entry for the node below the existing nodes. Node entries
928 don't have to be in nodeid order, but it will help keep you sane. So
929 the nodelist now looks like this:
930 nodelist {
932 node {
933 nodeid: 1
934 ring0_addr: 10.24.38.101
935 name: node1
936 }
937 node {
938 nodeid: 2
939 ring0_addr: 10.24.38.102
940 name: node2
941 }
943 node {
944 nodeid: 3
945 ring0_addr: 10.24.38.108
946 name: NEW
947 }
949 }
950 This file must then be copied onto all three nodes - the existing two
952 nodes, and the new one. On one of the existing corosync nodes, tell
953 corosync to re-read the updated config file into memory:
954 corosync-cfgtool -R
956 This command only needs to be run on one node in the cluster. You may
958 then start corosync on the NEW node and it should join the cluster. If
959 this doesn't work as expected then check the communications between all
960 three nodes is working, and check the syslog files on all nodes for
961 more information. It's important to note that the key bit of informa‐
962 tion about a node failing to join might be on a different node than you
963 expect.
964
967 This is the reverse procedure to 'Adding a node' above. First you need
968 to shut down the node you will be removing from the cluster.
969 corosync-cfgtool -H
971 Then delete the nodelist stanza from corosync.conf and finally update
975 corosync on the remaining nodes by running
976 corosync-cfgtool -R
978 on one of them.
980
983 corosync resolves ringX_addr names/IP addresses using the getad‐
984 drinfo(3) call with respect of totem.ip_version setting.
985 getaddrinfo() function uses a sophisticated algorithm to sort node ad‐
987 dresses into a preferred order and corosync always chooses the first
988 address in that list of the required family. As such it is essential
989 that your DNS or /etc/hosts files are correctly configured so that all
990 addresses for ringX appear on the same network (or are reachable with
991 minimal hops) and over the same IP protocol. If this is not the case
992 then some nodes might not be able to join the cluster. It is possible
993 to override the search order used by getaddrinfo() using the configura‐
994 tion file /etc/gai.conf(5) if necessary, but this is not recommended.
995 If there is any doubt about the order of addresses returned from getad‐
997 drinfo() then it might be simpler to use IP addresses (v4 or v6) in the
998 ringX_addr field.
999
1002 /etc/corosync/corosync.conf
1003 The corosync executive configuration file.
1004
1007 corosync_overview(7), votequorum(5), corosync-qdevice(8), logrotate(8)
1008 getaddrinfo(3) gai.conf(5)
1009corosync Man Page 2022-10-20 COROSYNC_CONF(5)