1HOLLAND(1) Holland HOLLAND(1)
2
6 holland - Holland Documentation
7
9 Introduction to Holland
10 Holland is an Open Source backup framework originally developed by
11 Rackspace and written in Python. It's goal is to help facilitate back‐
12 ing up databases with greater configurability, consistency, and ease.
13 Holland currently focuses on MySQL, however future development will
14 include other database platforms and even non-database related applica‐
15 tions. Because of it's plugin structure, Holland can be used to backup
16 anything you want by whatever means you want.
17 Dependencies
19 The core Holland framework has the following dependencies (available on
20 any remotely modern Linux distribution):
21 · Python >= 2.6
23 · pkg_resources
25 · python-setuptools
27 · future
29 · six
31 · ConfigParser
33 · ArgParser
35 MySQL based plugins additional require the MySQLdb python connector:
37 · Python 2: MySQLdb(mysql-python)
39 · Python 3: MySQLdb(mysqlclient-python)
41 Note that other plugins may have additional dependency requirements.
43 Holland Command-Line Reference
45 Here are the commands available from the 'holland' command-line tool:
46 help (h)
48 Usage: holland <command> --help(-h)
49 Provides basic information about the provided command. If no command is
51 provided, it displays global help instead.
52 backup (bk)
54 Usage: holland backup [backup-set1, backup-set2, ..., backup-setN]
55 Runs the backup operation. If no backup-sets are specified, all active
57 backup-sets (those defined in the 'backupsets' variable in hol‐
58 land.conf) are backed up.
59 One or more backup-sets can be specified directly, in which case only
61 those backup-sets are backed up.
62 Additional Command Line Arguments:
64 --dry-run (-n): Can be used here to simulate, but not actually run, a
66 backup. This should be used when troubleshooting a particular error
67 before trying to run a real backup.
68 --no-lock (-f): Normally, only one instance of Holland can run at any
70 given time using lock-files. Using this flag causes the lock-files to
71 be ignored. This has some very clear use-cases but otherwise be mindful
72 of using this setting as it can cause backups to fail in some cases.
73 --abort-immediately: abort on the first backup-set that fails (assuming
75 multiple backupsets were specified)
76 Examples:
78 # holland bk --dry-run weekly: Attempts a dry-run of the weekly
80 backup-set.
81 # holland bk --no-lock --abort-immediately: Attempts a backup of all
83 the default backup-sets ignoring locks and aborting immediately if one
84 of the backup-sets fails.
85 list-backups (lb)
87 Usage: holland list-backups
88 Provides extended information about available backups.
90 list-plugins (lp)
92 Usage: holland list-plugins
93 Lists all the available (installed) plugins available to Holland.
95 mk-config (mc)
97 Usage: holland mk-config <provider>
98 Generates a template backup-set for a particular provider (such as
100 mysqldump). By default, the output is sent to standard out but can be
101 copied to a file, either by using the --file, --edit, or -name options
102 (see below).
103 Additional Command Line Arguments:
105 --edit: Load the file into the system text-editor for further modifica‐
107 tions.
108 --file=FILE (-f): Write the output directly to provided file.
110 --name=NAME: Creates a backup-set usable in Holland, which basically
112 means that a file is created of the provided name under the backup-set
113 directory.
114 --provider: Indicates that the default provider configuration should be
116 outputted instead. This is really only used when creating a provider
117 config specifically - it should not be used for backup-sets.
118 Examples:
120 # holland mk-config mysql-lvm > mysql-lvm.conf: Output the default con‐
122 figuration for MySQL-LVM backups and write the contents out to
123 mysql-lvm.conf in the current working directory.
124 # holland mc mysqldump --name=Bob --edit: Create a backup-set using the
126 mysqldump provider named Bob and allow interactive editing of the
127 backup-set before saving the file.
128 purge (pg)
130 Usage: holland purge <backup-set>/<backup-id>
131 Purges old backups by specifying the backup-set name and set-id.
133 For example: # holland purge mybackups/20090502_155438: Purge one of
135 the backups taken on May 2nd, 2009 from the mybackups backup-set.
136 Usage and Implementation Overview
138 Because Holland is very pluggable, it may first seem a bit confusing
139 when it comes to configuring Holland to do something useful. Out of the
140 box, Holland is designed to backup MySQL databases using the mysqldump
141 provider. This is the simplest setup, and may be sufficient for most
142 people. However, others may wish to have more fine-grained control over
143 their backups and/or use another method other than mysqldump.
144 For instance, one can configure a backup set to backup certain data‐
146 bases using mysqldump, others using the mysql-lvm plugins etc. All this
147 is done by a mix of providers and backup-sets.
148 Backup-Sets
150 Each backup-set implements exactly one provider and will inherit the
151 default values of that provider. These values can be overridden to
152 adjust the behavior of the backup set. This includes defining what
153 databases or tables to include (or exclude) in the backup, the type of
154 compression used (if any), what locking method to use, among other
155 things.
156 Providers
158 Providers essentially provide a backup service for use in a backup set.
159 · mysqldump
161 Uses the mysqldump utility to backup databases.
162 · MySQL + LVM
164 Backup MySQL databases using LVM snapshots which allows for near
165 lockless or fully lockless (when transactional engines are used)
166 backups.
167 · mysqldump + LVM
169 This plugin creates an LVM snapshot, starts a mysql instance using
170 the snapshot as it's datadir, and then use the mysqldump utility
171 to backup the databases.
172 · XtraBackup
174 New in version 1.0.8.
175 Backup MySQL databases using Percona's XtraBackup tool. This pro‐
178 vides a near lockless backup when using the InnoDB storage engine.
179 · pgdump
181 Backup PostgreSQL databases using the pgdump utility.
182 · mariabackup
184 New in version 1.1.0.
185 Backup MySQL databases using MariaDB's mariabackup tool.
188 · mongodump
190 New in version 1.1.0.
191 This plugin performs logical backups of a MongoDB by using the
194 mongodump utility.
195 · Example
197 This is used solely as a template for designing providers. It oth‐
198 erwise does nothing.
199 As Holland is a framework, it can actually backup most anything as long
201 as there is a provider for it. This includes things that have nothing
202 to do with databases. The idea is to present an easy to use and clear
203 method of backing up and restoring backups no matter the source.
204 Holland Config Files
206 By default, Holland's configuration files reside in /etc/holland. The
207 main configuration file is holland.conf, however there are a number of
208 other configuration files for configuring default settings for
209 providers and for configuring backup sets.
210 Each configuration file has one ore more sections, defined by square
212 brackets Underneath each section, one or more configuration option can
213 be specified. These options are in a standard "option = value" format.
214 Comments are prefixed by the # sign.
215 Note that many settings have default values and, as a result, can
217 either be commented out or omitted entirely.
218 holland.conf - main config
220 The main configuration file (usually /etc/holland/holland.conf) defines
221 both global settings as well as the active backup sets. It is divided
222 into two sections [holland] and [logging].
223 [holland]
225 plugin-dirs
226 Defines where the plugins can be found. This can be a comma-sep‐
227 arated list but usually does not need to be modified.
228 backup_directory
230 Top-level directory where backups are held.
231 backupsets
233 A comma-separated list of all the backup sets Holland should
234 backup. Each backup set is defined in /etc/holland/back‐
235 upsets/<name>.conf by default.
236 umask Sets the umask of the resulting backup files.
238 path Defines a path for holland and its spawned processes
240 [logging]
242 filename
243 The log file itself.
244 level Sets the verbosity of Holland's logging process. Available
246 options are debug, info, warning, error, and critical
247 format Define the format of the log message. This options should be in
249 the format defined by the 'logging' python library.
250 New in version 1.1.0.
252 Provider Configs
255 These files control the global settings / defaults for the providers
256 used by the backup-sets. Many of these global settings can be overrid‐
257 den if defined in a backup-set. Note that each provider's configuration
258 file should begin with [provider-name].
259 mysqldump Provider Configuration [mysqldump]
261 Backs up a MySQL database using the mysqldump tool.
262 [mysqldump]
264 mysql-binpath = /path/to/mysql/bin
265 Defines the location of the MySQL binary utilities. If not provided,
266 Holland will use whatever is in the path.
267 lock-method = flush-lock | lock-tables | single-transaction |
269 auto-detect | none
270 Defines which lock method to use. By default, auto-detect will be
271 used.
272 · flush-lock
274 flush-lock will place a global lock on all tables involved in
275 the backup regardless of whether or not they are in the
276 backup-set. If file-per-database is enabled, then flush-lock
277 will lock all tables for every database being backed up. In
278 other words, this option may not make much sense when using
279 file-per-database.
280 · lock-tables
282 lock-tables will lock all tables involved in the backup. If
283 file-per-database is enabled, then lock-tables will only lock
284 all the tables associated with that database.
285 · single-transaction
287 Forces the use of --single-transaction which enabled
288 semi-transparent backups of transactional tables. Forcing this
289 can cause inconsistencies with non-transactional tables, how‐
290 ever. While non-transactional tables will still lock, they
291 will only lock when they are actually being backed up. Use
292 this setting with extreme caution when backing non-transac‐
293 tional tables.
294 · auto-detect
296 Let Holland decide which option to use by checking to see if a
297 database or backup-set only contains transactional tables. If
298 so, --single-transaction will be used. Otherwise,
299 --lock-tables will be used.
300 · none
302 Does absolutely no explicit locking when backing up the data‐
303 bases or backup-set. This should only be used when backing up
304 a slave and only after the slave has been turned off (ie, this
305 can be used with the stop-slave option).
306 exclude-invalid-views = yes | no (default: no)
308 Whether to automate exclusion of invalid views that would otherwise
309 cause mysqldump to fail. This adds additional overhead so this
310 option is not enabled by default.
311 When enabled, thos option will scan the INFORMATION_SCHEMA.VIEWS ta‐
313 ble and execute SHOW FIELDS against each view. If a view is detects
314 as invalid, an ignore-table option will be added to exclude the ta‐
315 ble. Additionally, the plugin will attempt to save the view defin‐
316 ion to 'invalid_views.sql' in the backupset's backup directory.
317 New in version 1.0.8.
319 dump-routines = yes | no (default: yes)
322 Whether or not to backup routines in the backup set directly. Rou‐
323 tines are stored in the 'mysql' database, but it can sometimes be
324 convenient to include them in a backup-set directly.
325 Changed in version 1.0.8: This option now enabled by default.
327 dump-events = yes | no
330 Whether or not to dump events explicitly. Like routines, events are
331 stored in the 'mysql' database. Nonetheless, it can sometimes be
332 convenient to include them in the backup-set directly.
333 Note: This feature requires MySQL 5.1 or later. The mysqldump plugin
335 will automatically disable events if the version of mysqldump is too
336 old.
337 Changed in version 1.0.8: This option is now enabled by default
339 stop-slave = yes | no
342 Stops the SQL_THREAD during the backup. This means that writes from
343 the master will continue to spool but will not be replayed. This
344 helps avoid lock wait timeouts among things while still allowing
345 data to be spooled from the master.
346 Note that previous versions of Holland prior to 1.0.6 simply ran a
348 STOP SLAVE instead, which suspends both replication threads.
349 bin-log-position = yes | no
351 Record the binary log name and position at the time of the backup.
352 The information provied by this option is collected just before
353 locking the database.
354 Note that if both 'stop-slave' and 'bin-log-position' are enabled,
356 Holland will grab the master binary log name and position at the
357 time of the backup which can be useful in using the backup to create
358 slaves or for point in time recovery using the master's binary log.
359 This information is found within the 'backup.conf' file located in
360 the backup-set destination directory (/var/spool/hol‐
361 land/<backup-set>/<backup> by default). For example:
362 [mysql:replication]
364 slave_master_log_pos = 4512
365 slave_master_log_file = 260792-mmm-agent1-bin-log.000001
366 flush-logs = yes | no
368 Whether or not to run FLUSH LOGS in MySQL with the backup. When
369 FLUSH LOGS is actually executed depends on which if database filter‐
370 ing is being used and whether or not file-per-database is enabled.
371 Generally speaking, it does not make sense to use flush-logs with
372 file-per-database since the binary logs will not be consistent with
373 the backup.
374 file-per-database = yes | no
376 Whether or not to split up each database into its own file. Note
377 that it can be more consistent an efficient to backup all databases
378 into one file, however this means that restore a single database can
379 be difficult if multiple databases are defined in the backup set.
380 arg-per-database = JSON object
382 If file-per-database is enable this argument is ued to specify
383 mysqldump arguments per database. It takes a JSON object with the
384 database names as keys. Example: {"employee1": "--no-data"} Adds
385 the '--no-data' argument to the mysqldump command when backing up
386 the 'employee1' database
387 New in version 1.0.9.
389 additional-options = <mysqldump argument>[, <mysqldump argument>]
392 Can optionally specify additional options directly to mysqldump if
393 there is no native Holland option available. This option accepts a
394 comma delimited list of arguments to pass on the commandline.
395 extra-defaults = yes | no (default: no)
397 This option controls whether mysqldump will only read options as set
398 by holland or if additional options from global config files are
399 read. By default, the plugin only uses optons as set in the back‐
400 upset config and includes authentication credentials only from the
401 [client] section in ~/.my.cnf.
402 estimate-method = plugin | const:<size> (default: plugin)
404 This option will skip some of the heavyweight queries necessary to
405 calculate the size of tables to be backed up. If a constant size is
406 specified, then only table names are evaluated and only if table
407 filtering is being used. Additionally, engines will be looked up via
408 SHOW CREATE TABLE if lock-method = auto-detect, in order for the
409 plugin to determine if tables are using a transactional storage
410 engine. With 'plugin', the default behavior of reading both size
411 information and table names from the information schema is used,
412 which may be slow particularly for a large number of tables.
413 Database and Table filtering
415 Database and Table filtering
416 databases = <glob>
417 exclude-databases = <glob>
419 tables = <glob>
421 exclude-tables = <glob>
423 The above options accepts GLOBs in comma-separated lists. Multiple fil‐
425 tering options can be specified. When filtering on tables, be sure to
426 include both the database and table name.
427 Be careful with quotes. Normally these are not needed, but when quotes
429 are necessary, be sure to only quote each filtering statement, as
430 opposed to putting quotes around all statements.
431 Below are a few examples of how these can be applied:
433 Default (backup everything):
435 databases = *
437 tables = *
438 Using database inclusion and exclusions:
440 databases = drupal*, smf_forum,
442 exclude-databases = drupal5
443 Including Tables:
445 tables = phpBB.sucks, drupal6.node*, smf_forum.*
447 Excluding Tables:
449 exclude-tables = mydb.uselesstable1, x_cart.*, *.sessions
451 [compression]
453 Specify various compression settings, such as compression utility, com‐
454 pression level, etc.
455 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
457 gpg | zstd
458 Define which compression method to use. Note that some methods may
459 not be available by default on every system and may need to be com‐
460 piled or installed.
461 For gpg compression, a key should already exist(gpg --gen-key) and
463 default-recipient must be configured in ~/.gnupg/gpg.conf.
464 inline = yes | no
466 Whether or not to pipe the output of the backup command into the
467 compression utility. Enabling this is recommended since it usually
468 only marginally impacts performance, particularly when using a lower
469 compression level.
470 level = 0-9
472 Specify the compression ratio. The lower the number, the lower the
473 compression ratio, but the faster the backup will take. Generally,
474 setting the lever to 1 or 2 results in favorable compression of tex‐
475 tual data and is noticeably faster than the higher levels. Setting
476 the level to 0 effectively disables compression.
477 bin-path = <full path to utility>
479 This only needs to be defined if the compression utility is not in
480 the usual places or not in the system path.
481 options = <string>
483 Add commandline options to the configuration compression command.
484 options = "-Q4"
486 split = yes | no
488 Defautls to no. If set the backup will be piped through the split
489 command. This may be useful for user's with large databases, as
490 some backup systems perform better with many smaller files instead
491 of 1 large one. This defaults to 1GB file size, so this option isn't
492 helpful if your dumps are smaller than that.
493 For python2.6, this option will be disabled if the subprocess32 mod‐
495 ule isn't avaiable.
496 New in version 1.1.13.
498 MySQL connection info [mysql:client]
501 These are optional and, if left undefined, Holland will try to login
502 using the standard .my.cnf conventions.
503 user = <user>
505 The user to connect to MySQL as.
507 password = <password>
509 The password for the MySQL user
511 socket = <socket>
513 The socket file to connect to MySQL with.
515 host = <host>
517 This would be used for connecting to MySQL remotely.
519 port = <port>
521 Used if MySQL is running on a port other than 3306.
523 MySQL LVM Provider Configuration [mysql-lvm]
525 Creates an LVM snapshot of a running MySQL instance and performs a
526 binary-based backup with minimal locking. MySQL must be running on an
527 LVM volume with reserved space for snapshots. It is highly recommended
528 that this volume be separate from the one storing the resulting back‐
529 ups.
530 [mysql-lvm]
532 snapshot-size = <size-in-MB>
533 The size of the snapshot itself. By default it is 20% of the size of
534 the MySQL LVM mount or the remaining free-space in the Volume Group
535 (if there is less than 20% available) up to 15GB. If snapshot-size
536 is defined, the number represents the size of the snapshot in
537 megabytes.
538 snapshot-name = <name>
540 The name of the snapshot, the default being the name of the MySQL
541 LVM volume + "_snapshot" (ie Storage-MySQL_snapshot)
542 snapshot-mountpoint = <path>
544 Where to mount the snapshot. By default a randomly generated direc‐
545 tory under /tmp is used.
546 innodb-recovery = yes | no (default: no)
548 Whether or not to run an InnoDB recovery operation. This avoids
549 needing to do so during a restore, though will make the backup
550 process itself take longer.
551 force-innodb-backup = yes | no (default: no)
553 Whether to attempt a backup even if the mysql-lvm plugin thinks it
554 cannot obtain a good backup. This can occur when innodb data files
555 are outside of the mysql datadir or exist on entirely separate logi‐
556 cal volumes.
557 lock-tables = yes | no (default: yes)
559 Whether or not to run a FLUSH TABLES WITH READ LOCK to grab various
560 bits of information (such as the binary log name and position). Dis‐
561 abling this requires that binary logging is disabled and InnoDB is
562 being used exclusively. Otherwise, it is possible that the backup
563 could contain crashed tables.
564 extra-flush-tables = yes | no (default: yes)
566 Whether or not to run a FLUSH TABLES before running the full FLUSH
567 TABLES WITH READ LOCK. Should make the FLUSH TABLES WITH READ LOCK
568 operation a bit faster.
569 archive-method = tar | dir (default: tar)
571 Create a tar file of the datadir, or just copy it.
572 [tar]
574 exclude = pattern[, pattern...]
575 Patterns to exclude from archive. These should be relative paths and
577 are almost always relative to the mysql data directory. For instance
578 to exclude binary logs in the data directory from the backup you might
579 specify: exclude = ./bin-log.*, mysql.sock
580 pre-args = <string>
582 Additional arguments to append to the tar commandline before the backup
584 path is specified. This should be the full string as you might specify
585 on the commandline. Shell globbing is not supported.
586 For instance you might add the /etc/my.cnf to the tar archive via:
588 pre-args = -C /etc ./my.cnf
589 post-args = <string>
591 Additional arguments to append to the tar commandline after the backup
593 path is specified. This should be a string exactly as you might spec‐
594 ify on the commandline. Shell globbing is not evaluated.
595 [compression]
597 Specify various compression settings, such as compression utility, com‐
598 pression level, etc.
599 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
601 gpg | zstd
602 Define which compression method to use. Note that some methods may
603 not be available by default on every system and may need to be com‐
604 piled or installed.
605 For gpg compression, a key should already exist(gpg --gen-key) and
607 default-recipient must be configured in ~/.gnupg/gpg.conf.
608 inline = yes | no
610 Whether or not to pipe the output of the backup command into the
611 compression utility. Enabling this is recommended since it usually
612 only marginally impacts performance, particularly when using a lower
613 compression level.
614 level = 0-9
616 Specify the compression ratio. The lower the number, the lower the
617 compression ratio, but the faster the backup will take. Generally,
618 setting the lever to 1 or 2 results in favorable compression of tex‐
619 tual data and is noticeably faster than the higher levels. Setting
620 the level to 0 effectively disables compression.
621 bin-path = <full path to utility>
623 This only needs to be defined if the compression utility is not in
624 the usual places or not in the system path.
625 options = <string>
627 Add commandline options to the configuration compression command.
628 options = "-Q4"
630 split = yes | no
632 Defautls to no. If set the backup will be piped through the split
633 command. This may be useful for user's with large databases, as
634 some backup systems perform better with many smaller files instead
635 of 1 large one. This defaults to 1GB file size, so this option isn't
636 helpful if your dumps are smaller than that.
637 For python2.6, this option will be disabled if the subprocess32 mod‐
639 ule isn't avaiable.
640 New in version 1.1.13.
642 MySQL connection info [mysql:client]
645 These are optional and, if left undefined, Holland will try to login
646 using the standard .my.cnf conventions.
647 user = <user>
649 The user to connect to MySQL as.
651 password = <password>
653 The password for the MySQL user
655 socket = <socket>
657 The socket file to connect to MySQL with.
659 host = <host>
661 This would be used for connecting to MySQL remotely.
663 port = <port>
665 Used if MySQL is running on a port other than 3306.
667 mysqldump LVM Provider Configuration [mysqldump-lvm]
669 Backs up one or more MySQL databases by creating an LVM snapshot and
670 then starting a instance of MySQL on top of it to then perform a mysql‐
671 dump. This effectively produces a non-blocking logical backup.
672 [mysql-lvm]
674 snapshot-size = <size-in-MB>
675 The size of the snapshot itself. By default it is 20% of the size of
676 the MySQL LVM mount or the remaining free-space in the Volume Group
677 (if there is less than 20% available) up to 15GB. If snapshot-size
678 is defined, the number represents the size of the snapshot in
679 megabytes.
680 snapshot-name = <name>
682 The name of the snapshot, the default being the name of the MySQL
683 LVM volume + "_snapshot" (ie Storage-MySQL_snapshot)
684 snapshot-mountpoint = <path>
686 Where to mount the snapshot. By default a randomly generated direc‐
687 tory under /tmp is used.
688 innodb-recovery = yes | no (default: no)
690 Whether or not to run an InnoDB recovery operation. This avoids
691 needing to do so during a restore, though will make the backup
692 process itself take longer.
693 lock-tables = yes | no (default: yes)
695 Whether or not to run a FLUSH TABLES WITH READ LOCK to grab various
696 bits of information (such as the binary log name and position). Dis‐
697 abling this requires that binary logging is disabled and InnoDB is
698 being used exclusively. Otherwise, it is possible that the backup
699 could contain crashed tables.
700 extra-flush-tables = yes | no (default: yes)
702 Whether or not to run a FLUSH TABLES before running the full FLUSH
703 TABLES WITH READ LOCK. Should make the FLUSH TABLES WITH READ LOCK
704 operation a bit faster.
705 [mysqld]
707 mysqld-exe = <path>[, <path>...] (default: mysqld in PATH,
708 /usr/libexec/mysqld)
709 This provides a list of locations where the mysqld process to use
710 might be found. This is searched in order of entries in this list.
711 user = <name>
713 The --user parameter to use with mysqld.
714 innodb-buffer-pool-size = <size> (default: 128M)
716 How large to size the innodb-buffer-pool-size.
717 tmpdir = <path> (default: system tempdir)
719 Path to the --tmpdir that mysqld should use.
720 log-error = <path> (default: tempdir/holland_lvm.log)
722 Define path for mysqld's error log. The default location get cleaned
723 up by Holland after the backup is complete. This settings allows the
724 user to define the log file in another location and can be useful
725 for debugging issue with the MySQL instance running on the snapshot.
726 New in version 1.0.9.
728 [mysqldump]
731 mysqldump-lvm supports almost all of the options from the mysqldump
732 plugin. --master-data is not supported, as the mysqld process will not
733 read binary logs, so this plugin will automatically disable
734 bin-log-position, if set.
735 Binary log information from SOHW MASTER STATUS and SHOW SLAVE STATUS is
737 recorded in the ${backup_directory}/backup.conf file under the
738 [mysql:replication] section.
739 [compression]
741 Specify various compression settings, such as compression utility, com‐
742 pression level, etc.
743 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
745 gpg | zstd
746 Define which compression method to use. Note that some methods may
747 not be available by default on every system and may need to be com‐
748 piled or installed.
749 For gpg compression, a key should already exist(gpg --gen-key) and
751 default-recipient must be configured in ~/.gnupg/gpg.conf.
752 inline = yes | no
754 Whether or not to pipe the output of the backup command into the
755 compression utility. Enabling this is recommended since it usually
756 only marginally impacts performance, particularly when using a lower
757 compression level.
758 level = 0-9
760 Specify the compression ratio. The lower the number, the lower the
761 compression ratio, but the faster the backup will take. Generally,
762 setting the lever to 1 or 2 results in favorable compression of tex‐
763 tual data and is noticeably faster than the higher levels. Setting
764 the level to 0 effectively disables compression.
765 bin-path = <full path to utility>
767 This only needs to be defined if the compression utility is not in
768 the usual places or not in the system path.
769 options = <string>
771 Add commandline options to the configuration compression command.
772 options = "-Q4"
774 split = yes | no
776 Defautls to no. If set the backup will be piped through the split
777 command. This may be useful for user's with large databases, as
778 some backup systems perform better with many smaller files instead
779 of 1 large one. This defaults to 1GB file size, so this option isn't
780 helpful if your dumps are smaller than that.
781 For python2.6, this option will be disabled if the subprocess32 mod‐
783 ule isn't avaiable.
784 New in version 1.1.13.
786 MySQL connection info [mysql:client]
789 These are optional and, if left undefined, Holland will try to login
790 using the standard .my.cnf conventions.
791 user = <user>
793 The user to connect to MySQL as.
795 password = <password>
797 The password for the MySQL user
799 socket = <socket>
801 The socket file to connect to MySQL with.
803 host = <host>
805 This would be used for connecting to MySQL remotely.
807 port = <port>
809 Used if MySQL is running on a port other than 3306.
811 Xtrabackup Provider Configuration [xtrabackup]
813 Backs up a MySQL instance using Percona's Xtrabackup tool.
814 [xtrabackup]
816 global-defaults = <path> (default: /etc/my.cnf)
817 The MySQL configuration file for xtrabackup to parse. This is
818 !include'd into the my.cnf the xtrabackup plugin generates
819 innobackupex = <name> (default: innobackupex-1.5.1)
821 The path to the innobackupex script to run. If this is a relative
822 path this will be found in holland's environment PATH as configured
823 in /etc/holland/holland.conf.
824 Changed in version 1.1.12: For xtrabackup version 8.0 or greater, this
826 option is ignored
827 ibbackup = <name>
830 The path to the ibbackup command to use. By default, no --ibbackup
831 option is pass to the innobackupex script. Usually innobackupex
832 will detect this by itself and this should not need to be set.
833 stream = tar|xbstream|yes|no (default: tar)
835 Whether to generate a streaming backup.
836 Changed in version 1.0.8: 'tar' and 'xbstream' are now valid options.
838 The old stream = yes is now equivalent to stream = tar and stream = no
839 disables streaming entirely and will result in a normal directory copy
840 with xtrabackup
841 Changed in version 1.1.12: For xtrabackup version 8.0 or
844 greater,'xstream' will be used unless this value is set to 'no'
845 apply-logs = yes | no (default: yes)
848 Whether to run innobackupex --apply-logs at the end of the backup.
849 This is only supported when performing a non-streaming, non-com‐
850 pressed backup. In this case, even if apply-logs = yes (the
851 default), the prepare stage will be skipped. Even with an uncom‐
852 pressed, non-streaming backup you may want to disable apply-logs if
853 you wish to use incremental backups.
854 New in version 1.0.8.
856 slave-info = yes | no (default: yes)
859 Whether to enable the --slave-info innobackupex option
860 safe-slave-backup = yes | no (default: yes)
862 Whether to enable the --safe-slave-backup innobackupex option.
863 no-lock = yes | no (default: no)
865 Whether to enable the --no-lock innobackupex option
866 tmpdir = <path> (default: ${backup_directory})
868 The path for the innobackupex --tmpdir option. By default this will
869 use the current holland backup directory to workaround the following
870 bug: https://bugs.launchpad.net/percona-xtrabackup/+bug/1007446
871 New in version 1.0.8.
873 additional-options = <option>[, <option>...]
876 A list of additional options to pass to innobackupex. This is a
877 comma separated list of options.
878 pre-command = <command-string>
880 A command to run prior to running this xtrabackup run. This can be
881 used, for instance, to generate a mysqldump schema dump prior to
882 running xtrabackup. instances of ${backup_directory} will be
883 replaced with the current holland backup directory where the xtra‐
884 backup data will be stored.
885 [compression]
887 Specify various compression settings, such as compression utility, com‐
888 pression level, etc.
889 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
891 gpg | zstd
892 Define which compression method to use. Note that some methods may
893 not be available by default on every system and may need to be com‐
894 piled or installed.
895 For gpg compression, a key should already exist(gpg --gen-key) and
897 default-recipient must be configured in ~/.gnupg/gpg.conf.
898 inline = yes | no
900 Whether or not to pipe the output of the backup command into the
901 compression utility. Enabling this is recommended since it usually
902 only marginally impacts performance, particularly when using a lower
903 compression level.
904 level = 0-9
906 Specify the compression ratio. The lower the number, the lower the
907 compression ratio, but the faster the backup will take. Generally,
908 setting the lever to 1 or 2 results in favorable compression of tex‐
909 tual data and is noticeably faster than the higher levels. Setting
910 the level to 0 effectively disables compression.
911 bin-path = <full path to utility>
913 This only needs to be defined if the compression utility is not in
914 the usual places or not in the system path.
915 options = <string>
917 Add commandline options to the configuration compression command.
918 options = "-Q4"
920 split = yes | no
922 Defautls to no. If set the backup will be piped through the split
923 command. This may be useful for user's with large databases, as
924 some backup systems perform better with many smaller files instead
925 of 1 large one. This defaults to 1GB file size, so this option isn't
926 helpful if your dumps are smaller than that.
927 For python2.6, this option will be disabled if the subprocess32 mod‐
929 ule isn't avaiable.
930 New in version 1.1.13.
932 MySQL connection info [mysql:client]
935 These are optional and, if left undefined, Holland will try to login
936 using the standard .my.cnf conventions.
937 user = <user>
939 The user to connect to MySQL as.
941 password = <password>
943 The password for the MySQL user
945 socket = <socket>
947 The socket file to connect to MySQL with.
949 host = <host>
951 This would be used for connecting to MySQL remotely.
953 port = <port>
955 Used if MySQL is running on a port other than 3306.
957 pgdump Provider Configuration [pgdump]
959 Backs up a PostgreSQL instance using the pgdump utility.
960 [pgdump]
962 format = custom | tar | plain (default: custom)
963 Defines the --format option for pg_dump. This defaults to --for‐
964 mat=custom. The custom format is required for pg_restore to do par‐
965 tial restore as well as enabling parallel restores. If set to cus‐
966 tom, the --compress option will be passed to pgdump
967 additional-options = <command-string>
969 Pass additional options to the pg_dump command
970 Only the 'level' option will be used if 'format=custom'
972 [compression]
974 Specify various compression settings, such as compression utility, com‐
975 pression level, etc.
976 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
978 gpg | zstd
979 Define which compression method to use. Note that some methods may
980 not be available by default on every system and may need to be com‐
981 piled or installed.
982 For gpg compression, a key should already exist(gpg --gen-key) and
984 default-recipient must be configured in ~/.gnupg/gpg.conf.
985 inline = yes | no
987 Whether or not to pipe the output of the backup command into the
988 compression utility. Enabling this is recommended since it usually
989 only marginally impacts performance, particularly when using a lower
990 compression level.
991 level = 0-9
993 Specify the compression ratio. The lower the number, the lower the
994 compression ratio, but the faster the backup will take. Generally,
995 setting the lever to 1 or 2 results in favorable compression of tex‐
996 tual data and is noticeably faster than the higher levels. Setting
997 the level to 0 effectively disables compression.
998 bin-path = <full path to utility>
1000 This only needs to be defined if the compression utility is not in
1001 the usual places or not in the system path.
1002 options = <string>
1004 Add commandline options to the configuration compression command.
1005 options = "-Q4"
1007 split = yes | no
1009 Defautls to no. If set the backup will be piped through the split
1010 command. This may be useful for user's with large databases, as
1011 some backup systems perform better with many smaller files instead
1012 of 1 large one. This defaults to 1GB file size, so this option isn't
1013 helpful if your dumps are smaller than that.
1014 For python2.6, this option will be disabled if the subprocess32 mod‐
1016 ule isn't avaiable.
1017 New in version 1.1.13.
1019 [pgauth]
1022 username = <name>
1023 Username for pg_dump to authenticate with
1024 password = <string>
1026 Password for pg_dump to authenticate with
1027 hostname = <string>
1029 Hostname for pg_dump to connect with
1030 port = <integer>
1032 TCP port for pg_dump to connect on
1033 mongodump Provider Configuration [mongodump]
1035 This plugin performs logical backups of a MongoDB by using the mongo‐
1036 dump utility.
1037 [mongodump]
1039 host = <string>
1040 Hostname for mongodump to connect with
1041 username = <name>
1043 Username for mongodump to authenticate with
1044 password = <string>
1046 Password for mongodump to authenticate with
1047 port = <integer>
1049 TCP port for mongodump to connect on
1050 uri = <string>
1052 Use a connection string instead of a host, username, and pass‐
1053 word
1054 New in version 1.1.14.
1056 [compression]
1059 Specify various compression settings, such as compression utility, com‐
1060 pression level, etc.
1061 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
1063 gpg | zstd
1064 Define which compression method to use. Note that some methods may
1065 not be available by default on every system and may need to be com‐
1066 piled or installed.
1067 For gpg compression, a key should already exist(gpg --gen-key) and
1069 default-recipient must be configured in ~/.gnupg/gpg.conf.
1070 inline = yes | no
1072 Whether or not to pipe the output of the backup command into the
1073 compression utility. Enabling this is recommended since it usually
1074 only marginally impacts performance, particularly when using a lower
1075 compression level.
1076 level = 0-9
1078 Specify the compression ratio. The lower the number, the lower the
1079 compression ratio, but the faster the backup will take. Generally,
1080 setting the lever to 1 or 2 results in favorable compression of tex‐
1081 tual data and is noticeably faster than the higher levels. Setting
1082 the level to 0 effectively disables compression.
1083 bin-path = <full path to utility>
1085 This only needs to be defined if the compression utility is not in
1086 the usual places or not in the system path.
1087 options = <string>
1089 Add commandline options to the configuration compression command.
1090 options = "-Q4"
1092 split = yes | no
1094 Defautls to no. If set the backup will be piped through the split
1095 command. This may be useful for user's with large databases, as
1096 some backup systems perform better with many smaller files instead
1097 of 1 large one. This defaults to 1GB file size, so this option isn't
1098 helpful if your dumps are smaller than that.
1099 For python2.6, this option will be disabled if the subprocess32 mod‐
1101 ule isn't avaiable.
1102 New in version 1.1.13.
1104 Mariabackup Provider Configuration [mariabackup]
1107 Backs up a MySQL instance using mariabackup tool.
1108 [mariabackup]
1110 global-defaults = <path> (default: /etc/my.cnf)
1111 The MySQL configuration file for mariabackup to parse. This is
1112 !include'd into the my.cnf the mariabackup plugin generates
1113 innobackupex = <name> (default: innobackupex-1.5.1)
1115 The path to the innobackupex script to run. If this is a relative
1116 path this will be found in holland's environment PATH as configured
1117 in /etc/holland/holland.conf.
1118 ibbackup = <name>
1120 The path to the ibbackup command to use. By default, no --ibbackup
1121 option is pass to the innobackupex script. Usually innobackupex
1122 will detect this by itself and this should not need to be set.
1123 stream = mbstream(default: tar)
1125 Placeholder
1126 apply-logs = yes | no (default: yes)
1128 Whether to run innobackupex --apply-logs at the end of the backup.
1129 This is only supported when performing a non-streaming, non-com‐
1130 pressed backup. In this case, even if apply-logs = yes (the
1131 default), the prepare stage will be skipped. Even with an uncom‐
1132 pressed, non-streaming backup you may want to disable apply-logs if
1133 you wish to use incremental backups.
1134 slave-info = yes | no (default: yes)
1136 Whether to enable the --slave-info innobackupex option
1137 safe-slave-backup = yes | no (default: yes)
1139 Whether to enable the --safe-slave-backup innobackupex option.
1140 no-lock = yes | no (default: no)
1142 Whether to enable the --no-lock innobackupex option
1143 tmpdir = <path> (default: ${backup_directory})
1145 The path for the innobackupex --tmpdir option.
1146 additional-options = <option>[, <option>...]
1148 A list of additional options to pass to innobackupex. This is a
1149 comma separated list of options.
1150 pre-command = <command-string>
1152 A command to run prior to running this mariabackup run. This can be
1153 used, for instance, to generate a mysqldump schema dump prior to
1154 running mariabackup. instances of ${backup_directory} will be
1155 replaced with the current holland backup directory where the mari‐
1156 abackup data will be stored.
1157 [compression]
1159 Specify various compression settings, such as compression utility, com‐
1160 pression level, etc.
1161 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
1163 gpg | zstd
1164 Define which compression method to use. Note that some methods may
1165 not be available by default on every system and may need to be com‐
1166 piled or installed.
1167 For gpg compression, a key should already exist(gpg --gen-key) and
1169 default-recipient must be configured in ~/.gnupg/gpg.conf.
1170 inline = yes | no
1172 Whether or not to pipe the output of the backup command into the
1173 compression utility. Enabling this is recommended since it usually
1174 only marginally impacts performance, particularly when using a lower
1175 compression level.
1176 level = 0-9
1178 Specify the compression ratio. The lower the number, the lower the
1179 compression ratio, but the faster the backup will take. Generally,
1180 setting the lever to 1 or 2 results in favorable compression of tex‐
1181 tual data and is noticeably faster than the higher levels. Setting
1182 the level to 0 effectively disables compression.
1183 bin-path = <full path to utility>
1185 This only needs to be defined if the compression utility is not in
1186 the usual places or not in the system path.
1187 options = <string>
1189 Add commandline options to the configuration compression command.
1190 options = "-Q4"
1192 split = yes | no
1194 Defautls to no. If set the backup will be piped through the split
1195 command. This may be useful for user's with large databases, as
1196 some backup systems perform better with many smaller files instead
1197 of 1 large one. This defaults to 1GB file size, so this option isn't
1198 helpful if your dumps are smaller than that.
1199 For python2.6, this option will be disabled if the subprocess32 mod‐
1201 ule isn't avaiable.
1202 New in version 1.1.13.
1204 MySQL connection info [mysql:client]
1207 These are optional and, if left undefined, Holland will try to login
1208 using the standard .my.cnf conventions.
1209 user = <user>
1211 The user to connect to MySQL as.
1213 password = <password>
1215 The password for the MySQL user
1217 socket = <socket>
1219 The socket file to connect to MySQL with.
1221 host = <host>
1223 This would be used for connecting to MySQL remotely.
1225 port = <port>
1227 Used if MySQL is running on a port other than 3306.
1229 Backup-Set Configs
1231 Backup-Set configuration files largely inherit the configuration
1232 options of the specified provider. To define a provider for the backup
1233 set, you must put the following at the top of the backup set configura‐
1234 tion file:
1235 [holland:backup]
1237 plugin = <plugin>
1238 backups-to-keep = #
1239 estimated-size-factor = #
1240 historic-size = <yes|no>
1241 historic-size-factor = #
1242 historic-estimated-size-factor = #
1243 Configuration Options
1245 plugin = <plugin>
1246 This is the name of the provider that will be used for the
1247 backup-set. This is required in order for the backup-set to func‐
1248 tion.
1249 backups-to-keep = #
1251 Specifies the number of backups to keep for a backup-set. Defaults
1252 to retaining 1 backup.
1253 estimated-size-factor = #
1255 Specifies the scale factor when Holland decides if there is enough
1256 free space to perform a backup. The default is 1.0 and this number
1257 is multiplied against what each individual plugin reports its esti‐
1258 mated backup size when Holland is verifying sufficient free space
1259 for the backupset.
1260 As of Holland 1.1.1, the 'historic' backup size will be used. If
1262 Holland is unable to determine that size, it will default back to
1263 this.
1264 auto-purge-failures = yes | no
1266 Specifies whether to keep a failed backup or to automatically remove
1267 the backup directory. By default this is on with the intention that
1268 whatever process is calling holland will retry when a backup fails.
1269 This behavior can be disabled by setting auto-purge-failures = no
1270 when partial backups might be useful or when troubleshooting a
1271 backup failure.
1272 purge-policy = manual | before-backup | after-backup
1274 Specifies when to run the purge routine on a backupset. By default
1275 this is run after a new successful backup completes. Up to back‐
1276 ups-to-keep backups will be retained including the most recent.
1277 purge-policy = before-backup will run the purge routine just before
1279 a new backup starts. This will retain up to backups-to-keep backups
1280 before the new backup is even started allowing purging all previous
1281 backups if backups-to-keep is set to 0. This behavior is useful if
1282 some other process is retaining backups off-server and disk space is
1283 at a premium.
1284 purge-policy = manual will never run the purge routine automati‐
1286 cally. Either holland purge must be run externally or an explicit
1287 removal of desired backup directories can be done at some later
1288 time.
1289 before-backup-command = string
1291 Run a shell command before a backup starts. This allows a command
1292 to perform some action before the backup starts such as setting up
1293 an iptables rule (taking a mysql slave out of a load balancer) or
1294 aborting the backup based on some external condition.
1295 The backup will fail if this command exits with a non-zero status.
1297 New in version 1.0.7.
1299 after-backup-command = string
1302 Run a shell command after a backup completes. This allows a command
1303 to perform some action when a backup completes successfully such as
1304 sending out a success notification.
1305 The backup will fail if this command exits with a non-zero status.
1307 New in version 1.0.7.
1309 failed-backup-command = string
1312 Run a shell command if a backup starts. This allows some command to
1313 perform some action when a backup fails such as sending out a fail‐
1314 ure notification.
1315 The backup will fail if this command exits with a non-zero status.
1317 New in version 1.0.7.
1319 historic-size = yes | no
1322 Defaults to yes. Check for the 'backup.conf' file in the 'newest'
1323 spooled folder for the running "backupset". If the configuration
1324 file exists and contains 'estimated-size' and 'on-disk-size', these
1325 values will be used to decide if holland has enough free space for a
1326 backupset. This means Holland will use values from the previous
1327 backup to estimate the size of the next backup.
1328 New in version 1.1.1.
1330 historic-size-factor = #
1333 Defaults to '1.5'. If the estimated size of the database has changed
1334 by more than this multiple, the 'estimated-size-factor' value will
1335 be used to determine if there is sufficient free space for the back‐
1336 upset.
1337 New in version 1.1.1.
1339 historic-estimated-size-factor = #
1342 Defaults to '1.1'. Specifies the scale factor when Holland decides
1343 if there is enough free space to perform a backup. Holland will
1344 throw an error if the system has less free space than the last
1345 backup size multiplied by this value
1346 New in version 1.1.1.
1348 For all hook commands, Holland will perform simple text substitution on
1351 the three parameters:
1352 · hook - name of the hook being called (one of: before-backup-com‐
1354 mand, after-backup-command, failed-backup-command)
1355 · backupdir - path to the current backup directory (e.g.
1357 /var/spool/holland/mysqldump/YYYYmmdd_HHMMSS)
1358 · backupset - name of the backupset being run (e.g. 'mysql-lvm')
1360 For example:
1362 [holland:backup]
1364 plugin = mysqldump
1365 before-backup-command = /usr/local/bin/my-custom-script --hook ${hook} --backupset ${backupset} --backupdir ${backupdir}
1366 after-backup-command = echo ${backupset} completed successfully. Files are in ${backupdir}
1367 [mysqldump]
1369 ...
1370 Backup-Set files are defined in the "backupsets" directory which is, by
1372 default, /etc/holland/backupsets. The name of the backup-set is defined
1373 by its configuration filename and can really be most anything. That
1374 means backup-sets can be organized in any arbitrary way, although
1375 backup set files must end in .conf. The file extension is not part of
1376 the name of the backup-set.
1377 As noted above, in order for a backup-set to be active, it must be
1379 listed in the backupsets variable.
1380 Backups are placed under the directory defined in the backup_directory
1382 section of the main configuration file. Each backup resides under a
1383 directory corresponding to the backup-set name followed by a
1384 date-encoded directory.
1385
1387 Holland Core Team
1388
1390 2019, Holland Core Team
13911.1.20 Dec 05, 2019 HOLLAND(1)