1HOLLAND(1) Holland HOLLAND(1)
2
6 holland - Holland Documentation
7
9 Introduction to Holland
10 Holland is an Open Source backup framework originally developed by
11 Rackspace and written in Python. It's goal is to help facilitate back‐
12 ing up databases with greater configurability, consistency, and ease.
13 Holland currently focuses on MySQL, however future development will
14 include other database platforms and even non-database related applica‐
15 tions. Because of it's plugin structure, Holland can be used to backup
16 anything you want by whatever means you want.
17 Python2 support has been removed as of holland 1.2.0
19 Dependencies
21 The core Holland framework has the following dependencies (available on
22 any remotely modern Linux distribution):
23 · Python >= 3.6
25 · pkg_resources
27 · python-setuptools
29 · ConfigParser
31 MySQL based plugins additional require the MySQLdb python connector:
33 · pyMySQL
35 Note that other plugins may have additional dependency requirements.
37 Holland Command-Line Reference
39 Here are the commands available from the 'holland' command-line tool:
40 help (h)
42 Usage: holland <command> --help(-h)
43 Provides basic information about the provided command. If no command is
45 provided, it displays global help instead.
46 backup (bk)
48 Usage: holland backup [backup-set1, backup-set2, ..., backup-setN]
49 Runs the backup operation. If no backup-sets are specified, all active
51 backup-sets (those defined in the 'backupsets' variable in hol‐
52 land.conf) are backed up.
53 One or more backup-sets can be specified directly, in which case only
55 those backup-sets are backed up.
56 Additional Command Line Arguments:
58 --dry-run (-n): Can be used here to simulate, but not actually run, a
60 backup. This should be used when troubleshooting a particular error
61 before trying to run a real backup.
62 --no-lock (-f): Normally, only one instance of Holland can run at any
64 given time using lock-files. Using this flag causes the lock-files to
65 be ignored. This has some very clear use-cases but otherwise be mindful
66 of using this setting as it can cause backups to fail in some cases.
67 --abort-immediately: abort on the first backup-set that fails (assuming
69 multiple backupsets were specified)
70 Examples:
72 # holland bk --dry-run weekly: Attempts a dry-run of the weekly
74 backup-set.
75 # holland bk --no-lock --abort-immediately: Attempts a backup of all
77 the default backup-sets ignoring locks and aborting immediately if one
78 of the backup-sets fails.
79 list-backups (lb)
81 Usage: holland list-backups
82 Provides extended information about available backups.
84 list-plugins (lp)
86 Usage: holland list-plugins
87 Lists all the available (installed) plugins available to Holland.
89 mk-config (mc)
91 Usage: holland mk-config <provider>
92 Generates a template backup-set for a particular provider (such as
94 mysqldump). By default, the output is sent to standard out but can be
95 copied to a file, either by using the --file, --edit, or -name options
96 (see below).
97 Additional Command Line Arguments:
99 --edit: Load the file into the system text-editor for further modifica‐
101 tions.
102 --file=FILE (-f): Write the output directly to provided file.
104 --name=NAME: Creates a backup-set usable in Holland, which basically
106 means that a file is created of the provided name under the backup-set
107 directory.
108 --provider: Indicates that the default provider configuration should be
110 outputted instead. This is really only used when creating a provider
111 config specifically - it should not be used for backup-sets.
112 Examples:
114 # holland mk-config mysql-lvm > mysql-lvm.conf: Output the default con‐
116 figuration for MySQL-LVM backups and write the contents out to
117 mysql-lvm.conf in the current working directory.
118 # holland mc mysqldump --name=Bob --edit: Create a backup-set using the
120 mysqldump provider named Bob and allow interactive editing of the
121 backup-set before saving the file.
122 purge (pg)
124 Usage: holland purge <backup-set>/<backup-id>
125 Purges old backups by specifying the backup-set name and set-id.
127 For example: # holland purge mybackups/20090502_155438: Purge one of
129 the backups taken on May 2nd, 2009 from the mybackups backup-set.
130 Usage and Implementation Overview
132 Because Holland is very pluggable, it may first seem a bit confusing
133 when it comes to configuring Holland to do something useful. Out of the
134 box, Holland is designed to backup MySQL databases using the mysqldump
135 provider. This is the simplest setup, and may be sufficient for most
136 people. However, others may wish to have more fine-grained control over
137 their backups and/or use another method other than mysqldump.
138 For instance, one can configure a backup set to backup certain data‐
140 bases using mysqldump, others using the mysql-lvm plugins etc. All this
141 is done by a mix of providers and backup-sets.
142 Backup-Sets
144 Each backup-set implements exactly one provider and will inherit the
145 default values of that provider. These values can be overridden to
146 adjust the behavior of the backup set. This includes defining what
147 databases or tables to include (or exclude) in the backup, the type of
148 compression used (if any), what locking method to use, among other
149 things.
150 Providers
152 Providers essentially provide a backup service for use in a backup set.
153 · mysqldump
155 Uses the mysqldump utility to backup databases.
156 · MySQL + LVM
158 Backup MySQL databases using LVM snapshots which allows for near
159 lockless or fully lockless (when transactional engines are used)
160 backups.
161 · mysqldump + LVM
163 This plugin creates an LVM snapshot, starts a mysql instance using
164 the snapshot as it's datadir, and then use the mysqldump utility
165 to backup the databases.
166 · XtraBackup
168 New in version 1.0.8.
169 Backup MySQL databases using Percona's XtraBackup tool. This pro‐
172 vides a near lockless backup when using the InnoDB storage engine.
173 · pgdump
175 Backup PostgreSQL databases using the pgdump utility.
176 · mariabackup
178 New in version 1.1.0.
179 Backup MySQL databases using MariaDB's mariabackup tool.
182 · mongodump
184 New in version 1.1.0.
185 This plugin performs logical backups of a MongoDB by using the
188 mongodump utility.
189 · Example
191 This is used solely as a template for designing providers. It oth‐
192 erwise does nothing.
193 As Holland is a framework, it can actually backup most anything as long
195 as there is a provider for it. This includes things that have nothing
196 to do with databases. The idea is to present an easy to use and clear
197 method of backing up and restoring backups no matter the source.
198 Holland Config Files
200 By default, Holland's configuration files reside in /etc/holland. The
201 main configuration file is holland.conf, however there are a number of
202 other configuration files for configuring default settings for
203 providers and for configuring backup sets.
204 Each configuration file has one ore more sections, defined by square
206 brackets Underneath each section, one or more configuration option can
207 be specified. These options are in a standard "option = value" format.
208 Comments are prefixed by the # sign.
209 Note that many settings have default values and, as a result, can
211 either be commented out or omitted entirely.
212 holland.conf - main config
214 The main configuration file (usually /etc/holland/holland.conf) defines
215 both global settings as well as the active backup sets. It is divided
216 into two sections [holland] and [logging].
217 [holland]
219 plugin-dirs
220 Defines where the plugins can be found. This can be a comma-sep‐
221 arated list but usually does not need to be modified.
222 backup_directory
224 Top-level directory where backups are held.
225 backupsets
227 A comma-separated list of all the backup sets Holland should
228 backup. Each backup set is defined in /etc/holland/back‐
229 upsets/<name>.conf by default.
230 umask Sets the umask of the resulting backup files.
232 path Defines a path for holland and its spawned processes
234 [logging]
236 filename
237 The log file itself.
238 level Sets the verbosity of Holland's logging process. Available
240 options are debug, info, warning, error, and critical
241 format Define the format of the log message. This options should be in
243 the format defined by the 'logging' python library.
244 New in version 1.1.0.
246 Provider Configs
249 These files control the global settings / defaults for the providers
250 used by the backup-sets. Many of these global settings can be overrid‐
251 den if defined in a backup-set. Note that each provider's configuration
252 file should begin with [provider-name].
253 mysqldump Provider Configuration [mysqldump]
255 Backs up a MySQL database using the mysqldump tool.
256 [mysqldump]
258 mysql-binpath = /path/to/mysql/bin
259 Defines the location of the MySQL binary utilities. If not provided,
260 Holland will use whatever is in the path.
261 lock-method = flush-lock | lock-tables | single-transaction |
263 auto-detect | none
264 Defines which lock method to use. By default, auto-detect will be
265 used.
266 · flush-lock
268 flush-lock will place a global lock on all tables involved in
269 the backup regardless of whether or not they are in the
270 backup-set. If file-per-database is enabled, then flush-lock
271 will lock all tables for every database being backed up. In
272 other words, this option may not make much sense when using
273 file-per-database.
274 · lock-tables
276 lock-tables will lock all tables involved in the backup. If
277 file-per-database is enabled, then lock-tables will only lock
278 all the tables associated with that database.
279 · single-transaction
281 Forces the use of --single-transaction which enabled
282 semi-transparent backups of transactional tables. Forcing this
283 can cause inconsistencies with non-transactional tables, how‐
284 ever. While non-transactional tables will still lock, they
285 will only lock when they are actually being backed up. Use
286 this setting with extreme caution when backing non-transac‐
287 tional tables.
288 · auto-detect
290 Let Holland decide which option to use by checking to see if a
291 database or backup-set only contains transactional tables. If
292 so, --single-transaction will be used. Otherwise,
293 --lock-tables will be used.
294 · none
296 Does absolutely no explicit locking when backing up the data‐
297 bases or backup-set. This should only be used when backing up
298 a slave and only after the slave has been turned off (ie, this
299 can be used with the stop-slave option).
300 exclude-invalid-views = yes | no (default: no)
302 Whether to automate exclusion of invalid views that would otherwise
303 cause mysqldump to fail. This adds additional overhead so this
304 option is not enabled by default.
305 When enabled, thos option will scan the INFORMATION_SCHEMA.VIEWS ta‐
307 ble and execute SHOW FIELDS against each view. If a view is detects
308 as invalid, an ignore-table option will be added to exclude the ta‐
309 ble. Additionally, the plugin will attempt to save the view defin‐
310 ion to 'invalid_views.sql' in the backupset's backup directory.
311 New in version 1.0.8.
313 dump-routines = yes | no (default: yes)
316 Whether or not to backup routines in the backup set directly. Rou‐
317 tines are stored in the 'mysql' database, but it can sometimes be
318 convenient to include them in a backup-set directly.
319 Changed in version 1.0.8: This option now enabled by default.
321 dump-events = yes | no
324 Whether or not to dump events explicitly. Like routines, events are
325 stored in the 'mysql' database. Nonetheless, it can sometimes be
326 convenient to include them in the backup-set directly.
327 Note: This feature requires MySQL 5.1 or later. The mysqldump plugin
329 will automatically disable events if the version of mysqldump is too
330 old.
331 Changed in version 1.0.8: This option is now enabled by default
333 stop-slave = yes | no
336 Stops the SQL_THREAD during the backup. This means that writes from
337 the master will continue to spool but will not be replayed. This
338 helps avoid lock wait timeouts among things while still allowing
339 data to be spooled from the master.
340 Note that previous versions of Holland prior to 1.0.6 simply ran a
342 STOP SLAVE instead, which suspends both replication threads.
343 bin-log-position = yes | no
345 Record the binary log name and position at the time of the backup.
346 The information provied by this option is collected just before
347 locking the database.
348 Note that if both 'stop-slave' and 'bin-log-position' are enabled,
350 Holland will grab the master binary log name and position at the
351 time of the backup which can be useful in using the backup to create
352 slaves or for point in time recovery using the master's binary log.
353 This information is found within the 'backup.conf' file located in
354 the backup-set destination directory (/var/spool/hol‐
355 land/<backup-set>/<backup> by default). For example:
356 [mysql:replication]
358 slave_master_log_pos = 4512
359 slave_master_log_file = 260792-mmm-agent1-bin-log.000001
360 flush-logs = yes | no
362 Whether or not to run FLUSH LOGS in MySQL with the backup. When
363 FLUSH LOGS is actually executed depends on which if database filter‐
364 ing is being used and whether or not file-per-database is enabled.
365 Generally speaking, it does not make sense to use flush-logs with
366 file-per-database since the binary logs will not be consistent with
367 the backup.
368 file-per-database = yes | no
370 Whether or not to split up each database into its own file. Note
371 that it can be more consistent an efficient to backup all databases
372 into one file, however this means that restore a single database can
373 be difficult if multiple databases are defined in the backup set.
374 arg-per-database = JSON object
376 If file-per-database is enable this argument is ued to specify
377 mysqldump arguments per database. It takes a JSON object with the
378 database names as keys. Example: {"employee1": "--no-data"} Adds
379 the '--no-data' argument to the mysqldump command when backing up
380 the 'employee1' database
381 New in version 1.0.9.
383 additional-options = <mysqldump argument>[, <mysqldump argument>]
386 Can optionally specify additional options directly to mysqldump if
387 there is no native Holland option available. This option accepts a
388 comma delimited list of arguments to pass on the commandline.
389 extra-defaults = yes | no (default: no)
391 This option controls whether mysqldump will only read options as set
392 by holland or if additional options from global config files are
393 read. By default, the plugin only uses optons as set in the back‐
394 upset config and includes authentication credentials only from the
395 [client] section in ~/.my.cnf.
396 estimate-method = plugin | const:<size> (default: plugin)
398 This option will skip some of the heavyweight queries necessary to
399 calculate the size of tables to be backed up. If a constant size is
400 specified, then only table names are evaluated and only if table
401 filtering is being used. Additionally, engines will be looked up via
402 SHOW CREATE TABLE if lock-method = auto-detect, in order for the
403 plugin to determine if tables are using a transactional storage
404 engine. With 'plugin', the default behavior of reading both size
405 information and table names from the information schema is used,
406 which may be slow particularly for a large number of tables.
407 Database and Table filtering
409 Database and Table filtering
410 databases = <glob>
411 exclude-databases = <glob>
413 tables = <glob>
415 exclude-tables = <glob>
417 The above options accepts GLOBs in comma-separated lists. Multiple fil‐
419 tering options can be specified. When filtering on tables, be sure to
420 include both the database and table name.
421 Be careful with quotes. Normally these are not needed, but when quotes
423 are necessary, be sure to only quote each filtering statement, as
424 opposed to putting quotes around all statements.
425 Below are a few examples of how these can be applied:
427 Default (backup everything):
429 databases = *
431 tables = *
432 Using database inclusion and exclusions:
434 databases = drupal*, smf_forum,
436 exclude-databases = drupal5
437 Including Tables:
439 tables = phpBB.sucks, drupal6.node*, smf_forum.*
441 Excluding Tables:
443 exclude-tables = mydb.uselesstable1, x_cart.*, *.sessions
445 [compression]
447 Specify various compression settings, such as compression utility, com‐
448 pression level, etc.
449 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
451 gpg | zstd
452 Define which compression method to use. Note that some methods may
453 not be available by default on every system and may need to be com‐
454 piled or installed.
455 For gpg compression, a key should already exist(gpg --gen-key) and
457 default-recipient must be configured in ~/.gnupg/gpg.conf.
458 inline = yes | no
460 Whether or not to pipe the output of the backup command into the
461 compression utility. Enabling this is recommended since it usually
462 only marginally impacts performance, particularly when using a lower
463 compression level.
464 level = 0-9
466 Specify the compression ratio. The lower the number, the lower the
467 compression ratio, but the faster the backup will take. Generally,
468 setting the lever to 1 or 2 results in favorable compression of tex‐
469 tual data and is noticeably faster than the higher levels. Setting
470 the level to 0 effectively disables compression.
471 bin-path = <full path to utility>
473 This only needs to be defined if the compression utility is not in
474 the usual places or not in the system path.
475 options = <string>
477 Add commandline options to the configuration compression command.
478 options = "-Q4"
480 split = yes | no
482 Defautls to no. If set the backup will be piped through the split
483 command. This may be useful for user's with large databases, as
484 some backup systems perform better with many smaller files instead
485 of 1 large one. This defaults to 1GB file size, so this option isn't
486 helpful if your dumps are smaller than that.
487 For python2.6, this option will be disabled if the subprocess32 mod‐
489 ule isn't avaiable.
490 New in version 1.1.13.
492 MySQL connection info [mysql:client]
495 These are optional and, if left undefined, Holland will try to login
496 using the standard .my.cnf conventions.
497 user = <user>
499 The user to connect to MySQL as.
501 password = <password>
503 The password for the MySQL user
505 socket = <socket>
507 The socket file to connect to MySQL with.
509 host = <host>
511 This would be used for connecting to MySQL remotely.
513 port = <port>
515 Used if MySQL is running on a port other than 3306.
517 MySQL LVM Provider Configuration [mysql-lvm]
519 Creates an LVM snapshot of a running MySQL instance and performs a
520 binary-based backup with minimal locking. MySQL must be running on an
521 LVM volume with reserved space for snapshots. It is highly recommended
522 that this volume be separate from the one storing the resulting back‐
523 ups.
524 [mysql-lvm]
526 snapshot-size = <size-in-MB>
527 The size of the snapshot itself. By default it is 20% of the size of
528 the MySQL LVM mount or the remaining free-space in the Volume Group
529 (if there is less than 20% available) up to 15GB. If snapshot-size
530 is defined, the number represents the size of the snapshot in
531 megabytes.
532 snapshot-name = <name>
534 The name of the snapshot, the default being the name of the MySQL
535 LVM volume + "_snapshot" (ie Storage-MySQL_snapshot)
536 snapshot-mountpoint = <path>
538 Where to mount the snapshot. By default a randomly generated direc‐
539 tory under /tmp is used.
540 innodb-recovery = yes | no (default: no)
542 Whether or not to run an InnoDB recovery operation. This avoids
543 needing to do so during a restore, though will make the backup
544 process itself take longer.
545 force-innodb-backup = yes | no (default: no)
547 Whether to attempt a backup even if the mysql-lvm plugin thinks it
548 cannot obtain a good backup. This can occur when innodb data files
549 are outside of the mysql datadir or exist on entirely separate logi‐
550 cal volumes.
551 lock-tables = yes | no (default: yes)
553 Whether or not to run a FLUSH TABLES WITH READ LOCK to grab various
554 bits of information (such as the binary log name and position). Dis‐
555 abling this requires that binary logging is disabled and InnoDB is
556 being used exclusively. Otherwise, it is possible that the backup
557 could contain crashed tables.
558 extra-flush-tables = yes | no (default: yes)
560 Whether or not to run a FLUSH TABLES before running the full FLUSH
561 TABLES WITH READ LOCK. Should make the FLUSH TABLES WITH READ LOCK
562 operation a bit faster.
563 archive-method = tar | dir (default: tar)
565 Create a tar file of the datadir, or just copy it.
566 [tar]
568 exclude = pattern[, pattern...]
569 Patterns to exclude from archive. These should be relative paths and
571 are almost always relative to the mysql data directory. For instance
572 to exclude binary logs in the data directory from the backup you might
573 specify: exclude = ./bin-log.*, mysql.sock
574 pre-args = <string>
576 Additional arguments to append to the tar commandline before the backup
578 path is specified. This should be the full string as you might specify
579 on the commandline. Shell globbing is not supported.
580 For instance you might add the /etc/my.cnf to the tar archive via:
582 pre-args = -C /etc ./my.cnf
583 post-args = <string>
585 Additional arguments to append to the tar commandline after the backup
587 path is specified. This should be a string exactly as you might spec‐
588 ify on the commandline. Shell globbing is not evaluated.
589 [compression]
591 Specify various compression settings, such as compression utility, com‐
592 pression level, etc.
593 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
595 gpg | zstd
596 Define which compression method to use. Note that some methods may
597 not be available by default on every system and may need to be com‐
598 piled or installed.
599 For gpg compression, a key should already exist(gpg --gen-key) and
601 default-recipient must be configured in ~/.gnupg/gpg.conf.
602 inline = yes | no
604 Whether or not to pipe the output of the backup command into the
605 compression utility. Enabling this is recommended since it usually
606 only marginally impacts performance, particularly when using a lower
607 compression level.
608 level = 0-9
610 Specify the compression ratio. The lower the number, the lower the
611 compression ratio, but the faster the backup will take. Generally,
612 setting the lever to 1 or 2 results in favorable compression of tex‐
613 tual data and is noticeably faster than the higher levels. Setting
614 the level to 0 effectively disables compression.
615 bin-path = <full path to utility>
617 This only needs to be defined if the compression utility is not in
618 the usual places or not in the system path.
619 options = <string>
621 Add commandline options to the configuration compression command.
622 options = "-Q4"
624 split = yes | no
626 Defautls to no. If set the backup will be piped through the split
627 command. This may be useful for user's with large databases, as
628 some backup systems perform better with many smaller files instead
629 of 1 large one. This defaults to 1GB file size, so this option isn't
630 helpful if your dumps are smaller than that.
631 For python2.6, this option will be disabled if the subprocess32 mod‐
633 ule isn't avaiable.
634 New in version 1.1.13.
636 MySQL connection info [mysql:client]
639 These are optional and, if left undefined, Holland will try to login
640 using the standard .my.cnf conventions.
641 user = <user>
643 The user to connect to MySQL as.
645 password = <password>
647 The password for the MySQL user
649 socket = <socket>
651 The socket file to connect to MySQL with.
653 host = <host>
655 This would be used for connecting to MySQL remotely.
657 port = <port>
659 Used if MySQL is running on a port other than 3306.
661 mysqldump LVM Provider Configuration [mysqldump-lvm]
663 Backs up one or more MySQL databases by creating an LVM snapshot and
664 then starting a instance of MySQL on top of it to then perform a mysql‐
665 dump. This effectively produces a non-blocking logical backup.
666 [mysql-lvm]
668 snapshot-size = <size-in-MB>
669 The size of the snapshot itself. By default it is 20% of the size of
670 the MySQL LVM mount or the remaining free-space in the Volume Group
671 (if there is less than 20% available) up to 15GB. If snapshot-size
672 is defined, the number represents the size of the snapshot in
673 megabytes.
674 snapshot-name = <name>
676 The name of the snapshot, the default being the name of the MySQL
677 LVM volume + "_snapshot" (ie Storage-MySQL_snapshot)
678 snapshot-mountpoint = <path>
680 Where to mount the snapshot. By default a randomly generated direc‐
681 tory under /tmp is used.
682 innodb-recovery = yes | no (default: no)
684 Whether or not to run an InnoDB recovery operation. This avoids
685 needing to do so during a restore, though will make the backup
686 process itself take longer.
687 lock-tables = yes | no (default: yes)
689 Whether or not to run a FLUSH TABLES WITH READ LOCK to grab various
690 bits of information (such as the binary log name and position). Dis‐
691 abling this requires that binary logging is disabled and InnoDB is
692 being used exclusively. Otherwise, it is possible that the backup
693 could contain crashed tables.
694 extra-flush-tables = yes | no (default: yes)
696 Whether or not to run a FLUSH TABLES before running the full FLUSH
697 TABLES WITH READ LOCK. Should make the FLUSH TABLES WITH READ LOCK
698 operation a bit faster.
699 [mysqld]
701 mysqld-exe = <path>[, <path>...] (default: mysqld in PATH,
702 /usr/libexec/mysqld)
703 This provides a list of locations where the mysqld process to use
704 might be found. This is searched in order of entries in this list.
705 user = <name>
707 The --user parameter to use with mysqld.
708 innodb-buffer-pool-size = <size> (default: 128M)
710 How large to size the innodb-buffer-pool-size.
711 tmpdir = <path> (default: system tempdir)
713 Path to the --tmpdir that mysqld should use.
714 log-error = <path> (default: tempdir/holland_lvm.log)
716 Define path for mysqld's error log. The default location get cleaned
717 up by Holland after the backup is complete. This settings allows the
718 user to define the log file in another location and can be useful
719 for debugging issue with the MySQL instance running on the snapshot.
720 New in version 1.0.9.
722 [mysqldump]
725 mysqldump-lvm supports almost all of the options from the mysqldump
726 plugin. --master-data is not supported, as the mysqld process will not
727 read binary logs, so this plugin will automatically disable
728 bin-log-position, if set.
729 Binary log information from SOHW MASTER STATUS and SHOW SLAVE STATUS is
731 recorded in the ${backup_directory}/backup.conf file under the
732 [mysql:replication] section.
733 [compression]
735 Specify various compression settings, such as compression utility, com‐
736 pression level, etc.
737 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
739 gpg | zstd
740 Define which compression method to use. Note that some methods may
741 not be available by default on every system and may need to be com‐
742 piled or installed.
743 For gpg compression, a key should already exist(gpg --gen-key) and
745 default-recipient must be configured in ~/.gnupg/gpg.conf.
746 inline = yes | no
748 Whether or not to pipe the output of the backup command into the
749 compression utility. Enabling this is recommended since it usually
750 only marginally impacts performance, particularly when using a lower
751 compression level.
752 level = 0-9
754 Specify the compression ratio. The lower the number, the lower the
755 compression ratio, but the faster the backup will take. Generally,
756 setting the lever to 1 or 2 results in favorable compression of tex‐
757 tual data and is noticeably faster than the higher levels. Setting
758 the level to 0 effectively disables compression.
759 bin-path = <full path to utility>
761 This only needs to be defined if the compression utility is not in
762 the usual places or not in the system path.
763 options = <string>
765 Add commandline options to the configuration compression command.
766 options = "-Q4"
768 split = yes | no
770 Defautls to no. If set the backup will be piped through the split
771 command. This may be useful for user's with large databases, as
772 some backup systems perform better with many smaller files instead
773 of 1 large one. This defaults to 1GB file size, so this option isn't
774 helpful if your dumps are smaller than that.
775 For python2.6, this option will be disabled if the subprocess32 mod‐
777 ule isn't avaiable.
778 New in version 1.1.13.
780 MySQL connection info [mysql:client]
783 These are optional and, if left undefined, Holland will try to login
784 using the standard .my.cnf conventions.
785 user = <user>
787 The user to connect to MySQL as.
789 password = <password>
791 The password for the MySQL user
793 socket = <socket>
795 The socket file to connect to MySQL with.
797 host = <host>
799 This would be used for connecting to MySQL remotely.
801 port = <port>
803 Used if MySQL is running on a port other than 3306.
805 Xtrabackup Provider Configuration [xtrabackup]
807 Backs up a MySQL instance using Percona's Xtrabackup tool.
808 [xtrabackup]
810 global-defaults = <path> (default: /etc/my.cnf)
811 The MySQL configuration file for xtrabackup to parse. This is
812 !include'd into the my.cnf the xtrabackup plugin generates
813 innobackupex = <name> (default: innobackupex-1.5.1)
815 The path to the innobackupex script to run. If this is a relative
816 path this will be found in holland's environment PATH as configured
817 in /etc/holland/holland.conf.
818 Changed in version 1.1.12: For xtrabackup version 8.0 or greater, this
820 option is ignored
821 ibbackup = <name>
824 The path to the ibbackup command to use. By default, no --ibbackup
825 option is pass to the innobackupex script. Usually innobackupex
826 will detect this by itself and this should not need to be set.
827 stream = tar|xbstream|yes|no (default: tar)
829 Whether to generate a streaming backup.
830 Changed in version 1.0.8: 'tar' and 'xbstream' are now valid options.
832 The old stream = yes is now equivalent to stream = tar and stream = no
833 disables streaming entirely and will result in a normal directory copy
834 with xtrabackup
835 Changed in version 1.1.12: For xtrabackup version 8.0 or
838 greater,'xstream' will be used unless this value is set to 'no'
839 apply-logs = yes | no (default: yes)
842 Whether to run innobackupex --apply-logs at the end of the backup.
843 This is only supported when performing a non-streaming, non-com‐
844 pressed backup. In this case, even if apply-logs = yes (the
845 default), the prepare stage will be skipped. Even with an uncom‐
846 pressed, non-streaming backup you may want to disable apply-logs if
847 you wish to use incremental backups.
848 New in version 1.0.8.
850 slave-info = yes | no (default: yes)
853 Whether to enable the --slave-info innobackupex option
854 safe-slave-backup = yes | no (default: yes)
856 Whether to enable the --safe-slave-backup innobackupex option.
857 no-lock = yes | no (default: no)
859 Whether to enable the --no-lock innobackupex option
860 tmpdir = <path> (default: ${backup_directory})
862 The path for the innobackupex --tmpdir option. By default this will
863 use the current holland backup directory to workaround the following
864 bug: https://bugs.launchpad.net/percona-xtrabackup/+bug/1007446
865 New in version 1.0.8.
867 additional-options = <option>[, <option>...]
870 A list of additional options to pass to innobackupex. This is a
871 comma separated list of options.
872 pre-command = <command-string>
874 A command to run prior to running this xtrabackup run. This can be
875 used, for instance, to generate a mysqldump schema dump prior to
876 running xtrabackup. instances of ${backup_directory} will be
877 replaced with the current holland backup directory where the xtra‐
878 backup data will be stored.
879 [compression]
881 Specify various compression settings, such as compression utility, com‐
882 pression level, etc.
883 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
885 gpg | zstd
886 Define which compression method to use. Note that some methods may
887 not be available by default on every system and may need to be com‐
888 piled or installed.
889 For gpg compression, a key should already exist(gpg --gen-key) and
891 default-recipient must be configured in ~/.gnupg/gpg.conf.
892 inline = yes | no
894 Whether or not to pipe the output of the backup command into the
895 compression utility. Enabling this is recommended since it usually
896 only marginally impacts performance, particularly when using a lower
897 compression level.
898 level = 0-9
900 Specify the compression ratio. The lower the number, the lower the
901 compression ratio, but the faster the backup will take. Generally,
902 setting the lever to 1 or 2 results in favorable compression of tex‐
903 tual data and is noticeably faster than the higher levels. Setting
904 the level to 0 effectively disables compression.
905 bin-path = <full path to utility>
907 This only needs to be defined if the compression utility is not in
908 the usual places or not in the system path.
909 options = <string>
911 Add commandline options to the configuration compression command.
912 options = "-Q4"
914 split = yes | no
916 Defautls to no. If set the backup will be piped through the split
917 command. This may be useful for user's with large databases, as
918 some backup systems perform better with many smaller files instead
919 of 1 large one. This defaults to 1GB file size, so this option isn't
920 helpful if your dumps are smaller than that.
921 For python2.6, this option will be disabled if the subprocess32 mod‐
923 ule isn't avaiable.
924 New in version 1.1.13.
926 MySQL connection info [mysql:client]
929 These are optional and, if left undefined, Holland will try to login
930 using the standard .my.cnf conventions.
931 user = <user>
933 The user to connect to MySQL as.
935 password = <password>
937 The password for the MySQL user
939 socket = <socket>
941 The socket file to connect to MySQL with.
943 host = <host>
945 This would be used for connecting to MySQL remotely.
947 port = <port>
949 Used if MySQL is running on a port other than 3306.
951 pgdump Provider Configuration [pgdump]
953 Backs up a PostgreSQL instance using the pgdump utility.
954 [pgdump]
956 format = custom | tar | plain (default: custom)
957 Defines the --format option for pg_dump. This defaults to --for‐
958 mat=custom. The custom format is required for pg_restore to do par‐
959 tial restore as well as enabling parallel restores. If set to cus‐
960 tom, the --compress option will be passed to pgdump
961 additional-options = <command-string>
963 Pass additional options to the pg_dump command
964 Only the 'level' option will be used if 'format=custom'
966 [compression]
968 Specify various compression settings, such as compression utility, com‐
969 pression level, etc.
970 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
972 gpg | zstd
973 Define which compression method to use. Note that some methods may
974 not be available by default on every system and may need to be com‐
975 piled or installed.
976 For gpg compression, a key should already exist(gpg --gen-key) and
978 default-recipient must be configured in ~/.gnupg/gpg.conf.
979 inline = yes | no
981 Whether or not to pipe the output of the backup command into the
982 compression utility. Enabling this is recommended since it usually
983 only marginally impacts performance, particularly when using a lower
984 compression level.
985 level = 0-9
987 Specify the compression ratio. The lower the number, the lower the
988 compression ratio, but the faster the backup will take. Generally,
989 setting the lever to 1 or 2 results in favorable compression of tex‐
990 tual data and is noticeably faster than the higher levels. Setting
991 the level to 0 effectively disables compression.
992 bin-path = <full path to utility>
994 This only needs to be defined if the compression utility is not in
995 the usual places or not in the system path.
996 options = <string>
998 Add commandline options to the configuration compression command.
999 options = "-Q4"
1001 split = yes | no
1003 Defautls to no. If set the backup will be piped through the split
1004 command. This may be useful for user's with large databases, as
1005 some backup systems perform better with many smaller files instead
1006 of 1 large one. This defaults to 1GB file size, so this option isn't
1007 helpful if your dumps are smaller than that.
1008 For python2.6, this option will be disabled if the subprocess32 mod‐
1010 ule isn't avaiable.
1011 New in version 1.1.13.
1013 [pgauth]
1016 username = <name>
1017 Username for pg_dump to authenticate with
1018 password = <string>
1020 Password for pg_dump to authenticate with
1021 hostname = <string>
1023 Hostname for pg_dump to connect with
1024 port = <integer>
1026 TCP port for pg_dump to connect on
1027 mongodump Provider Configuration [mongodump]
1029 This plugin performs logical backups of a MongoDB by using the mongo‐
1030 dump utility.
1031 [mongodump]
1033 host = <string>
1034 Hostname for mongodump to connect with
1035 username = <name>
1037 Username for mongodump to authenticate with
1038 password = <string>
1040 Password for mongodump to authenticate with
1041 port = <integer>
1043 TCP port for mongodump to connect on
1044 uri = <string>
1046 Use a connection string instead of a host, username, and pass‐
1047 word
1048 New in version 1.1.14.
1050 [compression]
1053 Specify various compression settings, such as compression utility, com‐
1054 pression level, etc.
1055 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
1057 gpg | zstd
1058 Define which compression method to use. Note that some methods may
1059 not be available by default on every system and may need to be com‐
1060 piled or installed.
1061 For gpg compression, a key should already exist(gpg --gen-key) and
1063 default-recipient must be configured in ~/.gnupg/gpg.conf.
1064 inline = yes | no
1066 Whether or not to pipe the output of the backup command into the
1067 compression utility. Enabling this is recommended since it usually
1068 only marginally impacts performance, particularly when using a lower
1069 compression level.
1070 level = 0-9
1072 Specify the compression ratio. The lower the number, the lower the
1073 compression ratio, but the faster the backup will take. Generally,
1074 setting the lever to 1 or 2 results in favorable compression of tex‐
1075 tual data and is noticeably faster than the higher levels. Setting
1076 the level to 0 effectively disables compression.
1077 bin-path = <full path to utility>
1079 This only needs to be defined if the compression utility is not in
1080 the usual places or not in the system path.
1081 options = <string>
1083 Add commandline options to the configuration compression command.
1084 options = "-Q4"
1086 split = yes | no
1088 Defautls to no. If set the backup will be piped through the split
1089 command. This may be useful for user's with large databases, as
1090 some backup systems perform better with many smaller files instead
1091 of 1 large one. This defaults to 1GB file size, so this option isn't
1092 helpful if your dumps are smaller than that.
1093 For python2.6, this option will be disabled if the subprocess32 mod‐
1095 ule isn't avaiable.
1096 New in version 1.1.13.
1098 Mariabackup Provider Configuration [mariabackup]
1101 Backs up a MySQL instance using mariabackup tool.
1102 [mariabackup]
1104 global-defaults = <path> (default: /etc/my.cnf)
1105 The MySQL configuration file for mariabackup to parse. This is
1106 !include'd into the my.cnf the mariabackup plugin generates
1107 innobackupex = <name> (default: innobackupex-1.5.1)
1109 The path to the innobackupex script to run. If this is a relative
1110 path this will be found in holland's environment PATH as configured
1111 in /etc/holland/holland.conf.
1112 ibbackup = <name>
1114 The path to the ibbackup command to use. By default, no --ibbackup
1115 option is pass to the innobackupex script. Usually innobackupex
1116 will detect this by itself and this should not need to be set.
1117 stream = mbstream(default: tar)
1119 Placeholder
1120 apply-logs = yes | no (default: yes)
1122 Whether to run innobackupex --apply-logs at the end of the backup.
1123 This is only supported when performing a non-streaming, non-com‐
1124 pressed backup. In this case, even if apply-logs = yes (the
1125 default), the prepare stage will be skipped. Even with an uncom‐
1126 pressed, non-streaming backup you may want to disable apply-logs if
1127 you wish to use incremental backups.
1128 slave-info = yes | no (default: yes)
1130 Whether to enable the --slave-info innobackupex option
1131 safe-slave-backup = yes | no (default: yes)
1133 Whether to enable the --safe-slave-backup innobackupex option.
1134 no-lock = yes | no (default: no)
1136 Whether to enable the --no-lock innobackupex option
1137 tmpdir = <path> (default: ${backup_directory})
1139 The path for the innobackupex --tmpdir option.
1140 additional-options = <option>[, <option>...]
1142 A list of additional options to pass to innobackupex. This is a
1143 comma separated list of options.
1144 pre-command = <command-string>
1146 A command to run prior to running this mariabackup run. This can be
1147 used, for instance, to generate a mysqldump schema dump prior to
1148 running mariabackup. instances of ${backup_directory} will be
1149 replaced with the current holland backup directory where the mari‐
1150 abackup data will be stored.
1151 [compression]
1153 Specify various compression settings, such as compression utility, com‐
1154 pression level, etc.
1155 method = gzip| gzip-rsyncable | pigz | bzip2 | pbzip2 | lzop | lzma |
1157 gpg | zstd
1158 Define which compression method to use. Note that some methods may
1159 not be available by default on every system and may need to be com‐
1160 piled or installed.
1161 For gpg compression, a key should already exist(gpg --gen-key) and
1163 default-recipient must be configured in ~/.gnupg/gpg.conf.
1164 inline = yes | no
1166 Whether or not to pipe the output of the backup command into the
1167 compression utility. Enabling this is recommended since it usually
1168 only marginally impacts performance, particularly when using a lower
1169 compression level.
1170 level = 0-9
1172 Specify the compression ratio. The lower the number, the lower the
1173 compression ratio, but the faster the backup will take. Generally,
1174 setting the lever to 1 or 2 results in favorable compression of tex‐
1175 tual data and is noticeably faster than the higher levels. Setting
1176 the level to 0 effectively disables compression.
1177 bin-path = <full path to utility>
1179 This only needs to be defined if the compression utility is not in
1180 the usual places or not in the system path.
1181 options = <string>
1183 Add commandline options to the configuration compression command.
1184 options = "-Q4"
1186 split = yes | no
1188 Defautls to no. If set the backup will be piped through the split
1189 command. This may be useful for user's with large databases, as
1190 some backup systems perform better with many smaller files instead
1191 of 1 large one. This defaults to 1GB file size, so this option isn't
1192 helpful if your dumps are smaller than that.
1193 For python2.6, this option will be disabled if the subprocess32 mod‐
1195 ule isn't avaiable.
1196 New in version 1.1.13.
1198 MySQL connection info [mysql:client]
1201 These are optional and, if left undefined, Holland will try to login
1202 using the standard .my.cnf conventions.
1203 user = <user>
1205 The user to connect to MySQL as.
1207 password = <password>
1209 The password for the MySQL user
1211 socket = <socket>
1213 The socket file to connect to MySQL with.
1215 host = <host>
1217 This would be used for connecting to MySQL remotely.
1219 port = <port>
1221 Used if MySQL is running on a port other than 3306.
1223 Backup-Set Configs
1225 Backup-Set configuration files largely inherit the configuration
1226 options of the specified provider. To define a provider for the backup
1227 set, you must put the following at the top of the backup set configura‐
1228 tion file:
1229 [holland:backup]
1231 plugin = <plugin>
1232 backups-to-keep = #
1233 estimated-size-factor = #
1234 historic-size = <yes|no>
1235 historic-size-factor = #
1236 historic-estimated-size-factor = #
1237 Configuration Options
1239 plugin = <plugin>
1240 This is the name of the provider that will be used for the
1241 backup-set. This is required in order for the backup-set to func‐
1242 tion.
1243 backups-to-keep = #
1245 Specifies the number of backups to keep for a backup-set. Defaults
1246 to retaining 1 backup.
1247 estimated-size-factor = #
1249 Specifies the scale factor when Holland decides if there is enough
1250 free space to perform a backup. The default is 1.0 and this number
1251 is multiplied against what each individual plugin reports its esti‐
1252 mated backup size when Holland is verifying sufficient free space
1253 for the backupset.
1254 As of Holland 1.1.1, the 'historic' backup size will be used. If
1256 Holland is unable to determine that size, it will default back to
1257 this.
1258 auto-purge-failures = yes | no
1260 Specifies whether to keep a failed backup or to automatically remove
1261 the backup directory. By default this is on with the intention that
1262 whatever process is calling holland will retry when a backup fails.
1263 This behavior can be disabled by setting auto-purge-failures = no
1264 when partial backups might be useful or when troubleshooting a
1265 backup failure.
1266 purge-policy = manual | before-backup | after-backup
1268 Specifies when to run the purge routine on a backupset. By default
1269 this is run after a new successful backup completes. Up to back‐
1270 ups-to-keep backups will be retained including the most recent.
1271 purge-policy = before-backup will run the purge routine just before
1273 a new backup starts. This will retain up to backups-to-keep backups
1274 before the new backup is even started allowing purging all previous
1275 backups if backups-to-keep is set to 0. This behavior is useful if
1276 some other process is retaining backups off-server and disk space is
1277 at a premium.
1278 purge-policy = manual will never run the purge routine automati‐
1280 cally. Either holland purge must be run externally or an explicit
1281 removal of desired backup directories can be done at some later
1282 time.
1283 before-backup-command = string
1285 Run a shell command before a backup starts. This allows a command
1286 to perform some action before the backup starts such as setting up
1287 an iptables rule (taking a mysql slave out of a load balancer) or
1288 aborting the backup based on some external condition.
1289 The backup will fail if this command exits with a non-zero status.
1291 New in version 1.0.7.
1293 after-backup-command = string
1296 Run a shell command after a backup completes. This allows a command
1297 to perform some action when a backup completes successfully such as
1298 sending out a success notification.
1299 The backup will fail if this command exits with a non-zero status.
1301 New in version 1.0.7.
1303 failed-backup-command = string
1306 Run a shell command if a backup starts. This allows some command to
1307 perform some action when a backup fails such as sending out a fail‐
1308 ure notification.
1309 The backup will fail if this command exits with a non-zero status.
1311 New in version 1.0.7.
1313 historic-size = yes | no
1316 Defaults to yes. Check for the 'backup.conf' file in the 'newest'
1317 spooled folder for the running "backupset". If the configuration
1318 file exists and contains 'estimated-size' and 'on-disk-size', these
1319 values will be used to decide if holland has enough free space for a
1320 backupset. This means Holland will use values from the previous
1321 backup to estimate the size of the next backup.
1322 New in version 1.1.1.
1324 historic-size-factor = #
1327 Defaults to '1.5'. If the estimated size of the database has changed
1328 by more than this multiple, the 'estimated-size-factor' value will
1329 be used to determine if there is sufficient free space for the back‐
1330 upset.
1331 New in version 1.1.1.
1333 historic-estimated-size-factor = #
1336 Defaults to '1.1'. Specifies the scale factor when Holland decides
1337 if there is enough free space to perform a backup. Holland will
1338 throw an error if the system has less free space than the last
1339 backup size multiplied by this value
1340 New in version 1.1.1.
1342 For all hook commands, Holland will perform simple text substitution on
1345 the three parameters:
1346 · hook - name of the hook being called (one of: before-backup-com‐
1348 mand, after-backup-command, failed-backup-command)
1349 · backupdir - path to the current backup directory (e.g.
1351 /var/spool/holland/mysqldump/YYYYmmdd_HHMMSS)
1352 · backupset - name of the backupset being run (e.g. 'mysql-lvm')
1354 For example:
1356 [holland:backup]
1358 plugin = mysqldump
1359 before-backup-command = /usr/local/bin/my-custom-script --hook ${hook} --backupset ${backupset} --backupdir ${backupdir}
1360 after-backup-command = echo ${backupset} completed successfully. Files are in ${backupdir}
1361 [mysqldump]
1363 ...
1364 Backup-Set files are defined in the "backupsets" directory which is, by
1366 default, /etc/holland/backupsets. The name of the backup-set is defined
1367 by its configuration filename and can really be most anything. That
1368 means backup-sets can be organized in any arbitrary way, although
1369 backup set files must end in .conf. The file extension is not part of
1370 the name of the backup-set.
1371 As noted above, in order for a backup-set to be active, it must be
1373 listed in the backupsets variable.
1374 Backups are placed under the directory defined in the backup_directory
1376 section of the main configuration file. Each backup resides under a
1377 directory corresponding to the backup-set name followed by a
1378 date-encoded directory.
1379
1381 Holland Core Team
1382
1384 2021
13851.2.4 Mar 12, 2021 HOLLAND(1)