1CRIU(8) CRIU Manual CRIU(8)
2
6 criu - checkpoint/restore in userspace
7
9 criu command [option ...]
10
12 criu is a tool for checkpointing and restoring running applications. It
13 does this by saving their state as a collection of files (see the dump
14 command) and creating equivalent processes from those files (see the
15 restore command). The restore operation can be performed at a later
16 time, on a different system, or both.
17
19 Most of the true / false long options (the ones without arguments) can
20 be prefixed with --no- to negate the option (example: --display-stats
21 and --no-display-stats).
22 Common options
24 Common options are applicable to any command.
25 -v[v...], --verbosity
27 Increase verbosity up from the default level. Multiple v can be
28 used, each increasing verbosity by one level. Using long option
29 without argument increases verbosity by one level.
30 -vnum, --verbosity=num
32 Set verbosity level to num. The higher the level, the more output
33 is produced.
34 The following levels are available:
36 · -v0 no output;
38 · -v1 only errors;
40 · -v2 above plus warnings (this is the default level);
42 · -v3 above plus information messages and timestamps;
44 · -v4 above plus lots of debug.
46 --config file
48 Pass a specific configuration file to criu.
49 --no-default-config
51 Forbid parsing of default configuration files.
52 --pidfile file
54 Write root task, service or page-server pid into a file.
55 -o, --log-file file
57 Write logging messages to file.
58 --display-stats
60 During dump as well as during restore criu collects information
61 like the time required to dump or restore the process or the number
62 of pages dumped or restored. This information is always written to
63 the files stats-dump and stats-restore and can be easily displayed
64 using crit. The option --display-stats additionally prints out this
65 information on the console at the end of a dump or a restore.
66 -D, --images-dir path
68 Use path as a base directory where to look for sets of image files.
69 --prev-images-dir path
71 Use path as a parent directory where to look for sets of image
72 files. This option makes sense in case of incremental dumps.
73 -W, --work-dir dir
75 Use directory dir for putting logs, pidfiles and statistics. If not
76 specified, path from -D option is taken.
77 --close fd
79 Close file descriptor fd before performing any actions.
80 -L, --libdir path
82 Path to plugins directory.
83 --action-script script
85 Add an external action script to be executed at certain stages. The
86 environment variable CRTOOLS_SCRIPT_ACTION is available to the
87 script to find out which action is being executed, and its value
88 can be one of the following:
89 pre-dump
91 run prior to beginning a dump
92 post-dump
94 run upon dump completion
95 pre-restore
97 run prior to beginning a restore
98 post-restore
100 run upon restore completion
101 pre-resume
103 run when all processes and resources are restored but tasks are
104 stopped waiting for final kick to run. Must not fail.
105 post-resume
107 called at the very end, when everything is restored and pro‐
108 cesses were resumed
109 network-lock
111 run to lock network in a target network namespace
112 network-unlock
114 run to unlock network in a target network namespace
115 setup-namespaces
117 run once root task has just been created with required names‐
118 paces. Note it is an early stage of restore, when nothing is
119 restored yet, except for namespaces themselves
120 post-setup-namespaces
122 called after the namespaces are configured
123 orphan-pts-master
125 called after master pty is opened and unlocked. This hook can
126 be used only in the RPC mode, and the notification message con‐
127 tains a file descriptor for the master pty
128 -V, --version
130 Print program version and exit.
131 -h, --help
133 Print some help and exit.
134 pre-dump
136 Performs the pre-dump procedure, during which criu creates a snapshot
137 of memory changes since the previous pre-dump. Note that during this
138 criu also creates the fsnotify cache which speeds up the restore proce‐
139 dure. pre-dump requires at least -t option (see dump below). In addi‐
140 tion, page-server options may be specified.
141 --track-mem
143 Turn on memory changes tracker in the kernel. If the option is not
144 passed the memory tracker get turned on implicitly.
145 dump
147 Performs a checkpoint procedure.
148 -t, --tree pid
150 Checkpoint the whole process tree starting from pid.
151 -R, --leave-running
153 Leave tasks in running state after checkpoint, instead of killing.
154 This option is pretty dangerous and should be used only if you
155 understand what you are doing.
156 Note if task is about to run after been checkpointed, it can modify
158 TCP connections, delete files and do other dangerous actions.
159 Therefore, criu can not guarantee that the next restore action will
160 succeed. Most likely if this option is used, at least the file sys‐
161 tem snapshot must be made with the help of post-dump action script.
162 In other words, do not use it unless really needed.
164 -s, --leave-stopped
166 Leave tasks in stopped state after checkpoint, instead of killing.
167 --external type[id]:value
169 Dump an instance of an external resource. The generic syntax is
170 type of resource, followed by resource id (enclosed in literal
171 square brackets), and optional value (prepended by a literal
172 colon). The following resource types are currently supported: mnt,
173 dev, file, tty, unix. Syntax depends on type. Note to restore
174 external resources, either --external or --inherit-fd is used,
175 depending on resource type.
176 --external mnt[mountpoint]:name
178 Dump an external bind mount referenced by mountpoint, saving it to
179 image under the identifier name.
180 --external mnt[]:flags
182 Dump all external bind mounts, autodetecting those. Optional flags
183 can contain m to also dump external master mounts, s to also dump
184 external shared mounts (default behavior is to abort dumping if
185 such mounts are found). If flags are not provided, colon is
186 optional.
187 --external dev[major/minor]:name
189 Allow to dump a mount namespace having a real block device mounted.
190 A block device is identified by its major and minor numbers, and
191 criu saves its information to image under the identifier name.
192 --external file[mnt_id:inode]
194 Dump an external file, i.e. an opened file that is can not be
195 resolved from the current mount namespace, which can not be dumped
196 without using this option. The file is identified by mnt_id (a
197 field obtained from /proc/pid/fdinfo/N) and inode (as returned by
198 stat(2)).
199 --external tty[rdev:dev]
201 Dump an external TTY, identified by st_rdev and st_dev fields
202 returned by stat(2).
203 --external unix[id]
205 Tell criu that one end of a pair of UNIX sockets (created by sock‐
206 etpair(2)) with id is OK to be disconnected.
207 --freeze-cgroup
209 Use cgroup freezer to collect processes.
210 --manage-cgroups
212 Collect cgroups into the image thus they gonna be restored then.
213 Without this option, criu will not save cgroups configuration asso‐
214 ciated with a task.
215 --cgroup-props spec
217 Specify controllers and their properties to be saved into the image
218 file. criu predefines specifications for common controllers, but
219 since the kernel can add new controllers and modify their proper‐
220 ties, there should be a way to specify ones matched the kernel.
221 spec argument describes the controller and properties specification
223 in a simplified YAML form:
224 "c1":
226 - "strategy": "merge"
227 - "properties": ["a", "b"]
228 "c2":
229 - "strategy": "replace"
230 - "properties": ["c", "d"]
231 where c1 and c2 are controllers names, and a, b, c, d are their
233 properties.
234 Note the format: double quotes, spaces and new lines are required.
236 The strategy specifies what to do if a controller specified already
237 exists as a built-in one: criu can either merge or replace such.
238 For example, the command line for the above example should look
240 like this:
241 --cgroup-props "\"c1\":\n - \"strategy\": \"merge\"\n - \"properties\": [\"a\", \"b\"]\n \"c2\":\n - \"strategy\": \"replace\"\n - \"properties\": [\"c\", \"d\"]"
243 --cgroup-props-file file
245 Same as --cgroup-props, except the specification is read from the
246 file.
247 --cgroup-dump-controller name
249 Dump a controller with name only, skipping anything else that was
250 discovered automatically (usually via /proc). This option is useful
251 when one needs criu to skip some controllers.
252 --cgroup-props-ignore-default
254 When combined with --cgroup-props, makes criu substitute a prede‐
255 fined controller property with the new one shipped. If the option
256 is not used, the predefined properties are merged with the provided
257 ones.
258 --tcp-established
260 Checkpoint established TCP connections.
261 --skip-in-flight
263 This option skips in-flight TCP connections. If any TCP connections
264 that are not yet completely established are found, criu ignores
265 these connections, rather than errors out. The TCP stack on the
266 client side is expected to handle the re-connect gracefully.
267 --evasive-devices
269 Use any path to a device file if the original one is inaccessible.
270 --page-server
272 Send pages to a page server (see the page-server command).
273 --force-irmap
275 Force resolving names for inotify and fsnotify watches.
276 --auto-dedup
278 Deduplicate "old" data in pages images of previous dump. This
279 option implies incremental dump mode (see the pre-dump command).
280 -l, --file-locks
282 Dump file locks. It is necessary to make sure that all file lock
283 users are taken into dump, so it is only safe to use this for
284 enclosed containers where locks are not held by any processes out‐
285 side of dumped process tree.
286 --link-remap
288 Allows to link unlinked files back, if possible (modifies filesys‐
289 tem during restore).
290 --ghost-limit size
292 Set the maximum size of deleted file to be carried inside image. By
293 default, up to 1M file is allowed. Using this option allows to not
294 put big deleted files inside images. Argument size may be postfixed
295 with a K, M or G, which stands for kilo-, mega, and gigabytes,
296 accordingly.
297 -j, --shell-job
299 Allow one to dump shell jobs. This implies the restored task will
300 inherit session and process group ID from the criu itself. This
301 option also allows to migrate a single external tty connection, to
302 migrate applications like top. If used with dump command, it must
303 be specified with restore as well.
304 --cpu-cap [cap[,cap...]]
306 Specify CPU capabilities to write to an image file. The argument is
307 a comma-separated list of:
308 · none to ignore capabilities at all; the image will not be pro‐
310 duced on dump, neither any check performed on restore;
311 · fpu to check if FPU module is compatible;
313 · ins to check if CPU supports all instructions required;
315 · cpu to check if CPU capabilities are exactly matching;
317 · all for all above set.
319 By default the option is set to fpu and ins.
321 --cgroup-root [controller:]/newroot
323 Change the root for the controller that will be dumped. By default,
324 criu simply dumps everything below where any of the tasks live.
325 However, if a container moves all of its tasks into a cgroup direc‐
326 tory below the container engine’s default directory for tasks, per‐
327 missions will not be preserved on the upper directories with no
328 tasks in them, which may cause problems.
329 --lazy-pages
331 Perform the dump procedure without writing memory pages into the
332 image files and prepare to service page requests over the network.
333 When dump runs in this mode it presumes that lazy-pages daemon will
334 connect to it and fetch memory pages to lazily inject them into the
335 restored process address space. This option is intended for
336 post-copy (lazy) migration and should be used in conjunction with
337 restore with appropriate options.
338 restore
340 Restores previously checkpointed processes.
341 --inherit-fd fd[N]:resource
343 Inherit a file descriptor. This option lets criu use an already
344 opened file descriptor N for restoring a file identified by
345 resource. This option can be used to restore an external resource
346 dumped with the help of --external file, tty, and unix options.
347 The resource argument can be one of the following:
349 · tty[rdev:dev]
351 · pipe[inode]
353 · socket[inode]
355 · file[mnt_id:inode]
357 · path/to/file
359 Note that square brackets used in this option arguments are liter‐
361 als and usually need to be escaped from shell.
362 -d, --restore-detached
364 Detach criu itself once restore is complete.
365 -s, --leave-stopped
367 Leave tasks in stopped state after restore (rather than resuming
368 their execution).
369 -S, --restore-sibling
371 Restore root task as a sibling (makes sense only with
372 --restore-detached).
373 --log-pid
375 Write separate logging files per each pid.
376 -r, --root path
378 Change the root filesystem to path (when run in a mount namespace).
379 --external type[id]:value
381 Restore an instance of an external resource. The generic syntax is
382 type of resource, followed by resource id (enclosed in literal
383 square brackets), and optional value (prepended by a literal
384 colon). The following resource types are currently supported: mnt,
385 dev, veth, macvlan. Syntax depends on type. Note to restore exter‐
386 nal resources dealing with opened file descriptors (such as dumped
387 with the help of --external file, tty, and unix options), option
388 --inherit-fd should be used.
389 --external mnt[name]:mountpoint
391 Restore an external bind mount referenced in the image by name,
392 bind-mounting it from the host mountpoint to a proper mount point.
393 --external mnt[]
395 Restore all external bind mounts (dumped with the help of --exter‐
396 nal mnt[] auto-detection).
397 --external dev[name]:/dev/path
399 Restore an external mount device, identified in the image by name,
400 using the existing block device /dev/path.
401 --external veth[inner_dev]:outer_dev@bridge
403 Set the outer VETH device name (corresponding to inner_dev being
404 restored) to outer_dev. If optional @bridge is specified, outer_dev
405 is added to that bridge. If the option is not used, outer_dev will
406 be autogenerated by the kernel.
407 --external macvlan[inner_dev]:outer_dev
409 When restoring an image that have a MacVLAN device in it, this
410 option must be used to specify to which outer_dev (an existing net‐
411 work device in CRIU namespace) the restored inner_dev should be
412 bound to.
413 --manage-cgroups [mode]
415 Restore cgroups configuration associated with a task from the
416 image. Controllers are always restored in an optimistic way — if
417 already present in system, criu reuses it, otherwise it will be
418 created.
419 The mode may be one of the following:
421 none
423 Do not restore cgroup properties but require cgroup to pre-exist at
424 the moment of restore procedure.
425 props
427 Restore cgroup properties and require cgroup to pre-exist.
428 soft
430 Restore cgroup properties if only cgroup has been created by criu,
431 otherwise do not restore properties. This is the default if mode is
432 unspecified.
433 full
435 Always restore all cgroups and their properties.
436 strict
438 Restore all cgroups and their properties from the scratch, requir‐
439 ing them to not present in the system.
440 ignore
442 Don’t deal with cgroups and pretend that they don’t exist.
443 --cgroup-root [controller:]/newroot
445 Change the root cgroup the controller will be installed into.
446 No controller means that root is the default for all con‐
447 trollers not specified.
448 --tcp-established
450 Restore previously dumped established TCP connections. This
451 implies that the network has been locked between dump and
452 restore phases so other side of a connection simply notice a
453 kind of lag.
454 --tcp-close
456 Restore connected TCP sockets in closed state.
457 --veth-pair IN=OUT
459 Correspondence between outside and inside names of veth
460 devices.
461 -l, --file-locks
463 Restore file locks from the image.
464 --lsm-profile type:name
466 Specify an LSM profile to be used during restore. The type can
467 be either apparmor or selinux.
468 --auto-dedup
470 As soon as a page is restored it get punched out from image.
471 -j, --shell-job
473 Restore shell jobs, in other words inherit session and process
474 group ID from the criu itself.
475 --cpu-cap [cap[,cap...]]
477 Specify CPU capabilities to be present on the CPU the process
478 is restoring. To inverse a capability, prefix it with ^. This
479 option implies that --cpu-cap has been passed on dump as well,
480 except fpu option case. The cap argument can be the following
481 (or a set of comma-separated values):
482 all
484 Require all capabilities. This is default mode if --cpu-cap is
485 passed without arguments. Most safe mode.
486 cpu
488 Require the CPU to have all capabilities in image to match runtime
489 CPU.
490 fpu
492 Require the CPU to have compatible FPU. For example the process
493 might be dumped with xsave capability but attempted to restore
494 without it present on target CPU. In such case we refuse to pro‐
495 ceed. This is default mode if --cpu-cap is not present in command
496 line. Note this argument might be passed even if on the dump no
497 --cpu-cap have been specified because FPU frames are always encoded
498 into images.
499 ins
501 Require CPU compatibility on instructions level.
502 none
504 Ignore capabilities. Most dangerous mode. The behaviour is imple‐
505 mentation dependent. Try to not use it until really required.
506 For example, this option can be used in case --cpu-cap=cpu was used
508 during dump, and images are migrated to a less capable CPU and are
509 to be restored. By default, criu shows an error that CPU capabili‐
510 ties are not adequate, but this can be suppressed by using
511 --cpu-cap=none.
512 --weak-sysctls
514 Silently skip restoring sysctls that are not available. This
515 allows to restore on an older kernel, or a kernel configured
516 without some options.
517 --lazy-pages
519 Restore the processes without filling out the entire memory
520 contents. When this option is used, restore sets up the infra‐
521 structure required to fill memory pages either on demand when
522 the process accesses them or in the background without stopping
523 the restored process. This option requires running lazy-pages
524 daemon.
525 check
527 Checks whether the kernel supports the features needed by criu to dump
528 and restore a process tree.
529 There are three categories of kernel support, as described below. criu
531 check always checks Category 1 features unless --feature is specified
532 which only checks a specified feature.
533 Category 1
535 Absolutely required. These are features like support for
536 /proc/PID/map_files, NETLINK_SOCK_DIAG socket monitoring,
537 /proc/sys/kernel/ns_last_pid etc.
538 Category 2
540 Required only for specific cases. These are features like AIO
541 remap, /dev/net/tun and others that are only required if a process
542 being dumped or restored is using those.
543 Category 3
545 Experimental. These are features like task-diag that are used for
546 experimental purposes (mostly during development).
547 If there are no errors or warnings, criu prints "Looks good." and its
549 exit code is 0.
550 A missing Category 1 feature causes criu to print "Does not look good."
552 and its exit code is non-zero.
553 Missing Category 2 and 3 features cause criu to print "Looks good but
555 ..." and its exit code is be non-zero.
556 Without any options, criu check checks Category 1 features. This behav‐
558 ior can be changed by using the following options:
559 --extra
561 Check kernel support for Category 2 features.
562 --experimental
564 Check kernel support for Category 3 features.
565 --all
567 Check kernel support for Category 1, 2, and 3 features.
568 --feature name
570 Check a specific feature. If name is list, a list of valid kernel
571 feature names that can be checked will be printed.
572 page-server
574 Launches criu in page server mode.
575 --daemon
577 Runs page server as a daemon (background process).
578 --status-fd
580 Write \0 to the FD and close it once page-server is ready to handle
581 requests. The status-fd allows to not daemonize a process and get
582 its exit code at the end. It isn’t supposed to use --daemon and
583 --status-fd together.
584 --address address
586 Page server IP address or hostname.
587 --port number
589 Page server port number.
590 --ps-socket fd
592 Use provided file descriptor as socket for incoming connection. In
593 this case --address and --port are ignored. Useful for intercepting
594 page-server traffic e.g. to add encryption or authentication.
595 --lazy-pages
597 Serve local memory dump to a remote lazy-pages daemon. In this mode
598 the page-server reads local memory dump and allows the remote
599 lazy-pages daemon to request memory pages in random order.
600 --tls-cacert file
602 Specifies the path to a trusted Certificate Authority (CA) certifi‐
603 cate file to be used for verification of a client or server cer‐
604 tificate. The file must be in PEM format. When this option is used
605 only the specified CA is used for verification. Otherwise, the sys‐
606 tem’s trusted CAs and, if present, /etc/pki/CA/cacert.pem will be
607 used.
608 --tls-cacrl file
610 Specifies a path to a Certificate Revocation List (CRL) file which
611 contains a list of revoked certificates that should no longer be
612 trusted. The file must be in PEM format. When this option is not
613 specified, the file, if present, /etc/pki/CA/cacrl.pem will be
614 used.
615 --tls-cert file
617 Specifies a path to a file that contains a X.509 certificate to
618 present to the remote entity. The file must be in PEM format. When
619 this option is not specified, the default location
620 (/etc/pki/criu/cert.pem) will be used.
621 --tls-key file
623 Specifies a path to a file that contains TLS private key. The file
624 must be in PEM format. When this option is not the default location
625 (/etc/pki/criu/private/key.pem) will be used.
626 --tls
628 Use TLS to secure remote connections.
629 lazy-pages
631 Launches criu in lazy-pages daemon mode.
632 The lazy-pages daemon is responsible for managing user-level demand
634 paging for the restored processes. It gets information required to fill
635 the process memory pages from the restore and from the checkpoint
636 directory. When a restored process access certain memory page for the
637 first time, the lazy-pages daemon injects its contents into the process
638 address space. The memory pages that are not yet requested by the
639 restored processes are injected in the background.
640 exec
642 Executes a system call inside a destination task's context. This func‐
643 tionality is deprecated; please use Compel instead.
644 service
646 Launches criu in RPC daemon mode, where criu is listening for RPC com‐
647 mands over socket to perform. This is convenient for a case where dae‐
648 mon itself is running in a privileged (superuser) mode but clients are
649 not.
650 dedup
652 Starts pagemap data deduplication procedure, where criu scans over all
653 pagemap files and tries to minimize the number of pagemap entries by
654 obtaining the references from a parent pagemap image.
655 cpuinfo dump
657 Fetches current CPU features and write them into an image file.
658 cpuinfo check
660 Fetches current CPU features (i.e. CPU the criu is running on) and test
661 if they are compatible with the ones present in an image file.
662
664 Criu supports usage of configuration files to avoid the need of writing
665 every option on command line, which is useful especially with repeated
666 usage of same options. A specific configuration file can be passed with
667 the "--config file" option. If no file is passed, the default configu‐
668 ration files /etc/criu/default.conf and $HOME/.criu/default.conf are
669 parsed (if present on the system). If the environment variable
670 CRIU_CONFIG_FILE is set, it will also be parsed.
671 The options passed to CRIU via CLI, RPC or configuration file are eval‐
673 uated in the following order:
674 · apply_config(/etc/criu/default.conf)
676 · apply_config($HOME/.criu/default.conf)
678 · apply_config(CRIU_CONFIG_FILE)
680 · apply_config(--config file)
682 · apply_config(CLI) or apply_config(RPC)
684 · apply_config(RPC configuration file) (only for RPC mode)
686 Default configuration file parsing can be deactivated with
688 "--no-default-config" if needed. Parsed configuration files are merged
689 with command line options, which allows overriding boolean options.
690 Configuration file syntax
692 Comments are supported using '#' sign. The rest of the line is ignored.
693 Options are the same as command line options without the '--' prefix,
694 use one option per line (with corresponding argument if applicable,
695 divided by whitespaces). If needed, the argument can be provided in
696 double quotes (this should be needed only if the argument contains
697 whitespaces). In case this type of argument contains a literal double
698 quote as well, it can be escaped using the '\' sign. Usage of commands
699 is disallowed and all other escape sequences are interpreted literally.
700 Example of configuration file to illustrate syntax:
702 $ cat ~/.criu/default.conf
704 tcp-established
705 work-dir "/home/USERNAME/criu/my \"work\" directory"
706 #this is a comment
707 no-restore-sibling # this is another comment
708 Configuration files in RPC mode
710 Not only does criu evaluate configuration files in CLI mode, it also
711 evaluates configuration files in RPC mode. Just as in CLI mode the con‐
712 figuration file values are evaluated first. This means that any option
713 set via RPC will overwrite the configuration file setting. The user can
714 thus change criu's default behavior but it is not possible to change
715 settings which are explicitly set by the RPC client.
716 The RPC client can, however, specify an additional configuration file
718 which will be evaluated after the RPC options (see above for option
719 evaluation order). The RPC client can specify this additional configu‐
720 ration file via "req.opts.config_file = /path/to/file". The values from
721 this configuration file will overwrite all other configuration file
722 settings or RPC options. This can lead to undesired behavior of criu
723 and should only be used carefully.
724
726 To checkpoint a program with pid of 1234 and write all image files into
727 directory checkpoint:
728 criu dump -D checkpoint -t 1234
730 To restore this program detaching criu itself:
732 criu restore -d -D checkpoint
734
736 The CRIU team.
737
739 Copyright (C) 2011-2016, Parallels Holdings, Inc.
740criu 3.13 09/16/2019 CRIU(8)