1CRIU(8) CRIU Manual CRIU(8)
2
6 criu - checkpoint/restore in userspace
7
9 criu command [option ...]
10
12 criu is a tool for checkpointing and restoring running applications. It
13 does this by saving their state as a collection of files (see the dump
14 command) and creating equivalent processes from those files (see the
15 restore command). The restore operation can be performed at a later
16 time, on a different system, or both.
17
19 Most of the true / false long options (the ones without arguments) can
20 be prefixed with --no- to negate the option (example: --display-stats
21 and --no-display-stats).
22 Common options
24 Common options are applicable to any command.
25 -v[v...], --verbosity
27 Increase verbosity up from the default level. Multiple v can be
28 used, each increasing verbosity by one level. Using long option
29 without argument increases verbosity by one level.
30 -vnum, --verbosity=num
32 Set verbosity level to num. The higher the level, the more output
33 is produced.
34 The following levels are available:
36 · -v0 no output;
38 · -v1 only errors;
40 · -v2 above plus warnings (this is the default level);
42 · -v3 above plus information messages and timestamps;
44 · -v4 above plus lots of debug.
46 --config file
48 Pass a specific configuration file to criu.
49 --no-default-config
51 Forbid parsing of default configuration files.
52 --pidfile file
54 Write root task, service or page-server pid into a file.
55 -o, --log-file file
57 Write logging messages to file.
58 --display-stats
60 During dump as well as during restore criu collects information
61 like the time required to dump or restore the process or the number
62 of pages dumped or restored. This information is always written to
63 the files stats-dump and stats-restore and can be easily displayed
64 using crit. The option --display-stats additionally prints out this
65 information on the console at the end of a dump or a restore.
66 -D, --images-dir path
68 Use path as a base directory where to look for sets of image files.
69 --prev-images-dir path
71 Use path as a parent directory where to look for sets of image
72 files. This option makes sense in case of incremental dumps.
73 -W, --work-dir dir
75 Use directory dir for putting logs, pidfiles and statistics. If not
76 specified, path from -D option is taken.
77 --close fd
79 Close file descriptor fd before performing any actions.
80 -L, --libdir path
82 Path to plugins directory.
83 --action-script script
85 Add an external action script to be executed at certain stages. The
86 environment variable CRTOOLS_SCRIPT_ACTION is available to the
87 script to find out which action is being executed, and its value
88 can be one of the following:
89 pre-dump
91 run prior to beginning a dump
92 post-dump
94 run upon dump completion
95 pre-restore
97 run prior to beginning a restore
98 post-restore
100 run upon restore completion
101 pre-resume
103 run when all processes and resources are restored but tasks are
104 stopped waiting for final kick to run. Must not fail.
105 post-resume
107 called at the very end, when everything is restored and pro‐
108 cesses were resumed
109 network-lock
111 run to lock network in a target network namespace
112 network-unlock
114 run to unlock network in a target network namespace
115 setup-namespaces
117 run once root task has just been created with required names‐
118 paces. Note it is an early stage of restore, when nothing is
119 restored yet, except for namespaces themselves
120 post-setup-namespaces
122 called after the namespaces are configured
123 orphan-pts-master
125 called after master pty is opened and unlocked. This hook can
126 be used only in the RPC mode, and the notification message con‐
127 tains a file descriptor for the master pty
128 -V, --version
130 Print program version and exit.
131 -h, --help
133 Print some help and exit.
134 pre-dump
136 Performs the pre-dump procedure, during which criu creates a snapshot
137 of memory changes since the previous pre-dump. Note that during this
138 criu also creates the fsnotify cache which speeds up the restore proce‐
139 dure. pre-dump requires at least -t option (see dump below). In addi‐
140 tion, page-server options may be specified.
141 --track-mem
143 Turn on memory changes tracker in the kernel. If the option is not
144 passed the memory tracker get turned on implicitly.
145 dump
147 Performs a checkpoint procedure.
148 -t, --tree pid
150 Checkpoint the whole process tree starting from pid.
151 -R, --leave-running
153 Leave tasks in running state after checkpoint, instead of killing.
154 This option is pretty dangerous and should be used only if you
155 understand what you are doing.
156 Note if task is about to run after been checkpointed, it can modify
158 TCP connections, delete files and do other dangerous actions.
159 Therefore, criu can not guarantee that the next restore action will
160 succeed. Most likely if this option is used, at least the file sys‐
161 tem snapshot must be made with the help of post-dump action script.
162 In other words, do not use it unless really needed.
164 -s, --leave-stopped
166 Leave tasks in stopped state after checkpoint, instead of killing.
167 --external type[id]:value
169 Dump an instance of an external resource. The generic syntax is
170 type of resource, followed by resource id (enclosed in literal
171 square brackets), and optional value (prepended by a literal
172 colon). The following resource types are currently supported: mnt,
173 dev, file, tty, unix. Syntax depends on type. Note to restore
174 external resources, either --external or --inherit-fd is used,
175 depending on resource type.
176 --external mnt[mountpoint]:name
178 Dump an external bind mount referenced by mountpoint, saving it to
179 image under the identifier name.
180 --external mnt[]:flags
182 Dump all external bind mounts, autodetecting those. Optional flags
183 can contain m to also dump external master mounts, s to also dump
184 external shared mounts (default behavior is to abort dumping if
185 such mounts are found). If flags are not provided, colon is
186 optional.
187 --external dev[major/minor]:name
189 Allow to dump a mount namespace having a real block device mounted.
190 A block device is identified by its major and minor numbers, and
191 criu saves its information to image under the identifier name.
192 --external file[mnt_id:inode]
194 Dump an external file, i.e. an opened file that is can not be
195 resolved from the current mount namespace, which can not be dumped
196 without using this option. The file is identified by mnt_id (a
197 field obtained from /proc/pid/fdinfo/N) and inode (as returned by
198 stat(2)).
199 --external tty[rdev:dev]
201 Dump an external TTY, identified by st_rdev and st_dev fields
202 returned by stat(2).
203 --external unix[id]
205 Tell criu that one end of a pair of UNIX sockets (created by sock‐
206 etpair(2)) with id is OK to be disconnected.
207 --freeze-cgroup
209 Use cgroup freezer to collect processes.
210 --manage-cgroups
212 Collect cgroups into the image thus they gonna be restored then.
213 Without this option, criu will not save cgroups configuration asso‐
214 ciated with a task.
215 --cgroup-props spec
217 Specify controllers and their properties to be saved into the image
218 file. criu predefines specifications for common controllers, but
219 since the kernel can add new controllers and modify their proper‐
220 ties, there should be a way to specify ones matched the kernel.
221 spec argument describes the controller and properties specification
223 in a simplified YAML form:
224 "c1":
226 - "strategy": "merge"
227 - "properties": ["a", "b"]
228 "c2":
229 - "strategy": "replace"
230 - "properties": ["c", "d"]
231 where c1 and c2 are controllers names, and a, b, c, d are their
233 properties.
234 Note the format: double quotes, spaces and new lines are required.
236 The strategy specifies what to do if a controller specified already
237 exists as a built-in one: criu can either merge or replace such.
238 For example, the command line for the above example should look
240 like this:
241 --cgroup-props "\"c1\":\n - \"strategy\": \"merge\"\n - \"properties\": [\"a\", \"b\"]\n \"c2\":\n - \"strategy\": \"replace\"\n - \"properties\": [\"c\", \"d\"]"
243 --cgroup-props-file file
245 Same as --cgroup-props, except the specification is read from the
246 file.
247 --cgroup-dump-controller name
249 Dump a controller with name only, skipping anything else that was
250 discovered automatically (usually via /proc). This option is useful
251 when one needs criu to skip some controllers.
252 --cgroup-props-ignore-default
254 When combined with --cgroup-props, makes criu substitute a prede‐
255 fined controller property with the new one shipped. If the option
256 is not used, the predefined properties are merged with the provided
257 ones.
258 --tcp-established
260 Checkpoint established TCP connections.
261 --skip-in-flight
263 This option skips in-flight TCP connections. If any TCP connections
264 that are not yet completely established are found, criu ignores
265 these connections, rather than errors out. The TCP stack on the
266 client side is expected to handle the re-connect gracefully.
267 --evasive-devices
269 Use any path to a device file if the original one is inaccessible.
270 --page-server
272 Send pages to a page server (see the page-server command).
273 --force-irmap
275 Force resolving names for inotify and fsnotify watches.
276 --auto-dedup
278 Deduplicate "old" data in pages images of previous dump. This
279 option implies incremental dump mode (see the pre-dump command).
280 -l, --file-locks
282 Dump file locks. It is necessary to make sure that all file lock
283 users are taken into dump, so it is only safe to use this for
284 enclosed containers where locks are not held by any processes out‐
285 side of dumped process tree.
286 --link-remap
288 Allows to link unlinked files back, if possible (modifies filesys‐
289 tem during restore).
290 --ghost-limit size
292 Set the maximum size of deleted file to be carried inside image. By
293 default, up to 1M file is allowed. Using this option allows to not
294 put big deleted files inside images. Argument size may be postfixed
295 with a K, M or G, which stands for kilo-, mega, and gigabytes,
296 accordingly.
297 -j, --shell-job
299 Allow one to dump shell jobs. This implies the restored task will
300 inherit session and process group ID from the criu itself. This
301 option also allows to migrate a single external tty connection, to
302 migrate applications like top. If used with dump command, it must
303 be specified with restore as well.
304 --cpu-cap [cap[,cap...]]
306 Specify CPU capabilities to write to an image file. The argument is
307 a comma-separated list of:
308 · none to ignore capabilities at all; the image will not be pro‐
310 duced on dump, neither any check performed on restore;
311 · fpu to check if FPU module is compatible;
313 · ins to check if CPU supports all instructions required;
315 · cpu to check if CPU capabilities are exactly matching;
317 · all for all above set.
319 By default the option is set to fpu and ins.
321 --cgroup-root [controller:]/newroot
323 Change the root for the controller that will be dumped. By default,
324 criu simply dumps everything below where any of the tasks live.
325 However, if a container moves all of its tasks into a cgroup direc‐
326 tory below the container engine’s default directory for tasks, per‐
327 missions will not be preserved on the upper directories with no
328 tasks in them, which may cause problems.
329 --lazy-pages
331 Perform the dump procedure without writing memory pages into the
332 image files and prepare to service page requests over the network.
333 When dump runs in this mode it presumes that lazy-pages daemon will
334 connect to it and fetch memory pages to lazily inject them into the
335 restored process address space. This option is intended for
336 post-copy (lazy) migration and should be used in conjunction with
337 restore with appropriate options.
338 restore
340 Restores previously checkpointed processes.
341 --inherit-fd fd[N]:resource
343 Inherit a file descriptor. This option lets criu use an already
344 opened file descriptor N for restoring a file identified by
345 resource. This option can be used to restore an external resource
346 dumped with the help of --external file, tty, and unix options.
347 The resource argument can be one of the following:
349 · tty[rdev:dev]
351 · pipe[inode]
353 · socket[inode]
355 · file[mnt_id:inode]
357 · path/to/file
359 Note that square brackets used in this option arguments are liter‐
361 als and usually need to be escaped from shell.
362 -d, --restore-detached
364 Detach criu itself once restore is complete.
365 -s, --leave-stopped
367 Leave tasks in stopped state after restore (rather than resuming
368 their execution).
369 -S, --restore-sibling
371 Restore root task as a sibling (makes sense only with
372 --restore-detached).
373 --log-pid
375 Write separate logging files per each pid.
376 -r, --root path
378 Change the root filesystem to path (when run in a mount namespace).
379 --external type[id]:value
381 Restore an instance of an external resource. The generic syntax is
382 type of resource, followed by resource id (enclosed in literal
383 square brackets), and optional value (prepended by a literal
384 colon). The following resource types are currently supported: mnt,
385 dev, veth, macvlan. Syntax depends on type. Note to restore exter‐
386 nal resources dealing with opened file descriptors (such as dumped
387 with the help of --external file, tty, and unix options), option
388 --inherit-fd should be used.
389 --external mnt[name]:mountpoint
391 Restore an external bind mount referenced in the image by name,
392 bind-mounting it from the host mountpoint to a proper mount point.
393 --external mnt[]
395 Restore all external bind mounts (dumped with the help of --exter‐
396 nal mnt[] auto-detection).
397 --external dev[name]:/dev/path
399 Restore an external mount device, identified in the image by name,
400 using the existing block device /dev/path.
401 --external veth[inner_dev]:outer_dev@bridge
403 Set the outer VETH device name (corresponding to inner_dev being
404 restored) to outer_dev. If optional @bridge is specified, outer_dev
405 is added to that bridge. If the option is not used, outer_dev will
406 be autogenerated by the kernel.
407 --external macvlan[inner_dev]:outer_dev
409 When restoring an image that have a MacVLAN device in it, this
410 option must be used to specify to which outer_dev (an existing net‐
411 work device in CRIU namespace) the restored inner_dev should be
412 bound to.
413 --manage-cgroups [mode]
415 Restore cgroups configuration associated with a task from the
416 image. Controllers are always restored in an optimistic way — if
417 already present in system, criu reuses it, otherwise it will be
418 created.
419 The mode may be one of the following:
421 none
423 Do not restore cgroup properties but require cgroup to pre-exist at
424 the moment of restore procedure.
425 props
427 Restore cgroup properties and require cgroup to pre-exist.
428 soft
430 Restore cgroup properties if only cgroup has been created by criu,
431 otherwise do not restore properties. This is the default if mode is
432 unspecified.
433 full
435 Always restore all cgroups and their properties.
436 strict
438 Restore all cgroups and their properties from the scratch, requir‐
439 ing them to not present in the system.
440 --cgroup-root [controller:]/newroot
442 Change the root cgroup the controller will be installed into.
443 No controller means that root is the default for all con‐
444 trollers not specified.
445 --tcp-established
447 Restore previously dumped established TCP connections. This
448 implies that the network has been locked between dump and
449 restore phases so other side of a connection simply notice a
450 kind of lag.
451 --tcp-close
453 Restore connected TCP sockets in closed state.
454 --veth-pair IN=OUT
456 Correspondence between outside and inside names of veth
457 devices.
458 -l, --file-locks
460 Restore file locks from the image.
461 --lsm-profile type:name
463 Specify an LSM profile to be used during restore. The type can
464 be either apparmor or selinux.
465 --auto-dedup
467 As soon as a page is restored it get punched out from image.
468 -j, --shell-job
470 Restore shell jobs, in other words inherit session and process
471 group ID from the criu itself.
472 --cpu-cap [cap[,cap...]]
474 Specify CPU capabilities to be present on the CPU the process
475 is restoring. To inverse a capability, prefix it with ^. This
476 option implies that --cpu-cap has been passed on dump as well,
477 except fpu option case. The cap argument can be the following
478 (or a set of comma-separated values):
479 all
481 Require all capabilities. This is default mode if --cpu-cap is
482 passed without arguments. Most safe mode.
483 cpu
485 Require the CPU to have all capabilities in image to match runtime
486 CPU.
487 fpu
489 Require the CPU to have compatible FPU. For example the process
490 might be dumped with xsave capability but attempted to restore
491 without it present on target CPU. In such case we refuse to pro‐
492 ceed. This is default mode if --cpu-cap is not present in command
493 line. Note this argument might be passed even if on the dump no
494 --cpu-cap have been specified because FPU frames are always encoded
495 into images.
496 ins
498 Require CPU compatibility on instructions level.
499 none
501 Ignore capabilities. Most dangerous mode. The behaviour is imple‐
502 mentation dependent. Try to not use it until really required.
503 For example, this option can be used in case --cpu-cap=cpu was used
505 during dump, and images are migrated to a less capable CPU and are
506 to be restored. By default, criu shows an error that CPU capabili‐
507 ties are not adequate, but this can be suppressed by using
508 --cpu-cap=none.
509 --weak-sysctls
511 Silently skip restoring sysctls that are not available. This
512 allows to restore on an older kernel, or a kernel configured
513 without some options.
514 --lazy-pages
516 Restore the processes without filling out the entire memory
517 contents. When this option is used, restore sets up the infra‐
518 structure required to fill memory pages either on demand when
519 the process accesses them or in the background without stopping
520 the restored process. This option requires running lazy-pages
521 daemon.
522 check
524 Checks whether the kernel supports the features needed by criu to dump
525 and restore a process tree.
526 There are three categories of kernel support, as described below. criu
528 check always checks Category 1 features unless --feature is specified
529 which only checks a specified feature.
530 Category 1
532 Absolutely required. These are features like support for
533 /proc/PID/map_files, NETLINK_SOCK_DIAG socket monitoring,
534 /proc/sys/kernel/ns_last_pid etc.
535 Category 2
537 Required only for specific cases. These are features like AIO
538 remap, /dev/net/tun and others that are only required if a process
539 being dumped or restored is using those.
540 Category 3
542 Experimental. These are features like task-diag that are used for
543 experimental purposes (mostly during development).
544 If there are no errors or warnings, criu prints "Looks good." and its
546 exit code is 0.
547 A missing Category 1 feature causes criu to print "Does not look good."
549 and its exit code is non-zero.
550 Missing Category 2 and 3 features cause criu to print "Looks good but
552 ..." and its exit code is be non-zero.
553 Without any options, criu check checks Category 1 features. This behav‐
555 ior can be changed by using the following options:
556 --extra
558 Check kernel support for Category 2 features.
559 --experimental
561 Check kernel support for Category 3 features.
562 --all
564 Check kernel support for Category 1, 2, and 3 features.
565 --feature name
567 Check a specific feature. If name is list, a list of valid kernel
568 feature names that can be checked will be printed.
569 page-server
571 Launches criu in page server mode.
572 --daemon
574 Runs page server as a daemon (background process).
575 --status-fd
577 Write \0 to the FD and close it once page-server is ready to handle
578 requests. The status-fd allows to not daemonize a process and get
579 its exit code at the end. It isn’t supposed to use --daemon and
580 --status-fd together.
581 --address address
583 Page server IP address or hostname.
584 --port number
586 Page server port number.
587 --ps-socket fd
589 Use provided file descriptor as socket for incoming connection. In
590 this case --address and --port are ignored. Useful for intercepting
591 page-server traffic e.g. to add encryption or authentication.
592 --lazy-pages
594 Serve local memory dump to a remote lazy-pages daemon. In this mode
595 the page-server reads local memory dump and allows the remote
596 lazy-pages daemon to request memory pages in random order.
597 lazy-pages
599 Launches criu in lazy-pages daemon mode.
600 The lazy-pages daemon is responsible for managing user-level demand
602 paging for the restored processes. It gets information required to fill
603 the process memory pages from the restore and from the checkpoint
604 directory. When a restored process access certain memory page for the
605 first time, the lazy-pages daemon injects its contents into the process
606 address space. The memory pages that are not yet requested by the
607 restored processes are injected in the background.
608 exec
610 Executes a system call inside a destination task's context. This func‐
611 tionality is deprecated; please use Compel instead.
612 service
614 Launches criu in RPC daemon mode, where criu is listening for RPC com‐
615 mands over socket to perform. This is convenient for a case where dae‐
616 mon itself is running in a privileged (superuser) mode but clients are
617 not.
618 dedup
620 Starts pagemap data deduplication procedure, where criu scans over all
621 pagemap files and tries to minimize the number of pagemap entries by
622 obtaining the references from a parent pagemap image.
623 cpuinfo dump
625 Fetches current CPU features and write them into an image file.
626 cpuinfo check
628 Fetches current CPU features (i.e. CPU the criu is running on) and test
629 if they are compatible with the ones present in an image file.
630
632 Criu supports usage of configuration files to avoid the need of writing
633 every option on command line, which is useful especially with repeated
634 usage of same options. A specific configuration file can be passed with
635 the "--config file" option. If no file is passed, the default configu‐
636 ration files /etc/criu/default.conf and $HOME/.criu/default.conf are
637 parsed (if present on the system). If the environment variable
638 CRIU_CONFIG_FILE is set, it will also be parsed.
639 The options passed to CRIU via CLI, RPC or configuration file are eval‐
641 uated in the following order:
642 · apply_config(/etc/criu/default.conf)
644 · apply_config($HOME/.criu/default.conf)
646 · apply_config(CRIU_CONFIG_FILE)
648 · apply_config(--config file)
650 · apply_config(CLI) or apply_config(RPC)
652 · apply_config(RPC configuration file) (only for RPC mode)
654 Default configuration file parsing can be deactivated with
656 "--no-default-config" if needed. Parsed configuration files are merged
657 with command line options, which allows overriding boolean options.
658 Configuration file syntax
660 Comments are supported using '#' sign. The rest of the line is ignored.
661 Options are the same as command line options without the '--' prefix,
662 use one option per line (with corresponding argument if applicable,
663 divided by whitespaces). If needed, the argument can be provided in
664 double quotes (this should be needed only if the argument contains
665 whitespaces). In case this type of argument contains a literal double
666 quote as well, it can be escaped using the '\' sign. Usage of commands
667 is disallowed and all other escape sequences are interpreted literally.
668 Example of configuration file to illustrate syntax:
670 $ cat ~/.criu/default.conf
672 tcp-established
673 work-dir "/home/USERNAME/criu/my \"work\" directory"
674 #this is a comment
675 no-restore-sibling # this is another comment
676 Configuration files in RPC mode
678 Not only does criu evaluate configuration files in CLI mode, it also
679 evaluates configuration files in RPC mode. Just as in CLI mode the con‐
680 figuration file values are evaluated first. This means that any option
681 set via RPC will overwrite the configuration file setting. The user can
682 thus change criu's default behavior but it is not possible to change
683 settings which are explicitly set by the RPC client.
684 The RPC client can, however, specify an additional configuration file
686 which will be evaluated after the RPC options (see above for option
687 evaluation order). The RPC client can specify this additional configu‐
688 ration file via "req.opts.config_file = /path/to/file". The values from
689 this configuration file will overwrite all other configuration file
690 settings or RPC options. This can lead to undesired behavior of criu
691 and should only be used carefully.
692
694 To checkpoint a program with pid of 1234 and write all image files into
695 directory checkpoint:
696 criu dump -D checkpoint -t 1234
698 To restore this program detaching criu itself:
700 criu restore -d -D checkpoint
702
704 The CRIU team.
705
707 Copyright (C) 2011-2016, Parallels Holdings, Inc.
708criu 3.12 05/14/2019 CRIU(8)