1CRIU(8) CRIU Manual CRIU(8)
2
6 criu - checkpoint/restore in userspace
7
9 criu command [option ...]
10
12 criu is a tool for checkpointing and restoring running applications. It
13 does this by saving their state as a collection of files (see the dump
14 command) and creating equivalent processes from those files (see the
15 restore command). The restore operation can be performed at a later
16 time, on a different system, or both.
17
19 Most of the true / false long options (the ones without arguments) can
20 be prefixed with --no- to negate the option (example: --display-stats
21 and --no-display-stats).
22 Common options
24 Common options are applicable to any command.
25 -v[v...], --verbosity
27 Increase verbosity up from the default level. Multiple v can be
28 used, each increasing verbosity by one level. Using long option
29 without argument increases verbosity by one level.
30 -vnum, --verbosity=num
32 Set verbosity level to num. The higher the level, the more output
33 is produced.
34 The following levels are available:
36 · -v0 no output;
38 · -v1 only errors;
40 · -v2 above plus warnings (this is the default level);
42 · -v3 above plus information messages and timestamps;
44 · -v4 above plus lots of debug.
46 --pidfile file
48 Write root task, service or page-server pid into a file.
49 -o, --log-file file
51 Write logging messages to file.
52 --log-pid
54 Write separate logging files per each pid.
55 --display-stats
57 During dump as well as during restore criu collects information
58 like the time required to dump or restore the process or the number
59 of pages dumped or restored. This information is always written to
60 the files stats-dump and stats-restore and can be easily displayed
61 using crit. The option --display-stats additionally prints out this
62 information on the console at the end of a dump or a restore.
63 -D, --images-dir path
65 Use path as a base directory where to look for sets of image files.
66 --prev-images-dir path
68 Use path as a parent directory where to look for sets of image
69 files. This option makes sense in case of incremental dumps.
70 -W, --work-dir dir
72 Use directory dir for putting logs, pidfiles and statistics. If not
73 specified, path from -D option is taken.
74 --close fd
76 Close file descriptor fd before performing any actions.
77 -L, --libdir path
79 Path to plugins directory.
80 --action-script script
82 Add an external action script to be executed at certain stages. The
83 environment variable CRTOOLS_SCRIPT_ACTION is available to the
84 script to find out which action is being executed, and its value
85 can be one of the following:
86 pre-dump
88 run prior to beginning a dump
89 post-dump
91 run upon dump completion
92 pre-restore
94 run prior to beginning a restore
95 pre-resume
97 run when all processes and resources are restored but tasks are
98 stopped waiting for final kick to run. Must not fail.
99 post-restore
101 run upon restore completion
102 network-lock
104 run to lock network in a target network namespace
105 network-unlock
107 run to unlock network in a target network namespace
108 setup-namespaces
110 run once root task just been created with required namespaces.
111 Note it is an early stage of restore, when nothing is restored
112 yet except for namespaces themselves
113 -V, --version
115 Print program version and exit.
116 -h, --help
118 Print some help and exit.
119 pre-dump
121 Performs the pre-dump procedure, during which criu creates a snapshot
122 of memory changes since the previous pre-dump. Note that during this
123 criu also creates the fsnotify cache which speeds up the restore proce‐
124 dure. pre-dump requires at least -t option (see dump below). In addi‐
125 tion, page-server options may be specified.
126 --track-mem
128 Turn on memory changes tracker in the kernel. If the option is not
129 passed the memory tracker get turned on implicitly.
130 dump
132 Performs a checkpoint procedure.
133 -t, --tree pid
135 Checkpoint the whole process tree starting from pid.
136 -R, --leave-running
138 Leave tasks in running state after checkpoint, instead of killing.
139 This option is pretty dangerous and should be used only if you
140 understand what you are doing.
141 Note if task is about to run after been checkpointed, it can modify
143 TCP connections, delete files and do other dangerous actions.
144 Therefore, criu can not guarantee that the next restore action will
145 succeed. Most likely if this option is used, at least the file sys‐
146 tem snapshot must be made with the help of post-dump action script.
147 In other words, do not use it unless really needed.
149 -s, --leave-stopped
151 Leave tasks in stopped state after checkpoint, instead of killing.
152 --external type[id]:value
154 Dump an instance of an external resource. The generic syntax is
155 type of resource, followed by resource id (enclosed in literal
156 square brackets), and optional value (prepended by a literal semi‐
157 colon). The following resource types are currently supported: mnt,
158 dev, file, tty, unix. Syntax depends on type. Note to restore
159 external resources, either --external or --inherit-fd is used,
160 depending on resource type.
161 --external mnt[mountpoint]:name
163 Dump an external bind mount referenced by mountpoint, saving it to
164 image under the identifier name.
165 --external mnt[]:flags
167 Dump all external bind mounts, autodetecting those. Optional flags
168 can contain m to also dump external master mounts, s to also dump
169 external shared mounts (default behavior is to abort dumping if
170 such mounts are found). If flags are not provided, semicolon is
171 optional.
172 --external dev[major/minor]:name
174 Allow to dump a mount namespace having a real block device mounted.
175 A block device is identified by its major and minor numbers, and
176 criu saves its information to image under the identifier name.
177 --external file[mnt_id:inode]
179 Dump an external file, i.e. an opened file that is can not be
180 resolved from the current mount namespace, which can not be dumped
181 without using this option. The file is identified by mnt_id (a
182 field obtained from /proc/pid/fdinfo/N) and inode (as returned by
183 stat(2)).
184 --external tty[rdev:dev]
186 Dump an external TTY, identified by st_rdev and st_dev fields
187 returned by stat(2).
188 --external unix[id]
190 Tell criu that one end of a pair of UNIX sockets (created by sock‐
191 etpair(2)) with id is OK to be disconnected.
192 --freeze-cgroup
194 Use cgroup freezer to collect processes.
195 --manage-cgroups
197 Collect cgroups into the image thus they gonna be restored then.
198 Without this option, criu will not save cgroups configuration asso‐
199 ciated with a task.
200 --cgroup-props spec
202 Specify controllers and their properties to be saved into the image
203 file. criu predefines specifications for common controllers, but
204 since the kernel can add new controllers and modify their proper‐
205 ties, there should be a way to specify ones matched the kernel.
206 spec argument describes the controller and properties specification
208 in a simplified YAML form:
209 "c1":
211 - "strategy": "merge"
212 - "properties": ["a", "b"]
213 "c2":
214 - "strategy": "replace"
215 - "properties": ["c", "d"]
216 where c1 and c2 are controllers names, and a, b, c, d are their
218 properties.
219 Note the format: double quotes, spaces and new lines are required.
221 The strategy specifies what to do if a controller specified already
222 exists as a built-in one: criu can either merge or replace such.
223 For example, the command line for the above example should look
225 like this:
226 --cgroup-props "\"c1\":\n - \"strategy\": \"merge\"\n - \"properties\": [\"a\", \"b\"]\n \"c2\":\n - \"strategy\": \"replace\"\n - \"properties\": [\"c\", \"d\"]"
228 --cgroup-props-file file
230 Same as --cgroup-props, except the specification is read from the
231 file.
232 --cgroup-dump-controller name
234 Dump a controller with name only, skipping anything else that was
235 discovered automatically (usually via /proc). This option is useful
236 when one needs criu to skip some controllers.
237 --cgroup-props-ignore-default
239 When combined with --cgroup-props, makes criu substitute a prede‐
240 fined controller property with the new one shipped. If the option
241 is not used, the predefined properties are merged with the provided
242 ones.
243 --tcp-established
245 Checkpoint established TCP connections.
246 --skip-in-flight
248 This option skips in-flight TCP connections. If any TCP connections
249 that are not yet completely established are found, criu ignores
250 these connections, rather than errors out. The TCP stack on the
251 client side is expected to handle the re-connect gracefully.
252 --tcp-close
254 Restore connected TCP sockets in closed state.
255 --evasive-devices
257 Use any path to a device file if the original one is inaccessible.
258 --page-server
260 Send pages to a page server (see the page-server command).
261 --force-irmap
263 Force resolving names for inotify and fsnotify watches.
264 --auto-dedup
266 Deduplicate "old" data in pages images of previous dump. This
267 option implies incremental dump mode (see the pre-dump command).
268 -l, --file-locks
270 Dump file locks. It is necessary to make sure that all file lock
271 users are taken into dump, so it is only safe to use this for
272 enclosed containers where locks are not held by any processes out‐
273 side of dumped process tree.
274 --link-remap
276 Allows to link unlinked files back, if possible (modifies filesys‐
277 tem during restore).
278 --ghost-limit size
280 Set the maximum size of deleted file to be carried inside image. By
281 default, up to 1M file is allowed. Using this option allows to not
282 put big deleted files inside images. Argument size may be postfixed
283 with a K, M or G, which stands for kilo-, mega, and gigabytes,
284 accordingly.
285 -j, --shell-job
287 Allow one to dump shell jobs. This implies the restored task will
288 inherit session and process group ID from the criu itself. This
289 option also allows to migrate a single external tty connection, to
290 migrate applications like top. If used with dump command, it must
291 be specified with restore as well.
292 --cpu-cap [cap[,cap...]]
294 Specify CPU capabilities to write to an image file. The argument is
295 a comma-separated list of none, fpu, cpu, ins, all. If the argument
296 is omitted or set to none, capabilities will not be written, which
297 is the default behavior.
298 --cgroup-root [controller:]/newroot
300 Change the root for the controller that will be dumped. By default,
301 criu simply dumps everything below where any of the tasks live.
302 However, if a container moves all of its tasks into a cgroup direc‐
303 tory below the container engine’s default directory for tasks, per‐
304 missions will not be preserved on the upper directories with no
305 tasks in them, which may cause problems.
306 --lazy-pages
308 Perform the dump procedure without writing memory pages into the
309 image files and prepare to service page requests over the network.
310 When dump runs in this mode it presumes that lazy-pages daemon will
311 connect to it and fetch memory pages to lazily inject them into the
312 restored process address space. This option is intended for
313 post-copy (lazy) migration and should be used in conjunction with
314 restore with appropriate options.
315 restore
317 Restores previously checkpointed processes.
318 --inherit-fd fd[N]:resource
320 Inherit a file descriptor. This option lets criu use an already
321 opened file descriptor N for restoring a file identified by
322 resource. This option can be used to restore an external resource
323 dumped with the help of --external file, tty, and unix options.
324 The resource argument can be one of the following:
326 · tty[rdev:dev]
328 · pipe[inode]
330 · socket[inode]
332 · file[mnt_id:inode]
334 · path/to/file
336 Note that square brackets used in this option arguments are liter‐
338 als and usually need to be escaped from shell.
339 -d, --restore-detached
341 Detach criu itself once restore is complete.
342 -s, --leave-stopped
344 Leave tasks in stopped state after restore (rather than resuming
345 their execution).
346 -S, --restore-sibling
348 Restore root task as a sibling (makes sense only with
349 --restore-detached).
350 -r, --root path
352 Change the root filesystem to path (when run in a mount namespace).
353 --external type[id]:value
355 Restore an instance of an external resource. The generic syntax is
356 type of resource, followed by resource id (enclosed in literal
357 square brackets), and optional value (prepended by a literal semi‐
358 colon). The following resource types are currently supported: mnt,
359 dev, veth, macvlan. Syntax depends on type. Note to restore exter‐
360 nal resources dealing with opened file descriptors (such as dumped
361 with the help of --external file, tty, and unix options), option
362 --inherit-fd should be used.
363 --external mnt[name]:mountpoint
365 Restore an external bind mount referenced in the image by name,
366 bind-mounting it from the host mountpoint to a proper mount point.
367 --external mnt[]
369 Restore all external bind mounts (dumped with the help of --exter‐
370 nal mnt[] auto-detection).
371 --external dev[name]:/dev/path
373 Restore an external mount device, identified in the image by name,
374 using the existing block device /dev/path.
375 --external veth[inner_dev]:outer_dev@bridge
377 Set the outer VETH device name (corresponding to inner_dev being
378 restored) to outer_dev. If optional @bridge is specified, outer_dev
379 is added to that bridge. If the option is not used, outer_dev will
380 be autogenerated by the kernel.
381 --external macvlan[inner_dev]:outer_dev
383 When restoring an image that have a MacVLAN device in it, this
384 option must be used to specify to which outer_dev (an existing net‐
385 work device in CRIU namespace) the restored inner_dev should be
386 bound to.
387 --manage-cgroups [mode]
389 Restore cgroups configuration associated with a task from the
390 image. Controllers are always restored in an optimistic way — if
391 already present in system, criu reuses it, otherwise it will be
392 created.
393 The mode may be one of the following:
395 none
397 Do not restore cgroup properties but require cgroup to pre-exist at
398 the moment of restore procedure.
399 props
401 Restore cgroup properties and require cgroup to pre-exist.
402 soft
404 Restore cgroup properties if only cgroup has been created by criu,
405 otherwise do not restore properties. This is the default if mode is
406 unspecified.
407 full
409 Always restore all cgroups and their properties.
410 strict
412 Restore all cgroups and their properties from the scratch, requir‐
413 ing them to not present in the system.
414 --cgroup-root [controller:]/newroot
416 Change the root cgroup the controller will be installed into.
417 No controller means that root is the default for all con‐
418 trollers not specified.
419 --tcp-established
421 Restore previously dumped established TCP connections. This
422 implies that the network has been locked between dump and
423 restore phases so other side of a connection simply notice a
424 kind of lag.
425 --veth-pair IN=OUT
427 Correspondence between outside and inside names of veth
428 devices.
429 -l, --file-locks
431 Restore file locks from the image.
432 --auto-dedup
434 As soon as a page is restored it get punched out from image.
435 -j, --shell-job
437 Restore shell jobs, in other words inherit session and process
438 group ID from the criu itself.
439 --cpu-cap [cap[,cap...]]
441 Specify CPU capabilities to be present on the CPU the process
442 is restoring. To inverse a capability, prefix it with ^. This
443 option implies that --cpu-cap has been passed on dump as well,
444 except fpu option case. The cap argument can be the following
445 (or a set of comma-separated values):
446 all
448 Require all capabilities. This is default mode if --cpu-cap is
449 passed without arguments. Most safe mode.
450 cpu
452 Require the CPU to have all capabilities in image to match runtime
453 CPU.
454 fpu
456 Require the CPU to have compatible FPU. For example the process
457 might be dumped with xsave capability but attempted to restore
458 without it present on target CPU. In such case we refuse to pro‐
459 ceed. This is default mode if --cpu-cap is not present in command
460 line. Note this argument might be passed even if on the dump no
461 --cpu-cap have been specified because FPU frames are always encoded
462 into images.
463 ins
465 Require CPU compatibility on instructions level.
466 none
468 Ignore capabilities. Most dangerous mode. The behaviour is imple‐
469 mentation dependent. Try to not use it until really required.
470 For example, this option can be used in case --cpu-cap=cpu was used
472 during dump, and images are migrated to a less capable CPU and are
473 to be restored. By default, criu shows an error that CPU capabili‐
474 ties are not adequate, but this can be suppressed by using
475 --cpu-cap=none.
476 --weak-sysctls
478 Silently skip restoring sysctls that are not available. This
479 allows to restore on an older kernel, or a kernel configured
480 without some options.
481 --lazy-pages
483 Restore the processes without filling out the entire memory
484 contents. When this option is used, restore sets up the infra‐
485 structure required to fill memory pages either on demand when
486 the process accesses them or in the background without stopping
487 the restored process. This option requires running lazy-pages
488 daemon.
489 check
491 Checks whether the kernel supports the features needed by criu to dump
492 and restore a process tree.
493 There are three categories of kernel support, as described below. criu
495 check always checks Category 1 features unless --feature is specified
496 which only checks a specified feature.
497 Category 1
499 Absolutely required. These are features like support for
500 /proc/PID/map_files, NETLINK_SOCK_DIAG socket monitoring,
501 /proc/sys/kernel/ns_last_pid etc.
502 Category 2
504 Required only for specific cases. These are features like AIO
505 remap, /dev/net/tun and others that are only required if a process
506 being dumped or restored is using those.
507 Category 3
509 Experimental. These are features like task-diag that are used for
510 experimental purposes (mostly during development).
511 If there are no errors or warnings, criu prints "Looks good." and its
513 exit code is 0.
514 A missing Category 1 feature causes criu to print "Does not look good."
516 and its exit code is non-zero.
517 Missing Category 2 and 3 features cause criu to print "Looks good but
519 ..." and its exit code is be non-zero.
520 Without any options, criu check checks Category 1 features. This behav‐
522 ior can be changed by using the following options:
523 --extra
525 Check kernel support for Category 2 features.
526 --experimental
528 Check kernel support for Category 3 features.
529 --all
531 Check kernel support for Category 1, 2, and 3 features.
532 --feature name
534 Check a specific feature. If name is list, a list of valid kernel
535 feature names that can be checked will be printed.
536 page-server
538 Launches criu in page server mode.
539 --daemon
541 Runs page server as a daemon (background process).
542 --status_fd
544 Write \\0 to the FD and close it once page-server is ready to han‐
545 dle requests. The status-fd allows to not daemonize a process and
546 get its exit code at the end. It isn’t supposed to use --daemon and
547 --status-fd together.
548 --address address
550 Page server IP address.
551 --port number
553 Page server port number.
554 --ps-socket fd
556 Use provided file descriptor as socket for incoming connection. In
557 this case --address and --port are ignored. Useful for intercepting
558 page-server traffic e.g. to add encryption or authentication.
559 --lazy-pages
561 Serve local memory dump to a remote lazy-pages daemon. In this mode
562 the page-server reads local memory dump and allows the remote
563 lazy-pages deamon to request memory pages in random order.
564 lazy-pages
566 Launches criu in lazy-pages daemon mode.
567 The lazy-pages daemon is responsible for managing user-level demand
569 paging for the restored processes. It gets information required to fill
570 the process memory pages from the restore and from the checkpont direc‐
571 tory. When a restored process access certain memory page for the first
572 time, the lazy-pages daemon injects its contents into the process
573 address space. The memory pages that are not yet requested by the
574 restored processes are injected in the background.
575 exec
577 Executes a system call inside a destination task's context. This func‐
578 tionality is deprecated; please use Compel instead.
579 service
581 Launches criu in RPC daemon mode, where criu is listening for RPC com‐
582 mands over socket to perform. This is convenient for a case where dae‐
583 mon itself is running in a privileged (superuser) mode but clients are
584 not.
585 dedup
587 Starts pagemap data deduplication procedure, where criu scans over all
588 pagemap files and tries to minimize the number of pagemap entries by
589 obtaining the references from a parent pagemap image.
590 cpuinfo dump
592 Fetches current CPU features and write them into an image file.
593 cpuinfo check
595 Fetches current CPU features (i.e. CPU the criu is running on) and test
596 if they are compatible with the ones present in an image file.
597
599 To checkpoint a program with pid of 1234 and write all image files into
600 directory checkpoint:
601 criu dump -D checkpoint -t 1234
603 To restore this program detaching criu itself:
605 criu restore -d -D checkpoint
607
609 The CRIU team.
610
612 Copyright (C) 2011-2016, Parallels Holdings, Inc.
613criu 3.10 05/14/2019 CRIU(8)