1CRIU(8)                           CRIU Manual                          CRIU(8)
2
3
4

NAME

6       criu - checkpoint/restore in userspace
7

SYNOPSIS

9       criu command [option ...]
10

DESCRIPTION

12       criu is a tool for checkpointing and restoring running applications. It
13       does this by saving their state as a collection of files (see the  dump
14       command)  and  creating  equivalent processes from those files (see the
15       restore command). The restore operation can be  performed  at  a  later
16       time, on a different system, or both.
17

OPTIONS

19       Most  of the true / false long options (the ones without arguments) can
20       be prefixed with --no- to negate the option  (example:  --display-stats
21       and --no-display-stats).
22
23   Common options
24       Common options are applicable to any command.
25
26       -v[v...], --verbosity
27           Increase  verbosity  up  from  the default level. Multiple v can be
28           used, each increasing verbosity by one  level.  Using  long  option
29           without argument increases verbosity by one level.
30
31       -vnum, --verbosity=num
32           Set  verbosity  level to num. The higher the level, the more output
33           is produced.
34
35           The following levels are available:
36
37           ·   -v0 no output;
38
39           ·   -v1 only errors;
40
41           ·   -v2 above plus warnings (this is the default level);
42
43           ·   -v3 above plus information messages and timestamps;
44
45           ·   -v4 above plus lots of debug.
46
47       --config file
48           Pass a specific configuration file to criu.
49
50       --no-default-config
51           Forbid parsing of default configuration files.
52
53       --pidfile file
54           Write root task, service or page-server pid into a file.
55
56       -o, --log-file file
57           Write logging messages to file.
58
59       --display-stats
60           During dump as well as during  restore  criu  collects  information
61           like the time required to dump or restore the process or the number
62           of pages dumped or restored. This information is always written  to
63           the  files stats-dump and stats-restore and can be easily displayed
64           using crit. The option --display-stats additionally prints out this
65           information on the console at the end of a dump or a restore.
66
67       -D, --images-dir path
68           Use path as a base directory where to look for sets of image files.
69
70       --prev-images-dir path
71           Use  path  as  a  parent  directory where to look for sets of image
72           files. This option makes sense in case of incremental dumps.
73
74       -W, --work-dir dir
75           Use directory dir for putting logs, pidfiles and statistics. If not
76           specified, path from -D option is taken.
77
78       --close fd
79           Close file descriptor fd before performing any actions.
80
81       -L, --libdir path
82           Path to plugins directory.
83
84       --action-script script
85           Add an external action script to be executed at certain stages. The
86           environment variable  CRTOOLS_SCRIPT_ACTION  is  available  to  the
87           script  to  find  out which action is being executed, and its value
88           can be one of the following:
89
90           pre-dump
91               run prior to beginning a dump
92
93           post-dump
94               run upon dump completion
95
96           pre-restore
97               run prior to beginning a restore
98
99           post-restore
100               run upon restore completion
101
102           pre-resume
103               run when all processes and resources are restored but tasks are
104               stopped waiting for final kick to run. Must not fail.
105
106           post-resume
107               called  at  the  very end, when everything is restored and pro‐
108               cesses were resumed
109
110           network-lock
111               run to lock network in a target network namespace
112
113           network-unlock
114               run to unlock network in a target network namespace
115
116           setup-namespaces
117               run once root task has just been created with  required  names‐
118               paces.  Note  it  is an early stage of restore, when nothing is
119               restored yet, except for namespaces themselves
120
121           post-setup-namespaces
122               called after the namespaces are configured
123
124           orphan-pts-master
125               called after master pty is opened and unlocked. This  hook  can
126               be used only in the RPC mode, and the notification message con‐
127               tains a file descriptor for the master pty
128
129       -V, --version
130           Print program version and exit.
131
132       -h, --help
133           Print some help and exit.
134
135   pre-dump
136       Performs the pre-dump procedure, during which criu creates  a  snapshot
137       of  memory  changes  since the previous pre-dump. Note that during this
138       criu also creates the fsnotify cache which speeds up the restore proce‐
139       dure.  pre-dump  requires at least -t option (see dump below). In addi‐
140       tion, page-server options may be specified.
141
142       --track-mem
143           Turn on memory changes tracker in the kernel. If the option is  not
144           passed the memory tracker get turned on implicitly.
145
146   dump
147       Performs a checkpoint procedure.
148
149       -t, --tree pid
150           Checkpoint the whole process tree starting from pid.
151
152       -R, --leave-running
153           Leave  tasks in running state after checkpoint, instead of killing.
154           This option is pretty dangerous and should  be  used  only  if  you
155           understand what you are doing.
156
157           Note if task is about to run after been checkpointed, it can modify
158           TCP connections, delete  files  and  do  other  dangerous  actions.
159           Therefore, criu can not guarantee that the next restore action will
160           succeed. Most likely if this option is used, at least the file sys‐
161           tem snapshot must be made with the help of post-dump action script.
162
163           In other words, do not use it unless really needed.
164
165       -s, --leave-stopped
166           Leave tasks in stopped state after checkpoint, instead of killing.
167
168       --external type[id]:value
169           Dump  an  instance  of  an external resource. The generic syntax is
170           type of resource, followed by  resource  id  (enclosed  in  literal
171           square  brackets),  and  optional  value  (prepended  by  a literal
172           colon). The following resource types are currently supported:  mnt,
173           dev,  file,  tty,  unix.  Syntax  depends  on type. Note to restore
174           external resources, either  --external  or  --inherit-fd  is  used,
175           depending on resource type.
176
177       --external mnt[mountpoint]:name
178           Dump  an external bind mount referenced by mountpoint, saving it to
179           image under the identifier name.
180
181       --external mnt[]:flags
182           Dump all external bind mounts, autodetecting those. Optional  flags
183           can  contain  m to also dump external master mounts, s to also dump
184           external shared mounts (default behavior is  to  abort  dumping  if
185           such  mounts  are  found).  If  flags  are  not  provided, colon is
186           optional.
187
188       --external dev[major/minor]:name
189           Allow to dump a mount namespace having a real block device mounted.
190           A  block  device  is identified by its major and minor numbers, and
191           criu saves its information to image under the identifier name.
192
193       --external file[mnt_id:inode]
194           Dump an external file, i.e. an opened  file  that  is  can  not  be
195           resolved  from the current mount namespace, which can not be dumped
196           without using this option. The file  is  identified  by  mnt_id  (a
197           field  obtained  from /proc/pid/fdinfo/N) and inode (as returned by
198           stat(2)).
199
200       --external tty[rdev:dev]
201           Dump an external TTY,  identified  by  st_rdev  and  st_dev  fields
202           returned by stat(2).
203
204       --external unix[id]
205           Tell  criu that one end of a pair of UNIX sockets (created by sock‐
206           etpair(2)) with id is OK to be disconnected.
207
208       --freeze-cgroup
209           Use cgroup freezer to collect processes.
210
211       --manage-cgroups
212           Collect cgroups into the image thus they gonna  be  restored  then.
213           Without this option, criu will not save cgroups configuration asso‐
214           ciated with a task.
215
216       --cgroup-props spec
217           Specify controllers and their properties to be saved into the image
218           file.   criu  predefines specifications for common controllers, but
219           since the kernel can add new controllers and modify  their  proper‐
220           ties, there should be a way to specify ones matched the kernel.
221
222           spec argument describes the controller and properties specification
223           in a simplified YAML form:
224
225               "c1":
226                - "strategy": "merge"
227                - "properties": ["a", "b"]
228               "c2":
229                - "strategy": "replace"
230                - "properties": ["c", "d"]
231
232           where c1 and c2 are controllers names, and a, b,  c,  d  are  their
233           properties.
234
235           Note  the format: double quotes, spaces and new lines are required.
236           The strategy specifies what to do if a controller specified already
237           exists as a built-in one: criu can either merge or replace such.
238
239           For  example,  the  command  line for the above example should look
240           like this:
241
242               --cgroup-props "\"c1\":\n - \"strategy\": \"merge\"\n - \"properties\": [\"a\", \"b\"]\n \"c2\":\n - \"strategy\": \"replace\"\n - \"properties\": [\"c\", \"d\"]"
243
244       --cgroup-props-file file
245           Same as --cgroup-props, except the specification is read  from  the
246           file.
247
248       --cgroup-dump-controller name
249           Dump  a  controller with name only, skipping anything else that was
250           discovered automatically (usually via /proc). This option is useful
251           when one needs criu to skip some controllers.
252
253       --cgroup-props-ignore-default
254           When  combined  with --cgroup-props, makes criu substitute a prede‐
255           fined controller property with the new one shipped. If  the  option
256           is not used, the predefined properties are merged with the provided
257           ones.
258
259       --tcp-established
260           Checkpoint established TCP connections.
261
262       --skip-in-flight
263           This option skips in-flight TCP connections. If any TCP connections
264           that  are  not  yet  completely established are found, criu ignores
265           these connections, rather than errors out. The  TCP  stack  on  the
266           client side is expected to handle the re-connect gracefully.
267
268       --evasive-devices
269           Use any path to a device file if the original one is inaccessible.
270
271       --page-server
272           Send pages to a page server (see the page-server command).
273
274       --force-irmap
275           Force resolving names for inotify and fsnotify watches.
276
277       --auto-dedup
278           Deduplicate  "old"  data  in  pages  images  of previous dump. This
279           option implies incremental dump mode (see the pre-dump command).
280
281       -l, --file-locks
282           Dump file locks. It is necessary to make sure that  all  file  lock
283           users  are  taken  into  dump,  so  it is only safe to use this for
284           enclosed containers where locks are not held by any processes  out‐
285           side of dumped process tree.
286
287       --link-remap
288           Allows  to link unlinked files back, if possible (modifies filesys‐
289           tem during restore).
290
291       --ghost-limit size
292           Set the maximum size of deleted file to be carried inside image. By
293           default,  up to 1M file is allowed. Using this option allows to not
294           put big deleted files inside images. Argument size may be postfixed
295           with  a  K,  M  or  G, which stands for kilo-, mega, and gigabytes,
296           accordingly.
297
298       -j, --shell-job
299           Allow one to dump shell jobs. This implies the restored  task  will
300           inherit  session  and  process  group ID from the criu itself. This
301           option also allows to migrate a single external tty connection,  to
302           migrate  applications  like top. If used with dump command, it must
303           be specified with restore as well.
304
305       --cpu-cap [cap[,cap...]]
306           Specify CPU capabilities to write to an image file. The argument is
307           a comma-separated list of:
308
309           ·   none  to ignore capabilities at all; the image will not be pro‐
310               duced on dump, neither any check performed on restore;
311
312           ·   fpu to check if FPU module is compatible;
313
314           ·   ins to check if CPU supports all instructions required;
315
316           ·   cpu to check if CPU capabilities are exactly matching;
317
318           ·   all for all above set.
319
320           By default the option is set to fpu and ins.
321
322       --cgroup-root [controller:]/newroot
323           Change the root for the controller that will be dumped. By default,
324           criu  simply  dumps  everything  below where any of the tasks live.
325           However, if a container moves all of its tasks into a cgroup direc‐
326           tory below the container engine’s default directory for tasks, per‐
327           missions will not be preserved on the  upper  directories  with  no
328           tasks in them, which may cause problems.
329
330       --lazy-pages
331           Perform  the  dump  procedure without writing memory pages into the
332           image files and prepare to service page requests over the  network.
333           When dump runs in this mode it presumes that lazy-pages daemon will
334           connect to it and fetch memory pages to lazily inject them into the
335           restored  process  address  space.  This  option  is  intended  for
336           post-copy (lazy) migration and should be used in  conjunction  with
337           restore with appropriate options.
338
339   restore
340       Restores previously checkpointed processes.
341
342       --inherit-fd fd[N]:resource
343           Inherit  a  file  descriptor.  This option lets criu use an already
344           opened file  descriptor  N  for  restoring  a  file  identified  by
345           resource.  This  option can be used to restore an external resource
346           dumped with the help of --external file, tty, and unix options.
347
348           The resource argument can be one of the following:
349
350           ·   tty[rdev:dev]
351
352           ·   pipe[inode]
353
354           ·   socket[inode]
355
356           ·   file[mnt_id:inode]
357
358           ·   path/to/file
359
360           Note that square brackets used in this option arguments are  liter‐
361           als and usually need to be escaped from shell.
362
363       -d, --restore-detached
364           Detach criu itself once restore is complete.
365
366       -s, --leave-stopped
367           Leave  tasks  in  stopped state after restore (rather than resuming
368           their execution).
369
370       -S, --restore-sibling
371           Restore  root  task  as  a   sibling   (makes   sense   only   with
372           --restore-detached).
373
374       --log-pid
375           Write separate logging files per each pid.
376
377       -r, --root path
378           Change the root filesystem to path (when run in a mount namespace).
379
380       --external type[id]:value
381           Restore  an instance of an external resource. The generic syntax is
382           type of resource, followed by  resource  id  (enclosed  in  literal
383           square  brackets),  and  optional  value  (prepended  by  a literal
384           colon). The following resource types are currently supported:  mnt,
385           dev,  veth, macvlan. Syntax depends on type. Note to restore exter‐
386           nal resources dealing with opened file descriptors (such as  dumped
387           with  the  help  of --external file, tty, and unix options), option
388           --inherit-fd should be used.
389
390       --external mnt[name]:mountpoint
391           Restore an external bind mount referenced in  the  image  by  name,
392           bind-mounting it from the host mountpoint to a proper mount point.
393
394       --external mnt[]
395           Restore  all external bind mounts (dumped with the help of --exter‐
396           nal mnt[] auto-detection).
397
398       --external dev[name]:/dev/path
399           Restore an external mount device, identified in the image by  name,
400           using the existing block device /dev/path.
401
402       --external veth[inner_dev]:outer_dev@bridge
403           Set  the  outer  VETH device name (corresponding to inner_dev being
404           restored) to outer_dev. If optional @bridge is specified, outer_dev
405           is  added to that bridge. If the option is not used, outer_dev will
406           be autogenerated by the kernel.
407
408       --external macvlan[inner_dev]:outer_dev
409           When restoring an image that have a  MacVLAN  device  in  it,  this
410           option must be used to specify to which outer_dev (an existing net‐
411           work device in CRIU namespace) the  restored  inner_dev  should  be
412           bound to.
413
414       --manage-cgroups [mode]
415           Restore  cgroups  configuration  associated  with  a  task from the
416           image. Controllers are always restored in an optimistic  way  —  if
417           already  present  in  system,  criu reuses it, otherwise it will be
418           created.
419
420       The mode may be one of the following:
421
422       none
423           Do not restore cgroup properties but require cgroup to pre-exist at
424           the moment of restore procedure.
425
426       props
427           Restore cgroup properties and require cgroup to pre-exist.
428
429       soft
430           Restore  cgroup properties if only cgroup has been created by criu,
431           otherwise do not restore properties. This is the default if mode is
432           unspecified.
433
434       full
435           Always restore all cgroups and their properties.
436
437       strict
438           Restore  all cgroups and their properties from the scratch, requir‐
439           ing them to not present in the system.
440
441           --cgroup-root [controller:]/newroot
442               Change the root cgroup the controller will be  installed  into.
443               No  controller  means  that  root  is  the default for all con‐
444               trollers not specified.
445
446           --tcp-established
447               Restore previously dumped  established  TCP  connections.  This
448               implies  that  the  network  has  been  locked between dump and
449               restore phases so other side of a connection  simply  notice  a
450               kind of lag.
451
452           --tcp-close
453               Restore connected TCP sockets in closed state.
454
455           --veth-pair IN=OUT
456               Correspondence   between  outside  and  inside  names  of  veth
457               devices.
458
459           -l, --file-locks
460               Restore file locks from the image.
461
462           --lsm-profile type:name
463               Specify an LSM profile to be used during restore. The type  can
464               be either apparmor or selinux.
465
466           --auto-dedup
467               As soon as a page is restored it get punched out from image.
468
469           -j, --shell-job
470               Restore  shell jobs, in other words inherit session and process
471               group ID from the criu itself.
472
473           --cpu-cap [cap[,cap...]]
474               Specify CPU capabilities to be present on the CPU  the  process
475               is  restoring.  To inverse a capability, prefix it with ^. This
476               option implies that --cpu-cap has been passed on dump as  well,
477               except  fpu  option case. The cap argument can be the following
478               (or a set of comma-separated values):
479
480       all
481           Require all capabilities. This is  default  mode  if  --cpu-cap  is
482           passed without arguments. Most safe mode.
483
484       cpu
485           Require  the CPU to have all capabilities in image to match runtime
486           CPU.
487
488       fpu
489           Require the CPU to have compatible FPU.  For  example  the  process
490           might  be  dumped  with  xsave  capability but attempted to restore
491           without it present on target CPU. In such case we  refuse  to  pro‐
492           ceed.  This  is default mode if --cpu-cap is not present in command
493           line. Note this argument might be passed even if  on  the  dump  no
494           --cpu-cap have been specified because FPU frames are always encoded
495           into images.
496
497       ins
498           Require CPU compatibility on instructions level.
499
500       none
501           Ignore capabilities. Most dangerous mode. The behaviour  is  imple‐
502           mentation dependent. Try to not use it until really required.
503
504           For example, this option can be used in case --cpu-cap=cpu was used
505           during dump, and images are migrated to a less capable CPU and  are
506           to  be restored. By default, criu shows an error that CPU capabili‐
507           ties are  not  adequate,  but  this  can  be  suppressed  by  using
508           --cpu-cap=none.
509
510           --weak-sysctls
511               Silently  skip  restoring  sysctls that are not available. This
512               allows to restore on an older kernel, or  a  kernel  configured
513               without some options.
514
515           --lazy-pages
516               Restore  the  processes  without  filling out the entire memory
517               contents. When this option is used, restore sets up the  infra‐
518               structure  required  to fill memory pages either on demand when
519               the process accesses them or in the background without stopping
520               the  restored  process. This option requires running lazy-pages
521               daemon.
522
523   check
524       Checks whether the kernel supports the features needed by criu to  dump
525       and restore a process tree.
526
527       There  are three categories of kernel support, as described below. criu
528       check always checks Category 1 features unless --feature  is  specified
529       which only checks a specified feature.
530
531       Category 1
532           Absolutely   required.   These   are   features  like  support  for
533           /proc/PID/map_files,    NETLINK_SOCK_DIAG    socket     monitoring,
534           /proc/sys/kernel/ns_last_pid etc.
535
536       Category 2
537           Required  only  for  specific  cases.  These  are features like AIO
538           remap, /dev/net/tun and others that are only required if a  process
539           being dumped or restored is using those.
540
541       Category 3
542           Experimental.  These  are features like task-diag that are used for
543           experimental purposes (mostly during development).
544
545       If there are no errors or warnings, criu prints "Looks good."  and  its
546       exit code is 0.
547
548       A missing Category 1 feature causes criu to print "Does not look good."
549       and its exit code is non-zero.
550
551       Missing Category 2 and 3 features cause criu to print "Looks  good  but
552       ..." and its exit code is be non-zero.
553
554       Without any options, criu check checks Category 1 features. This behav‐
555       ior can be changed by using the following options:
556
557       --extra
558           Check kernel support for Category 2 features.
559
560       --experimental
561           Check kernel support for Category 3 features.
562
563       --all
564           Check kernel support for Category 1, 2, and 3 features.
565
566       --feature name
567           Check a specific feature. If name is list, a list of  valid  kernel
568           feature names that can be checked will be printed.
569
570   page-server
571       Launches criu in page server mode.
572
573       --daemon
574           Runs page server as a daemon (background process).
575
576       --status-fd
577           Write \0 to the FD and close it once page-server is ready to handle
578           requests. The status-fd allows to not daemonize a process  and  get
579           its  exit  code  at  the end. It isn’t supposed to use --daemon and
580           --status-fd together.
581
582       --address address
583           Page server IP address or hostname.
584
585       --port number
586           Page server port number.
587
588       --ps-socket fd
589           Use provided file descriptor as socket for incoming connection.  In
590           this case --address and --port are ignored. Useful for intercepting
591           page-server traffic e.g. to add encryption or authentication.
592
593       --lazy-pages
594           Serve local memory dump to a remote lazy-pages daemon. In this mode
595           the  page-server  reads  local  memory  dump  and allows the remote
596           lazy-pages daemon to request memory pages in random order.
597
598   lazy-pages
599       Launches criu in lazy-pages daemon mode.
600
601       The lazy-pages daemon is responsible  for  managing  user-level  demand
602       paging for the restored processes. It gets information required to fill
603       the process memory pages from  the  restore  and  from  the  checkpoint
604       directory.  When  a restored process access certain memory page for the
605       first time, the lazy-pages daemon injects its contents into the process
606       address  space.  The  memory  pages  that  are not yet requested by the
607       restored processes are injected in the background.
608
609   exec
610       Executes a system call inside a destination task's context. This  func‐
611       tionality is deprecated; please use Compel instead.
612
613   service
614       Launches  criu in RPC daemon mode, where criu is listening for RPC com‐
615       mands over socket to perform. This is convenient for a case where  dae‐
616       mon  itself is running in a privileged (superuser) mode but clients are
617       not.
618
619   dedup
620       Starts pagemap data deduplication procedure, where criu scans over  all
621       pagemap  files  and  tries to minimize the number of pagemap entries by
622       obtaining the references from a parent pagemap image.
623
624   cpuinfo dump
625       Fetches current CPU features and write them into an image file.
626
627   cpuinfo check
628       Fetches current CPU features (i.e. CPU the criu is running on) and test
629       if they are compatible with the ones present in an image file.
630

CONFIGURATION FILES

632       Criu supports usage of configuration files to avoid the need of writing
633       every option on command line, which is useful especially with  repeated
634       usage of same options. A specific configuration file can be passed with
635       the "--config file" option. If no file is passed, the default  configu‐
636       ration  files  /etc/criu/default.conf  and $HOME/.criu/default.conf are
637       parsed  (if  present  on  the  system).  If  the  environment  variable
638       CRIU_CONFIG_FILE is set, it will also be parsed.
639
640       The options passed to CRIU via CLI, RPC or configuration file are eval‐
641       uated in the following order:
642
643       ·   apply_config(/etc/criu/default.conf)
644
645       ·   apply_config($HOME/.criu/default.conf)
646
647       ·   apply_config(CRIU_CONFIG_FILE)
648
649       ·   apply_config(--config file)
650
651       ·   apply_config(CLI) or apply_config(RPC)
652
653       ·   apply_config(RPC configuration file) (only for RPC mode)
654
655       Default  configuration   file   parsing   can   be   deactivated   with
656       "--no-default-config"  if needed. Parsed configuration files are merged
657       with command line options, which allows overriding boolean options.
658
659   Configuration file syntax
660       Comments are supported using '#' sign. The rest of the line is ignored.
661       Options  are  the same as command line options without the '--' prefix,
662       use one option per line (with  corresponding  argument  if  applicable,
663       divided  by  whitespaces).  If  needed, the argument can be provided in
664       double quotes (this should be needed  only  if  the  argument  contains
665       whitespaces).  In  case this type of argument contains a literal double
666       quote as well, it can be escaped using the '\' sign. Usage of  commands
667       is disallowed and all other escape sequences are interpreted literally.
668
669       Example of configuration file to illustrate syntax:
670
671           $ cat ~/.criu/default.conf
672           tcp-established
673           work-dir "/home/USERNAME/criu/my \"work\" directory"
674           #this is a comment
675           no-restore-sibling   # this is another comment
676
677   Configuration files in RPC mode
678       Not  only  does  criu evaluate configuration files in CLI mode, it also
679       evaluates configuration files in RPC mode. Just as in CLI mode the con‐
680       figuration  file values are evaluated first. This means that any option
681       set via RPC will overwrite the configuration file setting. The user can
682       thus  change  criu's  default behavior but it is not possible to change
683       settings which are explicitly set by the RPC client.
684
685       The RPC client can, however, specify an additional  configuration  file
686       which  will  be  evaluated  after the RPC options (see above for option
687       evaluation order). The RPC client can specify this additional  configu‐
688       ration file via "req.opts.config_file = /path/to/file". The values from
689       this configuration file will overwrite  all  other  configuration  file
690       settings  or  RPC  options. This can lead to undesired behavior of criu
691       and should only be used carefully.
692

EXAMPLES

694       To checkpoint a program with pid of 1234 and write all image files into
695       directory checkpoint:
696
697               criu dump -D checkpoint -t 1234
698
699       To restore this program detaching criu itself:
700
701               criu restore -d -D checkpoint
702

AUTHOR

704       The CRIU team.
705
707       Copyright (C) 2011-2016, Parallels Holdings, Inc.
708
709
710
711criu 3.12                         04/26/2019                           CRIU(8)
Impressum