1crio(8) System Manager's Manual crio(8)
2
6 crio - OCI-based implementation of Kubernetes Container Runtime Inter‐
7 face
8
12 crio
13 [--absent-mount-sources-to-reject]=[value]
16 [--add-inheritable-capabilities]
17 [--additional-devices]=[value]
18 [--allowed-devices]=[value]
19 [--apparmor-profile]=[value]
20 [--big-files-temporary-dir]=[value]
21 [--bind-mount-prefix]=[value]
22 [--blockio-config-file]=[value]
23 [--cdi-spec-dirs]=[value]
24 [--cgroup-manager]=[value]
25 [--clean-shutdown-file]=[value]
26 [--cni-config-dir]=[value]
27 [--cni-default-network]=[value]
28 [--cni-plugin-dir]=[value]
29 [--config-dir|-d]=[value]
30 [--config|-c]=[value]
31 [--conmon-cgroup]=[value]
32 [--conmon-env]=[value]
33 [--conmon]=[value]
34 [--container-attach-socket-dir]=[value]
35 [--container-exits-dir]=[value]
36 [--ctr-stop-timeout]=[value]
37 [--decryption-keys-path]=[value]
38 [--default-capabilities]=[value]
39 [--default-env]=[value]
40 [--default-mounts-file]=[value]
41 [--default-runtime]=[value]
42 [--default-sysctls]=[value]
43 [--default-transport]=[value]
44 [--default-ulimits]=[value]
45 [--device-ownership-from-security-context]
46 [--drop-infra-ctr]
47 [--enable-criu-support]
48 [--enable-metrics]
49 [--enable-nri]
50 [--enable-pod-events]
51 [--enable-profile-unix-socket]
52 [--enable-tracing]
53 [--gid-mappings]=[value]
54 [--global-auth-file]=[value]
55 [--grpc-max-recv-msg-size]=[value]
56 [--grpc-max-send-msg-size]=[value]
57 [--help|-h]
58 [--hooks-dir]=[value]
59 [--image-volumes]=[value]
60 [--infra-ctr-cpuset]=[value]
61 [--insecure-registry]=[value]
62 [--internal-wipe]
63 [--irqbalance-config-file]=[value]
64 [--listen]=[value]
65 [--log-dir]=[value]
66 [--log-filter]=[value]
67 [--log-format]=[value]
68 [--log-journald]
69 [--log-level|-l]=[value]
70 [--log-size-max]=[value]
71 [--log]=[value]
72 [--metrics-cert]=[value]
73 [--metrics-collectors]=[value]
74 [--metrics-key]=[value]
75 [--metrics-port]=[value]
76 [--metrics-socket]=[value]
77 [--minimum-mappable-gid]=[value]
78 [--minimum-mappable-uid]=[value]
79 [--namespaces-dir]=[value]
80 [--no-pivot]
81 [--nri-config-file]=[value]
82 [--nri-listen]=[value]
83 [--nri-plugin-dir]=[value]
84 [--pause-command]=[value]
85 [--pause-image-auth-file]=[value]
86 [--pause-image]=[value]
87 [--pids-limit]=[value]
88 [--pinns-path]=[value]
89 [--profile-cpu]=[value]
90 [--profile-mem]=[value]
91 [--profile-port]=[value]
92 [--profile]
93 [--rdt-config-file]=[value]
94 [--read-only]
95 [--registry]=[value]
96 [--root|-r]=[value]
97 [--runroot]=[value]
98 [--runtimes]=[value]
99 [--seccomp-profile]=[value]
100 [--seccomp-use-default-when-empty]
101 [--selinux]
102 [--separate-pull-cgroup]=[value]
103 [--signature-policy]=[value]
104 [--stats-collection-period]=[value]
105 [--storage-driver|-s]=[value]
106 [--storage-opt]=[value]
107 [--stream-address]=[value]
108 [--stream-enable-tls]
109 [--stream-idle-timeout]=[value]
110 [--stream-port]=[value]
111 [--stream-tls-ca]=[value]
112 [--stream-tls-cert]=[value]
113 [--stream-tls-key]=[value]
114 [--tracing-endpoint]=[value]
115 [--tracing-sampling-rate-per-million]=[value]
116 [--uid-mappings]=[value]
117 [--version-file-persist]=[value]
118 [--version-file]=[value]
119 [--version|-v]
120
125 OCI-based implementation of Kubernetes Container Runtime Interface Dae‐
126 mon
127 crio is meant to provide an integration path between OCI conformant
130 runtimes and the kubelet. Specifically, it implements the Kubelet Con‐
131 tainer Runtime Interface (CRI) using OCI conformant runtimes. The scope
132 of crio is tied to the scope of the CRI.
133 1. Support multiple image formats including the existing
136 Docker and OCI image formats.
137 2. Support for multiple means to download images including
139 trust & image verification.
140 3. Container image management (managing image layers, overlay
142 filesystems, etc).
143 4. Container process lifecycle management.
145 5. Monitoring and logging required to satisfy the CRI.
147 6. Resource isolation as required by the CRI.
149 Usage:
153 crio [GLOBAL OPTIONS] command [COMMAND OPTIONS] [ARGUMENTS...]
156
161 --absent-mount-sources-to-reject="": A list of paths that, when absent
162 from the host, will cause a container creation to fail (as opposed to
163 the current behavior of creating a directory).
164 --add-inheritable-capabilities: Add capabilities to the inheritable
167 set, as well as the default group of permitted, bounding and effective.
168 --additional-devices="": Devices to add to the containers.
171 --allowed-devices="": Devices a user is allowed to specify with the
174 "io.kubernetes.cri-o.Devices" allowed annotation. (default:
175 "/dev/fuse")
176 --apparmor-profile="": Name of the apparmor profile to be used as the
179 runtime's default. This only takes effect if the user does not specify
180 a profile via the Kubernetes Pod's metadata annotation. (default: crio-
181 default)
182 --big-files-temporary-dir="": Path to the temporary directory to use
185 for storing big files, used to store image blobs and data streams re‐
186 lated to containers image management.
187 --bind-mount-prefix="": A prefix to use for the source of the bind
190 mounts. This option would be useful if you were running CRI-O in a con‐
191 tainer. And had / mounted on /host in your container. Then if you ran
192 CRI-O with the --bind-mount-prefix=/host option, CRI-O would add /host
193 to any bind mounts it is handed over CRI. If Kubernetes asked to have
194 /var/lib/foobar bind mounted into the container, then CRI-O would bind
195 mount /host/var/lib/foobar. Since CRI-O itself is running in a con‐
196 tainer with / or the host mounted on /host, the container would end up
197 with /var/lib/foobar from the host mounted in the container rather then
198 /var/lib/foobar from the CRI-O container.
199 --blockio-config-file="": Path to the blockio class configuration file
202 for configuring the cgroup blockio controller.
203 --cdi-spec-dirs="": Directories to scan for CDI Spec files. (default:
206 "/etc/cdi", "/var/run/cdi")
207 --cgroup-manager="": cgroup manager (cgroupfs or systemd). (default:
210 systemd)
211 --clean-shutdown-file="": Location for CRI-O to lay down the clean
214 shutdown file. It indicates whether we've had time to sync changes to
215 disk before shutting down. If not found, crio wipe will clear the stor‐
216 age directory. (default: /var/lib/crio/clean.shutdown)
217 --cni-config-dir="": CNI configuration files directory. (default:
220 /etc/cni/net.d/)
221 --cni-default-network="": Name of the default CNI network to select. If
224 not set or "", then CRI-O will pick-up the first one found in --cni-
225 config-dir.
226 --cni-plugin-dir="": CNI plugin binaries directory.
229 --config, -c="": Path to configuration file (default:
232 /etc/crio/crio.conf)
233 --config-dir, -d="": Path to the configuration drop-in directory.
236 This directory will be recursively iterated and each file gets ap‐
237 plied
238 to the configuration in their processing order. This means that a
239 configuration file named '00-default' has a lower priority than a
240 file
241 named '01-my-overwrite'.
242 The global config file, provided via '--config,-c' or per default
243 in
244 /etc/crio/crio.conf, always has a lower priority than the files in
245 the directory specified
246 by '--config-dir,-d'.
247 Besides that, provided command line parameters have a higher prior‐
248 ity
249 than any configuration file. (default: /etc/crio/crio.conf.d)
250 --conmon="": Path to the conmon binary, used for monitoring the OCI
253 runtime. Will be searched for using $PATH if empty. This option is dep‐
254 recated, and will be removed in the future.
255 --conmon-cgroup="": cgroup to be used for conmon process. This option
258 is deprecated and will be removed in the future.
259 --conmon-env="": Environment variable list for the conmon process, used
262 for passing necessary environment variables to conmon or the runtime.
263 This option is deprecated and will be removed in the future.
264 --container-attach-socket-dir="": Path to directory for container at‐
267 tach sockets. (default: /var/run/crio)
268 --container-exits-dir="": Path to directory in which container exit
271 files are written to by conmon. (default: /var/run/crio/exits)
272 --ctr-stop-timeout="": The minimal amount of time in seconds to wait
275 before issuing a timeout regarding the proper termination of the con‐
276 tainer. The lowest possible value is 30s, whereas lower values are not
277 considered by CRI-O. (default: 30)
278 --decryption-keys-path="": Path to load keys for image decryption. (de‐
281 fault: /etc/crio/keys/)
282 --default-capabilities="": Capabilities to add to the containers. (de‐
285 fault: "CHOWN", "DAC_OVERRIDE", "FSETID", "FOWNER", "SETGID", "SETUID",
286 "SETPCAP", "NET_BIND_SERVICE", "KILL")
287 --default-env="": Additional environment variables to set for all con‐
290 tainers.
291 --default-mounts-file="": Path to default mounts file.
294 --default-runtime="": Default OCI runtime from the runtimes config.
297 (default: runc)
298 --default-sysctls="": Sysctls to add to the containers.
301 --default-transport="": A prefix to prepend to image names that cannot
304 be pulled as-is. (default: docker://)
305 --default-ulimits="": Ulimits to apply to containers by default
308 (name=soft:hard).
309 --device-ownership-from-security-context: Set devices' uid/gid owner‐
312 ship from runAsUser/runAsGroup.
313 --drop-infra-ctr: Determines whether pods are created without an infra
316 container, when the pod is not using a pod level PID namespace.
317 --enable-criu-support: Enable CRIU integration, requires that the criu
320 binary is available in $PATH.
321 --enable-metrics: Enable metrics endpoint for the server on local‐
324 host:9090.
325 --enable-nri: Enable NRI (Node Resource Interface) support. (default:
328 false)
329 --enable-pod-events: If true, CRI-O starts sending the container events
332 to the kubelet
333 --enable-profile-unix-socket: Enable pprof profiler on crio unix domain
336 socket.
337 --enable-tracing: Enable OpenTelemetry trace data exporting.
340 --gid-mappings="": Specify the GID mappings to use for the user name‐
343 space.
344 --global-auth-file="": Path to a file like /var/lib/kubelet/config.json
347 holding credentials necessary for pulling images from secure reg‐
348 istries.
349 --grpc-max-recv-msg-size="": Maximum grpc receive message size in
352 bytes. (default: 83886080)
353 --grpc-max-send-msg-size="": Maximum grpc receive message size. (de‐
356 fault: 83886080)
357 --help, -h: show help
360 --hooks-dir="": Set the OCI hooks directory path (may be set multiple
363 times)
364 If one of the directories does not exist, then CRI-O will automati‐
365 cally
366 skip them.
367 Each '*.json' file in the path configures a hook for CRI-O
368 containers. For more details on the syntax of the JSON files and
369 the semantics of hook injection, see 'oci-hooks(5)'. CRI-O
370 currently support both the 1.0.0 and 0.1.0 hook schemas, although
371 the 0.1.0 schema is deprecated.
372 This option may be set multiple times; paths from later options
373 have higher precedence ('oci-hooks(5)' discusses directory
374 precedence).
375 For the annotation conditions, CRI-O uses the Kubernetes
376 annotations, which are a subset of the annotations passed to the
377 OCI runtime. For example, 'io.kubernetes.cri-o.Volumes' is part of
378 the OCI runtime configuration annotations, but it is not part of
379 the Kubernetes annotations being matched for hooks.
380 For the bind-mount conditions, only mounts explicitly requested by
381 Kubernetes configuration are considered. Bind mounts that CRI-O
382 inserts by default (e.g. '/dev/shm') are not considered. (default:
383 "/usr/share/containers/oci/hooks.d")
384 --image-volumes="": Image volume handling ('mkdir', 'bind', or 'ig‐
387 nore')
388 1. mkdir: A directory is created inside the container root filesys‐
389 tem for
390 the volumes.
391 2. bind: A directory is created inside container state directory
392 and bind
393 mounted into the container for the volumes. 3. ignore: All
394 volumes are just ignored and no action is taken. (default: mkdir)
395 --infra-ctr-cpuset="": CPU set to run infra containers, if not speci‐
398 fied CRI-O will use all online CPUs to run infra containers.
399 --insecure-registry="": Enable insecure registry communication, i.e.,
402 enable un-encrypted and/or untrusted communication.
403 1. List of insecure registries can contain an element with CIDR no‐
404 tation to
405 specify a whole subnet.
406 2. Insecure registries accept HTTP or accept HTTPS with certifi‐
407 cates from
408 unknown CAs.
409 3. Enabling '--insecure-registry' is useful when running a local
410 registry.
411 However, because its use creates security vulnerabilities, it
412 should ONLY
413 be enabled for testing purposes. For increased security, users
414 should add
415 their CA to their system's list of trusted CAs instead of using
416 '--insecure-registry'.
417 --internal-wipe: Whether CRI-O should wipe containers after a reboot
420 and images after an upgrade when the server starts. If set to false,
421 one must run crio wipe to wipe the containers and images in these situ‐
422 ations. This option is deprecated, and will be removed in the future.
423 --irqbalance-config-file="": The irqbalance service config file which
426 is used by CRI-O. (default: /etc/sysconfig/irqbalance)
427 --listen="": Path to the CRI-O socket. (default:
430 /var/run/crio/crio.sock)
431 --log="": Set the log file path where internal debug information is
434 written.
435 --log-dir="": Default log directory where all logs will go unless di‐
438 rectly specified by the kubelet. (default: /var/log/crio/pods)
439 --log-filter="": Filter the log messages by the provided regular ex‐
442 pression. For example 'request.*' filters all gRPC requests.
443 --log-format="": Set the format used by logs: 'text' or 'json'. (de‐
446 fault: text)
447 --log-journald: Log to systemd journal (journald) in addition to kuber‐
450 netes log file.
451 --log-level, -l="": Log messages above specified level: trace, debug,
454 info, warn, error, fatal or panic. (default: info)
455 --log-size-max="": Maximum log size in bytes for a container. If it is
458 positive, it must be >= 8192 to match/exceed conmon read buffer. This
459 option is deprecated. The Kubelet flag '--container-log-max-size'
460 should be used instead. (default: -1)
461 --metrics-cert="": Certificate for the secure metrics endpoint.
464 --metrics-collectors="": Enabled metrics collectors. (default: "opera‐
467 tions", "operations_latency_microseconds_total", "operations_la‐
468 tency_microseconds", "operations_errors", "image_pulls_by_digest", "im‐
469 age_pulls_by_name", "image_pulls_by_name_skipped", "image_pulls_fail‐
470 ures", "image_pulls_successes", "image_pulls_layer_size", "im‐
471 age_layer_reuse", "containers_oom_total", "containers_oom", "pro‐
472 cesses_defunct", "operations_total", "operations_latency_seconds", "op‐
473 erations_latency_seconds_total", "operations_errors_total", "im‐
474 age_pulls_bytes_total", "image_pulls_skipped_bytes_total", "im‐
475 age_pulls_failure_total", "image_pulls_success_total", "image_layer_re‐
476 use_total", "containers_oom_count_total", "containers_seccomp_noti‐
477 fier_count_total")
478 --metrics-key="": Certificate key for the secure metrics endpoint.
481 --metrics-port="": Port for the metrics endpoint. (default: 9090)
484 --metrics-socket="": Socket for the metrics endpoint.
487 --minimum-mappable-gid="": Specify the lowest host GID which can be
490 specified in mappings for a pod that will be run as a UID other than 0.
491 (default: -1)
492 --minimum-mappable-uid="": Specify the lowest host UID which can be
495 specified in mappings for a pod that will be run as a UID other than 0.
496 (default: -1)
497 --namespaces-dir="": The directory where the state of the managed name‐
500 spaces gets tracked. Only used when manage-ns-lifecycle is true. (de‐
501 fault: /var/run)
502 --no-pivot: If true, the runtime will not use pivot_root, but instead
505 use MS_MOVE.
506 --nri-config-file="": NRI configuration file to use. (default:
509 "/etc/nri/nri.conf")
510 --nri-listen="": Socket to listen on for externally started NRI plugins
513 to connect to. (default: "/var/run/nri.sock")
514 --nri-plugin-dir="": Directory to scan for pre-installed NRI plugins to
517 start automatically. (default: "/opt/nri/plugins")
518 --pause-command="": Path to the pause executable in the pause image.
521 (default: /pause)
522 --pause-image="": Image which contains the pause executable. (default:
525 registry.k8s.io/pause:3.6)
526 --pause-image-auth-file="": Path to a config file containing creden‐
529 tials for --pause-image.
530 --pids-limit="": Maximum number of processes allowed in a container.
533 This option is deprecated. The Kubelet flag '--pod-pids-limit' should
534 be used instead. (default: 0)
535 --pinns-path="": The path to find the pinns binary, which is needed to
538 manage namespace lifecycle. Will be searched for in $PATH if empty.
539 --profile: Enable pprof remote profiler on localhost:6060.
542 --profile-cpu="": Write a pprof CPU profile to the provided path.
545 --profile-mem="": Write a pprof memory profile to the provided path.
548 --profile-port="": Port for the pprof profiler. (default: 6060)
551 --rdt-config-file="": Path to the RDT configuration file for configur‐
554 ing the resctrl pseudo-filesystem.
555 --read-only: Setup all unprivileged containers to run as read-only. Au‐
558 tomatically mounts the containers' tmpfs on /run, /tmp and /var/tmp.
559 --registry="": Registry to be prepended when pulling unqualified im‐
562 ages. Can be specified multiple times.
563 --root, -r="": The CRI-O root directory. (default: /var/lib/contain‐
566 ers/storage)
567 --runroot="": The CRI-O state directory. (default: /run/contain‐
570 ers/storage)
571 --runtimes="": OCI runtimes, format is 'runtime_name:runtime_path:run‐
574 time_root:runtime_type:privileged_without_host_devices:runtime_con‐
575 fig_path'.
576 --seccomp-profile="": Path to the seccomp.json profile to be used as
579 the runtime's default. If not specified, then the internal default sec‐
580 comp profile will be used.
581 --seccomp-use-default-when-empty: Use the default seccomp profile when
584 an empty one is specified.
585 --selinux: Enable selinux support.
588 --separate-pull-cgroup="": [EXPERIMENTAL] Pull in new cgroup.
591 --signature-policy="": Path to signature policy JSON file.
594 --stats-collection-period="": The number of seconds between collecting
597 pod and container stats. If set to 0, the stats are collected on-demand
598 instead. (default: 0)
599 --storage-driver, -s="": OCI storage driver.
602 --storage-opt="": OCI storage driver option.
605 --stream-address="": Bind address for streaming socket. (default:
608 127.0.0.1)
609 --stream-enable-tls: Enable encrypted TLS transport of the stream
612 server.
613 --stream-idle-timeout="": Length of time until open streams terminate
616 due to lack of activity.
617 --stream-port="": Bind port for streaming socket. If the port is set to
620 '0', then CRI-O will allocate a random free port number. (default: 0)
621 --stream-tls-ca="": Path to the x509 CA(s) file used to verify and au‐
624 thenticate client communication with the encrypted stream. This file
625 can change and CRI-O will automatically pick up the changes within 5
626 minutes.
627 --stream-tls-cert="": Path to the x509 certificate file used to serve
630 the encrypted stream. This file can change and CRI-O will automatically
631 pick up the changes within 5 minutes.
632 --stream-tls-key="": Path to the key file used to serve the encrypted
635 stream. This file can change and CRI-O will automatically pick up the
636 changes within 5 minutes.
637 --tracing-endpoint="": Address on which the gRPC tracing collector will
640 listen. (default: 0.0.0.0:4317)
641 --tracing-sampling-rate-per-million="": Number of samples to collect
644 per million OpenTelemetry spans. Set to 1000000 to always sample. (de‐
645 fault: 0)
646 --uid-mappings="": Specify the UID mappings to use for the user name‐
649 space.
650 --version, -v: print the version
653 --version-file="": Location for CRI-O to lay down the temporary version
656 file. It is used to check if crio wipe should wipe containers, which
657 should always happen on a node reboot. (default: /var/run/crio/version)
658 --version-file-persist="": Location for CRI-O to lay down the persis‐
661 tent version file. It is used to check if crio wipe should wipe images,
662 which should only happen when CRI-O has been upgraded. (default:
663 /var/run/crio/version)
664
669 Generate bash, fish or zsh completions.
670
673 Generate the man page documentation.
674
677 Generate the markdown documentation.
678 --help, -h: show help
681 help, h
684 Shows a list of commands or help for one command
685
688 Outputs a commented version of the configuration file that could be
689 used by CRI-O. This allows you to save you current configuration setup
690 and then load it later with --config. Global options will modify the
691 output.
692 --default: Output the default configuration (without taking into ac‐
695 count any configuration options).
696 --migrate-defaults, -m="": Migrate the default config from a specified
699 version.
700 To run a config migration, just select the input config via the
701 global
702 '--config,-c' command line argument, for example:
703 crio -c /etc/crio/crio.conf.d/00-default.conf config -m 1.17
705 The migration will print converted configuration options to stderr
706 and will
707 output the resulting configuration to stdout.
708 Please note that the migration will overwrite any fields that have
709 changed
710 defaults between versions. To save a custom configuration change,
711 it should
712 be in a drop-in configuration file instead.
713 Possible values: "1.17" (default: 1.17)
714
717 display detailed version information
718 --json, -j: print JSON instead of text
721 --verbose, -v: print verbose information (for example all golang depen‐
724 dencies)
725
728 wipe CRI-O's container and image storage
729 --force, -f: force wipe by skipping the version check
732
735 Shows a list of commands or help for one command
736
739 crio.conf (/etc/crio/crio.conf)
740 cri-o configuration file for all of the available command-line op‐
741 tions for
742 the crio(8) program, but in a TOML format that can be more easily
743 modified
744 and versioned.
745 policy.json (/etc/containers/policy.json)
748 Signature verification policy files are used to specify policy, e.g.
749 trusted
750 keys, applicable when deciding whether to accept an image, or indi‐
751 vidual
752 signatures of that image, as valid.
753 registries.conf (/etc/containers/registries.conf)
756 Registry configuration file specifies registries which are consulted
757 when
758 completing image names that do not include a registry or domain por‐
759 tion.
760 storage.conf (/etc/containers/storage.conf)
763 Storage configuration file specifies all of the available container
764 storage
765 options for tools using shared container storage.
766
770 All command-line options may also be specified as environment vari‐
771 ables. The options detailed in this section, however, can only be set
772 via environment variables.
773 KUBENSMNT: Path to a bind-mounted mount namespace that CRI-O should
776 join before launching any containers. If the path does not exist, or
777 does not point to a mount namespace bindmount, CRI-O will run in its
778 parent's mount namespace and log a warning that the requested namespace
779 was not joined.
780
784 crio.conf(5), crio.conf.d(5), oci-hooks(5), policy.json(5), reg‐
785 istries.conf(5), storage.conf(5)
786 crio(8)