1crio.conf(5)(Kubernetes) Daemon crio.conf(5)(Kubernetes)
2Aleksa Sarai OCTOBER 2016
6
9 crio.conf - configuration file of the CRI-O OCI Kubernetes Container
10 Runtime daemon
11
15 The CRI-O configuration file specifies all of the available configura‐
16 tion options and command-line flags for the crio(8) OCI Kubernetes Con‐
17 tainer Runtime daemon ⟨./crio.8.md⟩, but in a TOML format that can be
18 more easily modified and versioned.
19 CRI-O supports partial configuration reload during runtime, which can
22 be done by sending SIGHUP to the running process. Currently supported
23 options in crio.conf are explicitly marked with 'This option supports
24 live configuration reload'.
25 The containers-registries.conf(5) file can be reloaded as well by send‐
28 ing SIGHUP to the crio process.
29 The default crio.conf is located at /etc/crio/crio.conf.
32
36 The TOML format ⟨https://github.com/toml-lang/toml⟩ is used as the en‐
37 coding of the configuration file. Every option and subtable listed here
38 is nested under a global "crio" table. No bare options are used. The
39 format of TOML can be simplified to:
40 [table]
43 option = value
44 [table.subtable1]
46 option = value
47 [table.subtable2]
49 option = value
50
54 CRI-O reads its storage defaults from the containers-storage.conf(5)
55 file located at /etc/containers/storage.conf. Modify this storage con‐
56 figuration if you want to change the system's defaults. If you want to
57 modify storage just for CRI-O, you can change the storage configuration
58 options here.
59 root="/var/lib/containers/storage"
62 Path to the "root directory". CRI-O stores all of its data, including
63 containers images, in this directory.
64 runroot="/var/run/containers/storage"
67 Path to the "run directory". CRI-O stores all of its state in this
68 directory.
69 storage_driver="overlay"
72 Storage driver used to manage the storage of images and containers.
73 Please refer to containers-storage.conf(5) to see all available storage
74 drivers.
75 storage_option=[]
78 List to pass options to the storage driver. Please refer to contain‐
79 ers-storage.conf(5) to see all available storage options.
80 log_dir="/var/log/crio/pods"
83 The default log directory where all logs will go unless directly
84 specified by the kubelet. The log directory specified must be an abso‐
85 lute directory.
86 version_file="/var/run/crio/version"
89 Location for CRI-O to lay down the temporary version file.
90 It is used to check if crio wipe should wipe containers, which should
91 always happen on a node reboot
92 version_file_persist=""
95 Location for CRI-O to lay down the persistent version file.
96 It is used to check if crio wipe should wipe images, which should
97 only happen when CRI-O has been upgraded
98 internal_wipe=true
101 This option is currently DEPRECATED, and will be removed in the fu‐
102 ture.
103 Whether CRI-O should wipe containers after a reboot and images after
104 an upgrade when the server starts.
105 If set to false, one must run crio wipe to wipe the containers and
106 images in these situations.
107 clean_shutdown_file="/var/lib/crio/clean.shutdown"
110 Location for CRI-O to lay down the clean shutdown file.
111 It is used to check whether crio had time to sync before shutting
112 down.
113 If not found, crio wipe will clear the storage directory.
114
117 The crio.api table contains settings for the kubelet/gRPC interface.
118 listen="/var/run/crio/crio.sock"
121 Path to AF_LOCAL socket on which CRI-O will listen.
122 stream_address="127.0.0.1"
125 IP address on which the stream server will listen.
126 stream_port="0"
129 The port on which the stream server will listen. If the port is set
130 to "0", then CRI-O will allocate a random free port number.
131 stream_enable_tls=false
134 Enable encrypted TLS transport of the stream server.
135 stream_idle_timeout=""
138 Length of time until open streams terminate due to lack of activity.
139 stream_tls_cert=""
142 Path to the x509 certificate file used to serve the encrypted stream.
143 This file can change and CRI-O will automatically pick up the changes
144 within 5 minutes.
145 stream_tls_key=""
148 Path to the key file used to serve the encrypted stream. This file
149 can change and CRI-O will automatically pick up the changes within 5
150 minutes.
151 stream_tls_ca=""
154 Path to the x509 CA(s) file used to verify and authenticate client
155 communication with the encrypted stream. This file can change and CRI-O
156 will automatically pick up the changes within 5 minutes.
157 grpc_max_send_msg_size=83886080
160 Maximum grpc send message size in bytes. If not set or <=0, then CRI-
161 O will default to 80 * 1024 * 1024.
162 grpc_max_recv_msg_size=83886080
165 Maximum grpc receive message size. If not set or <= 0, then CRI-O
166 will default to 80 * 1024 * 1024.
167
170 The crio.runtime table contains settings pertaining to the OCI runtime
171 used and options for how to set up and manage the OCI runtime.
172 default_runtime="runc"
175 The name of the OCI runtime to be used as the default. This option
176 supports live configuration reload.
177 default_ulimits=[]
180 A list of ulimits to be set in containers by default, specified as
181 "=:", for example:"nofile=1024:2048". If nothing is set here, settings
182 will be inherited from the CRI-O daemon.
183 no_pivot=false
186 If true, the runtime will not use pivot_root, but instead use
187 MS_MOVE.
188 decryption_keys_path="/etc/crio/keys/"
191 Path where the keys required for image decryption are located
192 conmon=""
195 Path to the conmon binary, used for monitoring the OCI runtime. Will
196 be searched for using $PATH if empty.
197 This option is currently deprecated, and will be replaced with Run‐
198 timeHandler.MonitorPath.
199 conmon_cgroup=""
202 Cgroup setting for conmon
203 This option is currently deprecated, and will be replaced with Run‐
204 timeHandler.MonitorCgroup.
205 conmon_env=[]
208 Environment variable list for the conmon process, used for passing
209 necessary environment variables to conmon or the runtime.
210 This option is currently deprecated, and will be replaced with Run‐
211 timeHandler.MonitorEnv.
212 default_env=[]
215 Additional environment variables to set for all the containers. These
216 are overridden if set in the container image spec or in the container
217 runtime configuration.
218 selinux=false
221 If true, SELinux will be used for pod separation on the host.
222 seccomp_profile=""
225 Path to the seccomp.json profile which is used as the default seccomp
226 profile for the runtime. If not specified, then the internal default
227 seccomp profile will be used.
228 seccomp_use_default_when_empty=true
231 Changes the meaning of an empty seccomp profile. By default (and ac‐
232 cording to CRI spec), an empty profile means unconfined.
233 This option tells CRI-O to treat an empty profile as the default pro‐
234 file, which might increase security.
235 apparmor_profile=""
238 Used to change the name of the default AppArmor profile of CRI-O. The
239 default profile name is "crio-default".
240 blockio_config_file=""
243 Path to the blockio class configuration file for configuring the
244 cgroup blockio controller.
245 cdi_spec_dirs=[]
248 Directories to scan for Container Device Interface Specifications to
249 enable CDI device injection. For more details about CDI and the syntax
250 of CDI Spec files please refer to https://github.com/container-orches‐
251 trated-devices/container-device-interface.
252 Directories later in the list have precedence over earlier ones. The
255 default directory list is:
256 cdi_spec_dirs = [
259 "/etc/cdi",
260 "/var/run/cdi",
261 ]
262 irqbalance_config_file="/etc/sysconfig/irqbalance"
266 Used to change irqbalance service config file which is used by CRI-O.
267 For CentOS/SUSE, this file is located at /etc/sysconfig/irqbalance.
268 For Ubuntu, this file is located at /etc/default/irqbalance.
269 rdt_config_file=""
272 Path to the RDT configuration file for configuring the resctrl
273 pseudo-filesystem.
274 cgroup_manager="systemd"
277 Cgroup management implementation used for the runtime.
278 default_capabilities=[]
281 List of default capabilities for containers. If it is empty or com‐
282 mented out, only the capabilities defined in the container json file by
283 the user/kube will be added.
284 The default list is:
287 default_capabilities = [
290 "CHOWN",
291 "DAC_OVERRIDE",
292 "FSETID",
293 "FOWNER",
294 "SETGID",
295 "SETUID",
296 "SETPCAP",
297 "NET_BIND_SERVICE",
298 "KILL",
299 ]
300 add_inheritable_capabilities=false
304 Add capabilities to the inheritable set, as well as the default group
305 of permitted, bounding and effective.
306 If capabilities are expected to work for non-root users, this option
307 should be set.
308 default_sysctls=[]
311 List of default sysctls. If it is empty or commented out, only the
312 sysctls defined in the container json file by the user/kube will be
313 added.
314 One example would be allowing ping inside of containers. On systems
317 that support /proc/sys/net/ipv4/ping_group_range, the default list
318 could be:
319 default_sysctls = [
322 "net.ipv4.ping_group_range = 0 2147483647",
323 ]
324 allowed_devices=[]
328 List of devices on the host that a user can specify with the "io.ku‐
329 bernetes.cri-o.Devices" allowed annotation.
330 additional_devices=[]
333 List of additional devices. Specified as "::", for example: "--addi‐
334 tional-devices=/dev/sdc:/dev/xvdc:rwm". If it is empty or commented
335 out, only the devices defined in the container json file by the
336 user/kube will be added.
337 hooks_dir=["path", ...]
340 Each *.json file in the path configures a hook for CRI-O containers.
341 For more details on the syntax of the JSON files and the semantics of
342 hook injection, see oci-hooks(5). CRI-O currently support both the
343 1.0.0 and 0.1.0 hook schemas, although the 0.1.0 schema is deprecated.
344 Paths listed later in the array have higher precedence (oci-hooks(5)
347 discusses directory precedence).
348 For the annotation conditions, CRI-O uses the Kubernetes annotations,
351 which are a subset of the annotations passed to the OCI runtime. For
352 example, io.kubernetes.cri-o.Volumes is part of the OCI runtime config‐
353 uration annotations, but it is not part of the Kubernetes annotations
354 being matched for hooks.
355 For the bind-mount conditions, only mounts explicitly requested by Ku‐
358 bernetes configuration are considered. Bind mounts that CRI-O inserts
359 by default (e.g. /dev/shm) are not considered.
360 default_mounts=[]
363 List of default mounts for each container. Deprecated: this option
364 will be removed in future versions in favor of default_mounts_file.
365 default_mounts_file=""
368 Path to the file specifying the defaults mounts for each container.
369 The format of the config is /SRC:/DST, one mount per line. Notice that
370 CRI-O reads its default mounts from the following two files:
371 1) `/etc/containers/mounts.conf` (i.e., default_mounts_file): This is the override file, where users can either add in their own default mounts, or override the default mounts shipped with the package.
374 2) `/usr/share/containers/mounts.conf`: This is the default file read for mounts. If you want CRI-O to read from a different, specific mounts file, you can change the default_mounts_file. Note, if this is done, CRI-O will only add mounts it finds in this file.
376 pids_limit=0
380 Maximum number of processes allowed in a container.
381 This option is deprecated. The Kubelet flag --pod-pids-limit should
382 be used instead.
383 log_filter=""
386 Filter the log messages by the provided regular expression. This op‐
387 tion supports live configuration reload. For example 'request:.*' fil‐
388 ters all gRPC requests.
389 log_level="info"
392 Changes the verbosity of the logs based on the level it is set to.
393 Options are fatal, panic, error, warn, info, debug, and trace. This op‐
394 tion supports live configuration reload.
395 log_size_max=-1
398 Maximum size allowed for the container log file. Negative numbers in‐
399 dicate that no size limit is imposed. If it is positive, it must be >=
400 8192 to match/exceed conmon's read buffer. The file is truncated and
401 re-opened so the limit is never exceeded.
402 This option is deprecated. The Kubelet flag --container-log-max-size
403 should be used instead.
404 log_to_journald=false
407 Whether container output should be logged to journald in addition to
408 the kuberentes log file.
409 container_exits_dir="/var/run/crio/exits"
412 Path to directory in which container exit files are written to by
413 conmon.
414 container_attach_socket_dir="/var/run/crio"
417 Path to directory for container attach sockets.
418 bind_mount_prefix=""
421 A prefix to use for the source of the bind mounts. This option would
422 be useful when running CRI-O in a container and the / directory on the
423 host is mounted as /host in the container. Then if CRI-O runs with the
424 --bind-mount-prefix=/host option, CRI-O would add the /host directory
425 to any bind mounts it hands over to CRI. If Kubernetes asked to have
426 /var/lib/foobar bind mounted into the container, then CRI-O would bind
427 mount /host/var/lib/foobar. Since CRI-O itself is running in a con‐
428 tainer with / or the host mounted on /host, the container would end up
429 with /var/lib/foobar from the host mounted in the container rather than
430 /var/lib/foobar from the CRI-O container.
431 read_only=false
434 If set to true, all containers will run in read-only mode.
435 uid_mappings=""
438 The UID mappings for the user namespace of each container. A range is
439 specified in the form containerUID:HostUID:Size. Multiple ranges must
440 be separated by comma.
441 minimum_mappable_uid=-1
444 The lowest host UID which can be specified in mappings supplied, ei‐
445 ther as part of a uid_mappings or as part of a request received over
446 CRI, for a pod that will be run as a UID other than 0.
447 gid_mappings=""
450 The GID mappings for the user namespace of each container. A range is
451 specified in the form containerGID:HostGID:Size. Multiple ranges must
452 be separated by comma.
453 minimum_mappable_gid=-1
456 The lowest host GID which can be specified in mappings supplied, ei‐
457 ther as part of a gid_mappings or as part of a request received over
458 CRI, for a pod that will be run as a UID other than 0.
459 ctr_stop_timeout=30
462 The minimal amount of time in seconds to wait before issuing a time‐
463 out regarding the proper termination of the container.
464 drop_infra_ctr=true
467 Determines whether we drop the infra container when a pod does not
468 have a private PID namespace, and does not use a kernel separating run‐
469 time (like kata).
470 Requies manage_ns_lifecycle to be true.
471 infra_ctr_cpuset=""
474 Determines the CPU set to run infra containers. If not specified,
475 the CRI-O will use all online CPUs to run infra containers.
476 You can specify CPUs in the Linux CPU list format.
477 To get better isolation for guaranteed pods, set this parameter to
478 be equal to kubelet reserved-cpus.
479 namespaces_dir="/var/run"
482 The directory where the state of the managed namespaces gets tracked.
483 Only used when manage_ns_lifecycle is true
484 pinns_path=""
487 The path to find the pinns binary, which is needed to manage name‐
488 space lifecycle
489 absent_mount_sources_to_reject=[]
492 A list of paths that, when absent from the host, will cause a con‐
493 tainer creation to fail (as opposed to the current behavior of creating
494 a directory).
495 device_ownership_from_security_context=false
498 Changes the default behavior of setting container devices uid/gid
499 from CRI's SecurityContext (RunAsUser/RunAsGroup) instead of taking
500 host's uid/gid.
501 enable_criu_support=false
504 Enable CRIU integration, requires that the criu binary is available
505 in $PATH. (default: false)
506 enable_pod_events=false Enable CRI-O to generate the container pod-
509 level events in order to optimize the performance of the Pod Lifecycle
510 Event Generator (PLEG) module in Kubelet.
511 CRIO.RUNTIME.RUNTIMES TABLE
514 The "crio.runtime.runtimes" table defines a list of OCI compatible run‐
515 times. The runtime to use is picked based on the runtime handler pro‐
516 vided by the CRI. If no runtime handler is provided, the runtime will
517 be picked based on the level of trust of the workload. This option sup‐
518 ports live configuration reload. This option supports live configura‐
519 tion reload.
520 runtime_path=""
523 Path to the OCI compatible runtime used for this runtime handler.
524 runtime_root=""
527 Root directory used to store runtime data
528 runtime_type="oci"
531 Type of the runtime used for this runtime handler. "oci", "vm"
532 runtime_config_path=""
535 Path to the runtime configuration file, should only be used with VM
536 runtime types
537 privileged_without_host_devices=false
540 Whether this runtime handler prevents host devices from being passed
541 to privileged containers.
542 allowed_annotations=[]
545 This field is currently DEPRECATED. If you'd like to use allowed_an‐
546 notations, please use a workload.
547 A list of experimental annotations this runtime handler is allowed to
548 process.
549 The currently recognized values are:
550 "io.kubernetes.cri-o.userns-mode" for configuring a user namespace
551 for the pod.
552 "io.kubernetes.cri-o.Devices" for configuring devices for the pod.
553 "io.kubernetes.cri-o.ShmSize" for configuring the size of /dev/shm.
554 "io.kubernetes.cri-o.UnifiedCgroup.$CTR_NAME" for configuring the
555 cgroup v2 unified block for a container.
556 "io.containers.trace-syscall" for tracing syscalls via the OCI sec‐
557 comp BPF hook.
558 CRIO.RUNTIME.WORKLOADS TABLE
561 The "crio.runtime.workloads" table defines a list of workloads - a way
562 to customize the behavior of a pod and container. A workload is chosen
563 for a pod based on whether the workload's activation_annotation is an
564 annotation on the pod.
565 activation_annotation=""
568 activation_annotation is the pod annotation that activates these
569 workload settings.
570 annotation_prefix=""
573 annotation_prefix is the way a pod can override a specific resource
574 for a container.
575 The full annotation must be of the form $annotation_prefix.$re‐
576 source/$ctrname = $value.
577 allowed_annotations=[]
580 allowed_annotations is a slice of experimental annotations that this
581 workload is allowed to process.
582 The currently recognized values are:
583 "io.kubernetes.cri-o.userns-mode" for configuring a user namespace
584 for the pod.
585 "io.kubernetes.cri-o.Devices" for configuring devices for the pod.
586 "io.kubernetes.cri-o.ShmSize" for configuring the size of /dev/shm.
587 "io.kubernetes.cri-o.UnifiedCgroup.$CTR_NAME" for configuring the
588 cgroup v2 unified block for a container.
589 "io.containers.trace-syscall" for tracing syscalls via the OCI sec‐
590 comp BPF hook.
591 "io.kubernetes.cri-o.seccompNotifierAction" for enabling the seccomp
592 notifier feature.
593 Using the seccomp notifier feature:
596 This feature can help you to debug seccomp related issues, for example
597 if blocked syscalls (permission denied errors) have negative impact on
598 the workload.
599 To be able to use this feature, configure a runtime which has the anno‐
602 tation "io.kubernetes.cri-o.seccompNotifierAction" in the allowed_anno‐
603 tations array.
604 It also requires at least runc 1.1.0 or crun 0.19 which support the no‐
607 tifier feature.
608 If everything is setup, CRI-O will modify chosen seccomp profiles for
611 containers if the annotation "io.kubernetes.cri-o.seccompNotifierAc‐
612 tion" is set on the Pod sandbox. CRI-O will then get notified if a con‐
613 tainer is using a blocked syscall and then terminate the workload after
614 a timeout of 5 seconds if the value of "io.kubernetes.cri-o.seccompNo‐
615 tifierAction=stop".
616 This also means that multiple syscalls can be captured during that pe‐
619 riod, while the timeout will get reset once a new syscall has been dis‐
620 covered.
621 This also means that the Pods "restartPolicy" has to be set to "Never",
624 otherwise the kubelet will restart the container immediately.
625 Please be aware that CRI-O is not able to get notified if a syscall
628 gets blocked based on the seccomp defaultAction, which is a general
629 runtime limitation.
630 CRIO.RUNTIME.WORKLOAD.RESOURCES TABLE
633 The resources table is a structure for overriding certain resources for
634 pods using this workload. This structure provides a default value, and
635 can be overridden by using the AnnotationPrefix.
636 cpushares="" Specifies the number of CPU shares this pod has access to.
639 cpuset="" Specifies the cpuset this pod has access to.
642
645 The crio.image table contains settings pertaining to the management of
646 OCI images.
647 CRI-O reads its configured registries defaults from the system wide
650 containers-registries.conf(5) located in /etc/containers/reg‐
651 istries.conf. If you want to modify just CRI-O, you can change the reg‐
652 istries configuration in this file. Otherwise, leave insecure_reg‐
653 istries and registries commented out to use the system's defaults from
654 /etc/containers/registries.conf.
655 default_transport="docker://"
658 Default transport for pulling images from a remote container storage.
659 global_auth_file=""
662 The path to a file like /var/lib/kubelet/config.json holding creden‐
663 tials necessary for pulling images from secure registries.
664 pause_image="registry.k8s.io/pause:3.6"
667 The image used to instantiate infra containers. This option supports
668 live configuration reload.
669 pause_image_auth_file=""
672 The path to a file like /var/lib/kubelet/config.json holding creden‐
673 tials specific to pulling the pause_image from above. This option sup‐
674 ports live configuration reload.
675 pause_command="/pause"
678 The command to run to have a container stay in the paused state. This
679 option supports live configuration reload.
680 signature_policy=""
683 Path to the file which decides what sort of policy we use when decid‐
684 ing whether or not to trust an image that we've pulled. It is not rec‐
685 ommended that this option be used, as the default behavior of using the
686 system-wide default policy (i.e., /etc/containers/policy.json) is most
687 often preferred. Please refer to containers-policy.json(5) for more de‐
688 tails.
689 image_volumes="mkdir"
692 Controls how image volumes are handled. The valid values are mkdir,
693 bind and ignore; the latter will ignore volumes entirely.
694 insecure_registries=[]
697 List of registries to skip TLS verification for pulling images.
698 registries=["docker.io"]
701 List of registries to be used when pulling an unqualified image. Note
702 support for this option has been dropped and it has no effect. Please
703 refer to containers-registries.conf(5) for configuring unqualified-
704 search registries.
705 big_files_temporary_dir=""
708 Path to the temporary directory to use for storing big files, used to
709 store image blobs and data streams related to containers image manage‐
710 ment.
711 separate_pull_cgroup=""
714 [EXPERIMENTAL] If its value is set, then images are pulled into the
715 specified cgroup. If its value is set to "pod", then the pod's cgroup
716 is used. It is currently supported only with the systemd cgroup man‐
717 ager.
718
721 The crio.network table containers settings pertaining to the management
722 of CNI plugins.
723 cni_default_network=""
726 The default CNI network name to be selected. If not set or "", then
727 CRI-O will pick-up the first one found in network_dir.
728 network_dir="/etc/cni/net.d/"
731 Path to the directory where CNI configuration files are located.
732 plugin_dirs=["/opt/cni/bin/",]
735 List of paths to directories where CNI plugin binaries are located.
736
739 The crio.metrics table containers settings pertaining to the Prometheus
740 based metrics retrieval.
741 enable_metrics=false
744 Globally enable or disable metrics support.
745 metrics_collectors=["operations", "operations_latency_microseconds_to‐
748 tal", "operations_latency_microseconds", "operations_errors", "im‐
749 age_pulls_by_digest", "image_pulls_by_name", "im‐
750 age_pulls_by_name_skipped", "image_pulls_failures", "image_pulls_suc‐
751 cesses", "image_pulls_layer_size", "image_layer_reuse", "contain‐
752 ers_oom_total", "containers_oom", "processes_defunct"]
753 Enabled metrics collectors
754 metrics_port=9090
757 The port on which the metrics server will listen.
758 metrics_socket=""
761 The socket on which the metrics server will listen.
762 metrics_cert=""
765 The certificate for the secure metrics server.
766 metrics_key=""
769 The certificate key for the secure metrics server.
770
773 [EXPERIMENTAL] The crio.tracing table containers settings pertaining to
774 the export of OpenTelemetry trace data.
775 enable_tracing=false
778 Globally enable or disable OpenTelemetry trace data exporting.
779 tracing_endpoint="0.0.0.0:4317"
782 Address on which the gRPC trace collector will listen.
783 tracing_sampling_rate_per_million=""
786 Number of samples to collect per million OpenTelemetry spans. Set to
787 1000000 to always sample.
788
791 The crio.stats table specifies all necessary configuration for report‐
792 ing container and pod stats.
793 stats_collection_period=0
796 The number of seconds between collecting pod and container stats. If
797 set to 0, the stats are collected on-demand instead.
798
801 The crio.nri table contains settings for controlling NRI (Node Resource
802 Interface) support in CRI-O. enable_nri=false
803 Enable CRI-O NRI support.
804 nri_config_file="/etc/nri/nri.conf"
807 NRI configuration file to use.
808 nri_plugin_dir="/opt/nri/plugins"
811 Directory to scan for pre-installed plugins to automatically start.
812 nri_listen="/var/run/nri.sock"
815 Socket to listen on for externally started NRI plugins to connect to.
816
820 crio.conf.d(5), containers-storage.conf(5), containers-policy.json(5),
821 containers-registries.conf(5), crio(8)
822
826 Aug 2018, Update to the latest state by Valentin Rothberg vroth‐
827 berg@suse.com ⟨mailto:vrothberg@suse.com⟩
828 Oct 2016, Originally compiled by Aleksa Sarai asarai@suse.de
831 ⟨mailto:asarai@suse.de⟩
832Runtime Container crio.conf(5)(Kubernetes)