Using sysctls in a Kubernetes Cluster

This document describes how to configure and use kernel parameters within a Kubernetes cluster using the sysctl interface.

Before you begin

You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a cluster, you can create one by using minikube or you can use one of these Kubernetes playgrounds:

• iximiuz Labs
• Killercoda
• KodeKloud

For some steps, you also need to be able to reconfigure the command line options for the kubelets running on your cluster.

Listing all Sysctl Parameters

In Linux, the sysctl interface allows an administrator to modify kernel parameters at runtime. Parameters are available via the /proc/sys/ virtual process file system. The parameters cover various subsystems such as:

• kernel (common prefix: kernel.)
• networking (common prefix: net.)
• virtual memory (common prefix: vm.)
• MDADM (common prefix: dev.)
• More subsystems are described in Kernel docs.

To get a list of all parameters, you can run

sudo sysctl -a

Safe and Unsafe Sysctls

Kubernetes classes sysctls as either safe or unsafe. In addition to proper namespacing, a safe sysctl must be properly isolated between pods on the same node. This means that setting a safe sysctl for one pod

• must not have any influence on any other pod on the node
• must not allow to harm the node's health
• must not allow to gain CPU or memory resources outside of the resource limits of a pod.

By far, most of the namespaced sysctls are not necessarily considered safe. The following sysctls are supported in the safe set:

• kernel.shm_rmid_forced;
• net.ipv4.ip_local_port_range;
• net.ipv4.tcp_syncookies;
• net.ipv4.ping_group_range (since Kubernetes 1.18);
• net.ipv4.ip_unprivileged_port_start (since Kubernetes 1.22);
• net.ipv4.ip_local_reserved_ports (since Kubernetes 1.27, needs kernel 3.16+);
• net.ipv4.tcp_keepalive_time (since Kubernetes 1.29, needs kernel 4.5+);
• net.ipv4.tcp_fin_timeout (since Kubernetes 1.29, needs kernel 4.6+);
• net.ipv4.tcp_keepalive_intvl (since Kubernetes 1.29, needs kernel 4.5+);
• net.ipv4.tcp_keepalive_probes (since Kubernetes 1.29, needs kernel 4.5+).
• net.ipv4.tcp_rmem (since Kubernetes 1.32, needs kernel 4.15+).
• net.ipv4.tcp_wmem (since Kubernetes 1.32, needs kernel 4.15+).
• net.ipv4.tcp_slow_start_after_idle (since Kubernetes 1.37, needs kernel 4.15+).
• net.ipv4.tcp_notsent_lowat (since Kubernetes 1.37, needs kernel 4.6+).

This list will be extended in future Kubernetes versions when the kubelet supports better isolation mechanisms.

Enabling Unsafe Sysctls

All safe sysctls are enabled by default.

All unsafe sysctls are disabled by default and must be allowed manually by the cluster admin on a per-node basis. Pods with disabled unsafe sysctls will be scheduled, but will fail to launch.

With the warning above in mind, the cluster admin can allow certain unsafe sysctls for very special situations such as high-performance or real-time application tuning. Unsafe sysctls are enabled on a node-by-node basis with a flag of the kubelet; for example:

kubelet --allowed-unsafe-sysctls \
  'kernel.msg*,net.core.somaxconn' ...

For Minikube, this can be done via the extra-config flag:

minikube start --extra-config="kubelet.allowed-unsafe-sysctls=kernel.msg*,net.core.somaxconn"...

Only namespaced sysctls can be enabled this way.

Setting Sysctls for a Pod

A number of sysctls are namespaced in today's Linux kernels. This means that they can be set independently for each pod on a node. Only namespaced sysctls are configurable via the pod securityContext within Kubernetes.

The following sysctls are known to be namespaced. This list could change in future versions of the Linux kernel.

• kernel.shm*,
• kernel.msg*,
• kernel.sem,
• fs.mqueue.*,
• Those net.* that can be set in container networking namespace. However, there are exceptions (e.g., net.netfilter.nf_conntrack_max and net.netfilter.nf_conntrack_expect_max can be set in container networking namespace but are unnamespaced before Linux 5.12.2).

Sysctls with no namespace are called node-level sysctls. If you need to set them, you must manually configure them on each node's operating system, or by using a DaemonSet with privileged containers.

Use the pod securityContext to configure namespaced sysctls. The securityContext applies to all containers in the same pod.

This example uses the pod securityContext to set a safe sysctl kernel.shm_rmid_forced and two unsafe sysctls net.core.somaxconn and kernel.msgmax. There is no distinction between safe and unsafe sysctls in the specification.

apiVersion: v1
kind: Pod
metadata:
  name: sysctl-example
spec:
  securityContext:
    sysctls:
    - name: kernel.shm_rmid_forced
      value: "0"
    - name: net.core.somaxconn
      value: "1024"
    - name: kernel.msgmax
      value: "65536"
  ...

It is good practice to consider nodes with special sysctl settings as tainted within a cluster, and only schedule pods onto them which need those sysctl settings. It is suggested to use the Kubernetes taints and toleration feature to implement this.

A pod with the unsafe sysctls will fail to launch on any node which has not enabled those two unsafe sysctls explicitly. As with node-level sysctls it is recommended to use taints and toleration feature or taints on nodes to schedule those pods onto the right nodes.

Setting Sysctls for All Pods

You can configure a default set of kernel parameters (sysctls) that the kubelet applies to all Pods running on a Linux Node, including static Pods. This is useful when Node administrators need to enforce consistent kernel parameter tuning across all workloads on a Node or within a Node group (for example, adjusting TCP buffer sizes for high-performance networking) without requiring every Pod specification to individually set securityContext.sysctls.

To use this feature, enable the DefaultPodSysctls feature gate for the kubelet and specify key-value pairs in the defaultPodSysctls field of your KubeletConfiguration.

The defaultPodSysctls field supports all namespaced sysctls (kernel.shm*, kernel.msg*, kernel.sem, kernel.domainname, fs.mqueue.*, net.*, and user.*), covering both safe and unsafe sysctls. Because these defaults are configured directly by the Node administrator on the kubelet, you do not need to allow-list unsafe sysctls in allowedUnsafeSysctls.

The following example configures the kubelet to apply default sysctls across multiple namespaced subsystems (networking, IPC, and user namespaces) to all Pods on the Node:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  DefaultPodSysctls: true
defaultPodSysctls:
  # Network namespace sysctls (skipped if Pod uses hostNetwork: true)
  net.ipv4.ip_forward: "1"
  net.ipv4.tcp_rmem: "4096 87380 16777216"
  net.ipv4.tcp_wmem: "4096 65536 16777216"
  net.core.somaxconn: "1024"
  # IPC namespace sysctls (skipped if Pod uses hostIPC: true)
  kernel.shmall: "1048576"
  kernel.msgmax: "65536"
  kernel.sem: "250 32000 32 128"
  fs.mqueue.msg_max: "1024"
  # User namespace sysctls (skipped if Pod shares the host user namespace)
  user.max_user_namespaces: "1000"

Precedence and Overriding

Values explicitly set in a Pod's spec.securityContext.sysctls always override the matching default values specified in the kubelet's defaultPodSysctls. Overrides are applied individually on a per-key basis: if a Pod specifies a value for a sysctl that is also defined in defaultPodSysctls, the Pod-level setting takes precedence for that specific sysctl, while other defaults continue to apply. Note that there are no groups of connected sysctl settings; if your workload overrides a sysctl that is part of a related group (for example, networking buffer sizes), the Pod specification must account for all related settings as needed.

Host Namespaces and Filtering

The kubelet applies default sysctls during Pod sandbox creation only if the Pod runs in a separate namespace for the corresponding subsystem. If a Pod shares a host namespace, default sysctls for that namespace are skipped for that Pod:

• net.* sysctls are skipped if the Pod uses host networking (hostNetwork: true).
• IPC sysctls (kernel.sem, kernel.msg*, kernel.shm*, fs.mqueue.*) are skipped if the Pod uses host IPC (hostIPC: true).
• user.* sysctls are skipped if the Pod shares the host user namespace (hostUsers: true or unset).
• UTS sysctls (kernel.domainname) are skipped if the Pod uses host networking (hostNetwork: true).

Validation and Limitations

The kubelet validates defaultPodSysctls during startup. Non-namespaced sysctls, invalid sysctl names, or duplicate keys will prevent the kubelet from starting.

In addition, certain net.* sysctls might be unnamespaced depending on your kernel version. Specifying an unnamespaced net.* sysctl in defaultPodSysctls will cause Pod sandbox creation to fail with a FailedCreatePodSandBox error. The Pod will keep retrying to create a sandbox forever. Ensure that all specified sysctls are namespaced on your Node's kernel.

Changes to defaultPodSysctls apply only to newly created Pods. The kubelet does not dynamically reconfigure existing Pods (see What Happens After a Node Restart). Existing Pods continue to run with the sysctls applied when their sandbox was created. If you want existing Pods to adopt the updated default sysctls, you must recreate those Pods (for example, by cordoning and draining the Node).