Certificate Management with kubeadm
Kubernetes v1.15 [stable]
Client certificates generated by kubeadm expire after 1 year. This page explains how to manage certificate renewals with kubeadm.
Before you begin
You should be familiar with PKI certificates and requirements in Kubernetes.
Using custom certificates
By default, kubeadm generates all the certificates needed for a cluster to run. You can override this behavior by providing your own certificates.
To do so, you must place them in whatever directory is specified by the
--cert-dir flag or the
certificatesDir field of kubeadm's
By default this is
If a given certificate and private key pair exists before running
kubeadm does not overwrite them. This means you can, for example, copy an existing
and kubeadm will use this CA for signing the rest of the certificates.
External CA mode
It is also possible to provide only the
ca.crt file and not the
ca.key file (this is only available for the root CA file, not other cert pairs).
If all other certificates and kubeconfig files are in place, kubeadm recognizes
this condition and activates the "External CA" mode. kubeadm will proceed without the
CA key on disk.
Instead, run the controller-manager standalone with
point to the CA certificate and key.
PKI certificates and requirements includes guidance on setting up a cluster to use an external CA.
Check certificate expiration
You can use the
check-expiration subcommand to check when certificates expire:
kubeadm certs check-expiration
The output is similar to this:
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED admin.conf Dec 30, 2020 23:36 UTC 364d no apiserver Dec 30, 2020 23:36 UTC 364d ca no apiserver-etcd-client Dec 30, 2020 23:36 UTC 364d etcd-ca no apiserver-kubelet-client Dec 30, 2020 23:36 UTC 364d ca no controller-manager.conf Dec 30, 2020 23:36 UTC 364d no etcd-healthcheck-client Dec 30, 2020 23:36 UTC 364d etcd-ca no etcd-peer Dec 30, 2020 23:36 UTC 364d etcd-ca no etcd-server Dec 30, 2020 23:36 UTC 364d etcd-ca no front-proxy-client Dec 30, 2020 23:36 UTC 364d front-proxy-ca no scheduler.conf Dec 30, 2020 23:36 UTC 364d no CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED ca Dec 28, 2029 23:36 UTC 9y no etcd-ca Dec 28, 2029 23:36 UTC 9y no front-proxy-ca Dec 28, 2029 23:36 UTC 9y no
The command shows expiration/residual time for the client certificates in the
/etc/kubernetes/pki folder and for the client certificate embedded in the KUBECONFIG files used by kubeadm (
Additionally, kubeadm informs the user if the certificate is externally managed; in this case, the user should take care of managing certificate renewal manually/using other tools.
kubeadmcannot manage certificates signed by an external CA.
On nodes created with
kubeadm init, prior to kubeadm version 1.17, there is a bug where you manually have to modify the contents of
kubeadm initfinishes, you should update
kubelet.confto point to the rotated kubelet client certificates, by replacing
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
Automatic certificate renewal
kubeadm renews all the certificates during control plane upgrade.
This feature is designed for addressing the simplest use cases; if you don't have specific requirements on certificate renewal and perform Kubernetes version upgrades regularly (less than 1 year in between each upgrade), kubeadm will take care of keeping your cluster up to date and reasonably secure.
Note: It is a best practice to upgrade your cluster frequently in order to stay secure.
If you have more complex requirements for certificate renewal, you can opt out from the default behavior by passing
kubeadm upgrade apply or to
kubeadm upgrade node.
Warning: Prior to kubeadm version 1.17 there is a bug where the default value for
kubeadm upgrade nodecommand. In that case, you should explicitly set
Manual certificate renewal
You can renew your certificates manually at any time with the
kubeadm certs renew command.
This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in
After running the command you should restart the control plane Pods. This is required since
dynamic certificate reload is currently not supported for all components and certificates.
Static Pods are managed by the local kubelet
and not by the API Server, thus kubectl cannot be used to delete and restart them.
To restart a static Pod you can temporarily remove its manifest file from
and wait for 20 seconds (see the
fileCheckFrequency value in KubeletConfiguration struct.
The kubelet will terminate the Pod if it's no longer in the manifest directory.
You can then move the file back and after another
fileCheckFrequency period, the kubelet will recreate
the Pod and the certificate renewal for the component can complete.
Warning: If you are running an HA cluster, this command needs to be executed on all the control-plane nodes.
certs renewuses the existing certificates as the authoritative source for attributes (Common Name, Organization, SAN, etc.) instead of the kubeadm-config ConfigMap. It is strongly recommended to keep them both in sync.
kubeadm certs renew provides the following options:
The Kubernetes certificates normally reach their expiration date after one year.
--csr-onlycan be used to renew certificates with an external CA by generating certificate signing requests (without actually renewing certificates in place); see next paragraph for more information.
It's also possible to renew a single certificate instead of all.
Renew certificates with the Kubernetes certificates API
This section provides more details about how to execute manual certificate renewal using the Kubernetes certificates API.
Caution: These are advanced topics for users who need to integrate their organization's certificate infrastructure into a kubeadm-built cluster. If the default kubeadm configuration satisfies your needs, you should let kubeadm manage certificates instead.
Set up a signer
The Kubernetes Certificate Authority does not work out of the box. You can configure an external signer such as cert-manager, or you can use the built-in signer.
The built-in signer is part of
To activate the built-in signer, you must pass the
If you're creating a new cluster, you can use a kubeadm configuration file:
apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration controllerManager: extraArgs: cluster-signing-cert-file: /etc/kubernetes/pki/ca.crt cluster-signing-key-file: /etc/kubernetes/pki/ca.key
Create certificate signing requests (CSR)
See Create CertificateSigningRequest for creating CSRs with the Kubernetes API.
Renew certificates with external CA
This section provide more details about how to execute manual certificate renewal using an external CA.
To better integrate with external CAs, kubeadm can also produce certificate signing requests (CSRs). A CSR represents a request to a CA for a signed certificate for a client. In kubeadm terms, any certificate that would normally be signed by an on-disk CA can be produced as a CSR instead. A CA, however, cannot be produced as a CSR.
Create certificate signing requests (CSR)
You can create certificate signing requests with
kubeadm certs renew --csr-only.
Both the CSR and the accompanying private key are given in the output.
You can pass in a directory with
--csr-dir to output the CSRs to the specified location.
--csr-dir is not specified, the default certificate directory (
/etc/kubernetes/pki) is used.
Certificates can be renewed with
kubeadm certs renew --csr-only.
kubeadm init, an output directory can be specified with the
A CSR contains a certificate's name, domains, and IPs, but it does not specify usages. It is the responsibility of the CA to specify the correct cert usages when issuing a certificate.
opensslthis is done with the
cfsslyou specify usages in the config file.
After a certificate is signed using your preferred method, the certificate and the private key must be copied to the PKI directory (by default
Certificate authority (CA) rotation
Kubeadm does not support rotation or replacement of CA certificates out of the box.
For more information about manual rotation or replacement of CA, see manual rotation of CA certificates.
Enabling signed kubelet serving certificates
By default the kubelet serving certificate deployed by kubeadm is self-signed. This means a connection from external services like the metrics-server to a kubelet cannot be secured with TLS.
To configure the kubelets in a new kubeadm cluster to obtain properly signed serving
certificates you must pass the following minimal configuration to
apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration serverTLSBootstrap: true
If you have already created the cluster you must adapt it by doing the following:
- Find and edit the
kubelet-config-1.23ConfigMap in the
kube-systemnamespace. In that ConfigMap, the
kubeletkey has a KubeletConfiguration document as its value. Edit the KubeletConfiguration document to set
- On each node, add the
serverTLSBootstrap: truefield in
/var/lib/kubelet/config.yamland restart the kubelet with
systemctl restart kubelet
serverTLSBootstrap: true will enable the bootstrap of kubelet serving
certificates by requesting them from the
certificates.k8s.io API. One known limitation
is that the CSRs (Certificate Signing Requests) for these certificates cannot be automatically
approved by the default signer in the kube-controller-manager -
This will require action from the user or a third party controller.
These CSRs can be viewed using:
kubectl get csr NAME AGE SIGNERNAME REQUESTOR CONDITION csr-9wvgt 112s kubernetes.io/kubelet-serving system:node:worker-1 Pending csr-lz97v 1m58s kubernetes.io/kubelet-serving system:node:control-plane-1 Pending
To approve them you can do the following:
kubectl certificate approve <CSR-name>
By default, these serving certificate will expire after one year. Kubeadm sets the
true, which means that close
to expiration a new set of CSRs for the serving certificates will be created and must
be approved to complete the rotation. To understand more see
If you are looking for a solution for automatic approval of these CSRs it is recommended that you contact your cloud provider and ask if they have a CSR signer that verifies the node identity with an out of band mechanism.
Caution: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects. This page follows CNCF website guidelines by listing projects alphabetically. To add a project to this list, read the content guide before submitting a change.
Third party custom controllers can be used:
Such a controller is not a secure mechanism unless it not only verifies the CommonName in the CSR but also verifies the requested IPs and domain names. This would prevent a malicious actor that has access to a kubelet client certificate to create CSRs requesting serving certificates for any IP or domain name.