DEV Community: Ham

Fixing Expired Certificates In Kubernetes for Worker Nodes

Ham — Sat, 31 Oct 2026 09:53:55 +0000

By default, when you setup your Kubernetes cluster, the certificates expires after one year.

The process to check is exactly like the process explained in https://dev.to/hatr0/fixing-expired-certificates-in-kubernetes-1oe2.
I have copied and paste all the needed commands here so no need to visit the post.

From the control plane node, if you issue the following command:

kubectl get nodes

You'll notice some of your worker nodes have a STATUS of NotReady

Let's do some further checking. SSH to one of the worker nodes that has a status of NotReady.

The below commands requires 'root' privileged, so if you are login as non-root, you will need to prepend sudo before the commands.

Check your kubelet service, is it failing to start (systemctl status kubelet)?

Let's check logs related with kubelet (journalctl | grep kubelet), you'll noticed the belong error messages.

You should see an entry about bootstrap client certificate is expired.

Since the kubelet service on worker nodes talks to control plane, an expired license will prevent the kubelet from authenticating with the API server.

Let's verify our issue with the 'openssl' command.

Let's check kubelet client certificate with the following command:

openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text | grep 'Not After'

Not After : Feb 16 00:38:07 2023 GMT

If the date shown is in the past, the certificate has expired.

Let's renew certificates.

If your cluster has more than one worker node, be sure to check certificate expiration with the command above on all nodes. If certificate has expired, run the following commands on all expired worker nodes in the cluster.

First, let's backup our certificates just in case we need them.

cp /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.backup
cp -r /var/lib/kubelet/pki /var/lib/kubelet/pki.backup

Next, let's remove the expired certificates

rm /etc/kubernetes/kubelet.conf
rm /var/lib/kubelet/pki/kubelet-client* - Note: Wildcard might not work due to security restrictions, so delete each file individually.

After you delete the expired certificates, let's generate a new kubelet configuration.

On a working control plane node, let's first create a working config file since that is needed to create a new kubelet configuration file.

Create a file name temp-config.yaml
Add the following to the file and save the file. You can verify and match the entries with the current running cluster with the following command: kubectl get cm kubeadm-config -n kube-system -o=jsonpath="{.data.ClusterConfiguration}" This information can be found at https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/

# Temporary config.yaml file to generate new kubelet configuration file
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
# Cluster name in the kubeconfig 
clusterName: "kubernetes"
# The server IP/DNS name of the cluster in the kubeconfig
controlPlaneEndpoint: "IP or DNS name:6443"
# Cluster CA key and certificate will be loaded from this local directory
certficatesDir: "/etc/kubernetes/pki"

Now with the config.yaml file created, let's create the configuration file, substitute with the actual hostname of the worker node you are fixing, execute the following:

export NODE=<worker-node-name>
kubeadm kubeconfig user --org system:nodes --client-name system:node:${NODE} --config temp-config.yaml > kubelet.conf

Once the kubelet.conf file is generated, let's transfer this to the worker node. Substitute with the worker node IP or DNS name you are fixing.

scp kubelet.conf <worker-node>:/tmp/kubelet.conf
ssh <worker-node>
mv /tmp/kubelet.conf /etc/kubernetes/kubelet.conf
chown root:root /etc/kubernetes/kubelet.conf

Now still logged in on worker node, restart the kubectl service
systemctl restart kubelet

Verify by checking kubelet service and issue kubectl commands.

Getting Cilium to work on Ubuntu Cloud Image

Ham — Fri, 03 Jan 2025 08:07:56 +0000

How to get Cilium working on Ubuntu Cloud Images Focal (20.04) or Jammy (22.04).

If you are running one of the Ubuntu Cloud Images and you are trying to install Cilium as your CNI network plugin on your Kubernetes cluster. You might have noticed that you get CrashLoopBackOff from your cilium pods when issuing a kubectl get pods -n kube-system

Upon further troubleshooting on the problem pod with a kubectl logs cilium-jgcdm -n kube-system you might see the below messages.

There is a very good chance you are missing some kernel configuration options. Have a look at https://docs.cilium.io/en/stable/operations/system_requirements/#linux-kernel for more information.

The base requirements are:

On a Ubuntu system running 20.04, you can check your kernel configurations with:
cat /lib/modules/$(uname -r)/build/.config - Note: if you are not running as root, prepend sudo before the command.

You can look for specific config options by prepending a pipe to grep:
cat /lib/modules/$(uname -r)/build/.config | grep -i config_bpf_jit.

The Fix

After checking, you will noticed that certain options are not enabled. To solve, let's replace our kernel variant with the 'generic' version. If not running as root, prepend sudo.

apt update && apt install linux-generic or sudo apt update && sudo apt install linux-generic -y if not running as root.

Now let's build initramfs for the kernel. You will want to use the generic kernel version that was installed, you should see the version from the install screen. You can also check under '/boot' directory with the command ls /boot:

Note: at the time of this writing, '5.4.0-204-generic' is the installed kernel version.
update-initramfs -u -k 5.4.0-204-generic or sudo update-initramfs -u -k 5.4.0-204-generic for non-root user.

Note: If you want to remove 'linux-kvm' variant, issue
sudo apt purge linux-kvm
If you plan to keep the 'linux-kvm' kernel around, don't issue the apt purge command and move any files ending in '-kvm' in the /boot directory to a different folder.
mkdir /boot/kvm-kernel
mv /boot/*-kvm /boot/kvm-kernel/

Now, let's update grub
update-grub or sudo update-grub

Reboot the system
reboot or sudo reboot

Repeat the above steps for each of the nodes you have in the cluster.

After installing 'generic' kernel and rebooting, check your cilium pods again and they all should be running.

kubectl get pods -n kube-system -owide - the '-owide' option will show which node the pods are running on.

Fixing Expired Certificates In Kubernetes

Ham — Fri, 03 Jan 2025 06:20:32 +0000

By default, when you setup your Kubernetes cluster, the certificates expires after one year.

If it's been a while since you started up your Kubernetes cluster and you try to issue kubectl commands and notice connection refused errors.

The below commands requires 'root' privileged, so if you are login as non-root, you will need to prepend sudo before the commands.

Troubleshooting further, you noticed that your kubelet service is failing to start (systemctl status kubelet).

Upon checking logs related with kubelet (journalctl | grep kubelet), you noticed the belong error messages.

There are several ways to verify our issue, let's issue kubeadm certs check-expiration.

Bingo!

An alternative to using kubeadm would be to use 'openssl' command.
Let's check a specific certificate, in this case the API server certificate:

"openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text | grep 'Not After'"

Not After : Nov 07 01:16:00 2024 GMT

As we see, the API server certificate has expired, you can also check the other certificates to verify:

/etc/kubernetes/pki/front-proxy-client.crt
/etc/kubernetes/pki/etcd/peer.crt
/etc/kubernetes/pki/etcd/server.crt
/etc/kubernetes/pki/etcd/healthcheck-client.crt
/etc/kubernetes/pki/apiserver-etcd-client.crt
/etc/kubernetes/pki/apiserver-kubelet-client.crt

Let's renew certificates.

If your cluster has more than one control-plane node, be sure to run the following commands on all control-plane nodes in the cluster.

First, let's backup our certificates just in case we need them.

cp -R /etc/kubernetes/pki /etc/kubernetes/pki.backup

Manually renew certificates to fix our issues.

Now let's renew our certificates with

kubeadm certs renew all

This will renew all certificates, if only a specific certificate has expired, you can renew the specific certificate with the following command:

kubeadm certs renew <specific server/service> - where would be for example, apiserver, so kubeadm certs renew apiserver

Verify new certificate installed correctly.

kubeadm certs check-expiration or use openssl command from above

Restart kubelet service

systemctl restart kubelet

Once the renew process is complete, we will need to restart all the control plane pods. One way to restart is moving it's manifest file out and wait for for about 20 seconds before moving the file back in to the 'manifests' folder. This will will recreate the Pod to use the new certificates.

cd /etc/kubernetes/manifests
mv *.yaml .. - Move all yaml files one level up
mv ../*.yaml . - After 20 seconds, move all yaml file back to current directory

Copy the administrator certificates

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

This should get your cluster running again. You can verify with kubectl get pods and not get an error.

Update kubeconfig files - only if you renew certificates individually

If you updated individual certificates and after restarting kubelet service, things are still not working, you might need to update kubeconfig files. Do the following:

kubeadm init phase kubeconfig admin - this will update admin user (admin.conf).
kubeadm init phase kubeconfig controller-manager - this will update controller manager.
kubeadm init phase kubeconfig scheduler - this will update the scheduler.

If this still doesn't work, before continuing to troubleshoot, reboot the system to make sure all services are restarted correctly.

sudo reboot or sudo shutdown -r now
systemctl status kubelet - After reboot, login and check status again. if fail, view section on "Additional Things To Do..." below.

Renew automatically via Kubernetes Upgrade
If you also need to update your Kubernetes version, you can apply the upgrade and all certificates will get renewed.

Let's run the following command to check current version and what available upgrades version.

kubeadm upgrade plan

Find the version to upgrade to and issue the following to upgrade.

kubeadm upgrade apply <version> - substitute the with the version number to update to.

Additional Things To Do If The Steps Above Doesn't Work

If for some reason the above doesn't work, you will need to do the following. Make a backup of /etc/kubernetes/pki/

cp -R /etc/kubernetes/pki /etc/kubernetes/pki-backup

Delete the following files from /etc/kubernetes/pki

rm apiserver.crt \ apiserver-etcd-client.key\ apiserver-kubelet-client.crt\ front-proxy-ca.crt\ front-proxy-client.crt\ front-proxy-client.key\ front-proxy-ca.key\ apiserver-kubelet-client.key\ apiserver.key\ apiserver-etcd-client.crt

Remove all .crt and .key files from /etc/kubernetes/pki/etcd

rm /etc/kubernetes/pki/etcd/*.crt
rm /etc/kubernetes/pki/etcd/*.key

Then create certs with

kubeadm init phase certs all --apiserver-advertise-address <IP> --substitute your cluster IP.

Backup\Move conf files from /etc/kubernetes

mkdir conf-backup
mv admin.conf\ controller-manager.conf\ kubelet.conf\ scheduler.conf /etc/kubernetes/conf-backup/

Create new conf files

kubeadm init phase kubeconfig all

Finally, restart kubelet service or reboot system.

systemctl restart kubelet or reboot