Make metrics-server work out of the box with kubeadm

< Back

Make metrics-server work out of the box with kubeadm

By Romain Guichard at September 8, 2021

If you try to install metrics-server into a fresh, up to date, kubeadm cluster, it will fail with this error in the logs:

E0908 15:28:39.751310       1 scraper.go:139] "Failed to scrape node" err="Get \"https://10.68.14.125:10250/stats/summary?only_cpu_and_memory=true\": x509: cannot validate certificate for 10.68.14.125 because it doesn't contain any IP SANs" node="scw-sharp-cray"

metrics-server tries to reach the Kubelet API but the TLS connection fails because node’s IP is not part of the certificate SAN.

Why ? Because Kubelet certificates are self-signed, they are not signed by Kubernetes CA. And indeed, InternalIP is not part of the certificate SAN.

metrics-server can be configured to not use the InternalIP but rather the node hostname, or its ExternalIP. The argument --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname allows you to decide which one it should use. But this won’t solve our problem because if you use Hostname as first choice, you end up with a DNS resolution problem because your pod doesn’t know who “scw-sharp-cray” (the name of my node) is. Actually, even if the DNS resolution works, you will still face our first issue, because even the hostname of our node is not a certificate SAN…

Actually, InternalIP is a good choice. And the way to fix this is to signed Kubelet certificate with the APIServer.

From scratch

From scratch, you have to use serverTLSBootstrap: true in your kubeadm configfile.

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true

That’s it.

Running cluster

You basically have two things to do:

Edit the kubelet-config ConfigMap in the kube-system namespace. Edit the KubeletConfiguration document to set serverTLSBootstrap: true.
On each node, add the serverTLSBootstrap: true field in /var/lib/kubelet/config.yaml and restart the kubelet

Signed certificate

In each case, Kubelet will generate a CSR and submit it to the APIServer. You need to approve the CSR for each Kubelet on your cluster.

$ kubectl get csr
NAME        AGE     SIGNERNAME                                    REQUESTOR                         REQUESTEDDURATION   CONDITION
csr-7xhl8   4m15s   kubernetes.io/kube-apiserver-client-kubelet   system:node:scw-practical-kilby   <none>              Approved,Issued
csr-h24sp   4m4s    kubernetes.io/kubelet-serving                 system:node:scw-practical-kilby   <none>              Pending

$ kubectl certificate approve csr-h24sp

$ kubectl top node
NAME                  CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
scw-sharp-cray        271m         9%     1344Mi          35%

By default, these serving certificate will expire after one year. So, in one year, a new CSR will be generated by Kubelet and you will need to approve it, this is not automagic.

Rubber-stamp is an auto-approver operator for the Kubelet.

This article is part of a new take on articles where we focus on various power user issues we encounter and try to sum up our investigation (basicaly we read Github Issues and stackoverflow for you).

Don’t hesitate to reach us on Github or through our website if you need any help navigating the Kubernetes ecosystem for your projects.

Particule