Kubernetes Cluster Setup using Kubeadm

March 29, 2022 22 minute read

Kubernetes Cluster Setup using Kubeadm

Install Kubectl

lab@k8s1:~$ curl -LO https://dl.k8s.io/release/v1.22.0/bin/linux/amd64/kubectl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   154  100   154    0     0    890      0 --:--:-- --:--:-- --:--:--   885
100 44.7M  100 44.7M    0     0  22.7M      0  0:00:01  0:00:01 --:--:-- 32.5M
lab@k8s1:~$

lab@k8s1:~$ sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

lab@k8s1:~$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
lab@k8s1:~$

lab@k8s1:~$ sudo apt-get update
[sudo] password for lab:
Hit:1 https://brave-browser-apt-beta.s3.brave.com stable InRelease
Hit:2 https://apt.releases.hashicorp.com focal InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu focal InRelease
Hit:4 https://dl.google.com/linux/chrome/deb stable InRelease
Hit:5 https://brave-browser-apt-release.s3.brave.com stable InRelease
Get:7 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:8 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:9 http://packages.microsoft.com/repos/code stable InRelease
Hit:10 https://packages.microsoft.com/ubuntu/20.04/prod focal InRelease
Hit:6 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Get:11 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:12 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 DEP-11 Metadata [278 kB]
Get:13 http://us.archive.ubuntu.com/ubuntu focal-updates/universe amd64 DEP-11 Metadata [391 kB]
Get:14 http://us.archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 DEP-11 Metadata [944 B]
Get:15 http://us.archive.ubuntu.com/ubuntu focal-backports/main amd64 DEP-11 Metadata [7980 B]
Get:16 http://us.archive.ubuntu.com/ubuntu focal-backports/universe amd64 DEP-11 Metadata [30.7 kB]
Get:17 http://security.ubuntu.com/ubuntu focal-security/main amd64 DEP-11 Metadata [40.8 kB]
Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 DEP-11 Metadata [66.3 kB]
Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 DEP-11 Metadata [2464 B]
Fetched 1154 kB in 2s (632 kB/s)
Reading package lists... Done
lab@k8s1:~$

Kubeadm and Kubelet

lab@k8s1:~$ sudo apt-get install -y apt-transport-https ca-certificates curl
Reading package lists... Done
Building dependency tree
Reading state information... Done
ca-certificates is already the newest version (20210119~20.04.2).
curl is already the newest version (7.68.0-1ubuntu2.7).
apt-transport-https is already the newest version (2.0.6).
The following package was automatically installed and is no longer required:
  libfwupdplugin1
Use 'sudo apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
lab@k8s1:~$

lab@k8s1:~$ sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
lab@k8s1:~$

lab@k8s1:~$ echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main
lab@k8s1:~$

lab@k8s1:~$ sudo apt-get install kubeadm=1.22.0-00 kubelet=1.22.0-00 kubectl=1.22.0-00
Reading package lists... Done
Building dependency tree
Reading state information... Done
kubeadm is already the newest version (1.22.0-00).
kubelet is already the newest version (1.22.0-00).
kubelet set to manually installed.
The following package was automatically installed and is no longer required:
  libfwupdplugin1
Use 'sudo apt autoremove' to remove it.
The following packages will be DOWNGRADED:
  kubectl
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 1 not upgraded.
Need to get 9037 kB of archives.
After this operation, 311 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://packages.cloud.google.com/apt kubernetes-xenial/main amd64 kubectl amd64 1.22.0-00 [9037 kB]
Fetched 9037 kB in 1s (11.7 MB/s)
dpkg: warning: downgrading kubectl from 1.23.5-00 to 1.22.0-00
(Reading database ... 214950 files and directories currently installed.)
Preparing to unpack .../kubectl_1.22.0-00_amd64.deb ...
Unpacking kubectl (1.22.0-00) over (1.23.5-00) ...
Setting up kubectl (1.22.0-00) ...
lab@k8s1:~$

Hold the current versions

lab@k8s1:~$ sudo apt-mark hold kubelet kubeadm kubectl
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.
lab@k8s1:~$

Verify the current version

lab@k8s1:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"2021-08-04T18:02:08Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
lab@k8s1:~$

lab@k8s1:~$ kubelet --version
Kubernetes v1.22.0
lab@k8s1:~$

Initialise Kubeadm

lab@k8s1:~$ kubeadm init --apiserver-advertise-address=192.168.100.1
I0329 05:30:24.094513    2992 version.go:255] remote version is much newer: v1.23.5; falling back to: stable-1.22
[init] Using Kubernetes version: v1.22.8
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR IsPrivilegedUser]: user is not running as root
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

lab@k8s1:~$ sudo kubeadm init --apiserver-advertise-address=192.168.100.1
I0329 05:30:34.212748    3003 version.go:255] remote version is much newer: v1.23.5; falling back to: stable-1.22
[init] Using Kubernetes version: v1.22.8
[preflight] Running pre-flight checks
[preflight] WARNING: Couldn't create the interface used for talking to the container runtime: docker is required for container runtime: exec: "docker": executable file not found in $PATH
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
	[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
	[ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
lab@k8s1:~$

lab@k8s1:~$ sudo swapon --show
NAME      TYPE       SIZE USED PRIO
/dev/sda5 partition 14.9G   0B   -2

lab@k8s1:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:          1.9Gi       603Mi       747Mi       2.0Mi       626Mi       1.2Gi
Swap:          14Gi          0B        14Gi

lab@k8s1:~$ sudo swapoff -a

lab@k8s1:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:          1.9Gi       592Mi       757Mi       2.0Mi       627Mi       1.2Gi
Swap:            0B          0B          0B

lab@k8s1:~$ sudo vi /etc/fstab
lab@k8s1:~$ sudo swapon --show
lab@k8s1:~$

lab@k8s1:~$ sudo kubeadm init --apiserver-advertise-address=192.168.100.1
[sudo] password for lab:
I0329 06:02:53.248251    2036 version.go:255] remote version is much newer: v1.23.5; falling back to: stable-1.22
[init] Using Kubernetes version: v1.22.8
[preflight] Running pre-flight checks
[preflight] WARNING: Couldn't create the interface used for talking to the container runtime: docker is required for container runtime: exec: "docker": executable file not found in $PATH
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
lab@k8s1:~$

lab@k8s1:~$ lsmod | grep br_netfilter

lab@k8s1:~$ cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
> br_netfilter
> EOF
br_netfilter
lab@k8s1:~$

lab@k8s1:~$ cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
> net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-call-iptables = 1
> EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
lab@k8s1:~$ 

lab@k8s1:~$ sudo sysctl --system
* Applying /etc/sysctl.d/10-console-messages.conf ...
kernel.printk = 4 4 1 7
* Applying /etc/sysctl.d/10-ipv6-privacy.conf ...
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
* Applying /etc/sysctl.d/10-kernel-hardening.conf ...
kernel.kptr_restrict = 1
* Applying /etc/sysctl.d/10-link-restrictions.conf ...
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /etc/sysctl.d/10-magic-sysrq.conf ...
kernel.sysrq = 176
* Applying /etc/sysctl.d/10-network-security.conf ...
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
* Applying /etc/sysctl.d/10-ptrace.conf ...
kernel.yama.ptrace_scope = 1
* Applying /etc/sysctl.d/10-zeropage.conf ...
vm.mmap_min_addr = 65536
* Applying /etc/sysctl.d/30-brave.conf ...
* Applying /usr/lib/sysctl.d/30-tracker.conf ...
fs.inotify.max_user_watches = 65536
* Applying /usr/lib/sysctl.d/50-default.conf ...
net.ipv4.conf.default.promote_secondaries = 1
sysctl: setting key "net.ipv4.conf.all.promote_secondaries": Invalid argument
net.ipv4.ping_group_range = 0 2147483647
net.core.default_qdisc = fq_codel
fs.protected_regular = 1
fs.protected_fifos = 1
* Applying /usr/lib/sysctl.d/50-pid-max.conf ...
kernel.pid_max = 4194304
* Applying /etc/sysctl.d/99-sysctl.conf ...
net.ipv4.ip_forward = 1
* Applying /etc/sysctl.d/k8s.conf ...
* Applying /usr/lib/sysctl.d/protect-links.conf ...
fs.protected_fifos = 1
fs.protected_hardlinks = 1
fs.protected_regular = 2
fs.protected_symlinks = 1
* Applying /etc/sysctl.conf ...
net.ipv4.ip_forward = 1
lab@k8s1:~$

Install Docker

lab@k8s1:~$ sudo apt-get install \
>     ca-certificates \
>     curl \
>     gnupg \
>     lsb-release
Reading package lists... Done
Building dependency tree
Reading state information... Done
lsb-release is already the newest version (11.1.0ubuntu2).
lsb-release set to manually installed.
ca-certificates is already the newest version (20210119~20.04.2).
curl is already the newest version (7.68.0-1ubuntu2.7).
gnupg is already the newest version (2.2.19-3ubuntu2.1).
gnupg set to manually installed.
The following package was automatically installed and is no longer required:
  libfwupdplugin1
Use 'sudo apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
lab@k8s1:~$

lab@k8s1:~$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
lab@k8s1:~$

lab@k8s1:~$ echo \
>   "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
>   $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
lab@k8s1:~$

lab@k8s1:~$ sudo apt-get update
Hit:1 http://packages.microsoft.com/repos/code stable InRelease
Get:3 https://download.docker.com/linux/ubuntu focal InRelease [57.7 kB]
Hit:4 http://us.archive.ubuntu.com/ubuntu focal InRelease
Hit:5 https://apt.releases.hashicorp.com focal InRelease
Hit:6 https://packages.microsoft.com/ubuntu/20.04/prod focal InRelease
Hit:7 https://brave-browser-apt-release.s3.brave.com stable InRelease
Get:8 http://us.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Hit:9 https://dl.google.com/linux/chrome/deb stable InRelease
Hit:10 https://brave-browser-apt-beta.s3.brave.com stable InRelease
Get:11 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:2 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
Get:12 http://us.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:13 https://download.docker.com/linux/ubuntu focal/stable amd64 Packages [15.5 kB]
Get:14 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1349 kB]
Get:15 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [408 kB]
Get:16 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [547 kB]
Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [695 kB]
Fetched 3408 kB in 3s (1259 kB/s)
Reading package lists... Done
lab@k8s1:~$

lab@k8s1:~$ sudo apt-get install docker-ce docker-ce-cli containerd.io
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  libfwupdplugin1
Use 'sudo apt autoremove' to remove it.
The following additional packages will be installed:
  docker-ce-rootless-extras docker-scan-plugin git git-man liberror-perl pigz slirp4netns
Suggested packages:
  aufs-tools cgroupfs-mount | cgroup-lite git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk gitweb git-cvs
  git-mediawiki git-svn
The following NEW packages will be installed:
  containerd.io docker-ce docker-ce-cli docker-ce-rootless-extras docker-scan-plugin git git-man liberror-perl pigz slirp4netns
0 upgraded, 10 newly installed, 0 to remove and 4 not upgraded.
Need to get 102 MB of archives.
After this operation, 443 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 pigz amd64 2.4-1 [57.4 kB]
Get:2 https://download.docker.com/linux/ubuntu focal/stable amd64 containerd.io amd64 1.5.11-1 [22.9 MB]
Get:3 http://us.archive.ubuntu.com/ubuntu focal/main amd64 liberror-perl all 0.17029-1 [26.5 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 git-man all 1:2.25.1-1ubuntu3.2 [884 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 git amd64 1:2.25.1-1ubuntu3.2 [4554 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu focal/universe amd64 slirp4netns amd64 0.4.3-1 [74.3 kB]
Get:7 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-ce-cli amd64 5:20.10.14~3-0~ubuntu-focal [41.0 MB]
Get:8 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-ce amd64 5:20.10.14~3-0~ubuntu-focal [20.9 MB]
Get:9 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-ce-rootless-extras amd64 5:20.10.14~3-0~ubuntu-focal [7932 kB]
Get:10 https://download.docker.com/linux/ubuntu focal/stable amd64 docker-scan-plugin amd64 0.17.0~ubuntu-focal [3521 kB]
Fetched 102 MB in 4s (27.8 MB/s)
Selecting previously unselected package pigz.
(Reading database ... 214950 files and directories currently installed.)
Preparing to unpack .../0-pigz_2.4-1_amd64.deb ...
Unpacking pigz (2.4-1) ...
Selecting previously unselected package containerd.io.
Preparing to unpack .../1-containerd.io_1.5.11-1_amd64.deb ...
Unpacking containerd.io (1.5.11-1) ...
Selecting previously unselected package docker-ce-cli.
Preparing to unpack .../2-docker-ce-cli_5%3a20.10.14~3-0~ubuntu-focal_amd64.deb ...
Unpacking docker-ce-cli (5:20.10.14~3-0~ubuntu-focal) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../3-docker-ce_5%3a20.10.14~3-0~ubuntu-focal_amd64.deb ...
Unpacking docker-ce (5:20.10.14~3-0~ubuntu-focal) ...
Selecting previously unselected package docker-ce-rootless-extras.
Preparing to unpack .../4-docker-ce-rootless-extras_5%3a20.10.14~3-0~ubuntu-focal_amd64.deb ...
Unpacking docker-ce-rootless-extras (5:20.10.14~3-0~ubuntu-focal) ...
Selecting previously unselected package docker-scan-plugin.
Preparing to unpack .../5-docker-scan-plugin_0.17.0~ubuntu-focal_amd64.deb ...
Unpacking docker-scan-plugin (0.17.0~ubuntu-focal) ...
Selecting previously unselected package liberror-perl.
Preparing to unpack .../6-liberror-perl_0.17029-1_all.deb ...
Unpacking liberror-perl (0.17029-1) ...
Selecting previously unselected package git-man.
Preparing to unpack .../7-git-man_1%3a2.25.1-1ubuntu3.2_all.deb ...
Unpacking git-man (1:2.25.1-1ubuntu3.2) ...
Selecting previously unselected package git.
Preparing to unpack .../8-git_1%3a2.25.1-1ubuntu3.2_amd64.deb ...
Unpacking git (1:2.25.1-1ubuntu3.2) ...
Selecting previously unselected package slirp4netns.
Preparing to unpack .../9-slirp4netns_0.4.3-1_amd64.deb ...
Unpacking slirp4netns (0.4.3-1) ...
Setting up slirp4netns (0.4.3-1) ...
Setting up docker-scan-plugin (0.17.0~ubuntu-focal) ...
Setting up liberror-perl (0.17029-1) ...
Setting up containerd.io (1.5.11-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/containerd.service → /lib/systemd/system/containerd.service.
Setting up docker-ce-cli (5:20.10.14~3-0~ubuntu-focal) ...
Setting up pigz (2.4-1) ...
Setting up git-man (1:2.25.1-1ubuntu3.2) ...
Setting up docker-ce-rootless-extras (5:20.10.14~3-0~ubuntu-focal) ...
Setting up docker-ce (5:20.10.14~3-0~ubuntu-focal) ...
Created symlink /etc/systemd/system/multi-user.target.wants/docker.service → /lib/systemd/system/docker.service.
Created symlink /etc/systemd/system/sockets.target.wants/docker.socket → /lib/systemd/system/docker.socket.
Setting up git (1:2.25.1-1ubuntu3.2) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for systemd (245.4-4ubuntu3.15) ...
lab@k8s1:~$

lab@k8s1:~$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete
Digest: sha256:bfea6278a0a267fad2634554f4f0c6f31981eea41c553fdf5a83e95a41d40c38
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

lab@k8s1:~$

Kubeadm Init Again!

lab@desktop:~$ sudo kubeadm init --apiserver-advertise-address=192.168.100.1
[sudo] password for lab:
I0329 06:16:35.526055    2143 version.go:255] remote version is much newer: v1.23.5; falling back to: stable-1.22
[init] Using Kubernetes version: v1.22.8
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [desktop kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.100.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [desktop localhost] and IPs [192.168.100.1 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [desktop localhost] and IPs [192.168.100.1 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.

	Unfortunately, an error has occurred:
		timed out waiting for the condition

	This error is likely caused by:
		- The kubelet is not running
		- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

	If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
		- 'systemctl status kubelet'
		- 'journalctl -xeu kubelet'

	Additionally, a control plane component may have crashed or exited when started by the container runtime.
	To troubleshoot, list all containers using your preferred container runtimes CLI.

	Here is one example how you may list all Kubernetes containers running in docker:
		- 'docker ps -a | grep kube | grep -v pause'
		Once you have found the failing container, you can inspect its logs with:
		- 'docker logs CONTAINERID'

error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
lab@desktop:~$

lab@desktop:~$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Tue 2022-03-29 06:20:45 PDT; 7s ago
       Docs: https://kubernetes.io/docs/home/
    Process: 4211 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (c>
   Main PID: 4211 (code=exited, status=1/FAILURE)
lab@desktop:~$

lab@desktop:~$ systemctl start kubelet
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to start 'kubelet.service'.
Authenticating as: Lab User,,, (lab)
Password:
==== AUTHENTICATION COMPLETE ===
lab@desktop:~$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Tue 2022-03-29 06:22:38 PDT; 5s ago
       Docs: https://kubernetes.io/docs/home/
    Process: 5067 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (c>
   Main PID: 5067 (code=exited, status=1/FAILURE)
lab@desktop:~$ sudo systemctl restart kubelet
lab@desktop:~$

lab@desktop:~$ sudo kubeadm init --apiserver-advertise-address=192.168.100.1
I0329 06:24:46.304971    6084 version.go:255] remote version is much newer: v1.23.5; falling back to: stable-1.22
[init] Using Kubernetes version: v1.22.8
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml]: /etc/kubernetes/manifests/kube-apiserver.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml]: /etc/kubernetes/manifests/kube-controller-manager.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml]: /etc/kubernetes/manifests/kube-scheduler.yaml already exists
	[ERROR FileAvailable--etc-kubernetes-manifests-etcd.yaml]: /etc/kubernetes/manifests/etcd.yaml already exists
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
lab@desktop:~$ kubectl get pods -A
The connection to the server localhost:8080 was refused - did you specify the right host or port?
lab@desktop:~$

lab@desktop:~$ sudo kubeadm reset
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0329 06:25:19.020290    6411 reset.go:101] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get config map: Get "https://192.168.100.1:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s": dial tcp 192.168.100.1:6443: connect: connection refused
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W0329 06:25:20.805711    6411 removeetcdmember.go:80] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
lab@desktop:~$

lab@desktop:~$ journalctl -xeu kubelet
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit kubelet.service has finished successfully.
--
-- The job identifier is 6939.
Mar 29 06:28:03 desktop kubelet[7588]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Mar 29 06:28:03 desktop kubelet[7588]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.068028    7588 server.go:440] "Kubelet version" kubeletVersion="v1.22.0"
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.068375    7588 server.go:868] "Client rotation is on, will bootstrap in backgroun>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.070622    7588 certificate_store.go:130] Loading cert/key pair from "/var/lib/kub>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.071730    7588 dynamic_cafile_content.go:155] "Starting controller" name="client->
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.152544    7588 server.go:687] "--cgroups-per-qos enabled, but --cgroup-root was n>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.152850    7588 container_manager_linux.go:280] "Container manager verified user s>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.152933    7588 container_manager_linux.go:285] "Creating Container Manager object>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.152969    7588 topology_manager.go:133] "Creating topology manager with policy pe>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.152983    7588 container_manager_linux.go:320] "Creating device plugin manager" d>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.153023    7588 state_mem.go:36] "Initialized new in-memory state store"
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.153113    7588 kubelet.go:314] "Using dockershim is deprecated, please consider u>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.153172    7588 client.go:78] "Connecting to docker on the dockerEndpoint" endpoin>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.153196    7588 client.go:97] "Start docker client with request timeout" timeout=">
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.163972    7588 docker_service.go:566] "Hairpin mode is set but kubenet is not ena>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.164002    7588 docker_service.go:242] "Hairpin mode is set" hairpinMode=hairpin-v>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.164114    7588 cni.go:239] "Unable to update cni config" err="no networks found i>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.166800    7588 cni.go:239] "Unable to update cni config" err="no networks found i>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.166863    7588 docker_service.go:257] "Docker cri networking managed by the netwo>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.166983    7588 cni.go:239] "Unable to update cni config" err="no networks found i>
Mar 29 06:28:03 desktop kubelet[7588]: I0329 06:28:03.176824    7588 docker_service.go:264] "Docker Info" dockerInfo=&{ID:3S6T:RA2S:JGF>
Mar 29 06:28:03 desktop kubelet[7588]: E0329 06:28:03.176884    7588 server.go:294] "Failed to run kubelet" err="failed to run Kubelet:>
Mar 29 06:28:03 desktop systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit kubelet.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Mar 29 06:28:03 desktop systemd[1]: kubelet.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit kubelet.service has entered the 'failed' state with result 'exit-code'.
lines 2080-2121/2121 (END)

lab@desktop:~$ sudo vi /etc/docker/daemon.json
lab@desktop:~$ sudo cat /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}

lab@desktop:~$
lab@desktop:~$ sudo mkdir -p /etc/systemd/system/docker.service.d
lab@desktop:~$
lab@desktop:~$ sudo systemctl daemon-reload
lab@desktop:~$ sudo systemctl restart docker
lab@desktop:~$

Reset the partial work done by kubeadm init.

lab@desktop:~$ sudo kubeadm reset
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0329 06:39:13.574732   13753 reset.go:101] [reset] Unable to fetch the kubeadm-config ConfigMap from cluster: failed to get config map: configmaps "kubeadm-config" not found
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W0329 06:39:15.204593   13753 removeetcdmember.go:80] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/etcd /var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni]

The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually by using the "iptables" command.

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

The reset process does not clean your kubeconfig files and you must remove them manually.
Please, check the contents of the $HOME/.kube/config file.
lab@desktop:~$

Lets try one more time, now that we have fixed the cgroup issue.

lab@desktop:~$ sudo kubeadm init --apiserver-advertise-address=192.168.100.1
I0329 06:39:31.088163   14116 version.go:255] remote version is much newer: v1.23.5; falling back to: stable-1.22
[init] Using Kubernetes version: v1.22.8
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [desktop kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.100.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [desktop localhost] and IPs [192.168.100.1 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [desktop localhost] and IPs [192.168.100.1 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 8.002989 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.22" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node desktop as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node desktop as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: mxan6y.5ymzfnl54gx83qh0
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.100.1:6443 --token mxan6y.5ymzfnl54gx83qh0 \
	--discovery-token-ca-cert-hash sha256:f4ade04f6f47f155bbe145629b445ae31b8b72e009cb4b0f9a9e99b760c85cdb
lab@desktop:~$

Congratulations! We can see the Your Kubernetes control-plane has initialized successfully! Message confirming that our control-plane node installation is successful.

Let us follow the instructions given at the end of installation.

lab@desktop:~$ mkdir -p $HOME/.kube
lab@desktop:~$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
lab@desktop:~$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
lab@desktop:~$

Verify Control Plane Status

lab@desktop:~$ kubectl get nodes
NAME      STATUS     ROLES                  AGE    VERSION
desktop   NotReady   control-plane,master   5m7s   v1.22.0
lab@desktop:~$

lab@desktop:~$ kubectl get pods
No resources found in default namespace.

lab@desktop:~$ kubectl get pods -A
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   coredns-78fcd69978-hkvsl          0/1     Pending   0          5m7s
kube-system   coredns-78fcd69978-rvtkt          0/1     Pending   0          5m7s
kube-system   etcd-desktop                      1/1     Running   1          5m12s
kube-system   kube-apiserver-desktop            1/1     Running   1          5m10s
kube-system   kube-controller-manager-desktop   1/1     Running   1          5m10s
kube-system   kube-proxy-ck8n2                  1/1     Running   0          5m7s
kube-system   kube-scheduler-desktop            1/1     Running   1          5m12s
lab@desktop:~$

Twitter Facebook LinkedIn

Kubernetes

Blog

Kubernetes Cluster Setup using Kubeadm

Kubernetes Cluster Setup using Kubeadm

Install Kubectl

Kubeadm and Kubelet

Hold the current versions

Verify the current version

Initialise Kubeadm

Install Docker

Kubeadm Init Again!

Verify Control Plane Status

You May Also Enjoy

Vertex AI: Qwik Start

Dataprep: Qwik Start

Dataflow: Qwik Start - Templates

Dataflow: Qwik Start - Python