Kubernetes Cluster using kubeadm -Setup and Troubleshooting

Nowadays, we see that Kubernetes has become a common topic of discussion especially with the growing demand in cloud and microservices. It not only deploys services but is a platform that would manage the services and their workload. This made me curious to learn and try it out myself.

I used Ubuntu 16.04 which was available at my disposal and installed kubernetes.

sudo apt-get update && sudo apt-get install -y kubelet kubeadm kubectl docker.io

What I was more interested in was setting up a cluster which resembles a production grade system for developers.

So in this blog I would talk about a few key concepts and summarize how to setup a kubernetes cluster along with the troubleshooting steps that could be helpful.

Kubernetes

Kubernetes was build by google based on their rich experience running containers in production.

Kubernetes also known as k8s is an open-source container management platform for deploying and managing containerized services. It does not have the functionality to create or manage container images and to run containers, for that it needs to work with an external container source and runtime.

To understand Kubernetes, we must first understand two things.

Container And Orchestration

Docker

Whenever one asks about containers the most common and widely spoken container platform is docker. Docker has made a mark when it comes to building and running containers.

What are containers?

Containers are a way to package and isolate an application with everything it needs to run. As in they can have their own processes or services, their own networking interfaces, their own mounts just like virtual machines.

What is an image?

An image is a package or a template used to create one or more containers.

Containers are running instances of images that are isolated and have their own environments and set of processes.

Container Orchestration

The entire automated process of deploying and managing containers is known as Container Orchestration.

There are various advantages of Container Orchestration.

• Application becomes highly available as we have multiple instances of the application running on different nodes.
• The user traffic is load balanced across the various containers.
• When demand increases, deploy more instances of the applications seamlessly and within a matter of seconds.
• When we run out of hardware resources, we can scale the number of underlying nodes up or down without bringing down the application and all this can be easily done with a set of configuration files.

Now that we know what container and orchestration is we can define Kubernetes as a container orchestration tool which can manage and deploy hundreds and thousands of containers in a clustered environment.

Setup

Basic Terms:

As we focus on the cluster setup, let me highlight a few terms necessary during the setup.

Nodes:

• A node is a machine, physical or virtual on which Kubernetes is installed. Master node is one where we installed kubernetes and we will configure it as a Master or the control pane in terms of kubernetes. Worker node is a machine where kubernetes is installed and would be the place where containers will be launched by Kubernetes.
• The master watches over the nodes in the cluster and is responsible for the actual orchestration of containers on the worker nodes.

Kubectl:

• The kubectl tool is used to deploy and manage applications on a Kubernetes cluster.
• Also used to get cluster information, the status of other nodes in the cluster and to manage many other things.

Prerequisite:

Kubernetes requires to disable swap memory as it can cause stability issues within kubernetes impacting the scheduler. Kubernetes scheduler is responsible to determine on which node to deploy the pod.

##Execute on all nodes
sudo swapoff -a

Create a Cluster:

Since we are setting the network from a development perspective let us bootstrap the cluster without a DNS endpoint.

## Run only on master node
kubeadm init --pod-network-cidr=192.168.0.0/16

pod-network-cidr is used to set a Pod network add-on CIDR. If 192.168.0.0/16 is already used within your network please make sure that you select a different pod network CIDR.

The above init command will run some pre flight checks and setup the necessary components like api-server, controller-manager,scheduler required to manage the cluster. It will also print the next set of instruction to setup the cluster as mentioned below.

## Run only on master node
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

The above commands are used to setup the local kube config. Once this is done wait for some time and check if the pods are up by running the below command.

## Run only on master node
kubectl get pod --all-namespaces

The issue i faced was that the coredns were failing with a crashloopback, so i had to comment out the loop instruction by editing the coredns config map by running the instruction below. Then deleting the coredns pods and waiting for them to come up successfully.

## Run only on master node
kubectl edit cm coredns -n kube-system (Comment the loop)
kubectl get pods --all-namespaces (To check if coredns is running)

Apply the flannel CNI overlay network

Network overlays are used to address nodes, container, pods connectivity. It is an like a virtual network on the underlying physical infrastructure supporting connectivity.

## Run only on master node
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Join worker nodes:

The init command will also print below instruction of how to join a node to the master. The [token] will be printed as well. Instruction will look as mentioned below.

## Run only on worker nodes
sudo kubeadm join <master_node_ip>:6443 [token]

On Master node check the nodes:

## Run only on master node
kubectl get nodes -o wide
kubectl get pod --all-namespaces -o wide

Test pod creation:

To test the cluster lets create deploy nginx and check if the pod is up and running.

## Run only on master node
kubectl run nginx --image=nginx

To Bring down the cluster:

The delete all instruction will make sure all the deployments are deleted on the worker nodes.

## Run only on master node
kubectl delete all --all

To remove a worker node run the below command from the respective nodes.

## Run only on worker node
sudo kubeadm reset -f
## Run only on master node
kubectl get nodes
kubectl delete node <node name>

Troubleshooting

Let me put down a few commands i used while working on the cluster setup as i had 3 VMs which has different version of docker, kubernetes installed.

• Faced issue updating the repository. So i had to delete the existing repository list and update

## Merge List issue:
sudo rm /var/lib/apt/lists/* -vf
sudo apt-get update

• For missing key issue

## Missing key update
gpg -- keyserver hkp://keyserver.ubuntu.com:8 -- recv <KEY>
gpg -- export -- armor <KEY> | sudo apt-key add -

• Update docker

## update docker
sudo apt-get update
sudo apt-get upgrade docker-ce

• Verify br_filter module is loaded and nodes iptables is set correctly to see bridged traffic.

## Prerequisite for installing kubeadm
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl -- system

• If the init command does not pull the config then explicitly pull the configuration and then run the init command

## Run only on master node
kubeadm config images pull -- kubernetes-version=stable-1.19
kubeadm init -- pod-network-cidr=192.168.0.0/16

• when i ran the init command i got a warning [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.7. Latest validated version: 19.03
• Kubernetes has deprecated Docker as a container runtime after version 1.20, as docker does not implement the Container Runtime Interface. So the way it worked was Kubernetes had a docker shim which would serve as an interface between kubernetes and docker. But for now lets consider installing a lower version 1.19.16. Below commands would help us to install the required version.

## Update the correct version of kubeadm
apt list -a kubeadm
apt install -y kubeadm=1.19.16–00 kubelet=1.19.16–00 kubectl=1.19.16–00
apt remove -y kubeadm=1.23.1–00 kubelet=1.23.1–00 kubectl=1.23.1–00

Summary

That’s it!! With this we have configured a kubernetes cluster with a master and worker nodes.

We started with nodes having docker and kubernetes tools installed and our focus was mainly setting up a cluster.

Discussed briefly on what and why of container orchestration and what kubernetes is all about.

I hope this article is an informative one and helps to get started with Kubernetes cluster.