Kubernetes Hardway

Kubernetes

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications.


Kubernetes The Hard Way is optimized for learning, which means taking the long route to ensure you understand each task required to bootstrap a Kubernetes cluster.


Prerequisites

 Using Three servers

Master 192.168.1.1

Minion 192.168.1.2

Minion 192.168.1.3 

Make sure that each of these servers has at least 1 core and 2 GB of memory


Disable Firewall

Make the status of the SELINUX disabled.


Cluster Configuration

Infrastructure Private subnet IP range: 192.168.1.1/16

Flannel subnet IP range: 172.30.0.0/16 (You can choose any IP range just make sure it does not overlap with any other IP range)

Service Cluster IP range for Kubernetes: 10.254.0.0/16 (You can choose any IP range just make sure it does not overlap with any other IP range)

Kubernetes Service IP: 10.254.0.1 (First IP from service cluster IP range is always allocated to Kubernetes Service)

DNS service IP: 10.254.3.100 (You can use any IP from the service cluster IP range just make sure that the IP is not allocated to any other service)


Step 1: Create a Repo on all Host i.e. Master, Minion1, and Minion2.

# vi /etc/yum.repos.d/virt7-docker-common-release.repo

[virt7-docker-common-release]

name=virt7-docker-common-release

baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/

gpgcheck=0


# yum update –y

Step 2: installing kubernetes, etcd and flannel

# yum -y install --enablerepo=virt7-docker-common-release kubernetes etcd flannel


Step 3: configure Kubernetes components

Kubernetes Common Configuration (On All Nodes) includes master and nodes.

# vi /etc/kubernetes/config 

KUBE_ETCD_SERVERS="--etcd-servers=http://192.168.xxx.xxx:2379"

KUBE_LOGTOSTDERR="--logtostderr=true"

KUBE_LOG_LEVEL="--v=0"

KUBE_ALLOW_PRIV="--allow-privileged=false"

KUBE_MASTER="--master=http://192.168.xxx.xxx:8080"


etcd configuration (on master)

# vi /etc/etcd/etcd.conf


#[member]

ETCD_NAME=default

ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"

#[cluster]

ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

api server configuration (on master)


API Server handles the REST operations and acts as a front-end to the cluster’s shared state.

API Server Configuration is stored at /etc/kubernetes/apiserver.

Kubernetes uses certificates to authenticate API requests. Before configuring the API server, we need to generate certificates that can be used for authentication. Kubernetes provides ready-made scripts for generating these certificates.


# vi make-ca-cert.sh

#!/bin/bash

set -o errexit

set -o nounset

set -o pipefail

DEBUG="${DEBUG:-false}"

if [ "${DEBUG}" == "true" ]; then

set -x

fi

cert_ip=$1

extra_sans=${2:-}

cert_dir=${CERT_DIR:-/srv/kubernetes}

cert_group=${CERT_GROUP:-kube}


mkdir -p "$cert_dir"

use_cn=false

if [ "$cert_ip" == "_use_gce_external_ip_" ]; then

cert_ip=$(curl -s -H Metadata-Flavor:Google http://metadata.google.internal./computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip)

fi

if [ "$cert_ip" == "_use_aws_external_ip_" ]; then

  if cert_ip=$(curl -f -s http://169.254.169.254/latest/meta-data/public-ipv4); then

    :

  else

    cert_ip=$(curl -f -s http://169.254.169.254/latest/meta-data/local-ipv4)

  fi

fi

sans="IP:${cert_ip}"

if [[ -n "${extra_sans}" ]]; then

  sans="${sans},${extra_sans}"

fi

tmpdir=$(mktemp -d -t kubernetes_cacert.XXXXXX)

trap 'rm -rf "${tmpdir}"' EXIT

cd "${tmpdir}"

if [ -f ~/kube/easy-rsa.tar.gz ]; then

ln -s ~/kube/easy-rsa.tar.gz .

else

curl -L -O https://storage.googleapis.com/kubernetes-release/easy-rsa/easy

rsa.tar.gz > /dev/null 2>&1

fi

tar xzf easy-rsa.tar.gz > /dev/null 2>&1

cd easy-rsa-master/easyrsa3

./easyrsa init-pki > /dev/null 2>&1

./easyrsa --batch "--req-cn=$cert_ip@`date +%s`" build-ca nopass > /dev/null 2>&1

if [ $use_cn = "true" ]; then

    ./easyrsa build-server-full $cert_ip nopass > /dev/null 2>&1

    cp -p pki/issued/$cert_ip.crt "${cert_dir}/server.cert" > /dev/null 2>&1

    cp -p pki/private/$cert_ip.key "${cert_dir}/server.key" > /dev/null 2>&1

else

    ./easyrsa --subject-alt-name="${sans}" build-server-full kubernetes-master

nopass > /dev/null 2>&1

    cp -p pki/issued/kubernetes-master.crt "${cert_dir}/server.cert" > /dev/null 2>&1

    cp -p pki/private/kubernetes-master.key "${cert_dir}/server.key" > /dev/null 2>&1

fi

./easyrsa build-client-full kubecfg nopass > /dev/null 2>&1

cp -p pki/ca.crt "${cert_dir}/ca.crt"

cp -p pki/issued/kubecfg.crt "${cert_dir}/kubecfg.crt"

cp -p pki/private/kubecfg.key "${cert_dir}/kubecfg.key"

chgrp $cert_group "${cert_dir}/server.key" "${cert_dir}/server.cert"

"${cert_dir}/ca.crt"

chmod 660 "${cert_dir}/server.key" "${cert_dir}/server.cert" "${cert_dir}/ca.crt"


Now, run the script with following parameters to create certificates:

# bash make-ca-cert.sh "192.168.1.1" "IP:192.168.1.1,IP:10.254.0.1,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster.local"


Where 192.168.1.1 is the IP of the master server, 10.254.0.1 is the IP of the Kubernetes service. Now, we can configure the API server:

# vi /etc/kubernetes/apiserver

# Bind kube API server to this IP

KUBE_API_ADDRESS="--address=0.0.0.0"

# Port that Kube API server listens to.

KUBE_API_PORT="--port=8080"

# Port kubelet listen on

KUBELET_PORT="--kubelet-port=10250"

# Address range to use for services(Work unit of Kubernetes)

KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# default admission control policies

KUBE_ADMISSION_CONTROL="--admission><span "font-family:="" -apple-system;="" color:="" rgb(33,="" 37,="" 41);="" font-size:="" 18px;"="">control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"

# Add your own!

KUBE_API_ARGS="--client-ca-file=/srv/kubernetes/ca.crt --tls-cert-file=/srv/kubernetes/server.cert --tls-private-key-file=/srv/kubernetes/server.key" 

Controller Manager Configuration (On Master)

# vi /etc/kubernetes/controller-manager

# Add your own!

 KUBE_CONTROLLER_MANAGER_ARGS="--root-ca-file=/srv/kubernetes/ca.crt --service-account-private-key-file=/srv/kubernetes/server.key"


Kubelet Configuration (On Minions)

Kubelet is a node/minion agent that runs pods and makes sure that it is healthy. It also communicates pod details to Kubernetes Master. Kubelet configuration is stored

/etc/kubernetes/kubelet


[On Minion1]

vi /etc/kubernetes/kubelet

# kubelet bind ip address(Provide private ip of minion)

KUBELET_ADDRESS="--address=0.0.0.0"

# port on which kubelet listen

KUBELET_PORT="--port=10250"

# leave this blank to use the hostname of server

KUBELET_HOSTNAME="--hostname-override=192.168.1.2"

# Location of the api-server

KUBELET_API_SERVER="--api-servers=http://192.168.1.1:8080"

KUBELET_ARGS=""


[On Minion2]

vi /etc/kubernetes/kubelet 

# kubelet bind ip address(Provide private ip of minion)

KUBELET_ADDRESS="--address=0.0.0.0"

# port on which kubelet listen

KUBELET_PORT="--port=10250"

# leave this blank to use the hostname of server

KUBELET_HOSTNAME="--hostname-override=192.168.1.3"

# Location of the api-server

KUBELET_API_SERVER="--api-servers=http://192.168.1.1:8080"

KUBELET_ARGS=""


Create network configuration for Flannel in etcd. (On Master)

1.Start the etcd node on the master using the following command:

# systemctl start etcd

2. Create a new key in etcd to store Flannel configuration using   the  following command:

# etcdctl mkdir /kube-centos/network

3. Define the network configuration for Flannel

# etcdctl mk /kube-centos/network/config "{ \"Network\": \"172.30.0.0/16\",

\"SubnetLen\": 24, \"Backend\": { \"Type\": \"vxlan\" } }"

The above command allocates the 172.30.0.0/16 subnet to the Flannel network. A flannel

subnet of CIDR 24 is allocated to each server in Kubernetes cluster.


Step 4: start services on master and minion

On Master

systemctl enable kube-apiserver

systemctl start kube-apiserver

systemctl enable kube-controller-manager

systemctl start kube-controller-manager

systemctl enable kube-scheduler

systemctl start kube-scheduler

systemctl enable flanneld

systemctl start flanneld


On Minions

systemctl enable kube-proxy

systemctl start kube-proxy

systemctl enable kubelet

systemctl start kubelet

systemctl enable flanneld

systemctl start flanneld

systemctl enable docker

systemctl start docker




Relevant Blogs:

Kubernetes AWS Integration 

Kubernetes monitoring 

Service Now  

How-to Guide for Archiving to AWS S3 Glacier

Recent Comments

No comments

Leave a Comment