Kubernetes multi-master

Kubernetes clusters enable developers to deploy and manage entire groups of containers with one single API entry point, scheduler, authentication model, and naming scheme. But while single-master clusters can easily fail, multi-master clusters use multiple (usually at least three) master nodes – each of which has access to the same pool of worker nodes – to establish quorum in the case of a loss of one or more members.

Copy the certificates to the two other masters.

$ scp -r /etc/kubernetes/pki [email protected]:/home

$ scp -r /etc/kubernetes/pki [email protected]:/home

initializing the 192.168.1.91 master node

Remove the apiserver.crt and apiserver.key.

$ rm /home/pki/apiserver.*

Move the certificates to the /etc/Kubernetes directory.

$ mv /home/pki /etc/kubernetes/

Create the configuration file for kubeadm.

$ vim config.yaml

apiVersion: kubeadm.k8s.io/v1alpha3

kind: ClusterConfiguration

kubernetesVersion: stable

apiServerCertSANs:

- 192.168.1.93

controlPlaneEndpoint: "192.168.1.93:6443"

etcd:

  external:

    endpoints:

    - https://192.168.1.90:2379

    - https://192.168.1.91:2379

    - https://192.168.1.92:2379

    caFile: /etc/etcd/ca.pem

    certFile: /etc/etcd/kubernetes.pem

    keyFile: /etc/etcd/kubernetes-key.pem

networking:

  podSubnet: 10.30.0.0/24

apiServerExtraArgs:

  apiserver-count: "3"

Initialize the machine as a master node.

$ kubeadm init --config=config.yaml

Initializing the 192.168.1.92 master node

Remove the apiserver.crt and apiserver.key.

$ rm /home/pki/apiserver.*

Move the certificates to the /etc/Kubernetes directory.

$ mv /home/pki /etc/kubernetes/

Create the configuration file for kubeadm.

$ vim config.yaml

apiVersion: kubeadm.k8s.io/v1alpha3

kind: ClusterConfiguration

kubernetesVersion: stable

apiServerCertSANs:

- 192.168.1.93

controlPlaneEndpoint: "192.168.1.93:6443"

etcd:

  external:

    endpoints:

    - https://192.168.1.90:2379

    - https://192.168.1.91:2379

    - https://192.168.1.92:2379

    caFile: /etc/etcd/ca.pem

    certFile: /etc/etcd/kubernetes.pem

    keyFile: /etc/etcd/kubernetes-key.pem

networking:

  podSubnet: 10.30.0.0/24

apiServerExtraArgs:

  apiserver-count: "3"

Initialize the machine as a master node.

$ kubeadm init --config=config.yaml

Copy the "kubeadm join" command line printed as the result of the previous command.

installing the kubeadm for all kubernetes NODES

Installing Docker

Add the Docker repository key and repo

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -

$ add-apt-repository \

"deb https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \

$(lsb_release -cs) \

stable"

Update the list of packages.

$ apt-get update

Install Docker 17.03.

$ apt-get install -y docker-ce=$(apt-cache madison docker-ce | grep 17.03 | head -1 | awk '{print $3}')

Installing kubeadm, kubelet, and kubectl

 Add the Google repository key.

$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

Add the Google repository.

$ vim /etc/apt/sources.list.d/kubernetes.list

deb http://apt.kubernetes.io kubernetes-xenial main

Update the list of packages. and Install kubelet, kubeadm and kubectl.

$ apt-get update && apt-get install kubelet kubeadm kubectl –y

Disable the swap.

$ swapoff –a

$ sed -i '/ swap / s/^/#/' /etc/fstab

Execute the "kubeadm join" command that you copied from the last step of the initialization of the masters. (discovery token must be same on all master)  cert generate for load balancer IP address

$kubeadm join 192.168.1.93:6443 --token [your_token] --discovery-token-ca-cert-hash sha256:[your_token_ca_cert_hash]

Verifying that the workers joined the cluster on the master

 To one of the master nodes.

$ kubectl --kubeconfig /etc/kubernetes/admin.conf get nodes

The status of the nodes is NotReady as we haven't configured the networking overlay yet. 

Add permissions to the admin.conf file.

$ chmod +r /etc/kubernetes/admin.conf

 From the client machine, copy the configuration file

$ scp [email protected]:/etc/kubernetes/admin.conf .

Create the kubectl configuration directory.

$ mkdir ~/.kube

Move the configuration file to the configuration directory.

$ mv admin.conf ~/.kube/config

Modify the permissions of the configuration file

$ chmod 600 ~/.kube/config

Go back to the SSH session on the master and change back the permissions of the

configuration file

$ sudo chmod 600 /etc/kubernetes/admin.conf

check that you can access the Kubernetes API from the client machine.

$ kubectl get nodes

Deploying the overlay network

We are going to use Weavenet as the overlay network. You can also use static route or

another overlay network tool like Calico or Flannel.

Deploy the overlay network pods from the client(master) machine

$ kubectl apply -f https://git.io/weave-kube-1.6

Check that the pods are deployed properly

$ kubectl get pods -n kube-system

Check that the nodes are in the ready state.

$ kubectl get nodes

Installing Kubernetes add-ons

We will deploy two Kubernetes add-ons on our new cluster: the dashboard add-on to have a graphical view of the cluster, and the Heapster add-on to monitor our workload

Installing the Kubernetes dashboard

Create the Kubernetes dashboard manifest.

$ vim kubernetes-dashboard.yaml

# Copyright 2017 The Kubernetes Authors.

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#     http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# Configuration to deploy release version of the Dashboard UI compatible with

# Kubernetes 1.8.

# Example usage: kubectl create -f 

# ------------------- Dashboard Secret ------------------- #

apiVersion: v1

kind: Secret

metadata:

  labels:

    k8s-app: kubernetes-dashboard

  name: kubernetes-dashboard-certs

  namespace: kube-system

type: Opaque

---

# ------------------- Dashboard Service Account ------------------- #

apiVersion: v1

kind: ServiceAccountmetadata:

  labels:

    k8s-app: kubernetes-dashboard

  name: kubernetes-dashboard

  namespace: kube-system

---

# ------------------- Dashboard Role & Role Binding ------------------- #

kind: Role

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: kubernetes-dashboard-minimal

  namespace: kube-system

rules:

  # Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.

- apiGroups: [""]

  resources: ["secrets"]

  verbs: ["create"]

  # Allow Dashboard to create 'kubernetes-dashboard-settings' config map.

- apiGroups: [""]

  resources: ["configmaps"]

  verbs: ["create"]

  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.

- apiGroups: [""]

  resources: ["secrets"]

  resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard

certs"] 

 verbs: ["get", "update", "delete"]

  # Allow Dashboard to get and update 'Kubernetes-dashboard-settings' config map.

- apiGroups: [""]

  resources: ["configmaps"]

  resourceNames: ["kubernetes-dashboard-settings"]

  verbs: ["get", "update"]

  # Allow Dashboard to get metrics from heapster.

- apiGroups: [""]

  resources: ["services"]

  resourceNames: ["heapster"]

  verbs: ["proxy"]

- apiGroups: [""]

  resources: ["services/proxy"]

  resourceNames: ["heapster", "http:heapster:", "https:heapster:"]

  verbs: ["get"]

---

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

  name: kubernetes-dashboard-minimal

  namespace: kube-system

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: Role

  name: kubernetes-dashboard-minimal

subjects:

- kind: ServiceAccount

  name: kubernetes-dashboard

  namespace: kube-system

---

# ------------------- Dashboard Deployment ------------------- #

kind: Deployment

apiVersion: apps/v1beta2

metadata:

  labels:

    k8s-app: kubernetes-dashboard

  name: kubernetes-dashboard

  namespace: kube-system

spec:

  replicas: 1

  revisionHistoryLimit: 10

  selector:

    matchLabels:

      k8s-app: kubernetes-dashboard

  template:

    metadata:

      labels:

        k8s-app: kubernetes-dashboard

    spec:

      containers:

      - name: kubernetes-dashboard

        image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3

        ports:

        - containerPort: 8443

          protocol: TCP

        args:

          - --auto-generate-certificates

          # Uncomment the following line to manually specify Kubernetes API server

Host

          # If not specified, Dashboard will attempt to auto-discover the API server and

connect

          # to it. Uncomment only if the default does not work.

          # - --apiserver-host=http://my-address:port

        volumeMounts:

        - name: kubernetes-dashboard-certs

          mountPath: /certs

          # Create on-disk volume to store exec logs

        - mountPath: /tmp

          name: tmp-volume

        livenessProbe:

          httpGet:

            scheme: HTTPS

            path: /   

         port: 8443

          initialdelaySeconds: 30

          timeoutSeconds: 30

      volumes:

      - name: kubernetes-dashboard-certs

        secret:

          secretName: kubernetes-dashboard-certs

      - name: tmp-volume

        emptyDir: {}

      serviceAccountName: kubernetes-dashboard

      # Comment the following tolerations if Dashboard must not be deployed on master

      tolerations:

      - key: node-role.kubernetes.io/master

        effect: NoSchedule

---

# ------------------- Dashboard Service ------------------- #

kind: Service

apiVersion: v1

metadata:

  labels:

    k8s-app: kubernetes-dashboard

  name: kubernetes-dashboard

  namespace: kube-system

spec:

  ports:

    - port: 443

      targetPort: 8443

  selector:

    k8s-app: kubernetes-dashboard

Deploy the dashboard.

$ kubectl create -f kubernetes-dashboard.yaml

Installing Heapster

Create a manifest for Heapster

$ vim heapster.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

  name: heapster

  namespace: kube-system

---

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

  name: heapster

  namespace: kube-system

spec:

  replicas: 1

  template:

    metadata:

      labels:

        task: monitoring

        k8s-app: heapster

    spec:

      serviceAccountName: heapster

      containers:

      - name: heapster

        image: gcr.io/google_containers/heapster-amd64:v1.4.2

        imagePullPolicy: IfNotPresent

        command:

        - /heapster

        -

--source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true

apiVersion: v1

kind: Service

metadata:

  labels:

    task: monitoring

 # For use as a Cluster 

add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)    # If you are NOT using this as an addon, you should comment out this line.

    kubernetes.io/cluster-service: 'true'

    kubernetes.io/name: Heapster

  name: heapster

  namespace: kube-system

spec:

  ports:

  - port: 80

    targetPort: 8082

  selector:

    k8s-app: heapster

---

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1beta1

metadata:

  name: heapster

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: system:heapster

subjects:

- kind: ServiceAccount

  name: heapster

  namespace: kube-system

 Deploy Heapster.

$ kubectl create -f heapster.yaml

Edit the Heapster RBAC role and add the get permission on the nodes statistic at the end.

$ kubectl edit clusterrole system:heapster

...

- apiGroups:

  - ""

  resources:

  - nodes/stats

  verbs:

  - get

Accessing the Kubernetes dashboard

 Create an admin user manifest.

$ vim kubernetes-dashboard-admin.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

  name: admin-user

  namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRoleBinding

metadata:

  name: admin-user

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: cluster-admin

subjects:

- kind: ServiceAccount

  name: admin-user

  namespace: kube-system

Create the admin user

$ kubectl create -f kubernetes-dashboard-admin.yaml

Get the admin user token.

$ kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep

admin-user | awk '{print $1}')

Copy the token.

Start the proxy to access the dashboard.

$ kubectl proxy

Browse to  

http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy.

Select Token and paste the token

Install Kubernetes CLI on Windows 10

If you are looking to access Kubernetes Cluster from your windows machine. Look no further! we will show you how to install Kubernetes command-line utilities by leveraging the Chocolatey installer.

Note. we will be using Windows 10 to demonstrate.

Now let's go ahead and get started by opening PowerShell as administrator and execute the below command.

$ Set-ExecutionPolicy Bypass -Scope Process -Force; iex 

((NewObjectSystem.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

Now that Chocolatey has been installed, we will go ahead with the Kubernetes CLI setup.

Install Kubernetes CLI

Open PowerShell as an administrator and execute the below command

$ choco install kubernetes-cli

You will be prompted to confirm if you want to proceed forward with the installation. Go ahead

and say yes by typing Y and hit enter.

Connect to Kubernetes Cluster with Kubectl

Once you have install Kubernetes CLI. Go to your Kubernetes master node and copy the config file from ~/.kube/config to your windows machine to any location. We will move that file to the required location once we create the .kube directory on windows. Follow the below steps.

Open PowerShell as an administrator and execute the below commands.

$ cd ~

The above command will take you to your user home directory. In the user home directory create a folder called .kube. If it already exists, you can skip this step.

$ mkdir .kube

Once the above directory has been created, we need to copy the config file from the Kubernetes master node to the .kube folder. Earlier I mentioned copying the config file to your windows machine. Take that file and drop it on your under ~\.kube location path. In windows, the config file should be ALL FILES format

The above config file should be copied and paste into windows 10

Basic operations

After you have followed steps as shown above, let's go ahead and test connectivity with your

Kubernetes cluster.

$ kubectl.exe config get-clusters

If above command returns name of the cluster, then you have applied changes successfully. Below command will get information from your master node.

$ kubectl.exe version -o yaml

For me I received following output. Yours may vary depending on your cluster configuration.

clientVersion:

  buildDate: 2018-01-04T11:52:23Z

  compiler: gc

  gitCommit: 3a1c9449a956b6026f075fa3134ff92f7d55f812

  gitTreeState: clean

  gitVersion: v1.9.1

  goVersion: go1.9.2

  major: "1"

  minor: "9"

  platform: windows/amd64

serverVersion:

  buildDate: 2018-01-18T09:42:01Z

  compiler: gc

  gitCommit: 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b

  gitTreeState: clean

  gitVersion: v1.9.2

  goVersion: go1.9.2

  major: "1"

  minor: "9"

  platform: linux/amd64

Let's execute one more command to ensure we are successfully connected to the Kubernetes cluster.

$ kubectl.exe get nodes

If you received something similar to what I have received below, then you are fully connected to the cluster and you can go ahead and manage your cluster from a windows machine.

metal load balancer on master for glowing (external IP) on-premises

First, we need to apply the MetalLB manifest

$ kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml

Next, we want to check that the controller and the speaker are running. we can do this by using this command

$ kubectl get pods -n metallb-system

So next we will look at our configuration on master

# cat metalconfig.yaml

apiVersion: v1

kind: ConfigMap

metadata:

  namespace: metallb-system

  name: config

data:

  config: |

    address-pools:

    - name: my-ip-space

      protocol: layer2

      addresses:

      - 192.168.1.160-192.168.1.165

Apply the config for metallb

# kubectl apply –f metalconfig.yaml

Deploy a tomcat and service for checking

# cat tomcat.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

  name: tomcat-pod

spec:

  selector:

    matchLabels:

      run: tomcat-pod

  replicas: 3

  template:

    metadata:

      labels:

        run: tomcat-pod

    spec:

      containers:

      - name: tomcat

        image: tomcat:latest

        ports:

        - containerPort: 8080

---

apiVersion: v1

kind: Service

metadata:

  name: tomcat-pod

  labels:

    run: tomcat-pod

spec:

  type: LoadBalancer

  ports:

  - port: 8080

    targetPort: 8080

  selector:

    run: tomcat-pod

deploy the tomcat

# kubectl apply –f tomcat.yaml

NFS Persistent Volume

In this Lab we are going to provision an NFS server on Ubuntu 16.04 you can use your own NFS server on any platform

Installing NFS server on Ubuntu 16.04 (new machine)

To get the NFS server working you must install the server packages. run the commands below:

$ sudo apt-get update

$ sudo apt-get install nfs-kernel-server

Installing NFS client packages on the client systems. here we are going to use the client as All three master nodes. 

Install NFS client on all the master nodes using the following command, to access NFS mount points on the server, 

$ sudo apt-get update

$ sudo apt-get install nfs-common

After installing the client packages, switch to the server to configure a mount point to export to the client.

Creating the folder/directory to export (share) to the NFS clients, For this Lab, we’re creating a folder called nfs/kubedata in the /srv/ directory.so run,

$ sudo mkdir –p /srv/nfs/kubedata

Since we want this location to be viewed by all clients, we’re going to remove the restrictive permissions. To do that, change the folder permission to be owned by nobody in no group.

$ sudo chown nobody:nogroup /srv/nfs/kubedata

$ sudo chmod 777 /srv/nfs/kubedata

Configuring NFS Exports file

Now that the location is created on the host system, open the NFS export file and define the client access.

Access can be granted to a single client or entire network subnet. For this Lab, we’re allowing access to all clients.

NFS export file is at /etc/exports, open the export file by running the commands below:

$ vi /etc/exports

Then add the line below:

/srv/nfs/kubedata *(rw,sync,no_subtree_check,insecure)

The options in the setting above are: 

Export the shares by running the commands below

$ exportfs -v

/srv/nfs/kubedata          (rw,wdelay,insecure,root_squash,no_subtree_check,sec=sys,rw,root_squash,no_all_squash)  => output

$ sudo exportfs –rav

exporting *:/srv/nfs/kubedata   => output

Restart the NFS server by running the commands below.

$ sudo systemctl restart nfs-kernel-server

NFS server is ready to serve and share its storage… (IN master

$ showmount -e

Export list for zippyops:

/srv/nfs/kubedata *

apply persistent volume(PV) on any master

Create a persistent volume for NFS

# cat pv-nfs.yaml

apiVersion: v1

kind: PersistentVolume

metadata:

  name: pv-nfs-pv1

  labels:

    type: nfs

spec:

  storageClassName: manual

  capacity:

    storage: 3Gi

  accessModes:

    - ReadWriteMany

  nfs:

    server: 192.168.1.97

    path: "/srv/nfs/kubedata"

to create persistent volume

# kubectl apply –f pv-nfs.yaml

To claim the persistent volume

# cat pv-claim.yaml

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: pvc-nfs-pv1

spec:

  storageClassName: manual

  accessModes:

    - ReadWriteMany

  resources:

    requests:

      storage: 500Mi

After a claim, the persistent volume creates the deployment for Nginx and its service with include the volume mount

# cat nginx.yaml

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

  name: nginx

spec:

  replicas: 3

  selector:

    matchLabels:

      app: nginx

  template:

    metadata:

      labels:

        app: nginx

    spec:

      volumes:

      - name: www

        persistentVolumeClaim:

          claimName: pvc-nfs-pv1

      containers:

      - name: nginx

        image: nginx

        volumeMounts:

        - name: www

          mountPath: /usr/share/nginx/html

        ports:

        - name: http

          containerPort: 80

---

apiVersion: v1

kind: Service

metadata:

  name: nginx

spec:

  ports:

  - name: http

    port: 80

    protocol: TCP

    targetPort: 80

  selector:

    app: nginx

  type: LoadBalancer

to run pods with volume mount

# kubectl apply –f nginx.yaml

OUTPUT

Add HTML file in NFS server