k8s with metalLb and persistent volume

Prerequisites

we will use a standard Ubuntu 16.04 installation as a base image for the seven machines needed. The machines will all be configured on the same network, this network needs to have access to the Internet.

The first machine needed is the machine on which the HAProxy load balancer will be installed. We will assign IP 192.168.1.93 to this machine.

We also need three Kubernetes master nodes. These machines will have IPs192.168.1.90, 192.168.1.91, and 192.168.1.92.

Finally, we will also have three Kubernetes worker nodes with the IPs 192.168.1.94, 192.168.1.95, and 192.168.1.96. we are going to use three masters (a minimum of three is needed) embedded etcd and three worker nodes that will work fine. Use the unique machine to generate all the necessary certificates to manage the Kubernetes cluster. If you don't have a machine you can use the HAProxy machine to do the same thing.

Copy the certificates to the two other masters.

#scp -r /etc/kubernetes/pki [email protected]:/home

#scp -r /etc/kubernetes/pki [email protected]:/home

initializing the 192.168.1.91 master node

Remove the apiserver.crt and apiserver.key.

#rm /home/pki/apiserver.*

Move the certificates to the /etc/kubernetes directory.

#mv /home/pki /etc/kubernetes/

Create the configuration file for kubeadm.

#vim config.yaml

apiVersion: kubeadm.k8s.io/v1alpha3

kind: ClusterConfiguration

kubernetesVersion: stable

apiServerCertSANs:

- 192.168.1.93

controlPlaneEndpoint: "192.168.1.93:6443"

etcd:

  external:

    endpoints:

    - https://192.168.1.90:2379

    - https://192.168.1.91:2379

    - https://192.168.1.92:2379

    caFile: /etc/etcd/ca.pem

    certFile: /etc/etcd/kubernetes.pem

    keyFile: /etc/etcd/kubernetes-key.pem

networking:

  podSubnet: 10.30.0.0/24

apiServerExtraArgs:

  apiserver-count: "3"

Initialize the machine master node.

#kubeadm init --config=config.yaml

Deploying the overlay network

We are going to use Weavenet as the overlay network. You can also use static route or another overlay network tool like Calico or Flannel.

Deploy the overlay network pods from the client(master) machine

#kubectl apply -f https://git.io/weave-kube-1.6

Check that the pods are deployed properly

# kubectl get pods -n kube-system

Check that the nodes are in the ready state.

#kubectl get nodes

NAME                      STATUS     ROLES    AGE     VERSION

kubmaster1.zippyops.com   Ready      master   2d20h   v1.13.4

kubmaster2.zippyops.com   Ready      master   2d19h   v1.13.4

kubmaster3.zippyops.com   Ready      master   2d19h   v1.13.4

kubwork1.zippyops.com     Ready         2d19h   v1.13.4

kubwork2.zippyops.com     Ready         2d19h   v1.13.4

kubwork3.zippyops.com     Ready         2d19h   v1.13.4

Installing the Kubernetes dashboard

Create the Kubernetes dashboard manifest.

#vim kubernetes-dashboard.yaml

# Copyright 2017 The Kubernetes Authors.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

# Configuration to deploy release version of the Dashboard UI compatible with

# Kubernetes 1.8.

#

# Example usage: kubectl create -f 

# ------------------- Dashboard Secret ------------------- #

apiVersion: v1

kind: Secret

metadata:

  labels:

    k8s-app: kubernetes-dashboard

  name: kubernetes-dashboard-certs

  namespace: kube-system

type: Opaque

---

# ------------------- Dashboard Service Account ------------------- #

apiVersion: v1

kind: ServiceAccount

metadata:

  labels:

    k8s-app: kubernetes-dashboard

  name: kubernetes-dashboard

  namespace: kube-system

---

# ------------------- Dashboard Role & Role Binding ------------------- #

kind: Role

apiVersion: rbac.authorization.k8s.io/v1

metadata:

  name: kubernetes-dashboard-minimal

  namespace: kube-system

rules:

  # Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.

- apiGroups: [""]

  resources: ["secrets"]

  verbs: ["create"]

  # Allow Dashboard to create 'kubernetes-dashboard-settings' config map.

- apiGroups: [""]

  resources: ["configmaps"]

  verbs: ["create"]

  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.

- apiGroups: [""]

  resources: ["secrets"]

  resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"]

  verbs: ["get", "update", "delete"]

  # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.

- apiGroups: [""]

  resources: ["configmaps"]

  resourceNames: ["kubernetes-dashboard-settings"]

  verbs: ["get", "update"]

  # Allow Dashboard to get metrics from heapster.

- apiGroups: [""]

  resources: ["services"]

  resourceNames: ["heapster"]

  verbs: ["proxy"]

- apiGroups: [""]

  resources: ["services/proxy"]

  resourceNames: ["heapster", "http:heapster:", "https:heapster:"]

  verbs: ["get"]

---

apiVersion: rbac.authorization.k8s.io/v1

kind: RoleBinding

metadata:

  name: kubernetes-dashboard-minimal

  namespace: kube-system

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: Role

  name: kubernetes-dashboard-minimal

subjects:

- kind: ServiceAccount

  name: kubernetes-dashboard

  namespace: kube-system

---

# ------------------- Dashboard Deployment ------------------- #

kind: Deployment

apiVersion: apps/v1beta2

metadata:

  labels:

    k8s-app: kubernetes-dashboard

  name: kubernetes-dashboard

  namespace: kube-system

spec:

  replicas: 1

  revisionHistoryLimit: 10

  selector:

    matchLabels:

      k8s-app: kubernetes-dashboard

  template:

    metadata:

      labels:

        k8s-app: kubernetes-dashboard

    spec:

      containers:

      - name: kubernetes-dashboard

        image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3

        ports:

        - containerPort: 8443

          protocol: TCP

        args:

          - --auto-generate-certificates

          # Uncomment the following line to manually specify Kubernetes API server Host

          # If not specified, Dashboard will attempt to auto discover the API server and connect

          # to it. Uncomment only if the default does not work.

          # - --apiserver-host=http://my-address:port

        volumeMounts:

        - name: kubernetes-dashboard-certs

          mountPath: /certs

          # Create on-disk volume to store exec logs

        - mountPath: /tmp

          name: tmp-volume

        livenessProbe:

          httpGet:

            scheme: HTTPS

            path: /

            port: 8443

          initialDelaySeconds: 30

          timeoutSeconds: 30

      volumes:

      - name: kubernetes-dashboard-certs

        secret:

          secretName: kubernetes-dashboard-certs

      - name: tmp-volume

        emptyDir: {}

      serviceAccountName: kubernetes-dashboard

      # Comment the following tolerations if Dashboard must not be deployed on master

      tolerations:

      - key: node-role.kubernetes.io/master

        effect: NoSchedule

---

# ------------------- Dashboard Service ------------------- #

kind: Service

apiVersion: v1

metadata:

  labels:

    k8s-app: kubernetes-dashboard

  name: kubernetes-dashboard

  namespace: kube-system

spec:

  ports:

    - port: 443

      targetPort: 8443

  selector:

    k8s-app: kubernetes-dashboard

Deploy the dashboard.

#kubectl create -f kubernetes-dashboard.yaml

Installing Heapster

Create a manifest for Heapster

#vim heapster.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

  name: heapster

  namespace: kube-system

---

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

  name: heapster

  namespace: kube-system

spec:

  replicas: 1

  template:

    metadata:

      labels:

        task: monitoring

        k8s-app: heapster

    spec:

      serviceAccountName: heapster

      containers:

      - name: heapster

        image: gcr.io/google_containers/heapster-amd64:v1.4.2

        imagePullPolicy: IfNotPresent

        command:

        - /heapster

        - --source=kubernetes.summary_api:''?useServiceAccount=true&kubeletHttps=true&kubeletPort=10250&insecure=true

---

apiVersion: v1

kind: Service

metadata:

  labels:

    task: monitoring

    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)

    # If you are NOT using this as an addon, you should comment out this line.

    kubernetes.io/cluster-service: 'true'

    kubernetes.io/name: Heapster

  name: heapster

  namespace: kube-system

spec:

  ports:

  - port: 80

    targetPort: 8082

  selector:

    k8s-app: heapster

---

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1beta1

metadata:

  name: heapster

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: system:heapster

subjects:

- kind: ServiceAccount

  name: heapster

  namespace: kube-system

#kubectl create -f heapster.yaml

Edit the Heapster RBAC role and add the get permission on the nodes statistic at the end.

#kubectl edit clusterrole system:heapster

...

- apiGroups:

  - ""

  resources:

  - nodes/stats

  verbs:

  - get

Accessing the Kubernetes dashboard

Create an admin user manifest.

#vim kubernetes-dashboard-admin.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

  name: admin-user

  namespace: kube-system

---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRoleBinding

metadata:

  name: admin-user

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: cluster-admin

subjects:

- kind: ServiceAccount

  name: admin-user

  namespace: kube-system

Create the admin user

#kubectl create -f kubernetes-dashboard-admin.yaml

Get the admin user token.

#kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

Copy the token.

Start the proxy to access the dashboard.

#kubectl proxy

Browse to  http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

Select Token and paste the token

Install Kubernetes CLI on Windows 10

If you are looking to access Kubernetes Cluster from your windows machine. Look no further! I will show you how to install Kubernetes command-line utilities by leveraging the Chocolatey installer. Note. I will be using Windows 10 to demonstrate.

Now let's go ahead and get started by opening PowerShell as administrator and execute the below command.

#Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

Now that Chocolatey has been installed, we will go ahead with the Kubernetes CLI setup.

Install Kubernetes CLI

Open PowerShell as an administrator and execute the below command

#choco install kubernetes-cli

You will be prompted to confirm if you want to proceed forward with the installation. Go ahead and say yes by typing Y and hit enter.

Connect to Kubernetes Cluster with Kubectl

Once you have install Kubernetes CLI. Go to your Kubernetes master node and copy the config file from ~/.kube/config to your windows machine to any location. We will move that file to the required location once we create the .kube directory on windows. Follow the below steps.

Open PowerShell as an administrator and execute the below commands.

#cd ~

The above command will take you to your user's home directory. In the user home directory create a folder called .kube. If it already exists, you can skip this step.

#mkdir .kube

Once the above directory has been created, we need to copy the config file from the Kubernetes master node to the .kube folder. Earlier I mentioned copying config files to your windows machine. Take that file and drop it on your under ~\.kube location path. In windows, the config file should be ALL FILES format.

Basic operations

After you have followed the steps, as shown above, let's go ahead and test connectivity with your Kubernetes cluster.

#kubectl.exe config get-clusters

If the above command returns the name of the cluster, then you have applied changes successfully. The below command will get information from your master node.

#kubectl.exe version -o yaml

For me I received following output. Yours may vary depending on your cluster configuration.

clientVersion:

  buildDate: 2018-01-04T11:52:23Z

  compiler: gc

  gitCommit: 3a1c9449a956b6026f075fa3134ff92f7d55f812

  gitTreeState: clean

  gitVersion: v1.9.1

  goVersion: go1.9.2

  major: "1"

  minor: "9"

  platform: windows/amd64

serverVersion:

  buildDate: 2018-01-18T09:42:01Z

  compiler: gc

  gitCommit: 5fa2db2bd46ac79e5e00a4e6ed24191080aa463b

  gitTreeState: clean

  gitVersion: v1.9.2

  goVersion: go1.9.2

  major: "1"

  minor: "9"

  platform: linux/amd64

Let's execute one more command to ensure we are successfully connected to the Kubernetes cluster.

#kubectl.exe get nodes

If you received something similar to what I have received below, then you are fully connected to the cluster and you can go ahead and manage your cluster from a windows machine.

Browse to  http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

Metal load balancer on master for glowing (external IP) on-premises

First, we need to apply the MetalLB manifest

# kubectl apply –f https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml

Next, we want to check that the controller and the speaker are running. we can do this by using this command

#kubectl get pods -n metallb-system

So next we will look at our configuration on master

#cat metalconfig.yaml

apiVersion: v1

kind: ConfigMap

metadata:

  namespace: metallb-system

  name: config

data:

  config: |

    address-pools:

    - name: my-ip-space

      protocol: layer2

      addresses:

      - 192.168.1.160-192.168.1.165

Apply the config for metallb

#kubectl apply –f metalconfig.yaml

Deploy a tomcat and service for checking

#cat tomcat.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

  name: tomcat-pod

spec:

  selector:

    matchLabels:

      run: tomcat-pod

  replicas: 3

  template:

    metadata:

      labels:

        run: tomcat-pod

    spec:

      containers:

      - name: tomcat

        image: tomcat:latest

        ports:

        - containerPort: 8080

---

apiVersion: v1

kind: Service

metadata:

  name: tomcat-pod

  labels:

    run: tomcat-pod

spec:

  type: LoadBalancer

  ports:

  - port: 8080

    targetPort: 8080

  selector:

    run: tomcat-pod

deploy the tomcat

#kubectl apply –f tomcat.yaml

NFS Persistent Volume

we are going to provision an NFS server on Ubuntu 16.04 you can use your own NFS server on any platform

Installing NFS server on Ubuntu 16.04 (new machine)

To get the NFS server working you must install the server packages. run the commands below:

#sudo apt-get update

#sudo apt-get install nfs-kernel-server

Installing NFS client packages on the client systems. here we are going to use the client as All three master nodes. 

Install NFS client on all the master nodes using the following command, to access NFS mount points on the server, 

#sudo apt-get update

#sudo apt-get install nfs-common

After installing the client packages, switch to the server to configure a mount point to export to the client.

Creating the folder/directory to export (share) to the NFS clients, For this Lab, we’re creating a folder called nfs/kubedata in the /srv/ directory.so run,

#sudo mkdir –p /srv/nfs/kubedata

Since we want this location to be viewed by all clients, we’re going to remove the restrictive permissions. To do that, change the folder permission to be owned by nobody in no group.

#sudo chown nobody:nogroup /srv/nfs/kubedata

#sudo chmod 777 /srv/nfs/kubedata

Configuring NFS Exports file

Now that the location is created on the host system, open the NFS export file and define the client access.

Access can be granted to a single client or entire network subnet. For this Lab, we’re allowing access to all clients.

NFS export file is at /etc/exports, open the export file by running the commands below:

#vi /etc/exports

Then add the line below:

/srv/nfs/kubedata       *(rw,sync,no_subtree_check,insecure)

The options in the setting above are: rw= read/write, sync=write changes to disk before applying and no_subtree_check= prevents subtree checking insecure=no security, * = all ips.

Export the shares by running the commands below

#exportfs -v

/srv/nfs/kubedata               (rw,wdelay,insecure,root_squash,no_subtree_check,sec=sys,rw,root_squash,no_all_squash)      => output

#sudo exportfs –rav

exporting *:/srv/nfs/kubedata   => output

Restart the NFS server by running the commands below.

#sudo systemctl restart nfs-kernel-server

NFS server is ready to server ans share it storage… (IN master

#showmount -e

Export list for zippyops:

/srv/nfs/kubedata *