Kubernetes Performance Tuning: Make the Most of Your Clusters

In this article, learn the basics of Kubernetes performance and explore several best practices you can use to tune the performance of cluster resources.

Why Is Kubernetes Performance Tuning Needed?

As Kubernetes becomes a basic infrastructure for many organizations, performance tuning for Kubernetes clusters is becoming more important. Kubernetes is a highly scalable open-source platform for orchestrating containerized workloads in server environments. It enables declarative configuration and automation of computing resources.

In cloud systems, server costs can spike and in some cases grow unexpectedly. You should aim to maximize infrastructure utilization rather than just scaling to meet the needs of your workloads. Improving utilization is critical for cloud cost optimization. We’ll provide Kubernetes performance tuning tips that will help you optimize your environment to make Kubernetes more efficient, reduce network latency, and ensure your workloads run in an optimal and cost-effective manner.

Key Kubernetes Performance Metrics to Monitor

Kubernetes provides several tools that can help you monitor critical resources. Here are some of the metrics you should watch closely to discover and resolve performance issues. 

Memory Utilization

Monitoring memory utilization can help you gain insight into the cluster’s performance and ability to run workloads successfully. Ideally, you should monitor usage at the node and the pod level. Here’s how it can help:

<!--[if !supportLists]-->·         <!--[endif]-->Monitoring the pod level:  Can help identify pods that exceed memory usage limits and terminate them

<!--[if !supportLists]-->·         <!--[endif]-->Monitoring the node level: Can help identify nodes running low on available memory; in this case, the kubelet flags the node as under memory pressure and starts reclaiming resources.

The kubelet can evict pods to reclaim resources. Part of this process involves deleting pods from the node. The Control Plane is responsible for rescheduling the evicted pods on another node with sufficient resources. 

In cases when the pods’ memory usage exceeds the defined requests, the kubelet can prioritize these pods for eviction. Comparing requests with actual usage can help surface pods potentially vulnerable to eviction.

Monitoring memory utilization on nodes can also help learn when to scale the cluster. For example, when node-level usage is high, the cluster requires more nodes to share the workload.

Disk Utilization

Disk space is a non-compressible resource. Low disk space on the root volume can lead to issues with scheduling pods. Once the node’s remaining disk capacity exceeds a certain threshold, it is flagged under disk pressure.

You should also monitor the usage levels of volumes used by pods to learn of problems at the service or application level. Once volumes are provisioned and attached to a certain node, the node’s kubelet exposes several volume-level disk utilization metrics, including the volume’s capacity, available space, and utilization.

CPU Utilization

Monitoring CPU utilization can help you assess the performance of your cluster. You can compare the amount of CPU your pods use to their configured limits and requests. Additionally, you should monitor CPU utilization at the node level. A node experiencing a lack of CPU can throttle the CPU allocated to individual pods. 

Desired vs Current Pods

Production deployments typically involve using controllers to describe the desired cluster state and automate the process of creating pods. Kubernetes offers the following metrics to help monitor these aspects:

<!--[if !supportLists]-->·         <!--[endif]-->kube_deployment_spec_replicas: Reflects the number of desired pods

<!--[if !supportLists]-->·         <!--[endif]-->kube_deployment_status_replicas:  Reflects the number of currently running pods

Ideally, these numbers should match. The only exception is when you are in the deployment phase or another transitional phase. By comparing spec and status metrics, you can learn of cluster issues. For example, a big disparity between desired and running pods can indicate a bottleneck or a problem with the configuration. You can inspect pod logs to determine the cause.

Best Practices to Enhance Kubernetes Performance

Follow these best practices to fine-tune your Kubernetes performance:

Select the Right Persistent Storage and Service Quality

Different types of Kubernetes persistent volumes are better suited to different workloads. Like optimizing memory and CPU, you should choose the optimal hardware for persistent storage. SSD-based storage is better than HDD-based storage for read/write performance, while NVMe NVMe (Non-Volatile Memory Express) SSDs are best suited for heavy workloads.

Kubernetes supports various persistent volumes, offering an extensible, flexible storage approach. Kubernetes product vendors may use quality of service levels to extend the schema definitions of persistent volume claims. They prioritize the read/write quotas of volumes for a given deployment, providing higher throughput.

Strong hardware enables strong performance, although the better persistent storage technology does not necessarily improve overall performance. For example, if you use NVMe SSDs, you can expect better performance for read/write operations but not network latency.

Avoid Network Performance Issues by Deploying Clusters Close to Users

Kubernetes networking can be complex to set up and manage. One of the ways networking can affect performance is latency between cluster resources and Kubernetes users or application end-users. Locating Kubernetes nodes near the customer can enhance the user experience. Cloud providers usually offer several geographic zones, allowing systems operators to deploy Kubernetes clusters in different global locations. 

Before deploying your clusters across different zones, you must have a clear cluster management plan. Understand the provider-specific limitations regarding the zones you can use to optimize failure tolerance. For example, if you have an Azure-based deployment, you can assign specific zones to Azure Kubernetes Service (AKS). If you use Google Cloud, you can leverage Google Kubernetes Engine (GKE) to select multi-zone or region clusters (each option offers different benefits and drawbacks in terms of redundancy, cost, and proximity to the end-user).

Some organizations deploy locally and then adjust their deployment strategy according to user feedback. For this approach to work, you need to implement a monitoring system to identify latency issues before they impact end-users. 

Use Optimized, Lightweight Images

Building optimized images allows Kubernetes to pull them quickly and enables a more efficient deployment. There are various services and tools to scan images and optimize them.

An optimized image should:

Have a single application or function

Be lightweight (large images are less portable)

Have endpoints for readiness and health checks, allowing Kubernetes to respond in the event of downtime

Use a container-friendly operating system (e.g., CoreOS) to help the image resist misconfigurations

Employ multi-step builds to ensure you deploy the compiled application without the attached dev sources

Employ Multiple Masters

Using multiple master nodes in your Kubernetes cluster lets you achieve high availability. If you have more masters, there is a lower likelihood of a failure that brings down your cluster. 

Adding master nodes can also enhance performance because the critical Kubernetes components can leverage more hosting resources. For example, Kubernetes utilizes the shared resources of your masters to power the API server and scheduler. Thus, you can add one or several master nodes to boost your cluster’s performance.

Conclusion

In this article, I explained the basics of Kubernetes performance and provided several best practices you can use to tune the performance of cluster resources:

Closely monitor memory utilization, disk utilization, and CPU utilization.

Ensure your persistent storage performance matches the requirements workloads.

Deploy clusters closer to users to reduce latency.

Use optimized images with the minimal components needed to run your workload.

Run multiple Kubernetes masters to improve performance and availability.

I hope this will be useful as you learn to build faster, better optimized Kubernetes clusters.

Relevant Blogs

Security Matters: Vulnerability Scanning Done Right! 

Top 5 Recent Data Breaches: Causes and Lessons 

Application Security Predictions Over the Next 12 Months 

Why PKI Is Critical to Your Cloud Strategy