Why Pod Quality Of Service Didn’t Solve Our Queries

In Kubernetes, there are three classes of Quality of Service pod containers.

1. Guaranteed. Provides the highest quality and reliability. Pods with CPU and memory limits and requests belong to this class. This means that they are guaranteed specific resources.
2. Scalable class (Burstable). This class is suitable for pods where requests and resource limits are specified, but the requests are below the limits. In this case, pods can use more resources than specified in the recommendations, but only up to the limit level.
3. Best Effort class. This class is assigned to pods that do not specify limits/request resource limits. These pods can use the available resources in the tier that is currently available. This is the most vulnerable class.

If you don’t set resource limits for pods, they can be categorized as “best effort.” They will work, but they will need more resources at the first opportunity. In other words, constraints are necessary for the node to function correctly.

Following the practice of setting constraints and queries has several significant benefits to consider:

• The load is distributed evenly across the nodes, and resources are spent predictably.
• there is no problem with the “eviction” of high-priority pods;
• Cluster auto scaler can more accurately determine the resources needed to host the node.

To solve the problem of the lack of restrictions and requests, we decided to use the Gatekeeper policy. It requires users to specify limits and requests when running containers in pods explicitly. However, after a short trial period, we found that not all customers follow the new rules. This has made it challenging to use our services.

We couldn’t implement a mandatory policy for setting limits and requests. Instead, we decided to use a standard mechanism in Kubernetes called LimitRange.

Pod Quality of Service is convenient, but it is resource-intensive. When a project is dynamic, there is simply no time for it.

What Are The Advantages Of LimitRange And Shell-Operator?

LimitRange is a way to limit and set default values for resource usage (such as CPU and memory) in Kubernetes. Allows you to configure settings for different objects, such as pods or persistent storage requests (PersistentVolumeClaim), within a specific namespace.

LimitRange can also set default values for limits and resource requests for all containers running in this namespace.

In our case, after applying the LimitRange rule in the namespace, the values of limits and requests are automatically added to users’ containers, even if they do not specify them. This has made the process easier, but only within a single namespace.

To dynamically apply the LimitRange settings to all possible namespaces that customers can create, we used a tool called Shell-operator from Flint. Shell-operator allows us to monitor events in Kubernetes and run special scripts, such as hooks, in response to these events. We’ve set up a hand that automatically applies the LimitRange settings to each namespace you create.

LimitRange and Shell-operator are easier to use. It is more convenient for customers to set resource limits rather than calculate them to the byte.

Additional Security Measures

We used Gatekeeper to enforce security policies centrally, and it also helped fix some vulnerabilities in the underlying Kubernetes system. For example, there is a dangerous directive called hostPath, through which attackers could implement various threats:

• Mount Docker or CRI-O sockets in pods and, access all containers, perform various actions with them;
• Find a Kubeconfig file with administrative rights to the cluster and use it for your purposes.
• Mount their file with the keys, giving them unrestricted access to the cluster. Even mounting the /etc/shadow file could break password hashes.

To address all of these vulnerabilities, we applied a Gatekeeper policy called host filesystem. It allowed us to specify which directories can be mounted and set rules for accessing them.

In addition, we have restricted host PID and IPC settings in our clusters. Attackers could use these settings to access processes on the host, view pod environment variables, and even view file descriptors.

LimitRange can also set default values for limits and resource requests for all containers running in this namespace.

In our case, after applying the LimitRange rule in the namespace, the values of limits and requests are automatically added to users’ containers, even if they do not specify them. This has made the process easier, but only within a single namespace.

To dynamically apply the LimitRange settings to all possible namespaces that customers can create, we used a tool called Shell-operator from Flant. Shell-operator allows us to monitor events in Kubernetes and run special scripts, such as hooks, in response to these events. We’ve set up a hand that automatically applies the LimitRange settings to each namespace you create.

LimitRange and Shell-operator are easier to use. It is more convenient for customers to set resource limits rather than calculate them to the byte.

Conclusions From Our Experience

1. Configuring clusters for security is a requirement in projects at any level. Kubernetes has some vulnerabilities by default, and measures are needed to address them.
2. When working with Kubernetes, it’s essential to adhere to best practices. Ignoring them can cost the company a lot of time and resources to bring the cluster into compliance with security regulations.
3. When configuring the Gatekeeper, it is essential to turn off validation for the KubeSystem namespace. Otherwise, it may cause problems with cluster startup.
4. When implementing security policies, consider the Helm charts and their type. If the Gatekeeper policy is not configured correctly, it can affect the operation of Helm charts, which will lead to errors. It is also worth considering protection against DDoS attacks: their number and intensity are only growing.

Also Read: Impact Of Not Preparing Processes For Kubernetes Deployment