Skip to content

Foundations of Kubernetes security

In this lab, we will explore several security mechanisms in Kubernetes by implementing best practices at both the Linux and Kubernetes levels. We will first experiment with Linux capabilities and AppArmor, before moving to Role-Based Access Control (RBAC) in a Kubernetes cluster using Minikube. By the end of this lab, you will:

  1. Experiment with Linux capabilities to manage process permissions.
  2. Experiment with AppArmor to enforce security policies on applications.
  3. Set up a Kubernetes cluster using Minikube.
  4. Configure RBAC by creating roles, role bindings, and testing user permissions.
  5. Understand namespace-specific roles and cluster-wide roles (ClusterRoles).

The first two exercices will be conducted on your Linux environment, while the RBAC section will be executed within the Minikube cluster.

Linux Capabilities

We will use a small program that attempts to bind to port 900 and then exits. In Linux, ports 1-1023 are considered privileged, meaning only the root user can bind to them. A regular user running this program will encounter a permission error.

Our objective is to enable a non-root user to bind to port 900 without granting full root privileges.

Step 1: Writing the program

Create a C program tha binds to port 900: 

cat > bind_to_port_900.c << EOF
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>

#define PORT 900

int main() {
    int server_fd;
    struct sockaddr_in server_addr;

    // Create socket
    server_fd = socket(AF_INET, SOCK_STREAM, 0);
    if (server_fd < 0) {
        perror("Socket failed");
        exit(EXIT_FAILURE);
    }

    // Set up server address structure
    server_addr.sin_family = AF_INET;
    server_addr.sin_addr.s_addr = INADDR_ANY;
    server_addr.sin_port = htons(PORT);

    // Bind socket to port 900 (privileged port)
    if (bind(server_fd, (struct sockaddr *)&server_addr, sizeof(server_addr)) < 0) {
        perror("Bind failed");
        close(server_fd);
        exit(EXIT_FAILURE);
    }

    printf("Successfully bound to privileged port %d\n", PORT);

    // Close socket
    close(server_fd);

    return 0;
}
EOF

Step 2: Compiling and running the program

  1. Install GCC (if not already installed): 

    sudo apt install gcc

  2. Compile the program: 

    gcc bind_to_port_900.c -o bind_to_port_900

  3. Verify that the binary exists, you should see a file named bind_to_port_900: 

    ls

  4. Run it as root (this should work): 

    sudo ./bind_to_port_900

Expected output: 

Successfully bound to privileged port 900

  1. Run it as a normal user (this should fail): 
    ./bind_to_port_900

Expected output: 

Bind failed: Permission denied

Step 3: Granting capabilities

First check the capabilities manual 

man capabilities

Question 1

In the manual of capabilities, which capability seems adapted to allow you to bind to a privileged port?

We noticed the CAP_NET_BIND_SERVICE capability that enables to bind to privileged ports. Now, grant the necessary capability: 

sudo setcap 'cap_net_bind_service=+ep' ./bind_to_port_900

Verify that the capability was applied: 

getcap ./bind_to_port_900

Run the program again as a regular user. This time, it should succeed: 

./bind_to_port_900

AppArmor

AppArmor is a Linux security module that limits application permissions by enforcing policies defined in security profiles.

Step 1: Checking AppArmor status

Start AppArmor and verify if it is enabled on your system: 

sudo systemctl start apparmor
sudo systemctl status apparmor
sudo aa-status

Step 2: Creating a test script

We will create a script that interacts with system files: 

sudo mkdir /root/apparmor
cat <<EOF | sudo tee /root/apparmor/app.sh > /dev/null
#!/bin/bash
touch /tmp/file.txt
echo "New file created"

rm -f /tmp/file.txt
echo "New file removed"
EOF

Make the script executable and run it: 

sudo chmod +x /root/apparmor/app.sh
sudo /root/apparmor/app.sh

Step 3: Generating an AppArmor profile

  1. Install Apparmor utilities: 

    sudo apt install apparmor-utils -y

  2. Generate a profile: 

    sudo aa-genprof /root/apparmor/app.sh

Follow the on-screen prompts. - Press S to scan for execution. - Run the script in another terminal: 

sudo /root/apparmor/app.sh
- After running the script, return to the terminal where aa-genprof is running. For each capability identified by aa-genprof, press I for inheriting the rule. Save (press S) and exit the editor (press F for finish) when done.

Question 1

During the process, which capabilities were identified by aa-genprof when running the app.sh script? For each capability, to which instructions in the script it refers?

  1. Verify the profile is active: 
    sudo aa-status
sudo cat /etc/apparmor.d/root.apparmor.app.sh

The first command lists all loaded profiles, and the second displays the content of the newly created profile.

  1. Modify the script to add a ping command 
    cat <<EOF | sudo tee -a /root/apparmor/app.sh > /dev/null
ping localhost
EOF

Run it again (this should not work this time): 

sudo /root/apparmor/app.sh

  1. Disable the profile

To disable the profile, you can remove its symlink from the enabled directory and reload AppArmor. This will unload the profile, effectively disabling its enforcement: 

sudo ln -s /etc/apparmor.d/root.apparmor.app.sh /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/root.apparmor.app.sh

With the profile disabled, execute the script again (this should now work): 

sudo /root/apparmor/app.sh

Without the profile's restrictions, the script will have unrestricted access to system resources, which may pose security risks.

Role-Based Access Control (RBAC)

RBAC restricts access to Kubernetes resources based on user roles, allowing you to enforce the principle of least privilege.

Step 1: Starting Minikube

We first have to install Minikube and start a cluster. Follow the instructions from Lab 2 to install Minikube. You can refer to the Minikube setup this guide if needed.

Once Minikube is installed, you can start your Kubernetes cluster with the following command: 

minikube start --cni=calico

Step 2: Creating client certificates

To create a client certificate, you first need to have the openssl tool installed. You can run the follwoing command: 

sudo apt install openssl

We will first create a user account for the lab.

Kubernetes supports several user authentication methods. It also supports combining more than one to authenticate a user. If one of the chained methods fail, the user is not verified.

In this example, we will use only one authentication method, the X509 certificate to create a user account called test-user.

First, we need to create the client key: 

openssl genrsa -out test-user.key 2048

Then, we need to create a certificate signing request:

openssl req -new -key test-user.key -out test-user.csr -subj "/CN=test-user"

Next, you need to copy the certificate and key that exists in the Minikube cluster. Once you copy them to the root folder, you should have displayed the files: 

minikube ssh "sudo cat /var/lib/minikube/certs/ca.crt" > ca.crt
minikube ssh "sudo cat /var/lib/minikube/certs/ca.key" > ca.key

You can then check that test-user. and ca. files exist: 

ls -l

Now let's sign the user key and signing request with cluster certificate and key. 

openssl x509 -req -in test-user.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out test-user.crt -days 300

Step 3: Adding user's credentials to our kubeconfig file

kubectl config set-credentials test-user --client-certificate=test-user.crt --client-key=test-user.key

Testing permissions for the user

You can now create an image httpd and apache and verifying that it is running: 

kubectl run apache --image=httpd
kubectl get pod


kubectl --user=test-user get pods
Error from server (Forbidden): pods is forbidden: User "test-user" cannot list resource "pods" in API group "" in the namespace "default"

Question 1

Why the "test-user" user cannot access the given resource?

Step 4: Creating Role and Role Binding

A Role in Kubernetes is as a Group in other RBAC implementations. Instead of defining different authorization rules for each user, you can attach those rules to a group and add users to it. It is a more example, e.g., when users resign you only need to remove them from the group. Similarly, when a new user joins the company or gets transferred to another department, you need to change the roles they’re associated with.

Let’s create a role that enables our user to execute the get pods command:

vi role.yaml

Add the description of the role in this file: 

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: get-pods
rules:
 - apiGroups: ["*"]
   resources: ["pods"]
   verbs: ["list"]

And finally apply it: 

kubectl apply -f role.yaml

Now, we have a role that enables its users to list the pods on the default namespace. But, in order for the test-user user to be able to execute the get pods, it needs to get bound to this role.

Kubernetes offers the RoleBinding resource to link roles with their objects (for example, users).

Let’s describe a role binding that enables our user to execute the get pods command: 

cat <<EOF > role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: test-user-get-pods
subjects:
- kind: User
  name: test-user
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: get-pods
  apiGroup: rbac.authorization.k8s.io
EOF

Then let's create the object: 

kubectl apply -f role-binding.yaml

Now let’s see if test-user can list pods on the cluster:

kubectl --user=test-user get pods

And let’s try to delete the apache pod using the user test-user.

kubectl --user=test-user delete pods apache

As you can see, the user is not able to delete the pods, yet it was able to list them.

To understand why this behaviour happened, let’s have a look at the get-pods Role rules:

  • The apiGroups is an array that contains the different API namespaces that this rule applies to. For example, a Pod definition uses apiVersion: v1. In our case, we chose "[*]" which means any API namespace.
  • The resources is an array that defines which resources this rule applies to. For example, we could give this user access to pods, jobs, and deployments.
  • The verbs is an array that contains the allowed verbs. The verb in Kubernetes defines the type of action you need to apply to the resource. For example, the list verb is used against collections while get is used against a single resource. So, given the current access level granted to test-user, a command likekubectl --user=test-user get pods hostpath-pd will fail while kubectl --user=test-user get pods will get accepted. The reason is that the first command used the get verb because it requested information about a single pod. For more information about the different verbs used by Kubernetes, check the official documentation: text.

We want test-user to have read-only access to Pods (i.e., list, get, watch verbs) but allow them to manage Deployments (which will let them recreate and delete Pods indirectly, through rolling updates). To achieve this, we need to define a Kubernetes Role that grants access to Pods and Deployments, with the appropriate verbs for each.

Question 2

Create the role.yaml file to allow this.

Notice that in all the preceding examples, we didn’t specify a namespace, so our Role is applied to the default namespace. A Role is bound to the namespace defined in its configuration. So, if we changed the metadata of our Role to look like this:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: get-pods
  namespace: web

The test-user user wouldn’t have access to the pods or the deployments unless working in the web namespace.

But, sometimes you need to specify Roles that are not bound to a specific namespace but rather to the cluster as a whole. That’s when the ClusterRole comes into play.

Step 5: Cluster-Wide Authorization Using ClusterRoles

ClusterRoles work the same as Roles, but they are applied to the cluster as a whole. They are typically used with service accounts (accounts used and managed internally by the cluster). For example, the Kubernetes External DNS Incubator (https://github.com/kubernetes-incubator/external-dns) project uses a ClusterRole to gain the necessary permissions it needs to work. The External DNS Incubator can be used to utilize external DNS servers for Kubernetes service discovery. The application needs read-only access to Services and Ingresses on all namespaces, but it shouldn’t be granted any further privileges (like modifying or deleting resources). The ClusterRole for such an account should look as follows:

cat <<EOF > cluster-role-binding.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
 name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: external-dns
rules:
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["services"]
  verbs: ["get","watch","list"]
- apiGroups: ["rbac.authorization.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: default
roleRef:
  kind: ClusterRole
  name: external-dns
  apiGroup: rbac.authorization.k8s.io
EOF

Now create the cluster role binding: 

kubectl apply -f cluster-role-binding.yaml

The above definition contains three definitions: - A service account to use with the container running the application. - A ClusterRole that grants the read-only verbs to the Service and Ingress resources. - A ClusterRoleBinding which works that same as a RoleBinding but with ClusterRoles. The subject here is ServiceAccount rather than User, and its name is external-dns.

Another everyday use case with ClusterRoles is granting cluster administrators different privileges depending on their roles. For example, a junior cluster operator should have read-only access to resources to get acquainted; then more access can be granted later on.

Summary

  • Kubernetes uses RBAC to control different access levels to its resources depending on the rules set in Roles or ClusterRoles.
  • Roles and ClusterRoles use API namespaces, verbs and resources to secure access.
  • Roles and ClusterRoles are ineffective unless they are linked to a subject (User, serviceAccount...etc) through RoleBinding or ClusterRoleBinding.
  • Roles work within the constraints of a namespace. It would default to the “default” namespace if none was specified.
  • ClusterRoles are not bound to a specific namespace as they apply to the cluster as a whole.