RKE2

Defense in Depth with RKE2 and Sysdig

Jennifer Power

With the widespread adoption of Kubernetes throughout the DevOps space it is important to prioritize security using principles like defense in depth to safeguard your technology stack.

Kubernetes was created for developers to ease the friction of code deployment. Traditionally, additional security measures would mean additional time spent taking manual steps to enable and validate these measures. Making life easy for developers doesn’t mean that you must make security and operations complex. Because of the open nature and ease of integration with Kubernetes, think of security as something that can also be integrated into building and operating secure software.

Kubernetes and Container Security in Layers

There’s a false sense that containerized applications are inherently secure. Containers can make your environment more secure. Isolated, well-understood processes can have a smaller attack surface and more intelligent, more agile management than legacy VMs running a mess of code. While Kubernetes brings many security advantages due to the software-defined nature of the workloads, mistakes in configuration happen and vulnerabilities are always being discovered. Couple the velocity which software is deployed and the hands-off nature which your orchestrator spins up and down workloads and it can be very chaotic.

Container security requires a new approach. For a long time, container security was focused on container image creation. Image scanning is now table stakes for using containers in production. While image scanning is the first step in managing security risk for cloud-native workloads, runtime security is a critical component of a DevSecOps workflow. As cloud-native workloads are running production applications, the ability to monitor containers, Kubernetes, and cloud services as well as validate compliance is important for keeping customers happy and meeting industry regulations.

Several security threats, by their very nature, only manifest during runtime:

Zero-day vulnerabilities
Misconfiguration
Software bugs causing erratic behavior or resource leaking
Internal privilege escalation attempts

Container runtime security will protect you against these threats. At the same time, it will also help you verify that the other security barriers are effective and in-line with compliance standards by examining technical controls. Key aspects of runtime security are:

Continuous scanning: By continuously tracking your running images, you can detect which ones are affected by newly discovered vulnerabilities or aren’t compliant after a change to your agency’s policies.
Threat detection: Is your container doing what it’s supposed to do? By monitoring your container activity, you can immediately identify suspicious behavior.
Automatic incident response: Once a policy violation is detected, action needs to be taken as soon as possible. An automatic response will block security threats right when they are detected by killing or pausing the affected containers, notifying the relevant people about the incident.
Capturing forensics data: After an incident, you need to find the source to prevent it from happening again. Comprehensive forensics data is crucial for your investigation, as well as being able to correlate events from several sources with concrete Kubernetes resources (namespaces, deployments, containers, etc.)

Approaching security in layers is vital to any cybersecurity approach. You’re likely already doing this today with traditional workloads. When it comes to cloud computing, there are tools available to enable and allow compliance and security similarly.

Reduce Risk with the Right Tools (RKE2 & Sysdig)

RKE2 or RKE Government is an easy-to-install Kubernetes distribution specifically engineered to focus on the security and compliance requirements of the public sector.

The secure-by-default design of RKE2 includes Security-Enhanced Linux (SELinux) via containerd and the first fully FOSS FIPS-140-2 validated Kubernetes encryption module. RKE2 also provides defaults and configuration options that allow clusters to pass the Center for Information Security (CIS) Kubernetes Benchmark with a small amount of manual intervention. For additional hardening guidance, see the CIS Hardening Guide and CIS Self-Assessment Guide on the RKE2 documentation.

Building upon a solid foundation is critical. However, no Kubernetes security strategy is complete without instrumentation for cloud-native workloads. That is where tools like Sysdig’s Secure DevOps Platform come in. Sysdig is a unified platform for security, compliance, and monitoring that enables Cloud and DevOps teams to confidently run containers, Kubernetes, and cloud services. The Sysdig Secure DevOps Platform provides guardrails for developers to be effective and efficient while also maintaining your organization’s security posture throughout the container lifecycle. As previously mentioned, it’s now no longer enough to stop security after container image creation. Runtime protection is a requirement to have risk observability and awareness of the workloads running on your cluster in your cloud.

Sysdig Secure comes out of the box with policies to protect running workloads. The policies are also mapped to common frameworks like NIST, MITRE ATT&CK, and more.

Sysdig Secure leverages the Falco engine under the hood for runtime security and cloud threat detection. Sysdig Secure saves time in creating and maintaining policies. Falco, the open-source cloud-native runtime security project, is the de facto Kubernetes threat detection engine. Falco was created by Sysdig in 2016 and is the first runtime security project to join the CNCF (Cloud Native Computing Foundation) as an incubation-level project. Falco detects unexpected application behavior and alerts on threats at runtime. Falco and runtime security is just one component of Sysdig’s multi-layer protection provided with Sysdig Secure. For more information about all the ways Sysdig reduces risk and increases visibility across containers, Kubernetes, and the cloud, click here.

To see how easy it is to secure your agency’s container and Kubernetes running workloads, the tutorial below will show you how you can easily spin up a Rancher RKE2 cluster and protect it with Sysdig Secure.

Setting up RKE2 and Sysdig Secure

What do you need?

For the purposes of this tutorial, we will be using a single node cluster to act as both server and agent. You will also need the following tools available on your system:

Note: A CentOS machine was used for this tutorial and it must have the kernel-devel package installed

Kubectl
Helm
Git
Ansible 2.9.0+

To read more about the hardware requirements see the RKE2 documentation.

Installing RKE2

Let’s start with installing RKE2. You can install using the RKE2 Installation documentation, but we are going to use the Rancher Federal RKE2 installation Ansible playbook.

  # Clone the repo
  git clone https://github.com/rancherfederal/rke2-ansible.git
  # Create your own inventory file
  cp -R inventory/sample inventory/my-cluster
  # Add your IP under rke2_servers group
  vi inventory/my-cluster/hosts.ini
  # Run the playbook
  ansible-playbook site.yml -i inventory/my-cluster/hosts.ini

You can get the configuration file you need for cluster access on the node at /etc/rancher/rke2/rke2.yaml.

To see more information about the RKE2 Ansible setup and installation options, see the Rancher Federal documentation.

Install Sysdig

Link here:

https://sysdig.com/company/start-free/

Install the Sysdig Agent

There are few options when installing the Sysdig Agent, but we are going to use Helm here.

  helm repo add sysdig https://charts.sysdig.com
  helm repo update

  helm install sysdig-agent \
    --set sysdig.accessKey=<changeme> \
    --set sysdig.settings.collector=ingest-us2.app.sysdig.com \
    --set sysdig.settings.collector_port=6443 \
    --set sysdig.settings.k8s_cluster_name=k3s-multi sysdig/sysdig \
    --set nodeImageAnalyzer.settings.collectorEndpoint=https://us2.app.sysdig.com/internal/scanning/scanning-analysis-collector \
    --set nodeImageAnalyzer.settings.containerdSocketPath=unix:///run/k3s/containerd/containerd.sock \
    -n sysdig-agent --create-namespace

Note: Other options are available in the Sysdig Platform under Install the Agent.

Check that the pods are running

kubectl get pods –n sysdig-agent

Configure Sysdig

1) Setup Notification Channel

Sysdig gives you the option to receive alerts in several ways.

To configure, go to your account settings under Notifications and elect “Add a Notification Channel”.

2) Create a Policy

We are going to create a test policy with an existing rule so we can simulate an event. To do this, you will need to go to the Sysdig Platform under the Sysdig Secure page. Choose Policies on the left-hand toolbar and then Runtime policies.

To add a policy choose, Add Policy and then Workload Policy.

Fill in the information and leave the default options that are set. Import the rule from the library and add Launch Package Management Process in Container. Choose the notification channel you created the earlier step. (Allowing package managers in containers isn’t best practice. Sysdig can scan during the build phase to warn or stop container images that contain package mangers from ever getting to your Kubernetes cluster.)

By having this as a runtime policy, this allows agencies to increase risk observance and should a container get through with package managers enabled, provide information around what the package manager is doing.

Test Sysdig

On your system run the following command to launch a test pod with the command that will create an event:

kubectl run --rm -it testevent --image ubuntu --command /bin/bash
              apt-get update

Results

After running this test, a couple things should happen to let you know that Sysdig is doing its job. You should get a notification via the channel you setup during the configuration phase that an event has occurred. This event will also be logged and available for viewing on the Sysdig platform under Events.

Next Steps

To learn more about using RKE2 and Sysdig, please see the resources below:

RKE2 official documentation found here for RKE2 hardening and compliance information.

Sysdig official documentation found here

For more information around how Sysdig is extending OSS Falco within Sysdig Secure or about OSS Falco, please visit the links below:

Sysdig Secure with Falco

Falco OSS

Conclusion

Sysdig Secure provides a real-time look at the security posture in Kubernetes environments and extends Falco with the ability to react, block and report malicious behaviors. RKE2 and Sysdig make a great combination to add to your defense in depth Kubernetes implementation.

Special Thanks to Jennifer Power and Ryan O’Daniel for collaborating on this post.

“This publication was prepared or accomplished by the author in a personal capacity. All opinions expressed by the author of this publication are solely their current opinions and do not reflect the opinions of Rancher Federal, Inc., respective parent companies, or affiliates with which the author is affiliated. The author's opinions are based upon information they consider reliable, but neither Rancher Federal, Inc., nor its affiliates, nor the companies with which the author is affiliated, warrant its completeness or accuracy, and it should not be relied upon as such.”